Standard Field Mapping

Standard fields

In addition to supporting infinite custom fields, SparkLogs also defines the following standard fields:

• timestamp: The timestamp of the log message itself.
• ingested_timestamp: The timestamp when the log data was actually ingested into the system.
• event_index: The index of the ingested event within the batch of events submitted in one ingestion request. This allows to reconstruct the exact order of log events even if their timestamps are exactly the same.
• severity: The severity level of the log message.
• facility: The facility level of the log message, if any (usually only set for syslog data).
• subsource: Log filename, pod stream (e.g., stdout), Windows Event Log channel name, instrumentation scope name for OTLP logs. The SparkLogs Agent normalizes certain file paths to standardize OS directory paths (see below).
• subsource_raw: A reserved field name for the original, unnormalized subsource value. Collectors that normalize subsource (for example, collapsing user-specific path segments in a log file path) can preserve the exact original value here. It flows through as a regular custom field.
• source: The name of the source of the event (e.g., Kubernetes podname or hostname of the device that generated the log event).
• service: Logical service / workload identity (for example OTel service.name, ECS nested service.name, or Kubernetes workload names detected from resource attributes).
• app: A string field often set by log collectors for broader application grouping (for example ECS-style labels.application, service.namespace, or syslog appname).
• message: The original and unmodified string value of the log event.
• category: One or more category labels (separated by .), as extracted by category extraction.
• pattern: The pattern assigned by AutoClassify.
• trace_id: A globally unique ID that tracks a single request across distributed systems.
• span_id: An ID unique within a given trace that tracks a single operation within that trace.

The following reserved fields are also automatically populated based on the agent and organization that ingest the data:

• organization_id: The ID of the organization that owns the data.
• agent_id: The ID of the agent that ingested the data.

Inbound ingest control fields (for example __event_severity_cap, __event_curated) are stripped at ingest; see HTTPS+JSON ingest control fields.

Hash fields: stable anchors for grouping and automation

Several standard fields carry a companion hash field, computed automatically at ingestion:

Base field	Companion hash
source	source_hash
pattern	pattern_hash
subsource	subsource_hash
category	category_hash
service	service_hash
app	app_hash

A hash is a compact, stable identifier for its base value. Where the base value can be long (a full pattern, a normalized file path, a deep category), the hash gives you a short token that always refers to exactly that value, which makes it ideal as an anchor point: pin it in a saved filter, pass it between tools, or let automation and AI workflows group and correlate events by it without carrying the full text around. For example, AutoClassify assigns every event a pattern, and pattern_hash is the durable handle for that pattern: query results that group by pattern return both the pattern text and its hash, so a follow-up query can drill into one exact pattern by hash alone.

Behavior to rely on:

• Hashes are computed from the final base value after any normalization. The subsource/category/service/app/source hashes group values case-insensitively; pattern_hash identifies the exact pattern text.
• A hash is present only when its base field is non-empty.
• These field names are reserved: any values you send for them are replaced by the computed ones.
• Hashes are grouping keys, not security identifiers.

Subsource normalization by SparkLogs Agent

The SparkLogs Agent normalizes the subsource it stamps on events so that the same logical log stream groups identically across your whole fleet, no matter the drive letter, install location, locale, or which user profile a path runs through:

• Windows Event Log channels become a uniform dotted form, for example channel Microsoft-Windows-Ntfs/Operational becomes subsource eventlog.microsoft.windows.ntfs.operational. The exact channel name is preserved on the event in the channel field.
• Log file paths have well-known directory roots replaced by stable tokens, for example C:\Windows\Logs\CBS\CBS.log becomes <os>/logs/cbs/cbs.log, and when Program Files is installed to a non-standard drive (say D:\Program Files), paths under it still become <prog>/....
• User-specific path segments are collapsed: a path under a user profile becomes <userhome>/... (or <userlocalapp>/<userroaming> for AppData paths), so usernames do not fragment fleet-wide grouping and do not appear in the shared subsource facet.

The exact original path, with its real separators and real username, is always preserved in subsource_raw on the same event, so nothing is lost: subsource is for grouping across machines, subsource_raw is the ground truth for one machine.

Automatic detection of standard fields

Various logging systems and log collectors have widely different names for these standard fields, and for some sources, the mapping may be different for different types of events (e.g., for Google Cloud Platform events, the source field may be set by pod_name for K8s and by instance_id for VMs).

SparkLogs automatically detects and maps fields from many common log schemas, agents, platforms, and logger libraries.

Wire-format schemas:

• syslog (RFC 5424 / RFC 3164)
• OpenTelemetry semantic conventions
• Elastic Common Schema (ECS)
• Splunk HEC event format
• Heroku Logplex format

Agents and collectors:

• Vector
• Fluent Bit
• Fluentd (including fluent-plugin-kubernetes_metadata_filter)
• Filebeat / Elastic Beats
• Grafana Alloy (Loki label conventions)
• OpenTelemetry Collector (including the windowseventlog, journald, and k8sattributes components)
• Datadog Agent reserved attributes
• Splunk HEC clients

Platforms and data sources:

• Windows Event Log on Vector, OpenTelemetry Collector, and Winlogbeat
• Linux journald on Vector and OpenTelemetry Collector
• Docker container logs via Vector docker_logs
• Kubernetes on Vector, Fluent Bit, the Fluentd metadata plugin, OpenTelemetry Collector, Grafana Alloy, Filebeat, and Google Cloud Logging (k8s_container)
• AWS CloudTrail
• AWS Lambda via OpenTelemetry FaaS resource attributes
• AWS ECS / Fargate via OpenTelemetry aws.ecs.* attributes
• AWS EKS (Kubernetes shape, with aws.eks.cluster.arn as the cluster identifier)
• Google Cloud Logging with monitored resources including gce_instance, k8s_container, cloud_run_revision, cloud_function, and gae_app
• Azure Monitor for Azure Functions and Azure App Service
• Heroku Logplex
• Fly.io platform fields

Logger libraries:

• zap (Go)
• log4j / log4j2 (Java)

Standard-field mapping summary by source

The tables below list the vendor-emitted fields that populate each pivot for common sources. When a field is read from your ingestion configuration (not the log payload itself), the cell calls that out. Mapping is also performed for all other relevant fields (timestamp, message, etc.).

Wire-format schemas

Source	app	service	source	subsource
OpenTelemetry	k8s.cluster.name; else service.namespace	service.name; else workload names; else faas.name	k8s.namespace.name/k8s.pod.name; else k8s.pod.name; else faas.instance; else host.name	scope.name; else log.file.path; else stream
Elastic Common Schema (ECS)	orchestrator.cluster.name; else service.namespace	service.name	service.node.name; else kubernetes.pod.name; else host.name	log.file.path; else log.logger
Syslog (RFC 5424)	(empty)	appname	hostname	msgid; else procid

Agents and collectors

Source	app	service	source	subsource
Vector kubernetes_logs	kubernetes.pod_labels."app.kubernetes.io/part-of"	kubernetes.pod_labels."app.kubernetes.io/name"; else kubernetes.pod_owner	kubernetes.pod_namespace/kubernetes.pod_name	stream; else file
Vector windows_event_log	(empty)	(empty)	computer	channel
Vector journald	(empty)	_systemd_unit; else syslog_identifier	_hostname; else host	syslog_identifier; else _comm
Vector syslog	(empty)	appname	hostname; else host	msgid; else procid
Vector docker_logs	label.com.docker.compose.project	(container_name, fallback only)	host	stream; else container_id
Fluent Bit (kubernetes filter)	kubernetes.labels.app.kubernetes.io/part-of	kubernetes.labels.app.kubernetes.io/name	kubernetes.namespace_name/kubernetes.pod_name; else kubernetes.host	stream; else kubernetes.docker_id
Fluentd (in_tail + kubernetes metadata filter)	kubernetes.labels.app.kubernetes.io/part-of	kubernetes.labels.app.kubernetes.io/name	kubernetes.namespace_name/kubernetes.pod_name; else kubernetes.host	tailed_path; else tag
Filebeat / Elastic Beats	orchestrator.cluster.name; else service.namespace	service.name	kubernetes.pod.name; else host.name	log.file.path; else log.logger
Grafana Alloy / Loki labels	app	service_name; else job	namespace/pod; else pod; else instance	filename; else stream
Datadog Agent	(encoded in ddtags)	service / dd.service	host	ddsource
Splunk HEC	(from index via ingestion config)	sourcetype	host	the HEC source field, which is typically a file path

Platforms and data sources

Source	app	service	source	subsource
Windows Event Log (root-field shape)	(empty)	(empty)	Computer	Channel
macOS unified logging	reverse-DNS prefix of subsystem	subsystem	host	os.category
AWS CloudTrail	recipientAccountId	eventSource	composite of recipientAccountId and awsRegion	resources[0].ARN; else eventName
AWS Lambda	cloud.account.id	faas.name	faas.instance	scope.name
AWS ECS / Fargate	aws.ecs.cluster.name; else aws.ecs.cluster.arn	aws.ecs.task.family	aws.ecs.task.arn	aws.ecs.container.name (fallback only)
AWS EKS	aws.eks.cluster.arn (plus standard Kubernetes app candidates)	(Kubernetes shape)	(Kubernetes shape)	(Kubernetes shape)
Google Cloud Logging — GKE (k8s_container)	resource.labels.cluster_name	(resource.labels.container_name, fallback only)	composite of resource.labels.namespace_name and resource.labels.pod_name	(empty)
Google Cloud Logging — Cloud Run (cloud_run_revision)	resource.labels.project_id	resource.labels.service_name	resource.labels.instanceId	(empty)
Google Cloud Logging — Cloud Functions (cloud_function)	resource.labels.project_id	resource.labels.function_name	resource.labels.execution_id	(empty)
Google Cloud Logging — App Engine (gae_app)	resource.labels.project_id	resource.labels.module_id	gae_instance.instance_id	(empty)
Google Cloud Logging — Compute Engine (gce_instance)	resource.labels.project_id	(empty)	instance_id; else vm_name	(empty)
Azure Monitor (Functions / App Service)	(Resource Group, from ingestion config)	WEBSITE_SITE_NAME	WEBSITE_INSTANCE_ID	Category (e.g. AppServiceConsoleLogs, FunctionAppLogs)
Heroku Logplex	(Heroku app name, set by your drain configuration)	dyno type (web)	dyno (e.g. web.1)	the Heroku source field (app, heroku-router, etc.)
Fly.io	fly.app.name	process group	fly.machine_id; else fly.alloc_id	process group; else stream

tip

If no input field maps to a given pivot, that pivot is left empty rather than auto-filled from another pivot. This is expected for some SaaS-only data where no machine-level source exists. Empty values still work for filtering, grouping, and scoping queries, so the subsource / source / service / app / organization_id hierarchy for context exploration narrows naturally based on which pivots are populated.

A present empty string on a SparkLogs canonical root field (service, app, source, subsource, category_override) is authoritative: later vendor-alias mappings do not fill that pivot. JSON null or an omitted key still means unset.

Original vendor fields: copied vs moved

When SparkLogs detects a standard field from a vendor-specific field name, the value is either copied or moved into the standard field, depending on which standard field it populates:

• Pivot fields (source, service, app, subsource) and trace identifiers (trace_id, span_id, category) are copied. The original vendor field stays as a custom field so you can query on either the SparkLogs standard name or the original vendor name. For example, if your payload has pod, the value populates the source pivot AND the pod field remains queryable.
• Normalized fields (timestamp, severity, facility, message, pattern) are moved. The original vendor field is replaced by the normalized SparkLogs value (e.g. text "info" → numeric OTel severity), so a duplicate raw value would diverge from the canonical form.

One exception: if your payload uses the exact SparkLogs canonical name at root (e.g. you sent {service: "checkout"} directly), it's moved even though service is a pivot — keeping it would just duplicate the value against the standard pivot.

Mapping notes

A few specific mappings are worth highlighting:

• Syslog appname populates service. RFC 5424 APP-NAME is the program name (sshd, nginx, postgres) and is treated as a logical service identity.

• AWS CloudTrail eventSource populates service. Values like s3.amazonaws.com name the AWS service that was called, which matches the SaaS service convention.

• deployment.environment (prod/staging/dev) is not mapped to app. Environment is a separate axis. Any unmapped fields are stored as custom fields and you can filter on it as needed. You may want to map data from different environments to different organizations.

• container.name and container.id are fallback-only. Container name fills service only when no canonical service identity (service.name, deployment name, faas.name) is present. Container id fills subsource only when no canonical subsource (scope name, log file, stream, channel) is present.

• Specific workload identifiers beat host identifiers beat generic host / hostname. When a payload contains multiple candidates for source, the most-specific identifier wins deterministically:

  1. Workload-instance name (k8s.pod.name, aws.ecs.task.arn, faas.instance, dyno, bucket_name)
  2. Workload-instance UUID / hash (service.instance.id, fly.machine_id, kubernetes.pod.uid, Cloud Run instanceId)
  3. Underlying host / node name (host.name, kubernetes.host, computer, _HOSTNAME, vm_name)
  4. Underlying host UUID / hash (host.id, _machine_id, instance_id, ec2_instance_id)
  5. Generic root-level host or hostname (used only as a last resort, because different shippers fill these with different things: the Vector collector's host vs. the application's container hostname vs. the original log producer's host)

  Example: an OpenTelemetry SDK on EKS typically emits both k8s.pod.name and host.name. The pod name wins source (more specific) and host.name stays as a custom field. An ECS payload with both com.amazonaws.ecs.task-arn and host resolves source to the task ARN.

Dotted field names in JSON payloads

Many log schemas (OpenTelemetry resource attributes, ECS, Docker labels, and similar) ship flat JSON keys that contain dots, for example k8s.namespace.name, service.name, or host.name. In the raw payload those names are a single key string, not nested objects.

By default, SparkLogs normalizes eligible dotted keys into nested JSON when each event is ingested. The same path you use in LQL then matches the stored shape:

{"message": "payment ok", "k8s.namespace.name": "payments"}

becomes (conceptually):

{"message": "payment ok", "k8s": {"namespace": {"name": "payments"}}}

You can still filter with k8s.namespace.name="payments". Standard-field detection and pivot mapping run on the wire payload (flat dotted keys); normalization runs later so the stored event_json matches LQL path semantics. Vendor fields such as k8s.namespace.name and service.name continue to resolve to the same source, service, and app values described in the tables above.

What stays literal

Normalization applies only when the key is a valid dotted path (non-empty segments, no spaces or other characters that would require quoting in JSON key names, within supported depth). Keys such as a..b or com.amazonaws.ecs.container-name (hyphen in a segment) are left unchanged. If a scalar value already occupies a parent segment (for example "a": "busy" and "a.b": 99), the flat key a.b is kept as-is so existing data is not overwritten.

When to disable

If you intentionally use literal dotted keys as custom field names and do not want them nested, set the X-No-Normalize-Dotted-Field-Names HTTP header (or the matching no_normalize_dotted_field_names query parameter) to true on the ingest request. This applies to HTTPS+JSON, OTLP/HTTP, and other ingest APIs that document the shared request headers on the HTTPS+JSON parameters page.

Customizing severity mappings

For the field that is detected to be the severity field, it will interpret this field value from any supported text (case insensitive) or numeric value. Numeric severity values must be in the range 1-24 and are interpreted as defined by the OpenTelemetry standard for severity values. Standard textual severities include trace, debug, info, notice or display (info3), warn, error or fail, critical (error4), fatal, alert (fatal2), panic (fatal3), and emergency (fatal4).

If you have non-standard severity values, you can either transform these values before shipping the logs (e.g., using vector VRL transformations), or more conveniently by using the custom severity mapping feature.

To specify a custom mapping, use the X-Severity-Map HTTP header. This is a comma-delimited list of key=value pairs that specifies additional mappings from a custom severity level to a standard severity level.

For example, in the Node.js bunyan log library it uses numeric log levels from 10 (trace) to 60 (fatal). You could remap these with a X-Severity-Map HTTP header value of 10=TRACE,20=DEBUG,30=INFO,40=WARN,50=ERROR,60=FATAL.

Timestamp constraints

tip

When log data is ingested, if the timestamp is older than your configured retention period (or if you are using the SparkLogs cloud and the event is older than 50 days), then the timestamp will be set to ingested_timestamp and the original timestamp will be stored into a original_timestamp custom field.

Also, if there is no timestamp in the structured log data provided by the logging agent, then AutoExtract will attempt to determine the event timestamp from the log message. If nothing relevant is detected, it will use the current date/time for the ingested log event. If you prefer that AutoExtract uses the timestamp in the message first and then falls back to the timestamp sent by your log collector, then configure your log collector to use observed_timestamp for the timestamp field that it sends.