--- title: Normalize-for-Stream processor description: Detects whether a document is OpenTelemetry-compliant and if not - normalizes it as described below. If used in combination with the OTel-related mappings... url: https://www.elastic.co/elastic/docs-builder/docs/3028/reference/enrich-processor/normalize-for-stream products: - Elasticsearch applies_to: - Elastic Stack: Preview since 9.1 --- # Normalize-for-Stream processor Detects whether a document is OpenTelemetry-compliant and if not - normalizes it as described below. If used in combination with the OTel-related mappings such as the ones defined in `logs-otel@template`, the resulting document can be queried seamlessly by clients that expect either [ECS](https://www.elastic.co/guide/en/ecs/current/index.html) or OpenTelemetry-[Semantic-Conventions](https://github.com/open-telemetry/semantic-conventions) formats. This processor is in tech preview and is not available in our serverless offering. ## Detecting OpenTelemetry compliance The processor detects OpenTelemetry compliance by checking the following fields: - `resource` exists as a key and the value is a map - `resource` either doesn't contain an `attributes` field, or contains an `attributes` field of type map - `scope` is either missing or a map - `attributes` is either missing or a map - `body` is either missing or a map - `body` either doesn't contain a `text` field, or contains a `text` field of type `String` - `body` either doesn't contain a `structured` field, or contains a `structured` field that is not of type `String` If all of these conditions are met, the document is considered OpenTelemetry-compliant and is not modified by the processor. ## Normalization If the document is not OpenTelemetry-compliant, the processor normalizes it as follows: - Specific ECS fields are renamed to have their corresponding OpenTelemetry Semantic Conventions attribute names. These include the following: | ECS Field | Semantic Conventions Attribute | |-------------|--------------------------------| | `span.id` | `span_id` | | `trace.id` | `trace_id` | | `message` | `body.text` | | `log.level` | `severity_text` | The processor first looks for the nested form of the ECS field and if such does not exist, it looks for a top-level field with the dotted field name. - Other specific ECS fields that describe resources and have corresponding counterparts in the OpenTelemetry Semantic Conventions are moved to the `resource.attributes` map. Fields that are considered resource attributes are such that conform to the following conditions: - They are ECS fields that have corresponding counterparts (either with the same name or with a different name) in OpenTelemetry Semantic Conventions. - The corresponding OpenTelemetry attribute is defined in [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/model) within a group that is defined as `type: entity`. - All other fields, except for `@timestamp`, are moved to the `attributes` map. - All non-array entries of the `attributes` and `resource.attributes` maps are flattened. Flattening means that nested objects are merged into their parent object, and the keys are concatenated with a dot. See examples below. ## Examples If an OpenTelemetry-compliant document is detected, the processor does nothing. For example, the following document will stay unchanged: ```js { "resource": { "attributes": { "service.name": "my-service" } }, "scope": { "name": "my-library", "version": "1.0.0" }, "attributes": { "http.method": "GET" }, "body": { "text": "Hello, world!" } } ``` If a non-OpenTelemetry-compliant document is detected, the processor normalizes it. For example, the following document: ```js { "@timestamp": "2023-10-01T12:00:00Z", "service": { "name": "my-service", "version": "1.0.0", "environment": "production", "language": { "name": "python", "version": "3.8" } }, "log": { "level": "INFO" }, "message": "Hello, world!", "http": { "method": "GET", "url": { "path": "/api/v1/resource" }, "headers": [ { "name": "Authorization", "value": "Bearer token" }, { "name": "User-Agent", "value": "my-client/1.0" } ] }, "span" : { "id": "1234567890abcdef" }, "span.id": "abcdef1234567890", "trace.id": "abcdef1234567890abcdef1234567890" } ``` will be normalized into the following form: ```js { "@timestamp": "2023-10-01T12:00:00Z", "resource": { "attributes": { "service.name": "my-service", "service.version": "1.0.0", "service.environment": "production" } }, "attributes": { "service.language.name": "python", "service.language.version": "3.8", "http.method": "GET", "http.url.path": "/api/v1/resource", "http.headers": [ { "name": "Authorization", "value": "Bearer token" }, { "name": "User-Agent", "value": "my-client/1.0" } ] }, "severity_text": "INFO", "body": { "text": "Hello, world!" }, "span_id": "1234567890abcdef", "trace_id": "abcdef1234567890abcdef1234567890" } ``` ## Structured `message` field If the `message` field in the ingested document is structured as a JSON, the processor will determine whether it is in ECS format or not, based on the existence or absence of the `@timestamp` field. If the `@timestamp` field is present, the `message` field will be considered to be in ECS format, and its contents will be merged into the root of the document and then normalized as described above. The `@timestamp` from the `message` field will override the root `@timestamp` field in the resulting document. If the `@timestamp` field is absent, the `message` field will be moved to the `body.structured` field as is, without any further normalization. For example, if the `message` field is an ECS-JSON, as follows: ```js { "@timestamp": "2023-10-01T12:00:00Z", "message": "{\"@timestamp\":\"2023-10-01T12:01:00Z\",\"log.level\":\"INFO\",\"service.name\":\"my-service\",\"message\":\"The actual log message\",\"http\":{\"method\":\"GET\",\"url\":{\"path\":\"/api/v1/resource\"}}}" } ``` it will be normalized into the following form: ```js { "@timestamp": "2023-10-01T12:01:00Z", "severity_text": "INFO", "body": { "text": "The actual log message" }, "resource": { "attributes": { "service.name": "my-service" } }, "attributes": { "http.method": "GET", "http.url.path": "/api/v1/resource" } } ``` However, if the `message` field is not recognized as ECS format, as follows: ```js { "@timestamp": "2023-10-01T12:00:00Z", "log": { "level": "INFO" }, "service": { "name": "my-service" }, "tags": ["user-action", "api-call"], "message": "{\"root_cause\":\"Network error\",\"http\":{\"method\":\"GET\",\"url\":{\"path\":\"/api/v1/resource\"}}}" } ``` it will be normalized into the following form: ```js { "@timestamp": "2023-10-01T12:00:00Z", "severity_text": "INFO", "resource": { "attributes": { "service.name": "my-service" } }, "attributes": { "tags": ["user-action", "api-call"] }, "body": { "structured": { "root_cause": "Network error", "http": { "method": "GET", "url": { "path": "/api/v1/resource" } } } } } ```