--- title: ECS Logging with Pino description: This Node.js package provides a formatter for the pino logger, compatible with Elastic Common Schema (ECS) logging. In combination with the Filebeat shipper,... url: https://www.elastic.co/elastic/docs-builder/docs/3016/reference/ecs/logging/nodejs/pino products: - ECS Logging - ECS Logging Node.js --- # ECS Logging with Pino This Node.js package provides a formatter for the [pino](https://getpino.io) logger, compatible with [Elastic Common Schema (ECS) logging](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/ecs/logging/intro). In combination with the [Filebeat](https://www.elastic.co/beats/filebeat) shipper, you can [monitor all your logs](https://www.elastic.co/log-monitoring) in one place in the Elastic Stack. `pino` 6.x, 7.x, and 8.x versions are supported. ## Setup ### Step 1: Install ```cmd $ npm install @elastic/ecs-pino-format ``` ### Step 2: Configure ```js const { ecsFormat } = require('@elastic/ecs-pino-format'); const pino = require('pino'); const log = pino(ecsFormat(/* options */)); log.info('hi'); log.error({ err: new Error('boom') }, 'oops there is a problem'); // ... ``` See usage discussion and examples below. ### Step 3: Configure Filebeat The best way to collect the logs once they are ECS-formatted is with [Filebeat](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat): 1. Follow the [Filebeat quick start](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat/filebeat-installation-configuration) 2. Add the following configuration to your `filebeat.yaml` file. For Filebeat 7.16+ ```yaml filebeat.inputs: - type: filestream paths: /path/to/logs.json parsers: - ndjson: overwrite_keys: true add_error_key: true expand_keys: true processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~ ``` For Filebeat < 7.16 ```yaml filebeat.inputs: - type: log paths: /path/to/logs.json json.keys_under_root: true json.overwrite_keys: true json.add_error_key: true json.expand_keys: true processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~ ``` 1. Make sure your application logs to stdout/stderr. 2. Follow the [Run Filebeat on Kubernetes](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat/running-on-kubernetes) guide. 3. Enable [hints-based autodiscover](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat/configuration-autodiscover-hints) (uncomment the corresponding section in `filebeat-kubernetes.yaml`). 4. Add these annotations to your pods that log using ECS loggers. This will make sure the logs are parsed appropriately. ```yaml annotations: co.elastic.logs/json.overwrite_keys: true co.elastic.logs/json.add_error_key: true co.elastic.logs/json.expand_keys: true ``` 1. Make sure your application logs to stdout/stderr. 2. Follow the [Run Filebeat on Docker](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat/running-on-docker) guide. 3. Enable [hints-based autodiscover](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat/configuration-autodiscover-hints). 4. Add these labels to your containers that log using ECS loggers. This will make sure the logs are parsed appropriately. ```yaml labels: co.elastic.logs/json.overwrite_keys: true co.elastic.logs/json.add_error_key: true co.elastic.logs/json.expand_keys: true ``` For more information, see the [Filebeat reference](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat/configuring-howto-filebeat). ## Usage ```js const { ecsFormat } = require('@elastic/ecs-pino-format'); const pino = require('pino'); const log = pino(ecsFormat(/* options */)); log.info('Hello world'); const child = log.child({ module: 'foo' }); child.warn('From child'); ``` Running this will produce log output similar to the following: ```cmd {"log.level":"info","@timestamp":"2023-10-14T02:07:47.901Z","process.pid":56645,"host.hostname":"pink.local","ecs.version":"8.10.0","message":"Hello world"} {"log.level":"warn","@timestamp":"2023-10-14T02:07:47.901Z","process.pid":56645,"host.hostname":"pink.local","ecs.version":"8.10.0","module":"foo","message":"From child"} ``` ## Error Logging By default, the formatter will convert an `err` field that is an Error instance to [ECS Error fields](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/ecs/ecs-error). For example: ```js const { ecsFormat } = require('@elastic/ecs-pino-format'); const pino = require('pino'); const log = pino(ecsFormat()); const myErr = new Error('boom'); log.info({ err: myErr }, 'oops'); ``` will yield (pretty-printed for readability): ```cmd % node examples/error.js | jq . { "log.level": "info", "@timestamp": "2021-01-26T17:02:23.697Z", ... "error": { "type": "Error", "message": "boom", "stack_trace": "Error: boom\n at Object. (..." }, "message": "oops" } ``` This is analogous to and overrides [Pino’s default err serializer](https://getpino.io/#/docs/api?id=serializers-object). Special handling of the `err` field can be disabled via the `convertErr: false` option: ```js const log = pino(ecsFormat({ convertErr: false })); ``` ## HTTP Request and Response Logging With the `convertReqRes: true` option, the formatter will automatically convert Node.js core [request](https://nodejs.org/api/http.md#http_class_http_incomingmessage) and [response](https://nodejs.org/api/http.md#http_class_http_serverresponse) objects when passed as the `req` and `res` fields, respectively. (This option replaces the usage of `req` and `res` [Pino serializers](https://getpino.io/#/docs/api?id=pinostdserializers-object).) ```js const http = require('http'); const { ecsFormat } = require('@elastic/ecs-pino-format'); const pino = require('pino'); const log = pino(ecsFormat({ convertReqRes: true })); const server = http.createServer(function handler (req, res) { res.setHeader('Foo', 'Bar'); res.end('ok'); log.info({ req, res }, 'handled request'); }); server.listen(3000, () => { log.info('listening at http://localhost:3000'); } ``` This will produce logs with request and response info using [ECS HTTP fields](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/ecs/ecs-http). For example: ```cmd % node examples/http.js | jq . ... # run 'curl http://localhost:3000/' { "log.level": "info", "@timestamp": "2023-10-14T02:10:14.477Z", "process.pid": 56697, "host.hostname": "pink.local", "ecs.version": "8.10.0", "http": { "version": "1.1", "request": { "method": "GET", "headers": { "host": "localhost:3000", "user-agent": "curl/8.1.2", "accept": "*/*" } }, "response": { "status_code": 200, "headers": { "foo": "Bar" } } }, "url": { "full": "http://localhost:3000/", "path": "/" }, "client": { "address": "::ffff:127.0.0.1", "ip": "::ffff:127.0.0.1", "port": 49504 }, "user_agent": { "original": "curl/8.1.2" }, "message": "handled request" } ``` The [examples/ directory](https://github.com/elastic/ecs-logging-nodejs/tree/main/packages/ecs-pino-format/examples) shows sample programs using request and response logging: [with Express](https://github.com/elastic/ecs-logging-nodejs/tree/main/packages/ecs-pino-format/examples/express-simple.js), [with the pino-http middleware package](https://github.com/elastic/ecs-logging-nodejs/tree/main/packages/ecs-pino-format/examples/express-with-pino-http.js), etc. ## Log Correlation with APM This ECS log formatter integrates with [Elastic APM](https://www.elastic.co/apm). If your Node app is using the [Node.js Elastic APM Agent](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/apm/agents/nodejs), then a number of fields are added to log records to correlate between APM services or traces and logging data: - Log statements (e.g. `logger.info(...)`) called when there is a current tracing span will include [tracing fields](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/ecs/ecs-tracing)—`trace.id`, `transaction.id`, `span.id`. - A number of service identifier fields determined by or configured on the APM agent allow cross-linking between services and logs in Kibana—`service.name`, `service.version`, `service.environment`, `service.node.name`. - `event.dataset` enables [log rate anomaly detection](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/solutions/observability/logs/inspect-log-anomalies) in the Elastic Observability app. For example, running [examples/http-with-elastic-apm.js](https://github.com/elastic/ecs-logging-nodejs/blob/main/packages/ecs-pino-format/examples/http-with-elastic-apm.js) and `curl -i localhost:3000/` results in a log record with the following: ```cmd % node examples/http-with-elastic-apm.js | jq . ... "service.name": "http-with-elastic-apm", "service.version": "1.4.0", "service.environment": "development", "event.dataset": "http-with-elastic-apm", "trace.id": "9f338eae7211b7993b98929046aed21d", "transaction.id": "2afbef5642cc7a3f", ... ``` These IDs match trace data reported by the APM agent. Integration with Elastic APM can be explicitly disabled via the `apmIntegration: false` option, for example: ```js const log = pino(ecsFormat({ apmIntegration: false })); ``` ## Limitations and Considerations The [ecs-logging spec](https://github.com/elastic/ecs-logging/tree/main/spec) suggests that the first three fields in log records must be `@timestamp`, `log.level`, and `message`. Pino does not provide a mechanism to put the `message` field near the front. Given that ordering of ecs-logging fields is for **human readability** and does not affect interoperability, this is not considered a significant concern. The hooks that Pino currently provides do not enable this package to convert fields passed to `.child({ ... })`. This means that, even with the `convertReqRes` option, a call to `.child({ req })` will **not** convert that `req` to ECS HTTP fields. This is a slight limitation for users of [pino-http](https://github.com/pinojs/pino-http) which does this. ## Reference ### `ecsFormat([options])` - `options` `{type-object}` The following options are supported: - `convertErr` `{type-boolean}` Whether to convert a logged `err` field to ECS error fields. **Default:** `true`. - `convertReqRes` `{type-boolean}` Whether to logged `req` and `res` HTTP request and response fields to ECS HTTP, User agent, and URL fields. **Default:** `false`. - `apmIntegration` `{type-boolean}` Whether to enable APM agent integration. **Default:** `true`. - `serviceName` `{type-string}` A "service.name" value. If specified this overrides any value from an active APM agent. - `serviceVersion` `{type-string}` A "service.version" value. If specified this overrides any value from an active APM agent. - `serviceEnvironment` `{type-string}` A "service.environment" value. If specified this overrides any value from an active APM agent. - `serviceNodeName` `{type-string}` A "service.node.name" value. If specified this overrides any value from an active APM agent. - `eventDataset` `{type-string}` A "event.dataset" value. If specified this overrides the default of using `${serviceVersion}`. Create options for `pino(...)` that configures ECS Logging format output.