ECS Logging with Pino

This Node.js package provides a formatter for the pino logger, compatible with Elastic Common Schema (ECS) logging. In combination with the Filebeat shipper, you can monitor all your logs in one place in the Elastic Stack. pino 6.x, 7.x, and 8.x versions are supported.

$ npm install @elastic/ecs-pino-format

const { ecsFormat } = require('@elastic/ecs-pino-format');
const pino = require('pino');

const log = pino(ecsFormat(/* options */)); 1
log.info('hi');
log.error({ err: new Error('boom') }, 'oops there is a problem');
// ...

1. This will configure Pino’s formatters, messageKey and timestamp options.

See usage discussion and examples below.

The best way to collect the logs once they are ECS-formatted is with Filebeat:

filebeat.inputs:
- type: filestream 1
  paths: /path/to/logs.json
  parsers:
    - ndjson:
      overwrite_keys: true 2
      add_error_key: true 3
      expand_keys: true 4

processors: 5
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

filebeat.inputs:
- type: log
  paths: /path/to/logs.json
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.expand_keys: true

processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~

annotations:
  co.elastic.logs/json.overwrite_keys: true 1
  co.elastic.logs/json.add_error_key: true 2
  co.elastic.logs/json.expand_keys: true 3

labels:
  co.elastic.logs/json.overwrite_keys: true 1
  co.elastic.logs/json.add_error_key: true 2
  co.elastic.logs/json.expand_keys: true 3

For more information, see the Filebeat reference.

const { ecsFormat } = require('@elastic/ecs-pino-format');
const pino = require('pino');

const log = pino(ecsFormat(/* options */)); 1
log.info('Hello world');

const child = log.child({ module: 'foo' });
child.warn('From child');

1. See available options below.

Running this will produce log output similar to the following:

{"log.level":"info","@timestamp":"2023-10-14T02:07:47.901Z","process.pid":56645,"host.hostname":"pink.local","ecs.version":"8.10.0","message":"Hello world"}
{"log.level":"warn","@timestamp":"2023-10-14T02:07:47.901Z","process.pid":56645,"host.hostname":"pink.local","ecs.version":"8.10.0","module":"foo","message":"From child"}

By default, the formatter will convert an err field that is an Error instance to ECS Error fields. For example:

const { ecsFormat } = require('@elastic/ecs-pino-format');
const pino = require('pino');
const log = pino(ecsFormat());

const myErr = new Error('boom');
log.info({ err: myErr }, 'oops');

will yield (pretty-printed for readability):

% node examples/error.js | jq .
{
  "log.level": "info",
  "@timestamp": "2021-01-26T17:02:23.697Z",
  ...
  "error": {
    "type": "Error",
    "message": "boom",
    "stack_trace": "Error: boom\n    at Object.<anonymous> (..."
  },
  "message": "oops"
}

This is analogous to and overrides Pino’s default err serializer. Special handling of the err field can be disabled via the convertErr: false option:

const log = pino(ecsFormat({ convertErr: false }));

With the convertReqRes: true option, the formatter will automatically convert Node.js core request and response objects when passed as the req and res fields, respectively. (This option replaces the usage of req and res Pino serializers.)

const http = require('http');
const { ecsFormat } = require('@elastic/ecs-pino-format');
const pino = require('pino');

const log = pino(ecsFormat({ convertReqRes: true })); 1

const server = http.createServer(function handler (req, res) {
  res.setHeader('Foo', 'Bar');
  res.end('ok');
  log.info({ req, res }, 'handled request'); 2
});

server.listen(3000, () => {
  log.info('listening at http://localhost:3000');
}

1. use convertReqRes option
2. log with req and/or res fields

This will produce logs with request and response info using ECS HTTP fields. For example:

% node examples/http.js | jq .   1
...                               # run 'curl http://localhost:3000/'
{
  "log.level": "info",
  "@timestamp": "2023-10-14T02:10:14.477Z",
  "process.pid": 56697,
  "host.hostname": "pink.local",
  "ecs.version": "8.10.0",
  "http": {
    "version": "1.1",
    "request": {
      "method": "GET",
      "headers": {
        "host": "localhost:3000",
        "user-agent": "curl/8.1.2",
        "accept": "*/*"
      }
    },
    "response": {
      "status_code": 200,
      "headers": {
        "foo": "Bar"
      }
    }
  },
  "url": {
    "full": "http://localhost:3000/",
    "path": "/"
  },
  "client": {
    "address": "::ffff:127.0.0.1",
    "ip": "::ffff:127.0.0.1",
    "port": 49504
  },
  "user_agent": {
    "original": "curl/8.1.2"
  },
  "message": "handled request"
}

1. using jq for pretty printing

The examples/ directory shows sample programs using request and response logging: with Express, with the pino-http middleware package, etc.

This ECS log formatter integrates with Elastic APM. If your Node app is using the Node.js Elastic APM Agent, then a number of fields are added to log records to correlate between APM services or traces and logging data:

• Log statements (e.g. logger.info(...)) called when there is a current tracing span will include tracing fields — trace.id, transaction.id, span.id.
• A number of service identifier fields determined by or configured on the APM agent allow cross-linking between services and logs in Kibana — service.name, service.version, service.environment, service.node.name.
• event.dataset enables log rate anomaly detection in the Elastic Observability app.

For example, running examples/http-with-elastic-apm.js and curl -i localhost:3000/ results in a log record with the following:

% node examples/http-with-elastic-apm.js | jq .
...
  "service.name": "http-with-elastic-apm",
  "service.version": "1.4.0",
  "service.environment": "development",
  "event.dataset": "http-with-elastic-apm",
  "trace.id": "9f338eae7211b7993b98929046aed21d",
  "transaction.id": "2afbef5642cc7a3f",
...

These IDs match trace data reported by the APM agent.

Integration with Elastic APM can be explicitly disabled via the apmIntegration: false option, for example:

const log = pino(ecsFormat({ apmIntegration: false }));

The ecs-logging spec suggests that the first three fields in log records must be @timestamp, log.level, and message. Pino does not provide a mechanism to put the message field near the front. Given that ordering of ecs-logging fields is for human readability and does not affect interoperability, this is not considered a significant concern.

The hooks that Pino currently provides do not enable this package to convert fields passed to <logger>.child({ ... }). This means that, even with the convertReqRes option, a call to <logger>.child({ req }) will not convert that req to ECS HTTP fields. This is a slight limitation for users of pino-http which does this.

• options {{type-object}} The following options are supported:

  • convertErr {{type-boolean}} Whether to convert a logged err field to ECS error fields. Default: true.
  • convertReqRes {{type-boolean}} Whether to logged req and res HTTP request and response fields to ECS HTTP, User agent, and URL fields. Default: false.
  • apmIntegration {{type-boolean}} Whether to enable APM agent integration. Default: true.
  • serviceName {{type-string}} A "service.name" value. If specified this overrides any value from an active APM agent.
  • serviceVersion {{type-string}} A "service.version" value. If specified this overrides any value from an active APM agent.
  • serviceEnvironment {{type-string}} A "service.environment" value. If specified this overrides any value from an active APM agent.
  • serviceNodeName {{type-string}} A "service.node.name" value. If specified this overrides any value from an active APM agent.
  • eventDataset {{type-string}} A "event.dataset" value. If specified this overrides the default of using ${serviceVersion}.

Create options for pino(...) that configures ECS Logging format output.