ECS Logging with Morgan

This Node.js package provides a formatter for the morgan logging middleware — commonly used with Express — compatible with Elastic Common Schema (ECS) logging. In combination with the Filebeat shipper, you can monitor all your logs in one place in the Elastic Stack.

$ npm install @elastic/ecs-morgan-format

const app = require('express')();
const morgan = require('morgan');
const { ecsFormat } = require('@elastic/ecs-morgan-format');

app.use(morgan(ecsFormat(/* options */))); 1

// ...
app.get('/', function (req, res) {
  res.send('hello, world!');
})
app.listen(3000);

1. Pass the ECS formatter to morgan().

The best way to collect the logs once they are ECS-formatted is with Filebeat:

filebeat.inputs:
- type: filestream 1
  paths: /path/to/logs.json
  parsers:
    - ndjson:
      overwrite_keys: true 2
      add_error_key: true 3
      expand_keys: true 4

processors: 5
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

filebeat.inputs:
- type: log
  paths: /path/to/logs.json
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.expand_keys: true

processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~

annotations:
  co.elastic.logs/json.overwrite_keys: true 1
  co.elastic.logs/json.add_error_key: true 2
  co.elastic.logs/json.expand_keys: true 3

labels:
  co.elastic.logs/json.overwrite_keys: true 1
  co.elastic.logs/json.add_error_key: true 2
  co.elastic.logs/json.expand_keys: true 3

For more information, see the Filebeat reference.

const app = require('express')();
const morgan = require('morgan');
const { ecsFormat } = require('@elastic/ecs-morgan-format');

app.use(morgan(ecsFormat(/* options */))); 1

app.get('/', function (req, res) {
  res.send('hello, world!');
})
app.get('/error', function (req, res, next) {
  next(new Error('boom'));
})

app.listen(3000)

1. See available options below.

Running this script (the full example is here) and making a request (via curl -i localhost:3000/) will produce log output similar to the following:

% node examples/express.js | jq . 1
{
  "@timestamp": "2021-01-16T00:03:23.279Z",
  "log.level": "info",
  "message": "::1 - - [16/Jan/2021:00:03:23 +0000] \"GET / HTTP/1.1\" 200 13 \"-\" \"curl/7.64.1\"",
  "ecs.version": "8.10.0",
  "http": {
    "version": "1.1",
    "request": {
      "method": "GET",
      "headers": {
        "host": "localhost:3000",
        "accept": "*/*"
      }
    },
    "response": {
      "status_code": 200,
      "headers": {
        "x-powered-by": "Express",
        "content-type": "text/html; charset=utf-8",
        "etag": "W/\"d-HwnTDHB9U/PRbFMN1z1wps51lqk\""
      },
      "body": {
        "bytes": 13
      }
    }
  },
  "url": {
    "path": "/",
    "domain": "localhost",
    "full": "http://localhost:3000/"
  },
  "user_agent": {
    "original": "curl/7.64.1"
  }
}

1. piping to jq for pretty-printing

You can pass any format argument you would normally pass to morgan(), and the log "message" field will use the specified format. The default is combined.

const app = require('express')();
const morgan = require('morgan');
const { ecsFormat } = require('@elastic/ecs-morgan-format');

app.use(morgan(ecsFormat({ format: 'tiny' }))); 1
// ...

1. If "format" is the only option you are using, you may pass it as ecsFormat('tiny').

The log.level field will be "error" for response codes >= 500, otherwise "info". For example, running examples/express.js again, a curl -i localhost:3000/error will yield:

% node examples/express.js | jq .
{
  "@timestamp": "2021-01-18T17:52:12.810Z",
  "log.level": "error",
  "message": "::1 - - [18/Jan/2021:17:52:12 +0000] \"GET /error HTTP/1.1\" 500 1416 \"-\" \"curl/7.64.1\"",
  "http": {
    "response": {
      "status_code": 500,
  ...

This ECS log formatter integrates with Elastic APM. If your Node app is using the Node.js Elastic APM Agent, then a number of fields are added to log records to correlate between APM services or traces and logging data:

• Log statements (e.g. logger.info(...)) called when there is a current tracing span will include tracing fields — trace.id, transaction.id.
• A number of service identifier fields determined by or configured on the APM agent allow cross-linking between services and logs in Kibana — service.name, service.version, service.environment, service.node.name.
• event.dataset enables log rate anomaly detection in the Elastic Observability app.

For example, running examples/express-with-apm.js and curl -i localhost:3000/ results in a log record with the following:

% node examples/express-with-apm.js | jq .
{
  // The same fields as before, plus:
  "service.name": "express-with-elastic-apm",
  "service.version": "1.1.0",
  "service.environment": "development",
  "event.dataset": "express-with-elastic-apm",
  "trace.id": "116d46f667a7600deed9c41fa015f7de",
  "transaction.id": "b84fb72d7bf42866"
}

These IDs match trace data reported by the APM agent.

Integration with Elastic APM can be explicitly disabled via the apmIntegration: false option, for example:

app.use(morgan(ecsFormat({ apmIntegration: false })));

• options {type-object} The following options are supported:

  • format {type-string} A format name (e.g. combined), format function (e.g. morgan.combined), or a format string (e.g. :method :url :status). This is used to format the "message" field. Defaults to morgan.combined.
  • convertErr {type-boolean} Whether to convert a logged err field to ECS error fields. Default: true.
  • apmIntegration {type-boolean} Whether to enable APM agent integration. Default: true.
  • serviceName {type-string} A "service.name" value. If specified this overrides any value from an active APM agent.
  • serviceVersion {type-string} A "service.version" value. If specified this overrides any value from an active APM agent.
  • serviceEnvironment {type-string} A "service.environment" value. If specified this overrides any value from an active APM agent.
  • serviceNodeName {type-string} A "service.node.name" value. If specified this overrides any value from an active APM agent.
  • eventDataset {type-string} A "event.dataset" value. If specified this overrides the default of using ${serviceVersion}.

Create a formatter for morgan that emits in ECS Logging format.