--- title: ECS Logging with Morgan description: This Node.js package provides a formatter for the morgan logging middleware—commonly used with Express—compatible with Elastic Common Schema (ECS) logging... url: https://www.elastic.co/elastic/docs-builder/docs/3016/reference/ecs/logging/nodejs/morgan products: - ECS Logging - ECS Logging Node.js --- # ECS Logging with Morgan This Node.js package provides a formatter for the [morgan](https://github.com/expressjs/morgan#readme) logging middleware—commonly used with Express—compatible with [Elastic Common Schema (ECS) logging](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/ecs/logging/intro). In combination with the [Filebeat](https://www.elastic.co/beats/filebeat) shipper, you can [monitor all your logs](https://www.elastic.co/log-monitoring) in one place in the Elastic Stack. ## Setup ### Step 1: Install ```cmd $ npm install @elastic/ecs-morgan-format ``` ### Step 2: Configure ```js const app = require('express')(); const morgan = require('morgan'); const { ecsFormat } = require('@elastic/ecs-morgan-format'); app.use(morgan(ecsFormat(/* options */))); // ... app.get('/', function (req, res) { res.send('hello, world!'); }) app.listen(3000); ``` ### Step 3: Configure Filebeat The best way to collect the logs once they are ECS-formatted is with [Filebeat](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat): 1. Follow the [Filebeat quick start](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat/filebeat-installation-configuration) 2. Add the following configuration to your `filebeat.yaml` file. For Filebeat 7.16+ ```yaml filebeat.inputs: - type: filestream paths: /path/to/logs.json parsers: - ndjson: overwrite_keys: true add_error_key: true expand_keys: true processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~ ``` For Filebeat < 7.16 ```yaml filebeat.inputs: - type: log paths: /path/to/logs.json json.keys_under_root: true json.overwrite_keys: true json.add_error_key: true json.expand_keys: true processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~ ``` 1. Make sure your application logs to stdout/stderr. 2. Follow the [Run Filebeat on Kubernetes](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat/running-on-kubernetes) guide. 3. Enable [hints-based autodiscover](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat/configuration-autodiscover-hints) (uncomment the corresponding section in `filebeat-kubernetes.yaml`). 4. Add these annotations to your pods that log using ECS loggers. This will make sure the logs are parsed appropriately. ```yaml annotations: co.elastic.logs/json.overwrite_keys: true co.elastic.logs/json.add_error_key: true co.elastic.logs/json.expand_keys: true ``` 1. Make sure your application logs to stdout/stderr. 2. Follow the [Run Filebeat on Docker](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat/running-on-docker) guide. 3. Enable [hints-based autodiscover](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat/configuration-autodiscover-hints). 4. Add these labels to your containers that log using ECS loggers. This will make sure the logs are parsed appropriately. ```yaml labels: co.elastic.logs/json.overwrite_keys: true co.elastic.logs/json.add_error_key: true co.elastic.logs/json.expand_keys: true ``` For more information, see the [Filebeat reference](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/beats/filebeat/configuring-howto-filebeat). ## Usage ```js const app = require('express')(); const morgan = require('morgan'); const { ecsFormat } = require('@elastic/ecs-morgan-format'); app.use(morgan(ecsFormat(/* options */))); app.get('/', function (req, res) { res.send('hello, world!'); }) app.get('/error', function (req, res, next) { next(new Error('boom')); }) app.listen(3000) ``` Running this script (the full example is [here](https://github.com/elastic/ecs-logging-nodejs/blob/main/packages/ecs-morgan-format/examples/express.js)) and making a request (via `curl -i localhost:3000/`) will produce log output similar to the following: ```cmd % node examples/express.js | jq . { "@timestamp": "2021-01-16T00:03:23.279Z", "log.level": "info", "message": "::1 - - [16/Jan/2021:00:03:23 +0000] \"GET / HTTP/1.1\" 200 13 \"-\" \"curl/7.64.1\"", "ecs.version": "8.10.0", "http": { "version": "1.1", "request": { "method": "GET", "headers": { "host": "localhost:3000", "accept": "*/*" } }, "response": { "status_code": 200, "headers": { "x-powered-by": "Express", "content-type": "text/html; charset=utf-8", "etag": "W/\"d-HwnTDHB9U/PRbFMN1z1wps51lqk\"" }, "body": { "bytes": 13 } } }, "url": { "path": "/", "domain": "localhost", "full": "http://localhost:3000/" }, "user_agent": { "original": "curl/7.64.1" } } ``` ## Format options You can pass any [`format` argument](https://github.com/expressjs/morgan#morganformat-options) you would normally pass to `morgan()`, and the log "message" field will use the specified format. The default is [`combined`](https://github.com/expressjs/morgan#combined). ```js const app = require('express')(); const morgan = require('morgan'); const { ecsFormat } = require('@elastic/ecs-morgan-format'); app.use(morgan(ecsFormat({ format: 'tiny' }))); // ... ``` ## log.level The `log.level` field will be "error" for response codes >= 500, otherwise "info". For example, running [examples/express.js](https://github.com/elastic/ecs-logging-nodejs/blob/main/packages/ecs-morgan-format/examples/express.js) again, a `curl -i localhost:3000/error` will yield: ```cmd % node examples/express.js | jq . { "@timestamp": "2021-01-18T17:52:12.810Z", "log.level": "error", "message": "::1 - - [18/Jan/2021:17:52:12 +0000] \"GET /error HTTP/1.1\" 500 1416 \"-\" \"curl/7.64.1\"", "http": { "response": { "status_code": 500, ... ``` ## Log correlation with APM This ECS log formatter integrates with [Elastic APM](https://www.elastic.co/apm). If your Node app is using the [Node.js Elastic APM Agent](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/apm/agents/nodejs), then a number of fields are added to log records to correlate between APM services or traces and logging data: - Log statements (e.g. `logger.info(...)`) called when there is a current tracing span will include [tracing fields](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/reference/ecs/ecs-tracing)—`trace.id`, `transaction.id`. - A number of service identifier fields determined by or configured on the APM agent allow cross-linking between services and logs in Kibana—`service.name`, `service.version`, `service.environment`, `service.node.name`. - `event.dataset` enables [log rate anomaly detection](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/solutions/observability/logs/inspect-log-anomalies) in the Elastic Observability app. For example, running [examples/express-with-apm.js](https://github.com/elastic/ecs-logging-nodejs/blob/main/packages/ecs-morgan-format/examples/express-with-apm.js) and `curl -i localhost:3000/` results in a log record with the following: ```cmd % node examples/express-with-apm.js | jq . { // The same fields as before, plus: "service.name": "express-with-elastic-apm", "service.version": "1.1.0", "service.environment": "development", "event.dataset": "express-with-elastic-apm", "trace.id": "116d46f667a7600deed9c41fa015f7de", "transaction.id": "b84fb72d7bf42866" } ``` These IDs match trace data reported by the APM agent. Integration with Elastic APM can be explicitly disabled via the `apmIntegration: false` option, for example: ```js app.use(morgan(ecsFormat({ apmIntegration: false }))); ``` ## Reference ### `ecsFormat([options])` - `options` `{type-object}` The following options are supported: - `format` `{type-string}` A format **name** (e.g. *combined*), format function (e.g. `morgan.combined`), or a format string (e.g. *:method :url :status*). This is used to format the "message" field. Defaults to `morgan.combined`. - `convertErr` `{type-boolean}` Whether to convert a logged `err` field to ECS error fields. **Default:** `true`. - `apmIntegration` `{type-boolean}` Whether to enable APM agent integration. **Default:** `true`. - `serviceName` `{type-string}` A "service.name" value. If specified this overrides any value from an active APM agent. - `serviceVersion` `{type-string}` A "service.version" value. If specified this overrides any value from an active APM agent. - `serviceEnvironment` `{type-string}` A "service.environment" value. If specified this overrides any value from an active APM agent. - `serviceNodeName` `{type-string}` A "service.node.name" value. If specified this overrides any value from an active APM agent. - `eventDataset` `{type-string}` A "event.dataset" value. If specified this overrides the default of using `${serviceVersion}`. Create a formatter for morgan that emits in ECS Logging format.