Using Logstash with Elastic integrations

You can take advantage of the extensive, built-in capabilities of Elastic Integrations--such as managing data collection, transformation, and visualization—​and then use Logstash for additional data processing and output options. Logstash can further expand capabilities for use cases where you need additional processing, or if you need your data delivered to multiple destinations.

Elastic Integrations provide quick, end-to-end solutions for:

• ingesting data from a variety of data sources,
• ensuring compliance with the [Elastic Common Schema (ECS)]Elastic Common Schema (ECS)),
• getting the data into the Elastic Stack, and
• visualizing it with purpose-built dashboards.

Integrations are available for popular services and platforms, such as Nginx, AWS, and MongoDB, as well as many generic input types like log files. Each integration includes pre-packaged assets to help reduce the time between ingest and insights.

To see available integrations, go to the Kibana home page, and click Add Integrations. You can use the query bar to search for integrations you may want to use. When you find an integration for your data source, the UI walks you through adding and configuring it.

Logstash can run the ingest pipeline component of your Elastic integration when you use the Logstash filter-elastic_integration plugin in your Logstash pipeline.

Adding the filter-elastic_integration plugin as the first filter plugin keeps the pipeline’s behavior as close as possible to the behavior you’d expect if the bytes were processed by the integration in Elasticsearch. The more you modify an event before calling the elastic_integration filter, the higher the risk that the modifications will have meaningful effect in how the event is transformed.

Sample pipeline configuration

input {
  elastic_agent {
    port => 5044
  }
}

filter {
  elastic_integration{ 1
    cloud_id => "<cloud id>"
    cloud_auth => "<your_cloud-auth"
  }

  translate { 2
    source => "[http][host]"
    target => "[@metadata][tenant]"
    dictionary_path => "/etc/conf.d/logstash/tenants.yml"
  }
}

output { 3
  if [@metadata][tenant] == "tenant01" {
    elasticsearch {
      cloud_id => "<cloud id>"
      api_key => "<api key>"
    }
  } else if [@metadata][tenant] == "tenant02" {
    elasticsearch {
      cloud_id => "<cloud id>"
      api_key => "<api key>"
    }
  }
}

1. Use filter-elastic_integration as the first filter in your pipeline
2. You can use additional filters as long as they follow filter-elastic_integration
3. Sample config to output data to multiple destinations

Elastic Integrations are designed to work with data streams and ECS-compatible output. Be sure that these features are enabled in the output-elasticsearch plugin.

• Set data-stream to true.
  (Check out Data streams for additional data streams settings.)
• Set ecs-compatibility to v1 or v8.

Check out the output-elasticsearch plugin docs for additional settings.