--- title: Manage your data quality with Streams description: Monitor Streams data quality by tracking degraded documents, failed documents, quality scores, and ingestion trends over time. url: https://www.elastic.co/elastic/docs-builder/docs/3717/solutions/observability/streams/manage-data-quality products: - Elastic Cloud Enterprise - Elastic Cloud Hosted - Elastic Cloud Serverless - Elastic Cloud on Kubernetes - Elastic Observability - Elastic Stack - Elasticsearch - Kibana applies_to: - Elastic Cloud Serverless: Preview - Elastic Stack: Generally available since 9.2, Preview in 9.1 --- # Manage your data quality with Streams When documents fail during ingestion, you can lose data without knowing what went wrong. Streams preserves failed documents and gives you the tools to find, understand, and fix data quality issues without leaving the Streams UI. The Streams **Data quality** tab provides a single place to monitor and resolve data quality issues: - **Failed documents are preserved, not dropped**: When a processing error occurs, data lands in a [failure store](#streams-data-quality-failure) instead of being lost, so nothing is silently discarded. - **No separate infrastructure needed**. - **See exactly what is failing and why**: The **Data quality** tab shows failure counts, error types, and sample messages, so you can identify problems immediately rather than searching for them. - **Fix issues against the actual failing documents**: Instead of re-ingesting data from the source, you iterate on the processor in the **Processing** tab using the exact documents that failed, with real-time validation before deploying the fix. ## Find and fix data quality issues Use Streams to identify failing documents, trace the source of failure, and deploy a fix. 1. Open **Streams** from the navigation menu or use the [global search field](https://www.elastic.co/elastic/docs-builder/docs/3717/explore-analyze/find-and-organize/find-apps-and-objects). 2. Select your stream from the list. 3. Go to the **Data quality** tab. A stream's data quality (**Good**, **Degraded**, or **Poor**) is displayed at the top of each stream's page so you can assess health at a glance. Streams uses the following ES|QL queries to calculate the quality score for a stream: - All documents (including failed documents): `FROM , ::failures | STATS doc_count = COUNT(*)` - Failed documents only: `FROM ::failures | STATS failed_doc_count = COUNT(*)` - Degraded documents: `FROM METADATA _ignored | WHERE _ignored IS NOT NULL | STATS degraded_doc_count = COUNT(*)` Streams calculates data quality as follows: - **Good:** Both the **Degraded documents** percentage and the **Failed documents** percentage are 0. - **Degraded:** Either the **Degraded documents** percentage or the **Failed documents** percentage is greater than 0 and less than or equal to 3. - **Poor:** Either the **Degraded documents** percentage or the **Failed documents** percentage is greater than 3. The **Data quality** tab shows a breakdown of what's wrong and when it started: - **Failed documents**: Documents rejected during ingestion because of mapping conflicts or pipeline failures. Failed documents land in the [failure store](#streams-data-quality-failure) rather than being dropped. - **Degraded documents**: Documents that were ingested but with fields silently ignored because of mapping issues or values that exceeded configured limits. - **Quality score**: An overall health rating based on the percentage of failed and degraded documents. - **Trends over time**: A time-series chart showing when the problem started and whether it's getting worse. Use the chart to understand the scope of the issue before drilling into individual documents.To get notified when degraded or failed document percentages exceed a threshold, [create a data quality alert](#streams-data-quality-alert). Navigate to the **Processing** tab to edit the failing processor. The **Data preview** pane lets you test your changes against the actual documents that failed—so you can confirm the fix works against real data before deploying it.When the preview looks right, select **Save changes**. Streams applies the updated pipeline to all future incoming documents.Refer to [Process your documents](https://www.elastic.co/elastic/docs-builder/docs/3717/solutions/observability/streams/parse-and-process) for detailed guidance on editing processors and using the data preview. ## Failure store A [failure store](https://www.elastic.co/elastic/docs-builder/docs/3717/manage-data/data-store/data-streams/failure-store) is a secondary set of indices inside a data stream, dedicated to storing failed documents. Instead of losing documents that are rejected during ingestion, a failure store retains them in a `::failures` index, so you can review failed documents to understand what went wrong and how to fix it. For example, for a stream called `my-stream`, the failure store would be accessible by querying `my-stream::failures`. For more information on data quality, refer to the [data set quality](https://www.elastic.co/elastic/docs-builder/docs/3717/solutions/observability/data-set-quality-monitoring) documentation. ### Required permissions To view and modify failure store in Elastic Stack, you need the following data stream level privileges: - `read_failure_store` - `manage_failure_store` For more information, refer to [Granting privileges for data streams and aliases](https://www.elastic.co/elastic/docs-builder/docs/3717/deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases). ### Enable the failure store When the failure store is off, the **Failed documents** component shows an **Enable failure store** link. Select the link and set a retention period for your failure store. ## Create a data quality alert To get notified when the percentage of degraded documents in a stream exceeds a threshold, create an alert rule from the **Data quality** tab. 1. Open the **Data quality** tab for the stream you want to monitor. 2. Select **Create rule** `bell`. 3. [Define the conditions](/elastic/docs-builder/docs/3717/solutions/observability/incident-management/create-a-degraded-docs-rule#degraded-docs-rule-conditions) for your rule.