diff --git a/hugo/content/en/observability_pipelines/_index.md b/hugo/content/en/observability_pipelines/_index.md index 6e021e1fc13..c326e2fd86d 100644 --- a/hugo/content/en/observability_pipelines/_index.md +++ b/hugo/content/en/observability_pipelines/_index.md @@ -1,6 +1,6 @@ --- title: Observability Pipelines -description: Learn how Observability Pipelines lets you collect, process, and route logs, metrics, and traces within your own infrastructure to destinations such as Datadog, Amazon S3, Splunk, and Microsoft Sentinel. +description: Learn how Observability Pipelines lets you collect, process, and route logs and metrics within your own infrastructure to destinations such as Datadog, Amazon S3, Splunk, and Microsoft Sentinel. disable_toc: false further_reading: - link: "/observability_pipelines/configuration/explore_templates/" @@ -78,7 +78,7 @@ further_reading: {{< img src="observability_pipelines/op_marketecture_06042025.png" alt="A graphic showing data being aggregated from a variety of sources, processed and enriched by the observability pipelines worker in your own environment, and then being routed to the security, analytics, and storage destinations of your choice" style="width:100%;" >}} -Datadog Observability Pipelines allows you to collect and process {{< tooltip text="logs, metrics, and traces" tooltip="Contact your account manager to discuss use cases and pricing." >}} within your own infrastructure, and then route the data to different destinations. It gives you control over your observability data before it leaves your environment. +Datadog Observability Pipelines allows you to collect and process logs and metrics within your own infrastructure, and then route the data to different destinations. It gives you control over your observability data before it leaves your environment. With out-of-the-box templates, you can build pipelines that redact sensitive data, enrich data, filter out noisy events, and route data to destinations like Datadog, SIEM tools, or cloud storage. @@ -141,13 +141,6 @@ Observability Pipelines includes prebuilt templates for common data routing and |----------|-------------| | Metric Tag Governance | Manage the quality and volume of your metrics by keeping only the metrics you need, standardizing metrics tagging, and removing unwanted tags to prevent high cardinality. | -{{% /tab %}} -{{% tab "Traces" %}} - -| Template | Description | -|----------|-------------| -| Trace Sampling | Ingest, process, and route traces to control costs while retaining the traces you need for troubleshooting and analysis. | - {{% /tab %}} {{< /tabs >}} diff --git a/hugo/content/en/observability_pipelines/configuration/_index.md b/hugo/content/en/observability_pipelines/configuration/_index.md index c3902df85eb..c5dda65a352 100644 --- a/hugo/content/en/observability_pipelines/configuration/_index.md +++ b/hugo/content/en/observability_pipelines/configuration/_index.md @@ -20,7 +20,7 @@ further_reading: {{< img src="observability_pipelines/setup/pipeline_ui.png" alt="The pipelines page with a source going to two processors groups and two destinations" style="width:100%;" >}} -Observability Pipelines lets you collect and process {{< tooltip text="logs, metrics, and traces" tooltip="Contact your account manager to discuss use cases and pricing." >}} within your own infrastructure, and then route them to different destinations. A pipeline consists of three core components: +Observability Pipelines lets you collect and process logs and metrics within your own infrastructure, and then route them to different destinations. A pipeline consists of three core components: - [Source][1]: Receives data from a tool like the Datadog Agent. - [Processors][2]: Transform, enrich, or filter data. @@ -120,17 +120,6 @@ See [Metric Types][3] for more information. [3]: /metrics/types/?tab=gauge#metric-types [4]: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality -{{% /tab %}} - -{{% tab "Traces" %}} - -You can ingest, process, and send {{< tooltip text="traces" tooltip="Contact your account manager to request access." >}} to different destinations using the [Trace Sampling][1] template. - -See [Set Up Pipelines][2] for more information on setting up a source, processors, and destinations. - -[1]: /observability_pipelines/configuration/explore_templates/?tab=traces#trace-sampling -[2]: /observability_pipelines/configuration/set_up_pipelines/ - {{% /tab %}} {{< /tabs >}} diff --git a/hugo/content/en/observability_pipelines/configuration/explore_templates.md b/hugo/content/en/observability_pipelines/configuration/explore_templates.md index e7abc0d4381..d74a4a696d8 100644 --- a/hugo/content/en/observability_pipelines/configuration/explore_templates.md +++ b/hugo/content/en/observability_pipelines/configuration/explore_templates.md @@ -75,13 +75,6 @@ Metrics capture signals about your environment and offer insight into your syste To help you manage the quality and volume of your metrics, use the Metric Tag Governance template to process them in Observability Pipelines before sending them to your destinations. You can use processors to keep only the metrics you need, standardize metrics tagging, and remove unwanted tags to prevent high cardinality. -{{% /tab %}} -{{% tab "Traces" %}} - -### Trace Sampling - -Use the Trace Sampling template to ingest, process, and route your {{< tooltip text="traces" tooltip="Contact your account manager to request access." >}} telemetry data to control costs while retaining the traces you need for troubleshooting and analysis. - {{% /tab %}} {{< /tabs >}} diff --git a/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md b/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md index 689761be6ba..da1116bc773 100644 --- a/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md +++ b/hugo/content/en/observability_pipelines/configuration/install_the_worker/_index.mdoc.md @@ -46,7 +46,7 @@ For RHEL and CentOS, the Observability Pipelines Worker supports versions 8.0 or {% /if %} -The Observability Pipelines Worker is software that runs in your environment to centrally aggregate and process your {% tooltip contents="Contact your account manager to discuss use cases and pricing." %}logs, metrics, and traces{% /tooltip %}, and then route them to different destinations. +The Observability Pipelines Worker is software that runs in your environment to centrally aggregate and process your logs and metrics, and then route them to different destinations. {% if equals($platform, "kubernetes") %} diff --git a/hugo/content/en/observability_pipelines/configuration/live_capture.md b/hugo/content/en/observability_pipelines/configuration/live_capture.md index 0d4cd849f64..1adfbe91f7d 100644 --- a/hugo/content/en/observability_pipelines/configuration/live_capture.md +++ b/hugo/content/en/observability_pipelines/configuration/live_capture.md @@ -20,9 +20,6 @@ products: - name: Metrics icon: metrics url: /observability_pipelines/configuration/?tab=metrics#pipeline-types -- name: Traces - icon: apm - url: /observability_pipelines/configuration/?tab=traces#pipeline-types --- {{< product-availability >}} diff --git a/hugo/content/en/observability_pipelines/configuration/set_up_pipelines.md b/hugo/content/en/observability_pipelines/configuration/set_up_pipelines.md index 315b626a088..87ac349cc6a 100644 --- a/hugo/content/en/observability_pipelines/configuration/set_up_pipelines.md +++ b/hugo/content/en/observability_pipelines/configuration/set_up_pipelines.md @@ -110,31 +110,6 @@ See [Export a Pipeline Configuration to JSON or Terraform][14] if you want to pr [6]: /observability_pipelines/configuration/pipeline_simulation/ [11]: /observability_pipelines/search_syntax/metrics/ -{{% /tab %}} -{{% tab "Traces" %}} - -1. Navigate to [Observability Pipelines][1]. -1. Select the [Trace Sampling][2] template, or click {{< ui >}}New Pipeline{{< /ui >}} and select {{< ui >}}Traces Pipeline{{< /ui >}}. -1. Select and set up a [trace source][3]. -1. Select and set up [destinations][5] for your processed traces. -1. Click {{< ui >}}Edit{{< /ui >}} on a processor group to add, update, and validate your [processors][4] using [Pipeline Simulation][6]. - - **Notes**: - - For a pipeline canvas, there is a limit of 25 processor groups and a total of 150 processors. - - To copy a processor, click the copy icon for that processor and then paste it (`Cmd+V` on Mac, `Ctrl+V` on Windows or Linux). - -#### Add another processor group - -{{< img src="observability_pipelines/setup/another_processor_group.png" alt="The Pipelines page showing two processor groups sending logs to the same destination" style="width:100%;" >}} - -{{% observability_pipelines/set_up_pipelines/add_another_processor_group %}} - -[1]: https://app.datadoghq.com/observability-pipelines -[2]: /observability_pipelines/configuration/explore_templates/?tab=traces#trace-sampling -[3]: /observability_pipelines/sources/?tab=traces#sources -[4]: /observability_pipelines/processors/?tab=traces#processors -[5]: /observability_pipelines/destinations/?tab=traces#destinations -[6]: /observability_pipelines/configuration/pipeline_simulation/ - {{% /tab %}} {{< /tabs >}} diff --git a/hugo/content/en/observability_pipelines/configuration/update_existing_pipelines.md b/hugo/content/en/observability_pipelines/configuration/update_existing_pipelines.md index 88d1ea3cd9b..1d14f16fc00 100644 --- a/hugo/content/en/observability_pipelines/configuration/update_existing_pipelines.md +++ b/hugo/content/en/observability_pipelines/configuration/update_existing_pipelines.md @@ -208,14 +208,8 @@ On the Worker installation page: {{% /tab %}} {{% tab "OpenTelemetry" %}} -**Metrics** - {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry_metrics %}} -**Traces** - -{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry_traces %}} - {{% /tab %}} {{% tab "Prometheus" %}} diff --git a/hugo/content/en/observability_pipelines/destinations/_index.md b/hugo/content/en/observability_pipelines/destinations/_index.md index ddc9e21e108..efdf91bf6c7 100644 --- a/hugo/content/en/observability_pipelines/destinations/_index.md +++ b/hugo/content/en/observability_pipelines/destinations/_index.md @@ -1,6 +1,9 @@ --- title: Destinations disable_toc: false +aliases: + - /observability_pipelines/destinations/datadog_apm/ + - /observability_pipelines/destinations/opentelemetry/traces/ further_reading: - link: "logs/processing/pipelines" tag: "Documentation" @@ -9,7 +12,7 @@ further_reading: ## Overview -Use the Observability Pipelines Worker to send your processed {{< tooltip text="logs, metrics, and traces" tooltip="Contact your account manager to discuss use cases and pricing." >}} to different destinations. Most Observability Pipelines destinations send events in batches to the downstream integration. See [Event batching](#event-batching) for more information. Some Observability Pipelines destinations also have fields that support template syntax, so you can set these fields based on specific fields. See [Template syntax](#template-syntax) for more information. +Use the Observability Pipelines Worker to send your processed logs and metrics to different destinations. Most Observability Pipelines destinations send events in batches to the downstream integration. See [Event batching](#event-batching) for more information. Some Observability Pipelines destinations also have fields that support template syntax, so you can set these fields based on specific fields. See [Template syntax](#template-syntax) for more information. Select a destination in the left navigation menu to see more information about it. @@ -88,15 +91,6 @@ These are the available destinations: [5]: /observability_pipelines/destinations/opentelemetry/metrics [6]: /observability_pipelines/destinations/prometheus/ -{{% /tab %}} -{{% tab "Traces" %}} - -- [Datadog APM][1] -- [OpenTelemetry Traces][2] - -[1]: /observability_pipelines/destinations/datadog_apm/ -[2]: /observability_pipelines/destinations/opentelemetry/traces - {{% /tab %}} {{< /tabs >}} diff --git a/hugo/content/en/observability_pipelines/destinations/datadog_apm.md b/hugo/content/en/observability_pipelines/destinations/datadog_apm.md deleted file mode 100644 index e6329813d7b..00000000000 --- a/hugo/content/en/observability_pipelines/destinations/datadog_apm.md +++ /dev/null @@ -1,61 +0,0 @@ ---- -title: Datadog APM Destination -disable_toc: false -products: -- name: Traces - icon: apm - url: /observability_pipelines/configuration/?tab=traces#pipeline-types ---- - -{{< product-availability >}} - -## Overview - -Use Observability Pipelines' {{< tooltip text="Datadog APM destination" tooltip="Contact your account manager to request access." >}} to send traces to Datadog. - -## Setup - -Configure the Datadog APM destination when you [set up a pipeline][1] in the UI. - -### Optional buffering - -{{% observability_pipelines/destination_buffer %}} - -## Secret defaults - -{{% observability_pipelines/set_secrets_intro %}} - -{{< tabs >}} -{{% tab "Secrets Management" %}} - -There are no secret identifiers for this destination. - -{{% /tab %}} - -{{% tab "Environment Variables" %}} - -{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/datadog %}} - -{{% /tab %}} -{{< /tabs >}} - -## AWS PrivateLink - -To send traces from Observability Pipelines to Datadog using AWS PrivateLink, see [Connect to Datadog over AWS PrivateLink][7] for setup instructions. The two endpoints you need to set up are: - -- Traces: {{< region-param key=traces_endpoint_private_link code="true" >}} -- Remote Configuration: {{< region-param key=remote_config_endpoint_private_link code="true" >}} - -**Note**: The `obpipeline-intake.datadoghq.com` endpoint is used for Live Capture and is not available as a PrivateLink endpoint. - -## Health metrics - -See [Component metrics][5] and [Destination buffer metrics][6] for more information on metrics emitted by all destinations. - -[1]: /observability_pipelines/configuration/set_up_pipelines/ -[2]: https://app.datadoghq.com/observability-pipelines -[3]: /api/latest/observability-pipelines/ -[4]: https://registry.terraform.io/providers/datadog/datadog/latest/docs/resources/observability_pipeline -[5]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/#component-metrics -[6]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/?tab=destinations#buffer -[7]: /agent/guide/private-link/?tab=crossregionprivatelinkendpoints diff --git a/hugo/content/en/observability_pipelines/destinations/opentelemetry/metrics.md b/hugo/content/en/observability_pipelines/destinations/opentelemetry/metrics.md index 6a15b76be60..781a197ddf3 100644 --- a/hugo/content/en/observability_pipelines/destinations/opentelemetry/metrics.md +++ b/hugo/content/en/observability_pipelines/destinations/opentelemetry/metrics.md @@ -1,11 +1,14 @@ --- -title: OpenTelemetry Metrics Destination +title: OpenTelemetry Destination disable_toc: false -code_lang: metrics -type: multi-code-lang -weight: 1 +products: +- name: Metrics + icon: metrics + url: /observability_pipelines/configuration/?tab=metrics#pipeline-types --- +{{< product-availability >}} + ## Overview Use Observability Pipelines' {{< tooltip text=" OpenTelemetry destination" tooltip="Contact your account manager to request access." >}} to send metrics over HTTP/S to an OpenTelemetry (OTel) Collector or another OpenTelemetry Protocol (OTLP)-compatible endpoint. diff --git a/hugo/content/en/observability_pipelines/destinations/opentelemetry/traces.md b/hugo/content/en/observability_pipelines/destinations/opentelemetry/traces.md deleted file mode 100644 index f602a6ad7fb..00000000000 --- a/hugo/content/en/observability_pipelines/destinations/opentelemetry/traces.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -title: OpenTelemetry Traces Destination -disable_toc: false -code_lang: traces -type: multi-code-lang -weight: 2 ---- - -## Overview - -Use Observability Pipelines' {{< tooltip text="OpenTelemetry Traces destination" tooltip="Contact your account manager to request access." >}} to send traces to an OpenTelemetry (OTel) Collector. - -
You must use an OpenTelemetry source to use the OpenTelemetry Traces destination.
- -## Set up destination - -
For Secrets Management: Only enter the identifier for the HTTP/S Client URI and, if applicable, the TLS key pass. Do not enter the actual values.
- -Configure the OpenTelemetry Traces destination when you [set up a pipeline][3]. This section covers how to do so in the [UI][1], but you can also set up a pipeline using the [API][4] or with [Terraform][5]. - -After you select the OpenTelemetry Traces destination in the pipeline UI, enter the identifier for your HTTP/S Client URI Key. An example of the URI endpoint the identifier references: `http://localhost:4319/v1/traces`. If you leave the identifier field blank, the [default](#secret-defaults) is used. - -{{% observability_pipelines/secrets_env_var_note %}} - -### Optional settings - -#### Enable TLS - -{{% observability_pipelines/tls_settings %}} - -#### Buffering - -{{% observability_pipelines/destination_buffer %}} - -## Allow out-of-order samples - -The Worker doesn't always send metrics in the correct order for a given series because it doesn't reorder metrics. For example, if the first batch of metrics contains metrics with timestamps: `10:03`, `10:04`, `10:05` and the second batch contains metrics with timestamps: `10:01`, `10:02`, `10:06`, the Worker does not reorder those metrics before sending them out. - -Because the OTLP receiver rejects out-of-order samples, the Worker logs a Bad Request (`400`) error and the entire second batch of metrics gets dropped, even if the OTLP receiver accepted some of the valid metrics in the batch. - -Datadog recommends setting your OTLP receiver to allow out-of-order samples to prevent out-of-order samples from getting dropped. - -## Troubleshooting - -### Debug error logs - -If you see `400` or `500` error logs from this destination, you can enable debug logs to see the response returned by the server. To enable logs for this HTTP-based destination only and not every Worker module, set `VECTOR_LOG` to `info,vector::sinks::util::http=debug`: - -``` -docker run -i -e DD_API_KEY= \ - -e DD_OP_PIPELINE_ID= \ - -e VECTOR_LOG=info,vector::sinks::util::http=debug \ - datadog/observability-pipelines-worker run -``` - -See [Enable debug logs][6] for instruction on enabling full debug logs. - -## Secret defaults - -{{% observability_pipelines/set_secrets_intro %}} - -{{< tabs >}} -{{% tab "Secrets Management" %}} - -- HTTP/S Client URI endpoint identifier: - - References the HTTP/S URI endpoint to which the Worker sends OpenTelemetry data. An example of the URI endpoint the identifier references: `http://localhost:4319/v1/traces`. - - The default identifier is `DESTINATION_OTEL_HTTP_CLIENT_URI`. -- OpenTelemetry Traces TLS passphrase identifier (when TLS is enabled): - - The default identifier is `DESTINATION_OTEL_KEY_PASS`. - -{{% /tab %}} - -{{% tab "Environment Variables" %}} - -{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry_traces %}} - -{{% /tab %}} -{{< /tabs >}} - -[1]: https://app.datadoghq.com/observability-pipelines -[3]: /observability_pipelines/configuration/set_up_pipelines/ -[4]: /api/latest/observability-pipelines/ -[5]: https://registry.terraform.io/providers/datadog/datadog/latest/docs/resources/observability_pipeline -[6]: /observability_pipelines/monitoring_and_troubleshooting/troubleshooting/#enable-debug-logs diff --git a/hugo/content/en/observability_pipelines/guide/environment_variables.md b/hugo/content/en/observability_pipelines/guide/environment_variables.md index a6b14530b38..1c6c376e8a7 100644 --- a/hugo/content/en/observability_pipelines/guide/environment_variables.md +++ b/hugo/content/en/observability_pipelines/guide/environment_variables.md @@ -133,13 +133,8 @@ Some Observability Pipelines components require setting up environment variables {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opensearch %}} ### OpenTelemetry - -**Metrics** {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry_metrics %}} -**Traces** -{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry_traces %}} - ### Prometheus {{% observability_pipelines/configure_existing_pipelines/destination_env_vars/prometheus %}} diff --git a/hugo/content/en/observability_pipelines/processors/_index.md b/hugo/content/en/observability_pipelines/processors/_index.md index d0fd0d36c31..86590d3cd29 100644 --- a/hugo/content/en/observability_pipelines/processors/_index.md +++ b/hugo/content/en/observability_pipelines/processors/_index.md @@ -1,6 +1,8 @@ --- title: Processors disable_toc: false +aliases: + - /observability_pipelines/processors/tail_based_sampling/ further_reading: - link: https://www.datadoghq.com/blog/rehydrate-archived-logs-with-observability-pipelines tag: Blog @@ -14,7 +16,7 @@ further_reading:
The processors outlined in this documentation are specific to on-premises logging environments. To parse, structure, and enrich cloud-based logs, see the Log Management documentation.
-Use Observability Pipelines' processors to parse, structure, and enrich your {{< tooltip text="logs, metrics, and traces" tooltip="Contact your account manager to discuss use cases and pricing." >}}. When you create a pipeline in the UI, pre-selected processors are added to your processor group based on the selected template. You can add additional processors and delete any existing ones based on your processing needs. +Use Observability Pipelines' processors to parse, structure, and enrich your logs and metrics. When you create a pipeline in the UI, pre-selected processors are added to your processor group based on the selected template. You can add additional processors and delete any existing ones based on your processing needs. Processor groups are executed from top to bottom. The order of the processors is important because events are checked by each processor, but only events that match the processor's filters are processed. To modify the order of the processors, use the drag handle on the top left corner of the processor you want to move. @@ -84,21 +86,6 @@ These are the available processors: [4]: /observability_pipelines/processors/tag_allow_block_list/ [5]: /observability_pipelines/processors/tag_cardinality_control/ -{{% /tab %}} -{{% tab "Traces" %}} - -- [Custom Processor][1] -- [Filter][2] -- [Sample][3] -- [Sensitive Data Scanner][4] -- [Tail-Based Sampling][5] - -[1]: /observability_pipelines/processors/custom_processor/ -[2]: /observability_pipelines/processors/filter/ -[3]: /observability_pipelines/processors/sample/ -[4]: /observability_pipelines/processors/sensitive_data_scanner/ -[5]: /observability_pipelines/processors/tail_based_sampling/ - {{% /tab %}} {{< /tabs >}} diff --git a/hugo/content/en/observability_pipelines/processors/custom_processor.md b/hugo/content/en/observability_pipelines/processors/custom_processor.md index 134b322823f..0f0093ae25e 100644 --- a/hugo/content/en/observability_pipelines/processors/custom_processor.md +++ b/hugo/content/en/observability_pipelines/processors/custom_processor.md @@ -18,16 +18,13 @@ products: - name: Metrics icon: metrics url: /observability_pipelines/configuration/?tab=metrics#pipeline-types -- name: Traces - icon: apm - url: /observability_pipelines/configuration/?tab=traces#pipeline-types --- {{< product-availability >}} ## Overview -Use this processor with Vector Remap Language (VRL) to modify and enrich your logs, metrics, or {{< tooltip text="traces" tooltip="Contact your account manager to request access." >}}. VRL is an expression-oriented, domain specific language designed for transforming data. It features built-in functions for observability use cases. You can use custom functions in the following ways: +Use this processor with Vector Remap Language (VRL) to modify and enrich your logs or metrics. VRL is an expression-oriented, domain specific language designed for transforming data. It features built-in functions for observability use cases. You can use custom functions in the following ways: - Manipulate [arrays](#array), [strings](#string), and other data types. - Encode and decode values using [Codec](#codec). diff --git a/hugo/content/en/observability_pipelines/processors/filter.md b/hugo/content/en/observability_pipelines/processors/filter.md index 72d3f3719a8..065068c2564 100644 --- a/hugo/content/en/observability_pipelines/processors/filter.md +++ b/hugo/content/en/observability_pipelines/processors/filter.md @@ -15,16 +15,13 @@ products: - name: Metrics icon: metrics url: /observability_pipelines/configuration/?tab=metrics#pipeline-types -- name: Traces - icon: apm - url: /observability_pipelines/configuration/?tab=traces#pipeline-types --- {{< product-availability >}} ## Overview -This processor sends all logs, metrics, or {{< tooltip text="traces" tooltip="Contact your account manager to request access." >}} that match the filter query to the next step in the pipeline. Events that do not match the filter query are dropped and are not sent to any subsequent processors or destinations. +This processor sends all logs or metrics that match the filter query to the next step in the pipeline. Events that do not match the filter query are dropped and are not sent to any subsequent processors or destinations. **Note**: For all other processor queries, events that don't match the query are sent to the subsequent step in the pipeline. They are not dropped. @@ -32,7 +29,7 @@ This processor sends all logs, metrics, or {{< tooltip text="traces" tooltip="Co To set up the filter processor: -- Define a {{< ui >}}filter query{{< /ui >}}. See [Logs Search Syntax][1], [Metrics Search Syntax][2], or [APM Query Syntax][6] for more information. +- Define a {{< ui >}}filter query{{< /ui >}}. See [Logs Search Syntax][1] or [Metrics Search Syntax][2] for more information. - Events that match the query are sent to the next component. - Events that don't match the query are dropped. @@ -45,7 +42,6 @@ For [component metrics][3] and [processor buffer metrics][4] emitted by all proc [3]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/#component-metrics [4]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/#processor-buffer-metrics [5]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/ -[6]: /tracing/trace_explorer/query_syntax/ ## Further reading diff --git a/hugo/content/en/observability_pipelines/processors/sample.md b/hugo/content/en/observability_pipelines/processors/sample.md index de1b947f601..2a2424f8ce7 100644 --- a/hugo/content/en/observability_pipelines/processors/sample.md +++ b/hugo/content/en/observability_pipelines/processors/sample.md @@ -5,23 +5,20 @@ products: - name: Logs icon: logs url: /observability_pipelines/configuration/?tab=logs#pipeline-types -- name: Traces - icon: apm - url: /observability_pipelines/configuration/?tab=traces#pipeline-types --- {{< product-availability >}} ## Overview -This processor samples your logs or {{< tooltip text="traces" tooltip="Contact your account manager to request access." >}} for a representative subset at the rate that you define, dropping the remaining events. As an example, you can use this processor to sample 20% of events from a noisy non-critical service. +This processor samples your logs for a representative subset at the rate that you define, dropping the remaining events. As an example, you can use this processor to sample 20% of events from a noisy non-critical service. The sampling only applies to events that match your filter query and does not impact other events. If an event is dropped at this processor, the event is not sent to subsequent processors. ## Setup To set up the sample processor: -1. Define a {{< ui >}}filter query{{< /ui >}}. See [Logs Search Syntax][1] or [APM Query Syntax][2] for more information. +1. Define a {{< ui >}}filter query{{< /ui >}}. See [Logs Search Syntax][1] for more information. - Only events that match the specified filter query are sampled at the specified retention rate. - The sampled events and the events that do not match the filter query are sent to the next step in the pipeline. 1. Enter your desired sampling rate in the {{< ui >}}Retain{{< /ui >}} field. For example, entering `2` means 2% of events are retained out of all events that match the filter query. @@ -44,4 +41,3 @@ Then, 40% of events for each unique combination of `status` and `service` from ` - 40% of events with `status:error` and `service:core-web` are retained. [1]: /observability_pipelines/search_syntax/logs/ -[2]: /tracing/trace_explorer/query_syntax/ diff --git a/hugo/content/en/observability_pipelines/processors/sensitive_data_scanner.md b/hugo/content/en/observability_pipelines/processors/sensitive_data_scanner.md index 163d4ebd699..b82ec0407df 100644 --- a/hugo/content/en/observability_pipelines/processors/sensitive_data_scanner.md +++ b/hugo/content/en/observability_pipelines/processors/sensitive_data_scanner.md @@ -12,16 +12,13 @@ products: - name: Logs icon: logs url: /observability_pipelines/configuration/?tab=logs#pipeline-types -- name: Traces - icon: apm - url: /observability_pipelines/configuration/?tab=traces#pipeline-types --- {{< product-availability >}} ## Overview -The Sensitive Data Scanner processor scans logs or {{< tooltip text="traces" tooltip="Contact your account manager to request access." >}} to detect and redact or hash sensitive information such as PII, PCI, and custom sensitive data. You can pick from Datadog's library of predefined rules, or input custom Regex rules to scan for sensitive data. +The Sensitive Data Scanner processor scans logs to detect and redact or hash sensitive information such as PII, PCI, and custom sensitive data. You can pick from Datadog's library of predefined rules, or input custom Regex rules to scan for sensitive data. You can set up the pipeline and processor in the [UI](#set-up-the-processor-in-the-ui), [API][10], or [Terraform](#set-up-the-processor-using-terraform). @@ -31,7 +28,7 @@ See [Best practices to optimize performance](#best-practices-to-optimize-perform To set up the processor: -1. Define a {{< ui >}}filter query{{< /ui >}}. See [Logs Search Syntax][1] or [APM Query Syntax][17] for more information. +1. Define a {{< ui >}}filter query{{< /ui >}}. See [Logs Search Syntax][1] for more information. - Only events matching the filter are scanned and processed. - All events, regardless of whether they match the filter query, are sent to the next step in the pipeline. 1. Click {{< ui >}}Add Scanning Rule{{< /ui >}}. @@ -411,4 +408,3 @@ For [component metrics][13] and [processor buffer metrics][14] emitted by all pr [14]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/#processor-buffer-metrics [15]: /observability_pipelines/monitoring_and_troubleshooting/pipeline_usage_metrics/ [16]: https://app.datadoghq.com/dash/integration/32326/observability-pipelines-overview -[17]: /tracing/trace_explorer/query_syntax/ diff --git a/hugo/content/en/observability_pipelines/processors/tail_based_sampling.md b/hugo/content/en/observability_pipelines/processors/tail_based_sampling.md deleted file mode 100644 index fc7286f2ee2..00000000000 --- a/hugo/content/en/observability_pipelines/processors/tail_based_sampling.md +++ /dev/null @@ -1,43 +0,0 @@ ---- -title: Tail-Based Sampling Processor -disable_toc: false -products: -- name: Traces - icon: apm - url: /observability_pipelines/configuration/?tab=traces#pipeline-types ---- - -{{< product-availability >}} - -## Overview - -The Tail-based Sampling processor determines whether the Worker keeps a completed trace based on sampling policies that you define. This processor can sample based on the full context of a trace, such as its status code, latency, or associated events. - -## Setup - -
If you are using the Filter processor, do not filter out spans belonging to a trace because this can lead to incomplete traces and incorrect sampling.
- -To set up the tail-based sampling processor: - -1. Define a {{< ui >}}filter query{{< /ui >}}. See [APM Query Syntax][1] for more information. - - Only traces that match the specified filter query are evaluated against your sampling policy groups. -1. Click {{< ui >}}Manage sampling policies{{< /ui >}} to add at least one policy group. - -### Add a sampling policy group - -1. Enter a {{< ui >}}Policy Group Name{{< /ui >}}. -1. Click to add a policy, and select a {{< ui >}}Policy Type{{< /ui >}}: - - {{< ui >}}Condition{{< /ui >}}: Keep traces when any associated event matches the specified query. - - {{< ui >}}Status code{{< /ui >}}: Keep traces that have the selected status code. - - {{< ui >}}Latency{{< /ui >}}: Keep traces whose total duration falls within the specified bounds. - - {{< ui >}}Sampling rate{{< /ui >}}: Keep this percentage of traces, sampled consistently by trace ID. -1. (Optional) Click to add more policies to the policy group. A trace matches a policy group only if it satisfies all of the policies within that group. -1. (Optional) Repeat these steps to add more policy groups. -1. Click **Save**. - -**Notes**: -- A trace is only sampled if it matches any of the configured policy groups. -- If a trace matches a policy group, it's kept and sent to the next step in the pipeline. If a trace doesn't match any policy groups, it's dropped. -- If there are multiple policies in a policy group, a trace must match all policies to be sampled. - -[1]: /tracing/trace_explorer/query_syntax/ diff --git a/hugo/content/en/observability_pipelines/sources/_index.md b/hugo/content/en/observability_pipelines/sources/_index.md index 3b9822022fe..e4ae6840fdd 100644 --- a/hugo/content/en/observability_pipelines/sources/_index.md +++ b/hugo/content/en/observability_pipelines/sources/_index.md @@ -87,15 +87,6 @@ These are the available sources: [2]: /observability_pipelines/sources/opentelemetry/ [25]: /observability_pipelines/sources/prometheus/ -{{% /tab %}} -{{% tab "Traces" %}} - -- [Datadog Agent][1] -- [OpenTelemetry][2] - -[1]: /observability_pipelines/sources/datadog_agent/ -[2]: /observability_pipelines/sources/opentelemetry/ - {{% /tab %}} {{< /tabs >}} diff --git a/hugo/content/en/observability_pipelines/sources/datadog_agent.md b/hugo/content/en/observability_pipelines/sources/datadog_agent.md index 52133d606fd..09c3bad4572 100644 --- a/hugo/content/en/observability_pipelines/sources/datadog_agent.md +++ b/hugo/content/en/observability_pipelines/sources/datadog_agent.md @@ -8,9 +8,6 @@ products: - name: Metrics icon: metrics url: /observability_pipelines/configuration/?tab=metrics#pipeline-types -- name: Traces - icon: apm - url: /observability_pipelines/configuration/?tab=traces#pipeline-types further_reading: - link: https://www.datadoghq.com/blog/manage-metrics-cost-control-with-observability-pipelines tag: Blog @@ -21,7 +18,7 @@ further_reading: ## Overview -Use Observability Pipelines' Datadog Agent source to receive {{< tooltip text="logs, metrics, or traces" tooltip="Contact your account manager to discuss use cases and pricing." >}} from the Datadog Agent. +Use Observability Pipelines' Datadog Agent source to receive logs or metrics from the Datadog Agent. **Note**: If you are using the Datadog Distribution of OpenTelemetry (DDOT) Collector to collect logs or metrics, you must [use the OpenTelemetry source to send that data to Observability Pipelines][4]. @@ -149,61 +146,6 @@ datadog: [1]: /containers/docker/data_collected/ [2]: /containers/guide/container-discovery-management/?tab=helm#setting-environment-variables -{{% /tab %}} - -{{% tab "Traces" %}} - -Use the Agent configuration file or the Agent Helm chart values file to connect the Datadog Agent to the Observability Pipelines Worker. - -{{% collapse-content title="Agent configuration file" level="h4" expanded=false id="traces-agent-config-file" %}} - -To send Datadog Agent traces to the Observability Pipelines Worker, update your [Agent configuration file][1] with the following: - -``` -observability_pipelines_worker: - traces: - enabled: true - url: "http://:8484" - -``` - -`` is the host IP address or the load balancer URL associated with the Observability Pipelines Worker. -- For CloudFormation installs, use the `LoadBalancerDNS` CloudFormation output for the URL. -- For Kubernetes installs, you can use the internal DNS record of the Observability Pipelines Worker service. For example: `http://opw-observability-pipelines-worker.default.svc.cluster.local:`. - -**Note**: If the Worker is listening for logs or metrics on ports 8282 or 8383, you must use another port for traces, such as 8484. - -After you [restart the Agent][2], your observability data is sent to the Worker, processed by the pipeline, and delivered to Datadog. - -[1]: /agent/configuration/agent-configuration-files/ -[2]: /agent/configuration/agent-commands/#restart-the-agent - -{{% /collapse-content %}} - -{{% collapse-content title="Agent Helm values file" level="h4" expanded=false id="traces-agent-helm-values-file" %}} - -To send Datadog Agent traces to the Observability Pipelines Worker, update your Datadog Helm chart [datadog-values.yaml][1] with the following environment variables. See [Agent Environment Variables][2] for more information. - -``` -datadog: - env: - - name: DD_OBSERVABILITY_PIPELINES_WORKER_TRACES_ENABLED - value: true - - name: DD_OBSERVABILITY_PIPELINES_WORKER_TRACES_URL - value: "http://:8484" -``` - -`` is the host IP address or the load balancer URL associated with the Observability Pipelines Worker. - - For Kubernetes installs, you can use the internal DNS record of the Observability Pipelines Worker service. For example: `http://opw-observability-pipelines-worker.default.svc.cluster.local:`. - -**Note**: If the Worker is listening for logs or metrics on ports 8282 or 8383, you must use another port for traces, such as 8484. - -[1]: https://github.com/DataDog/helm-charts/blob/main/charts/datadog/values.yaml -[2]: https://docs.datadoghq.com/agent/guide/environment-variables/ - -{{% /collapse-content %}} - {{% /tab %}} {{< /tabs >}} diff --git a/hugo/content/en/observability_pipelines/sources/opentelemetry.md b/hugo/content/en/observability_pipelines/sources/opentelemetry.md index 2cfd12dd112..37a55de9e92 100644 --- a/hugo/content/en/observability_pipelines/sources/opentelemetry.md +++ b/hugo/content/en/observability_pipelines/sources/opentelemetry.md @@ -15,16 +15,13 @@ products: - name: Metrics icon: metrics url: /observability_pipelines/configuration/?tab=metrics#pipeline-types -- name: Traces - icon: apm - url: /observability_pipelines/configuration/?tab=traces#pipeline-types --- {{< product-availability >}} ## Overview -Use Observability Pipelines' OpenTelemetry (OTel) source to collect {{< tooltip text="logs, metrics, or traces" tooltip="Contact your account manager to discuss use cases and pricing." >}} from your OTel Collector through HTTP or gRPC. +Use Observability Pipelines' OpenTelemetry (OTel) source to collect logs or metrics from your OTel Collector through HTTP or gRPC. **Notes**: - If you are using the Datadog Distribution of OpenTelemetry (DDOT) Collector, use the OpenTelemetry source to [send data to Observability Pipelines](#send-data-from-the-datadog-distribution-of-opentelemetry-collector-to-observability-pipelines). @@ -163,40 +160,6 @@ Set the listener address environment variables to the following default values. - HTTP listener address: `worker:4318` - gRPC listener address: `worker:4317` -{{% /tab %}} -{{% tab "Traces" %}} - -### HTTP configuration example - -The Worker exposes the HTTP endpoint on port 4318, which is the default port. You can configure the port value in the Worker. - -For example, to configure an OTel trace exporter over HTTP in Python: - -```python - from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter - http_exporter = OTLPSpanExporter( - endpoint="http://worker:4318/v1/traces" - ) -``` - -### gRPC configuration example - -The Worker exposes the gRPC endpoint on port 4317, which is the default port. You can configure the port value in the Worker. - -For example, to configure an OTel trace exporter over gRPC in Python: - -```python - from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter - grpc_exporter = OTLPSpanExporter( - endpoint="grpc://worker:4317" - ) -``` - -Set the listener address environment variables to the following default values. If you configured different port values in the Worker, use those instead. - -- HTTP listener address: `worker:4318` -- gRPC listener address: `worker:4317` - {{% /tab %}} {{< /tabs >}} @@ -287,45 +250,6 @@ To send metrics from the Datadog Distribution of the OpenTelemetry (DDOT) Collec [8]: /observability_pipelines/processors/custom_processor [9]: https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/kubernetes_daemonset/?tab=helm#configure-the-opentelemetry-collector -{{% /tab %}} -{{% tab "Traces" %}} - -To send traces from the Datadog Distribution of the OpenTelemetry (DDOT) Collector: -1. Deploy the DDOT Collector using Helm. See [Install the DDOT Collector as a Kubernetes DaemonSet][5] for instructions. -1. [Set up a pipeline][6] on Observability Pipelines using the [OpenTelemetry source](#set-up-the-source-in-the-pipeline-ui). - 1. (Optional) Datadog recommends adding an [Edit Fields processor][7] to the pipeline that appends the field `op_otel_ddot:true`. - 1. When you install the Worker, for the OpenTelemetry source environment variables: - 1. Set your HTTP listener to `0.0.0.0:4318`. - 1. Set your gRPC listener to `0.0.0.0:4317`. - 1. After you install the Worker and deployed the pipeline, update the OpenTelemetry Collector's [`otel-config.yaml`][9] to include an exporter that sends traces to Observability Pipelines. For example: - ``` - exporters: - otlphttp: - endpoint: http://opw-observability-pipelines-worker..svc.cluster.local:4318 - ... - service: - pipelines: - traces: - exporters: [otlphttp] - ``` - Replace `` with the Kubernetes namespace where the Observability Pipelines Worker is deployed (for example, `default`). - 1. Redeploy the Datadog Agent with the updated [`otel-config.yaml`][9]. For example, if the Agent is installed in Kubernetes: - ``` - helm upgrade --install datadog-agent datadog/datadog \ - --values ./agent.yaml \ - --set-file datadog.otelCollector.config=./otel-config.yaml - ``` - -**Notes**: -- Traces sent from DDOT might have nested objects that prevent Datadog from parsing the traces correctly. To resolve this, Datadog recommends using the [Custom Processor][8] to flatten the nested `resource` object. -- If the DDOT Collector and the Observability Pipelines Worker are running on the same host, their default OTLP receiver ports (4317/4318) may conflict. In a typical Kubernetes deployment, the Collector and the Worker run in separate pods, so this is not an issue. - -[5]: /opentelemetry/setup/ddot_collector/install/kubernetes_daemonset/?tab=datadogoperator -[6]: /observability_pipelines/configuration/set_up_pipelines/ -[7]: /observability_pipelines/processors/edit_fields#add-field -[8]: /observability_pipelines/processors/custom_processor -[9]: https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/kubernetes_daemonset/?tab=helm#configure-the-opentelemetry-collector - {{% /tab %}} {{< /tabs >}} diff --git a/hugo/layouts/shortcodes/observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry_traces.md b/hugo/layouts/shortcodes/observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry_traces.md deleted file mode 100644 index f5687bfb0ae..00000000000 --- a/hugo/layouts/shortcodes/observability_pipelines/configure_existing_pipelines/destination_env_vars/opentelemetry_traces.md +++ /dev/null @@ -1,4 +0,0 @@ -- HTTP/S client URI endpoint, such as `http://localhost:4319/v1/traces`: - - The default environment variable is `DD_OP_DESTINATION_OTEL_HTTP_CLIENT_URI`. -- OpenTelemetry Traces TLS passphrase (when enabled): - - The default environment variable is `DD_OP_DESTINATION_OTEL_KEY_PASS`.