Datadog PII Masking: Sensitive Data Scanner vs In-Cluster Redaction

TL;DR: Datadog's Sensitive Data Scanner is a capable, rule-based masking product — but in its default cloud mode your raw data reaches Datadog before it's redacted, and it's a premium add-on. Masking PII inside your cluster with an open-source sidecar keeps unmasked data on your own infrastructure and adds entropy detection for secrets no regex knows about.

Two ways to do Datadog PII masking

If you ship Kubernetes logs to Datadog and need to mask emails, tokens, credit card numbers, or API keys, you have two broad options:

1. Datadog Sensitive Data Scanner (SDS) — Datadog's native, stream-based pattern-matching service that discovers, tags, and optionally redacts or hashes sensitive data.
2. In-cluster redaction — mask PII with a lightweight sidecar (like the open-source PII-Shield) so logs are already sanitized before the Datadog Agent ever collects them.

Both work. The right choice depends on where you want unmasked data to live, what you're willing to pay, and whether regex rules are enough for your threat model. Here's an honest, fact-checked breakdown.

Side-by-side comparison

Where does masking actually happen?

This is the single most important distinction, and it's worth being precise. Datadog SDS runs in one of two modes:

• Cloud mode (default): You submit logs and events to the Datadog backend, and redaction happens there before events are indexed and shown in the UI. In Datadog's own words, "the data leaves your environment before it gets redacted." Cloud-mode scanning is also sampled — configurable between 10% and 99%.
• Observability Pipelines: You run the Sensitive Data Scanner processor on-premises to redact logs "before they leave your premises." This closes the raw-data gap — but Observability Pipelines is its own product with its own infrastructure to deploy and operate.

An in-cluster sidecar sits earlier still: right next to the application, redacting each pod's output before it's ever collected. Unmasked data never leaves the pod, and there's no separate pipeline tier to run. If your compliance posture is "raw PII must never touch a third party," the place you mask matters more than the tool you use to mask it. See the Datadog sidecar guide for the ready-to-use pod configuration.

Detection: regex rules vs entropy

Datadog SDS detects sensitive data with a library of predefined rules (emails, credit cards, API keys, auth tokens, network/device info) plus custom rules you write as regex. The regex engine is a subset of PCRE2 — notably without backreferences or lookarounds — and you can add keyword proximity hints (by default, keywords must sit within 30 characters of a match) to cut false positives.

This is excellent for known patterns. The gap is unknown secrets: a freshly rotated internal token or a bespoke credential that matches no rule will pass straight through. PII-Shield adds Shannon entropy detection on top of a regex whitelist, so high-entropy strings get caught even when no one wrote a rule for them. In a fast-moving codebase this is the difference between "we mask what we remembered to configure" and "we mask secrets we didn't know existed."

What about cost?

Datadog's Sensitive Data Scanner is a premium add-on, billed separately from your base log ingestion — it is not part of the base plan, and public per-GB pricing isn't listed (it's quoted per account). Observability Pipelines, if you use it to redact on-prem, is likewise its own line item. An in-cluster open-source sidecar carries no license fee at all; its only cost is the modest compute it uses — around ~30Mi of memory per pod.

When to use which

This isn't "Datadog bad, sidecar good." They fit different situations. Here are four concrete scenarios to help you decide.

How in-cluster masking works with Datadog

The integration is straightforward. Your application writes its logs to a file on a shared in-pod volume instead of stdout. PII-Shield runs as a sidecar in the same pod, tails that file, redacts PII, and writes the sanitized result to its own stdout. The Datadog Agent, via Autodiscovery, is configured to collect the sidecar's stdout and ignore the raw application container — so only sanitized logs are ever picked up.

Pod
├── app container
│     └── writes logs → /shared/app.log   (not stdout)
│
├── pii-shield sidecar
│     ├── tails /shared/app.log
│     ├── entropy + regex redaction
│     └── writes sanitized → its own stdout ──┐
│                                             ▼
└── shared volume: /shared/          Datadog Agent (Autodiscovery)
                                     collects sidecar stdout only
                                              │
                                              ▼
                                     Datadog backend
                                     (no raw PII arrives)

The Datadog Agent configuration change is minimal — three lines in your pod spec to point the log collection path at the sanitized volume instead of the default container log path. Full YAML and Autodiscovery annotation examples are in the Datadog integration guide.

Common questions

Want to mask PII before it ever reaches Datadog?

Follow the Datadog sidecar guide for a ready-to-use pod config, or check out the PII-Shield repository on GitHub and drop a star if it helps.

Need enterprise support or custom compliance rules? → GitHub Sponsors

Not on Datadog? The same in-cluster masking pattern works for other stacks — see the Loki & Promtail guide, the ELK Stack guide, and the Fluentd & Fluent Bit guide.