Guide

How to mask PII in Kubernetes before sending logs to Datadog

TL;DR: route application logs to an in-pod shared volume, let PII-Shield redact them, and configure Datadog Autodiscovery to collect only the sidecar's sanitized stdout.

The Problem: Datadog Bills and GDPR Nightmares

If you are running applications in Kubernetes and shipping your logs to Datadog, you have probably faced two major headaches:

Cost & Zero-Trust Compliance: Datadog charges you based on the volume of logs ingested and indexed. While Datadog offers a built-in Sensitive Data Scanner, it is a premium feature billed separately. More importantly, PII-Shield operates inside your cluster, reducing the chance that open PII leaves your infrastructure before masking.
Compliance: Sending Personally Identifiable Information (PII) like emails, credit card numbers, or API keys to a third-party logging service can create GDPR and privacy compliance risk.

The more comprehensive your logs are for debugging, the higher your Datadog bill gets, and the bigger your risk of a privacy breach becomes.

Datadog's own Sensitive Data Scanner can do the same job, but in its default cloud mode it redacts after the data has already reached Datadog, and it is a premium add-on. If you are still choosing between the two, read the Sensitive Data Scanner vs in-cluster redaction comparison first. This guide assumes you have picked the sidecar and want it running.

The Standard Approach (And Why It Hurts)

The standard way to solve this is to configure the Datadog Agent to mask or scrub PII before it leaves your cluster.

However, this approach has significant drawbacks:

Complexity: Setting up custom parsing rules, regexes, and pipelines in the Datadog Agent configuration can be tedious and difficult to maintain.
High CPU Usage: Running heavy regex operations over massive volumes of text inside your log shipper consumes a lot of CPU resources. This can slow down your node's performance or require larger, more expensive compute instances.
Whack-a-Mole: You are constantly updating rules as your application output changes, which takes time and effort.

The Solution: PII-Shield as a Lightweight Sidecar

Instead of burdening your cluster-wide log shipper with heavy processing, you can mask PII before it even leaves the pod.

PII-Shield is a small, dependency-free tool written in Go. It acts as a sidecar container that sits right next to your application. It intercepts the logs in real-time, scrubs sensitive data using entropy detection and deterministic hashing, and then passes the clean logs forward.

By the time the Datadog Agent picks up the logs from the Kubernetes node, they are already completely sanitized.

Ready-to-Use Pod Configuration with Datadog Autodiscovery

Here is a practical example of how to inject PII-Shield as a sidecar. We use Datadog's Autodiscovery annotations (ad.datadoghq.com) to ensure the Datadog Agent correctly collects logs from the PII-Shield sidecar while explicitly ignoring the raw output of your main app, preventing any duplicate billing.

yaml

apiVersion: v1
kind: Secret
metadata:
  name: pii-shield-secret
type: Opaque
stringData:
  pii-salt: "replace-with-a-long-random-value"
---
apiVersion: v1
kind: Pod
metadata:
  name: my-app-with-pii-shield
  annotations:
    # Disable logging for the raw app container to prevent duplicates
    ad.datadoghq.com/my-app.logs: '[{"source": "my-app", "service": "billing", "log_processing_rules": [{"type": "exclude_at_match", "name": "exclude_all", "pattern": ".*"}]}]'
    # Explicitly collect clean logs from the sidecar
    ad.datadoghq.com/pii-shield-sidecar.logs: '[{"source": "pii-shield", "service": "billing"}]'
spec:
  containers:
    - name: my-app
      image: my-app-image:v1.0.0
      # Instead of writing directly to stdout, the app writes to a shared file or pipe
      command: ["/bin/sh", "-c"]
      args: ["./my-app-binary > /shared-logs/app.log 2>&1"]
      volumeMounts:
        - name: shared-logs
          mountPath: /shared-logs

    - name: pii-shield-sidecar
      image: thelisdeep/pii-shield:2.2.0
      env:
        - name: PII_SALT
          valueFrom:
            secretKeyRef:
              name: pii-shield-secret
              key: pii-salt
      # Scratch image: run the binary directly (no shell/tail). It watches the
      # file, scrubs each line, and writes the clean stream to stdout.
      command: ["/pii-shield"]
      args: ["--watch-file", "/shared-logs/app.log"]
      volumeMounts:
        - name: shared-logs
          mountPath: /shared-logs

  volumes:
    - name: shared-logs
      emptyDir: {}

Note: The thelisdeep/pii-shield image is multi-arch (supporting both amd64 and arm64), which is perfect if you are saving costs by running on ARM processors like AWS Graviton.

How does Datadog know what to read?
Because the main application now redirects its output to a file, its standard output (stdout) is empty. The Datadog Agent, which natively listens to stdout across all containers via Autodiscovery, will automatically pick up only the clean stream from the pii-shield-sidecar. There are no conflicts and no duplicate logs.

Production note: if your application writes sensitive data to stderr, redirect stderr into the same file as stdout (for example 2>&1) or route stderr through the same sanitization path.

Exporting Custom Metrics to Datadog

PII-Shield exposes Prometheus metrics on port :9090/metrics. You can easily push these into Datadog using the OpenMetrics integration. Simply add the following annotations to your pod to start tracking redaction rates and entropy scores in Datadog:

yaml

  annotations:
    ad.datadoghq.com/pii-shield-sidecar.checks: |
      {
        "openmetrics": {
          "init_config": {},
          "instances": [
            {
              "openmetrics_endpoint": "http://%%host%%:9090/metrics",
              "namespace": "pii_shield",
              "metrics": ["pii_redacted_total", "pii_entropy_alerts"]
            }
          ]
        }
      }

Why this is better:

Zero Configuration for Log Shippers: Datadog just receives clean logs. There are no complex pipeline rules to manage.
Bypass Premium Vendor Fees: Datadog's built-in Sensitive Data Scanner is a premium feature billed on top of your regular log volumes. Using an open-source in-cluster sidecar can reduce your dependency on vendor-side scrubbing.
Predictable Performance: PII-Shield utilizes low-allocation JSON parsing and consumes a mere ~30Mi of memory (footprint). For a sidecar running in every pod across your cluster, this negligible resource footprint is critical.
Easy Debugging: With deterministic hashing, user@email.com becomes something like [HIDDEN:a1b2c3]. You can still trace that same user across your Datadog logs for debugging without ever knowing their real email.

By putting the shield right where the data is generated, you protect your users' privacy and keep your observability bills in check.

Ready to secure your Kubernetes logs?

Check out the PII-Shield repository on GitHub, try out the Helm chart, and if you find it useful, consider dropping a star!