Zero-code PII sanitization for Elasticsearch, Logstash, and Kibana (ELK) in Kubernetes

TL;DR: redact logs inside the pod before Filebeat, Fluentd, or Logstash sends them to Elasticsearch, while keeping stable masked IDs for Kibana troubleshooting.

The ELK Stack (Elasticsearch, Logstash, Kibana) remains the undisputed heavy-weight champion of enterprise log management. However, its greatest strength—powerful full-text indexing and visualization—is also its biggest privacy liability.

If Personally Identifiable Information (PII) like emails, credit card data, or internal API tokens make it into Elasticsearch, that sensitive data becomes searchable across your observability stack. This can expose secrets through Kibana dashboards and create privacy remediation work that may require painful re-indexing to fix.

The Standard Logstash Approach: grok and gsub

The traditional defense mechanism in the ELK ecosystem is to filter data centrally using Logstash (or Fluentd/Filebeat processors) before it hits Elasticsearch. A typical Logstash pipeline relies heavily on the mutate filter and regex substitution:

filter {
  if [kubernetes][labels][app] == "payment-service" {
    mutate {
      gsub => [
        # Match emails
        "message", "[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+", "[REDACTED_EMAIL]",
        # Match generic API tokens
        "message", "Bearer [a-zA-Z0-9\-_]+\.[a-zA-Z0-9\-_]+\.[a-zA-Z0-9\-_]+", "Bearer [REDACTED_TOKEN]"
      ]
    }
  }
}

Why central Logstash filtering breaks down:

1. The CPU Bottleneck: Logstash is notoriously resource-intensive (running on the JVM). Forcing it to execute complex regex gsub operations over millions of log lines per minute creates severe bottlenecks, forcing you to scale up expensive central Logstash clusters.
2. The Whack-a-Mole Game: You are forced to maintain a colossal, ever-growing list of regex patterns for every new token format your microservices invent.
3. Loss of Kibana Tracing: If you blindly replace every email with [REDACTED], your support engineers lose the ability to track a user's journey through Kibana. You can't filter a dashboard to see the timeline of a specific user if all users look identical.

Ensuring JSON Integrity for Logstash

A common fear when using log scrubbers is that they might corrupt JSON structures, breaking downstream Grok patterns or Elasticsearch mapping. PII-Shield is fully JSON-aware. It surgically redacts values while keeping all keys and structural integrity perfectly intact.

// Before (Raw Application JSON):
{"level":"error", "context":{"user_id":123, "email":"admin@corp.com", "fail_count":5}}

// After (PII-Shield Output):
{"level":"error", "context":{"user_id":123, "email":"[REDACTED]", "fail_count":5}}

Your existing Logstash json filters will parse this output flawlessly.

The Edge Defense: PII-Shield Sidecar

Instead of battling regex bottlenecks at the center of your ELK pipeline, you can sanitize data at the absolute edge—within the Kubernetes pod itself—using PII-Shield.

PII-Shield intercepts the application's output, scrubs it with Go-based entropy detection, and then passes it safely to stdout for your DaemonSet (Filebeat or Fluentd) to collect and ship to Logstash.

The Result:

// What your app generated:
{"level":"info", "message":"Processing payout", "email":"john.doe@gmail.com", "stripe_key":"sk_live_51Mabc..."}

// What Filebeat ships to Logstash and Elasticsearch:
{"level":"info", "message":"Processing payout", "email":"[HIDDEN:e9f1a2]", "stripe_key":"[REDACTED:entropy]"}

How it works:

1. The App redirects its output to an ephemeral shared volume (like an emptyDir) instead of the main stdout.
2. PII-Shield (Sidecar) tails that ephemeral file, scrubs it using low-allocation Go routines and mathematical entropy detection (no regex needed for API keys!), and prints the clean logs to its own stdout.
3. Filebeat/Fluentd natively scrapes the sidecar's stdout just like any regular container log.

Your ELK pipeline needs zero configuration changes. Data arrives at Logstash already sanitized.

Kubernetes Implementation

Here is how you configure the pod natively. Since Filebeat reads /var/log/containers/*.log, it automatically discovers the sidecar's clean output.

apiVersion: v1
kind: Secret
metadata:
  name: pii-shield-secret
type: Opaque
stringData:
  pii-salt: "replace-with-a-long-random-value"
---
apiVersion: v1
kind: Pod
metadata:
  name: payment-service
  labels:
    app: payments
spec:
  containers:
    - name: payment-app
      image: payment-app:v3.0.0
      # The app writes its private output to an ephemeral pipe/file
      command: ["/bin/sh", "-c"]
      args: ["./payment-binary > /var/run/logs/app.log 2>&1"]
      volumeMounts:
        - name: log-volume
          mountPath: /var/run/logs

    - name: pii-shield-sidecar
      image: thelisdeep/pii-shield:2.2.0
      env:
        - name: PII_SALT
          valueFrom:
            secretKeyRef:
              name: pii-shield-secret
              key: pii-salt
      # Scratch image: run the binary directly (no shell/tail). Reads, scrubs,
      # and outputs to stdout for Filebeat to pick up.
      command: ["/pii-shield"]
      args: ["--watch-file", "/var/run/logs/app.log"]
      volumeMounts:
        - name: log-volume
          mountPath: /var/run/logs

  volumes:
    - name: log-volume
      emptyDir: {}

*Pro tip: The thelisdeep/pii-shield image is multi-arch (amd64/arm64).*

Filebeat Configuration Example

Here is a snippet of a filebeat.yml configuration that correctly harvests the cleaned logs from the PII-Shield sidecar and ships them to Elasticsearch:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      templates:
        - condition:
            equals:
              kubernetes.container.name: "pii-shield-sidecar"
          config:
            - type: container
              paths:
                - /var/log/containers/*-${data.kubernetes.container.id}.log
              json.keys_under_root: true
              json.add_error_key: true

output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  index: "clean-logs-%{+yyyy.MM.dd}"

Setting Up Kibana Index Patterns

Once your clean logs are in Elasticsearch, finding security events is simple:

1. Create an Index Pattern: In Kibana, go to Stack Management -> Index Patterns and create a pattern like clean-logs-*.
2. Search for Tags: Use KQL (Kibana Query Language) to quickly locate redacted data. For example: message: "[REDACTED]" or email: "[HIDDEN*".
3. Build Security Dashboards: Create visualizations based on the frequency of the [REDACTED:entropy] keyword to monitor how often your applications attempt to leak unknown secrets.

Why this is a practical ELK upgrade:

• Save Massive Logstash CPU Costs: By offloading regex string manipulation to distributed Go sidecars, your Logstash clusters only need to route and parse JSON, allowing you to downsize your central logging infrastructure.
• Deterministic Hashing for Kibana: PII-Shield replaces identifiers with stable HMAC hashes using the PII_SALT (e.g., [HIDDEN:e9f1a2]). In Kibana, you can now build dashboards or filter Discover search results by exactly email: "[HIDDEN:e9f1a2]" to track a workflow without legally compromising the user.
• Smart Entropy Detection: Unlike manual gsub regexes, PII-Shield mathematically calculates Shannon Entropy to automatically detect and redact unknown API keys and secrets in real-time.
• Micro-footprint at the Edge: PII-Shield uses <30Mi of memory, keeping your application pods lightweight.

Keep your Elasticsearch indices compliant and clean.

Check out the PII-Shield repository on GitHub and drop a star if this simplifies your ELK observability stack!