Mask PII in Grafana Alloy logs without a stage.replace pattern list

TL;DR: Alloy can mask log lines with loki.process and stage.replace, and it even hashes with a salt if you ask it to. What it cannot do is mask a value before that value is written to the node's disk — by the time Alloy reads the line, the raw secret has already been sitting in /var/log/pods. Redact inside the pod with a PII-Shield sidecar, and Alloy collects lines that were never sensitive.

Everyone is rewriting their log pipeline right now

Promtail reached end of life on 2 March 2026. Its features were merged into Grafana Alloy, and every Promtail deployment is being converted — by hand or with alloy convert — into Alloy components.

Which makes this a good moment to ask an awkward question about the pipeline being rewritten: if it masked personal data before, where exactly did that masking happen?

The standard approach: loki.process and stage.replace

Alloy's answer to redaction is a processing stage between the source and the write component. A typical Kubernetes setup discovers pods, tails their log files, runs them through loki.process, and writes to Loki:

discovery.kubernetes "pod" {
  role = "pod"
  selectors {
    role  = "pod"
    field = "spec.nodeName=" + coalesce(sys.env("HOSTNAME"), constants.hostname)
  }
}

discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pod.targets

  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }
}

loki.source.kubernetes "pod_logs" {
  targets    = discovery.relabel.pod_logs.output
  forward_to = [loki.process.redact.receiver]
}

loki.process "redact" {
  forward_to = [loki.write.default.receiver]

  stage.replace {
    expression = "([a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+)"
    replace    = "[REDACTED_EMAIL]"
  }

  stage.replace {
    expression = "(Bearer [A-Za-z0-9\\-_.]+)"
    replace    = "[REDACTED_TOKEN]"
  }

  // Hashing keeps correlation: the same value always yields the same digest.
  stage.replace {
    expression = "user_id=(\\d+)"
    replace    = "{{ Hash .Value \"a-long-random-salt\" }}"
  }
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Credit where it is due: that last stage is better than most collectors offer. Alloy's replace field takes a Go template, and the Hash function takes a salt and returns a SHA3-256 digest — so a masked value stays consistent across lines and your dashboards keep grouping by user without exposing anyone. If you are going to mask in the collector, mask like that, not with a constant.

Run that exact configuration, though, and watch what comes out the other end. Feed it one line carrying an email, an Authorization header and a Stripe key, and the email becomes [REDACTED_EMAIL], the header becomes [REDACTED_TOKEN], the same user_id hashes to the same digest on every line it appears in — and sk_live_4eC39HqLyjWDarjt ships to Loki exactly as it was written, because nothing in the list above covers it. That is the whole failure mode in one line of output: the stages you wrote work, and the secret you did not think of leaves silently.

Where this stops being enough:

1. The value has already left the pod. loki.source.kubernetes reads what the kubelet already wrote to the node — the raw line exists in /var/log/pods before any stage runs. Anything else on that node with read access to those files (a second collector, a debugging sidecar, a node agent, someone with kubectl debug) sees the unmasked value, and your log retention on the node is now a copy you did not plan for.
2. You have to enumerate the shapes. Every stage matches one RE2 expression. Emails, bearer tokens and your internal ID format are three stages; the API key format a vendor introduced last month is a fourth that nobody has written yet. A stage that does not match fails silently — there is no signal that a secret went past.
3. The cost lands on the shared collector. Alloy usually runs one instance per node for every pod on it. Each stage is evaluated against each line, so the work scales with total log volume multiplied by the length of your pattern list, on a process your whole node shares.

The alternative: redact inside the pod

Move the redaction one step earlier — into the pod that produced the line — and the three problems above disappear together. PII-Shield is a small Go sidecar that reads the application's log file, scrubs it, and prints clean lines to its own stdout. It finds secrets by Shannon entropy and key context rather than by a list of shapes, so an unfamiliar token format is still caught.

// What your app wrote:
{"level":"info","msg":"payment accepted","email":"john.doe@gmail.com","user_id":"88213","token":"sk_live_4eC39HqLyjWDarjt"}

// What Alloy reads from the node:
{"level":"info","msg":"payment accepted","email":"[HIDDEN:adc335]","user_id":"88213","token":"[HIDDEN:4a692a]"}

The marker is a salted HMAC of the value, so [HIDDEN:adc335] is the same person on every line and in every service that shares the salt — the same correlation the Hash template gives you, except the raw value never reached the node's disk to begin with.

Note what did not change in that line: user_id is still 88213. A bare number has no shape to recognise and user_id is not a secret-bearing key name, so the scanner leaves it alone rather than guessing. If that identifier is personal data in your threat model, it goes in as a named rule through PII_CUSTOM_REGEX_LIST — the same explicit decision the stage.replace above was making, taken once and applied in every pod instead of in each collector config.

Kubernetes implementation

The application writes to a shared emptyDir instead of stdout; the sidecar owns stdout for the pod. Alloy needs no knowledge of any of this.

apiVersion: v1
kind: Secret
metadata:
  name: pii-shield-secret
type: Opaque
stringData:
  pii-salt: "replace-with-a-long-random-value"
---
apiVersion: v1
kind: Pod
metadata:
  name: billing-service
  labels:
    app: billing
spec:
  containers:
    - name: billing-app
      image: billing-app:v2.1.0
      # The app writes its private output to a shared file
      command: ["/bin/sh", "-c"]
      args: ["./billing-binary > /var/run/logs/app.log 2>&1"]
      volumeMounts:
        - name: log-volume
          mountPath: /var/run/logs

    - name: pii-shield-sidecar
      image: thelisdeep/pii-shield:2.2.3
      env:
        - name: PII_SALT
          valueFrom:
            secretKeyRef:
              name: pii-shield-secret
              key: pii-salt
      # Scratch image: run the binary directly (no shell, no tail).
      command: ["/pii-shield"]
      args: ["--watch-file", "/var/run/logs/app.log"]
      volumeMounts:
        - name: log-volume
          mountPath: /var/run/logs

  volumes:
    - name: log-volume
      emptyDir: {}

Your Alloy config then shrinks back to collection and labelling — the redaction stages come out:

loki.source.kubernetes "pod_logs" {
  targets    = discovery.relabel.pod_logs.output
  forward_to = [loki.process.pod_logs.receiver]
}

loki.process "pod_logs" {
  forward_to = [loki.write.default.receiver]

  stage.static_labels {
    values = {
      cluster = "production",
    }
  }
}

Verify it is actually working

Check the sidecar's own output first. Querying Loki only tells you what arrived; it does not tell you whether the raw stream is still being written somewhere on the node because a mount path was wrong.

# This is exactly what Alloy will collect
kubectl logs billing-service -c pii-shield-sidecar --tail=50

# Nothing raw should survive
kubectl logs billing-service -c pii-shield-sidecar --tail=200 | grep -E "@|Bearer|sk_live" | grep -v "\[HIDDEN:"

# And the app container should now be quiet: its output goes to the file, not to stdout
kubectl logs billing-service -c billing-app --tail=20

The second command should print nothing. If it prints a line, that format is not being caught yet — report it as a scanner bug rather than assuming the pipeline is safe. The third should be empty too; if it is not, the application is still writing to stdout and Alloy is collecting the raw stream alongside the clean one.

Honest limitations

This covers the logs path only. Metrics and traces that your instrumentation sends straight to Alloy over OTLP never pass through the sidecar, so if span attributes carry personal data, handle those in Alloy's own processors.

Detection is not magic either. Entropy and key context catch values with a shape — tokens, keys, contact details, card numbers. A customer's name sitting in a free-text field has no shape, and no threshold finds it; that needs a model, which is a different cost. Your own internal identifier formats do have a shape, but it is yours, so it goes in as a rule (PII_CUSTOM_REGEX_LIST).

Nothing here retroactively cleans what has already been shipped to Loki, and none of it is a substitute for not logging secrets in the first place — it is the last line of defence for what slips through anyway. And if you already have a stage.replace list that works, there is no reason to delete it on day one: the sidecar in the pod and the stage in the collector compose fine, and running both while you build confidence costs you nothing but a few regex evaluations.

Common questions

Verified on 9 September 2026: the Alloy configuration and the manifest above were run as printed — Grafana Alloy v1.19.2, Kubernetes v1.35 (kind), PII-Shield 2.2.3. Alloy collected the sidecar's output with no configuration change of its own, and every entry it received carried markers instead of values.

Stop maintaining a pattern list inside your collector.

Check out the PII-Shield repository on GitHub and drop a star if this simplifies your Alloy pipeline!