Hardening Grafana: −68% CVEs, the Rest Proven Unreachable

A full teardown of how we hardened grafana/grafana:13.0.3 — what we deleted, what we proved harmless, and how to reproduce every step. 54 CVEs down to 17 (−68%): 37 physically removed, 17 proven unreachable with evidence.

Trivy reports 54 vulnerabilities, 29 of them HIGH, for grafana/grafana:13.0.3 — the current stable release at the time of writing. The reflexive fixes don't work here: upgrading to 13.1.0 makes things worse (83 findings), and no amount of apk upgrade touches CVEs baked into a Go binary. So this became a two-part job.

The split falls out of one question: which findings live in Grafana itself, and which came along with the Alpine base layer? It turns out 24 of the 29 HIGH are outdated curl/libcurl packages that Grafana never touches at runtime — those can simply be deleted. The rest sit in Go dependencies compiled into the Grafana binary: they can't be removed, but it can be proven that the vulnerable code is unreachable in the way Grafana uses those dependencies.

The end state: 54 → 17 (−68%). Roughly 37 vulnerabilities are physically gone (packages removed), and each of the 17 that remain has an unreachability proof attached. Below is the full walkthrough — Dockerfile, reproduction commands, and the parts worth stealing for your own images. The setup:

Scan before hardening

The official image was scanned with Trivy 0.71.2, vulnerability scanner only:

trivy image --scanners vuln --format json \
  --output grafana-13.0.3.json grafana/grafana:13.0.3

Trivy detected Alpine 3.24.1 and three target types: OS packages (alpine), static Go binaries (gobinary) and frontend-plugin npm manifests (node-pkg). The picture across both image variants:

Almost everything is fixable (52 of 54) — a reliable sign that most of it goes away by upgrading or removing packages. Breaking findings down by source shows where the pain actually lives:

Diagnosis: 24 of 29 HIGH are curl/libcurl (the same 12 CVEs counted once per package), not Grafana code. That's the biggest slice, and it's exactly what hardening removes.

Hardening the image

First: verify the packages really are unused

Deleting packages blindly is a reliable way to break a container. So before touching the Dockerfile, every assumption was checked against the live image:

• ldd $(which grafana) → “not a valid dynamic program”: the Grafana server is a static Go binary and needs no system .so libraries.
• /run.sh never calls curl/wget (grep comes back empty) — plugin installs go through grafana cli, which uses Go's own HTTP client.
• bash stays: /run.sh has a #!/bin/bash shebang and is the entrypoint.
• No setuid/setgid binaries (find / -perm /6000 is empty), and the default user is non-root (uid 472).

Conclusion: curl, wget, libcurl, c-ares can be removed, not merely patched.

The Dockerfile

ARG BASE_IMAGE
FROM ${BASE_IMAGE}

USER root

# 1. Patch every outdated Alpine package.
# 2. Fully remove curl/wget/libcurl/c-ares: the Grafana server is a
#    static Go binary and /run.sh never calls them. bash stays.
RUN apk update \
    && apk upgrade --no-cache \
    && apk del --no-cache curl wget libcurl c-ares \
    && rm -rf /var/cache/apk/*

USER grafana

A pleasant side effect: apk del also swept out curl's transitive dependencies — libpsl, nghttp2-libs, zstd-libs, brotli-libs, libidn2, libunistring. Less attack surface for free. The build:

docker build --build-arg BASE_IMAGE=grafana/grafana:13.0.3 \
  -f Dockerfile.hardened -t grafana-hardened:13.0.3 .

Runtime hardening

The image is only half the job. The container runs with runtime restrictions (verified: Grafana starts and is fully functional under all of them):

docker run -d --name grafana-hardened-1303 -p 3003:3000 \
  --read-only \
  --tmpfs /tmp --tmpfs /var/lib/grafana --tmpfs /var/log/grafana \
  --cap-drop ALL \
  --security-opt no-new-privileges:true \
  -e GF_SECURITY_ADMIN_USER=admin -e GF_SECURITY_ADMIN_PASSWORD=*** \
  grafana-hardened:13.0.3

Verifying nothing broke

Any hardening pass has to end with a functional check. A 13-check smoke test covered health, auth, datasource CRUD with health and query, dashboards, the plugin catalog, service accounts and tokens, the alerting provisioning API, internal metrics and frontend settings. The score: 13/13 PASS on the baseline and 13/13 PASS on the hardened container with read-only rootfs and dropped capabilities.

The results

Every OS-level vulnerability is gone — removed, not patched: apk info in the hardened image lists no curl, libcurl, c-ares, nghttp2 or brotli. The remaining 17 findings split across two Go binaries with different toolchains — a detail that matters for everything that follows:

Why the rest gets proven, not patched

The 17 remaining findings can't be fixed at the Dockerfile level: the vulnerable code is compiled into Grafana's static binary. A real fix needs an upstream Grafana release built with updated dependencies (tempo 2.10.3, prometheus 0.311.3) and the Go 1.26.5 toolchain — which doesn't exist yet as of the analysis date (even the 2026-07-09 nightlies ship the same unpatched dependencies). With no immediate fix available, the only honest path is to prove the vulnerable code unreachable and accept the residual risk on that basis.

Two properties, four methods

The whole Go-binary analysis rests on two facts: (1) dead-code elimination — the Go linker drops functions that are never statically called, so a missing symbol means guaranteed unreachability; and (2) “symbol present” ≠ “symbol called”. That yields four methods, in increasing order of rigor:

Trivy and govulncheck count different things

Don't be alarmed when the numbers disagree — the tools look at different layers.

“Trivy found 17” and “govulncheck found 351” is not a contradiction — they're different slices. govulncheck digs deeper (it sees hundreds of transitive CVEs), but the vast majority are unreachable and get filtered automatically by symbol analysis. Use both: Trivy for the inventory, govulncheck for reachability.

Why binary mode alone isn't enough: the x509 example

For CVE-2026-27145 (crypto/x509) in the plugin, the symbols Certificate.Verify / VerifyHostname are present — binary mode says “reachable”. Source mode proves the opposite: they are import-only, never called from the plugin's reachable code (they exist only as transitive baggage of net/http/crypto/tls). Had the analysis stopped at binary mode, the verdict would have been wrong. That's exactly why source mode was used for the plugin — it settles these disagreements in favor of the truth.

Unreachability proofs

Each finding class got the strongest method available; where possible, methods cross-confirm each other.

Tempo — DoS and S3 key disclosure (HIGH)

Both CVEs live in Tempo's HTTP server. go tool nm shows that only protobuf types (the trace/TraceQL data model) and pkg/pool are linked in; Tempo's server packages (config, status, query-frontend) are absent. Runtime confirms it: GET /status/config → 404. The vulnerable server isn't in the binary — unconditionally unreachable.

Prometheus — client_secret disclosure and XSS (HIGH/MED)

All three CVEs are in Prometheus's web server/UI. Checking for the host component — nm | grep -c prometheus/prometheus/web → 0; the endpoints /-/config, /api/v1/status/config, /graph all return 404. Grafana uses Prometheus as a PromQL/config library; the web server isn't linked. Unconditionally unreachable.

OpenFGA — Zanzana is off (MED/LOW)

OpenFGA ships inside Grafana as the experimental Zanzana authorization engine. Its OIDC authenticator isn't linked at all (nm → 0) — CVE-2026-55689 is unconditionally unreachable. Two more CVEs (iterator cache, MySQL) are linked, but on the live container no zanzana feature toggle is enabled (the default), so the OpenFGA server never starts and the code never enters the execution path. Conditionally unreachable (condition: zanzana stays off).

stdlib in the Elasticsearch plugin — proven by call graph (HIGH/MED)

The plugin is built with go1.26.3 (vulnerable by version) and stripped, so source mode was used: the plugin sources were cloned at the exact commit and govulncheck -mode=source produced the true call graph. All four stdlib CVEs (x509, mime, textproto, os.Root) came back import-only: the symbols exist as transitive baggage of net/http/crypto/tls but are never called from the plugin's reachable code. A call-graph proof beats any data-flow reasoning.

ECH — the weakest claim (UNK)

CVE-2026-42505 concerns de-anonymizing ECH handshakes. Source mode showed tls.Dialer.Dial is reachable (CheckHealth → net/http → SOCKS5 → tls.Dialer.Dial). But only the ECH branch is vulnerable — an option that requires explicitly setting EncryptedClientHelloConfigList, which Grafana never configures. Conditionally unreachable — and honestly the weakest link in the whole analysis: the TLS symbol is in the execution path, and the justification is configurational, not “code not present”.

openpgp — the package simply isn't there

GO-2026-5932 covers the abandoned x/crypto/openpgp (no fix will ever exist). The package isn't linked into either binary (nm/strings → 0). Unconditionally unreachable.

The full verdict matrix

Three verdicts are conditional

Eleven of the fourteen unique CVEs are unconditionally unreachable — the code isn't linked, or a call graph proves it's never invoked. Three hold only under configuration preconditions, and keeping those preconditions true is on whoever operates the deployment:

“Fixed” ≠ “unreachable”

This distinction is the crux of interpreting the results honestly. The OS vulnerabilities are actually fixed — the packages no longer exist. The 17 Go findings are NOT fixed: the code is still there; only its unreachability has been proven.

Practical consequences: scanners will keep flagging them (that's what VEX is for); calling them “fixed” would be wrong — the correct phrasing is “unreachable in this configuration”; and if the reachability analysis is mistaken, the vulnerability is live. Hence the emphasis on reproducibility and independent review.

VEX: the machine-readable verdict

All verdicts were exported as OpenVEX 0.2.0. One trap cost the first attempt:

trivy image --vex openvex-trivy.json \
  --show-suppressed grafana-hardened:13.0.3

# Without VEX:  Total: 17  (HIGH: 5, MEDIUM: 5, LOW: 1, UNKNOWN: 6)
# With VEX:     Total: 0   + Suppressed Vulnerabilities (17)

All 17 findings move to the Suppressed section with a not_affected status, a justification, and a pointer to the VEX source — nothing is hidden, everything is traceable.

How much to trust the VEX

Honest weaknesses that reduce confidence: VEX ≠ remediation (the code stays); the ECH justification is imprecise (tls.Dial is provably in the execution path); a coverage gap — go-pkcs12 (GO-2026-5052) is invisible to Trivy and absent from the VEX, so “Trivy+VEX = 0” ≠ “zero vulnerabilities”; and it's self-attestation — the VEX was written by the same party that ran the analysis, so an independent reviewer is required.

Supply chain

To make the artifact trustworthy, an SBOM was generated (Syft, SPDX 2.3 + CycloneDX 1.6, 704 components) along with a cosign signature over the digest and an in-toto SBOM attestation. Verification passes with the public key (exit 0), and the negative control holds — a foreign key is rejected (exit 12).

Scanner pitfalls (the cheat sheet)

The most valuable takeaway from this work is the list of places where a scanner quietly misleads you. All in one table:

Takeaways

1. Use the hardened image (17 residual findings, all with proofs) over the base 13.0.3 (54) — and especially over 13.1.0 (83).
2. Enforce conditions §A/§B automatically: a guard check that zanzana is off and ECH is not configured.
3. Wire VEX into CI (trivy --vex) with --show-suppressed — a green report that stays transparent.
4. Watch upstream Grafana: a release on Go 1.26.5 with updated tempo/prometheus will fix the residue physically — then the VEX entries can be retired.
5. Take go-pkcs12 (§C) to a definite verdict.
6. Set up independent review and formal risk acceptance (separation of duties).
7. For production supply chains — keyless signing + Rekor + SLSA provenance from CI.

The short version: hardening isn't “run the scanner until it shows zero”. It's two honest operations — physically remove everything the workload doesn't need, and attach evidence to whatever remains. The first shrinks the attack surface; the second turns a red report into a decision you can defend in front of an auditor.