FHIR teams rarely fail because they lack a validator. They fail because they cannot answer simple operational questions about their data.
A validator can tell you whether a resource conforms under a given configuration. It does not, by itself, create continuous history, drift timelines, or audit-ready evidence. Those require a different operating model: continuous data quality with reproducible records.
The previous article described a personal version of this problem — a designer who couldn't understand FHIR data, and engineers who confirmed they needed the same visibility into the data/profile/validation relationship. In conversations across more projects since, a broader pattern keeps surfacing. The teams that have real confidence in their data quality can answer three specific questions. Teams that can't answer them have a gap they may not have named yet.
The questions aren't about tooling. They're not about which validator you use or how your pipeline is structured. They're about what you actually know versus what you assume.
Here they are.
Question 1: Is your production data valid right now?
Not "did it pass validation when it was ingested?" Not "does your CI pipeline pass?" Right now. The data currently at rest in your FHIR server — is it conformant to the profiles your consumers depend on?
Most teams answer this with something like: "We validate in our pipeline, so yes."
That's not an answer to the question. Pipeline validation tells you whether data was conformant at ingestion time, against the profiles and terminology available at that moment. It doesn't tell you whether it's still conformant today.
For a German hospital that just received its ISiK Stage 3 Bestätigung, part of Question 1 has been tested — at the moment of testing, against the profiles and terminology current at that moment. The day after the certification, the operational question is open again. The day after the next gematik profile update, it's open with a wider gap.
Profiles are updated. Terminology servers receive new releases. FHIR servers get upgraded. Any of these can shift the conformance status of data that's already in production — without triggering a new pipeline run, without producing an error message, without anyone noticing.
The honest answer for most teams is: "We don't know. We assume it is."
That assumption is what continuous validation is designed to replace.
Question 2: When did that error first appear?
Something is failing. A partner is rejecting resources. An audit surfaces a conformance issue. A downstream system starts returning errors.
Can you tell when it started?
Imagine a partner integration team in another organization sends you a ticket: "As of three weeks ago, 12% of the resources you sent us fail our validation. They didn't before." The clock starts. You need to know: when did the change happen? Was it a terminology update? A profile revision? An infrastructure change in your stack? A specific deployment?
This question separates teams that produce evidence from teams that produce signals. A signal tells you something is broken now. Evidence tells you when it broke, what the state was before it broke, and what changed between then and now.
To answer this question, you need validation runs that are:
- Timestamped reproducibly — not just "we ran validation on Tuesday," but a result tied to a specific validator version, specific IG packages, specific terminology snapshot
- Comparable across time — so you can diff Tuesday's results against last month's and see what changed
- Run against production data consistently — so the history reflects reality, not a curated test environment
Most teams have CI logs. CI logs are not the same as validation history. They're point-in-time signals against test fixtures, not a continuous record of production data conformance.
If you can't answer "when did this first appear," you can't do root cause analysis. You can fix the symptom, but you don't know what caused it — which means you can't prevent the next one.
Question 3: Can you prove the conformance state of your data at a specific point in the past?
An auditor is reviewing a prior authorization transaction from six months ago. A partner organization asks what your data looked like at the time of a specific exchange. A regulator wants to see your conformance posture at the release of a specific clinical study.
In all three cases, the question is the same: not "are you conformant now?" but "what was your conformance state then — and can you prove it?"
This is the question that catches teams off guard most often. It's not about the current state. It's not even about when things changed. It's about reconstructing, with evidence, what was true at a specific point in time.
A passing CI build from six months ago is not an answer. A current conformance statement is not an answer. A claim that "we've always validated" is not an answer. What's needed is a reproducible artifact: a validation run from that exact moment, tied to the validator version, IG package versions, terminology snapshot, timestamp, input scope, and environment that were in effect at the time — so that anyone can independently verify what was true.
This isn't a hypothetical concern. As FHIR moves into regulated infrastructure — ISiK in Germany, EHDS across Europe, the CMS Interoperability and Prior Authorization Final Rule in the US — the question of historical conformance becomes the question that determines audit outcomes and partnership trust. The organizations that can answer it have evidence. The organizations that can't have hope.
Why these questions are hard to answer
None of these questions are exotic. They're the questions any responsible operator should be able to answer about a system they run.
They're hard because they require a different model than the one most FHIR teams operate with.
The standard model is: validate at ingestion, check in CI, certify against a conformance test suite. This is snapshot validation — it captures a moment in time and moves on. Validators are necessary in that model, but they are not the operating system for data quality history.
The questions above require something different: continuous validation with history. Validation that runs on a schedule against live data, produces comparable results over time, and maintains a record that outlasts the moment of measurement.
This is not a new idea. It's how observability works for application performance. You don't only check latency when you deploy — you monitor it continuously because the system changes in ways that have nothing to do with your code.
FHIR data quality changes in exactly the same ways.
The gap is not a tooling problem
The reason most teams can't answer these questions isn't that they lack the right validator or the right CI configuration. It's that they haven't treated data quality as an ongoing operational concern.
A FHIR server without continuous validation is like an application without monitoring. It might be running fine. It might not. You'll find out when something downstream breaks.
Of the three questions, the third is the one most teams are most exposed on — because it requires not just current observation, but a preserved history of past observations. It's also the question with the sharpest external consequence: it's what auditors ask, what partners ask when trust breaks, and what regulators will increasingly require.
If you can answer Q1, you have an operational baseline. If you can answer Q2, you have drift detection. If you can answer Q3, you have evidence — and evidence is the only thing that survives the moment.