RWD lakehouse
Last reviewed: 2026-06-28 — see the freshness policy.
Problem & context
An organization wants to run real-world-evidence studies across clinical and claims data from multiple sources, each with a different schema and vocabulary, with governed access to PHI and reproducible results acceptable to regulators.
Requirements (NFRs)
| ID | Requirement | Source |
|---|---|---|
| NFR-1 | Analysts query one model, not N source schemas | Research |
| NFR-2 | Column-level PHI access control + audit | HIPAA/HITRUST |
| NFR-3 | Reproducible: pinned data + vocabulary versions | RWE credibility |
| NFR-4 | Incremental ingest at TB scale (no full reloads) | Cost/scale |
| NFR-5 | Cross-org collaboration without moving PHI | Partnerships |
Architecture
flowchart LR
S1["FHIR $export"] & S2["Claims (X12)"] & S3["Labs"] --> Bronze["Bronze (raw, immutable)"]
Bronze --> Silver["Silver (conformed)"]
Vocab["OMOP vocabularies"] --> Gold
Silver --> Gold["Gold: OMOP CDM"]
Gold --> DQ["DQ (Achilles / DQD)"]
Gold --> RWE["Cohorts / RWE (ATLAS, HADES)"]
Gold --> Share["Clean room / secure share"]
Gov["Governance: masking, lineage, audit"] -.-> Bronze & Silver & Gold
Built from the medallion lakehouse, OMOP on cloud gold, governance & data contracts, and tokenized linkage for external sources. See Building a health data lake & lakehouse for the practical per-modality build guide this architecture assumes.
Key decisions & trade-offs
- OMOP as gold vs source-specific marts → OMOP for cross-source comparability and the OHDSI tool ecosystem; accept upfront mapping cost.
- Lakehouse vs warehouse → lakehouse for cheap raw retention + ML; Snowflake variant narrows the ops gap for SQL-first teams.
- Tokenization for external linkage → link trial/claims/EHR by patient token without exposing PII; manage residual re-identification risk on the linked set.
- Clean room / secure share for collaboration (NFR-5) → partners compute without seeing row-level PHI.
Compliance mapping
Unity Catalog / Horizon masking + row filters (NFR-2) · immutable bronze + pinned versions + lineage (NFR-3, GxP/21 CFR Part 11 integrity) · residency-pinned storage (regional) · audited access.
Cost
Storage-heavy but cheap object tiers for bronze; compute auto-suspends; biggest lever is co-locating compute with storage to avoid egress (see TCO).
Lab
hls-lakehouse-rwd — OMOP lakehouse on Databricks and Snowflake.