Databricks for HLS

Last reviewed: 2026-06-28 — see the freshness policy.

Learning objectives

After this chapter you will be able to:

• Explain the lakehouse model and why it suits HLS data engineering and ML.
• Use dbignite, Glow, and Unity Catalog in an HLS reference architecture.
• Decide when Databricks is the right layer — independently of the underlying cloud.

Databricks's HLS positioning

Databricks is not a cloud — it runs on AWS, Azure, and GCP — so you choose it for the lakehouse and ML layer rather than for managed clinical endpoints. Its Lakehouse for Healthcare and Life Sciences unifies clinical, genomic, claims, imaging, and operational data on Delta Lake, governed by Unity Catalog, with first-class ML/AI. It is the strongest fit when the center of gravity is data engineering, OMOP/RWE, genomics at scale, or ML — not transactional FHIR serving.

The lakehouse model

A lakehouse combines the cheap, open storage of a data lake with the reliability and performance of a warehouse, via Delta Lake (ACID transactions, schema enforcement, time travel on object storage). For HLS this matters because you have both petabyte-scale raw data (genomics, imaging) and the need for governed, query-ready tables (OMOP) in one place — without copying between a lake and a separate warehouse.

flowchart LR
  subgraph Medallion
    Bronze["Bronze<br/>raw (FHIR bundles, VCF, claims)"] --> Silver["Silver<br/>conformed"] --> Gold["Gold<br/>OMOP CDM / features"]
  end
  FHIR["FHIR bundles"] -->|dbignite| Bronze
  Genomics["VCF / BGEN / PLINK"] -->|Glow| Bronze
  Gold --> ML["Mosaic AI / MLflow<br/>(models)"]
  Gold --> BI["Databricks SQL / BI"]
  UC["Unity Catalog<br/>(governance, lineage, masking)"] -.-> Bronze & Silver & Gold

See Lakehouse vs warehouse for the deeper comparison.

The key HLS components

Reference architecture: RWD/OMOP lakehouse

flowchart LR
  EHR["FHIR $export"] -->|dbignite| Bronze["Bronze (Delta)"]
  Claims["Claims (X12)"] --> Bronze
  Labs["Lab feeds"] --> Bronze
  Bronze --> Silver["Silver (conformed)"]
  Silver --> OMOP["Gold: OMOP CDM (Delta)"]
  OMOP --> RWE["RWE / cohorts"]
  OMOP --> Pred["Risk prediction (Mosaic AI)"]
  UC["Unity Catalog — PHI governance"] -.-> Bronze & Silver & OMOP

This is the architecture behind the hls-lakehouse-rwd lab. Bronze→Silver→Gold (medallion), with OMOP CDM as Gold and Unity Catalog enforcing PHI access controls throughout.

Governance and HIPAA on Databricks

• Unity Catalog is the centerpiece: tag PHI columns, apply dynamic masking and row filters, and get lineage + audit for every access — the evidence a HITRUST assessor wants.
• Databricks offers a HIPAA-compliant deployment (with BAA) and inherits the underlying cloud's controls — but you must enable the compliance security profile and configure workspace isolation.
• Keep data in customer-owned cloud storage; use customer-managed keys; restrict cluster egress.

When Databricks is the right call

• The use case is OMOP/RWD, genomics at scale, or ML — not transactional FHIR serving.
• The team already runs Databricks for data engineering; extending into HLS is far lower friction than adopting a new managed service.
• You need fine-grained PHI governance (Unity Catalog) across a multi-modal data estate.
• You want portability across AWS/Azure/GCP — Databricks abstracts the underlying cloud.

A common enterprise pattern: a managed FHIR service (HealthLake / AHDS / Cloud Healthcare API) handles transactional interoperability, and Databricks handles the analytical lakehouse downstream of its $export.

Lab

hls-lakehouse-rwd — OMOP CDM lakehouse on Databricks (Delta + Unity Catalog) and Snowflake.

Check yourself

1. Why do you choose Databricks for a layer rather than as a cloud, and what layer is that?
2. Which two accelerators turn FHIR bundles and genomic VCFs into Delta tables, and why is landing both in one lakehouse valuable?
3. How does Unity Catalog make a HITRUST assessment of a PHI lakehouse easier?

Reference architectures

• Databricks Lakehouse for Healthcare & Life Sciences.
• Industry Solutions — OMOP CDM on the lakehouse (GitHub).
• dbignite — FHIR bundles → lakehouse/OMOP.
• Solution Accelerators — RWE suite, disease prediction, medical imaging, NLP.

Further reading

• Databricks Lakehouse for Healthcare and Life Sciences
• dbignite FHIR interoperability accelerator
• Glow — genomics on Spark
• Unity Catalog
• Databricks Solution Accelerators (HLS reference implementations)