When Microsoft announced Fabric at Build 2023, the headline was "unified analytics platform." That's true, and also about as descriptive as calling a car "a vehicle." What Fabric actually is at the infrastructure level is more interesting — and more specific — than the marketing suggests. After spending significant time in its internals, here are the things I wish someone had explained to me at the start.
OneLake: ADLS Gen2 with Strong Opinions
OneLake is built on top of Azure Data Lake Storage Gen2. That's not a criticism — ADLS Gen2 is excellent object storage and the right foundation. What Fabric adds is a strong, deliberate opinion about data organization and format.
Every Fabric item — lakehouses, warehouses, semantic models, real-time eventstreams — stores its data in OneLake. Automatically, without you doing anything. And by default, that data lands in Delta Parquet format. This design choice is load-bearing. The reason a Spark notebook, a SQL warehouse, and a Power BI Direct Lake report can all read the same table without copying it is that they all speak Delta Lake. There's one physical copy of the data and multiple engines pointing at it.
The organization within OneLake follows a hierarchy: tenant / workspace / item / data. Items are things like lakehouses or warehouses. Within a lakehouse, you get a Tables/ area for managed Delta tables and a Files/ area for raw files. Simple, but enforced consistently across the entire platform.
flowchart TB
subgraph COMP ["Compute Layer"]
direction LR
SP["🔥 Spark\nNotebooks / Jobs"]
WH["📋 SQL Warehouse\nT-SQL queries"]
PBI["📊 Power BI\nDirect Lake"]
RT["⚡ Real-Time\nAnalytics / KQL"]
end
subgraph OL ["OneLake — single storage tier (ADLS Gen2 underneath)"]
direction LR
DT["Delta Parquet files\n+ V-Order optimization"]
LOG["_delta_log/\ntransaction log"]
DT --- LOG
end
SP <-->|"read / write"| OL
WH <-->|"read / write"| OL
PBI -->|"frame + lazy\ncolumn load"| OL
RT <-->|"read / write"| OL
EXT["External data\nADLS · S3 · GCS"] -. "Shortcut\n(virtual pointer,\nno data copy)" .-> OL
One physical copy of each Delta table. Every compute engine reads from the same place. Shortcuts let external data participate without moving it.
The Delta Transaction Log: How Time Travel Actually Works
A lot of people know Delta Lake provides ACID transactions and time travel. Fewer understand the mechanism, which matters when you start hitting performance problems.
Every Delta table has a _delta_log/ directory alongside its Parquet data files. Inside are numbered JSON files — one per transaction — each recording what happened: which Parquet files were added, which were removed, any schema changes. A query like SELECT * FROM table VERSION AS OF 5 doesn't reach into the past; it replays the transaction log to version 5, identifying the exact set of Parquet files that constituted the table at that point, and scans those.
lakehouse/Tables/orders/_delta_log/
00000000000000000000.json ← initial table creation
00000000000000000001.json ← first INSERT batch
00000000000000000002.json ← UPDATE
00000000000000000003.json ← another INSERT
00000000000000000010.checkpoint.parquet ← checkpoint (compacted log)
Every 10 transactions (by default), Delta writes a checkpoint file — a Parquet snapshot of the entire log state up to that point. Subsequent reads only need to replay from the most recent checkpoint forward, not from transaction 0.
Why this matters practically: small, frequent writes are a problem in two ways. First, you accumulate lots of small Parquet data files, and each query has to open and scan more of them. Second, your transaction log grows steadily. Delta's OPTIMIZE command (or Fabric's automatic table maintenance) rewrites many small files into fewer large ones and rewrites the log. Skipping this on busy tables eventually shows up as slower-than-expected query times — the culprit isn't your data volume, it's your file count.
V-Order: Microsoft's Write-Time Secret
This is one of the least-discussed features of Fabric and one of the most impactful for Direct Lake workloads.
V-Order is an optimization Fabric applies at write time when creating Parquet files. It rearranges column data within the file to improve run-length encoding quality and to match the reading patterns of VertiPaq (Power BI's in-memory engine). The result: Parquet files that can be loaded into VertiPaq significantly faster than standard Parquet, with meaningfully better compression.
The performance difference is real — up to 50% faster for typical Power BI analytical queries on V-Ordered vs. standard Parquet. Fabric enables V-Order by default for all writes from native Fabric experiences: Spark notebooks, Data Factory pipelines, Warehouse operations.
The V-Order gap: Parquet files written by external tools — Databricks, Azure Data Factory pointed at external storage, anything writing to ADLS Gen2 that isn't Fabric-native — don't get V-Order applied. Direct Lake performance on those files will be noticeably worse. The fix is to run OPTIMIZE on those tables within Fabric after ingestion, which rewrites the files with V-Order applied. This trips up a lot of hybrid architectures where data engineering happens outside Fabric.
Direct Lake Framing: The Part Nobody Explains Clearly
Direct Lake gets described as "reads from OneLake without importing." Technically accurate, but the interesting part is what "framing" means — because it's fundamentally different from a Power BI refresh and understanding the difference matters for architecture decisions.
A Direct Lake semantic model holds a pointer to a specific version of each Delta table. Framing is the operation that updates those pointers to the latest Delta version. Fabric reads the _delta_log, identifies the current file set, and updates the metadata references. That's it. No data movement, no transcoding. It takes seconds even for large tables because it's a pure metadata operation.
What framing is not is loading data into memory. That happens lazily, as queries need column segments. The model progresses through states:
| State | Description | What happens on query |
|---|---|---|
| Cold | No columns in VertiPaq memory | First query transcodes Parquet → VertiPaq on the fly. Noticeable delay. |
| Semiwarm | Some columns cached, others not | Fast for cached columns, cold-load delay for anything uncached |
| Warm | All needed columns in memory | Import-equivalent speed |
| Hot | Warm + VertiScan result caches | Fastest possible — repeated identical queries served from cache |
When a framing event occurs (because the underlying Delta table was updated), Fabric doesn't evict your cached column segments. The old segments stay in memory and expire lazily as new data gets loaded on demand. This means Direct Lake handles continuously-updating tables very gracefully — there's no "refresh window" during which reports are slow or unavailable.
The cold state is the one to design around. A model that nobody has accessed in a while will be cold. The first user to hit it after a long idle period eats the transcoding latency. For business-critical dashboards, some teams warm their models proactively by triggering queries on a schedule — essentially keeping the model from going fully cold during expected usage hours.
Shortcuts: Virtual Pointers, Not Data Copies
Shortcuts are one of the most architecturally interesting features in Fabric, and also one of the most underused in practice.
A shortcut in a lakehouse appears as a table or folder. Queries treat it like any other lakehouse table. But the data doesn't actually live in that lakehouse — the shortcut is a virtual pointer to data stored elsewhere: another OneLake workspace, an Azure Data Lake Storage Gen2 container, an Amazon S3 bucket, or Google Cloud Storage. When a query touches a shortcut, Fabric reaches through the pointer to the source in real time. No copy is made, no ETL is involved, no storage is duplicated.
The practical architecture shift: instead of building pipelines to copy data from your engineering lakehouse into an analytics workspace, you create shortcuts. The analytics team gets a read view of the engineering team's data as it exists right now. Lineage is maintained. Storage costs stay flat. When data engineers update a table, the change is immediately visible through the shortcut.
Where shortcuts have limits: they're read-only. You can query through a shortcut but you can't write to it. Governance is also trickier — shortcuts don't automatically inherit sensitivity labels from their source. For Gold-layer tables that need active Spark writes or require fine-grained access control, managed tables in the workspace are the right call. Use shortcuts for cross-team data sharing; use managed tables for data you own and actively modify.
Medallion Architecture: The Standard Playbook (and Where Teams Go Wrong)
flowchart LR
SRC["Source Systems\nAPIs · DBs · Files · Streams"] --> B
subgraph B ["🥉 Bronze — Raw"]
B1["Exact copy of source\nAppend-only · No transforms\nGround truth for replays"]
end
B -->|"deduplicate\nschema conform\nquality checks"| S
subgraph S ["🥈 Silver — Clean"]
S1["Source-agnostic\nTechnically correct\nNo business rules yet"]
end
S -->|"apply business logic\naggregate · join domains\nencode definitions"| G
subgraph G ["🥇 Gold — Business-Ready"]
G1["Dimensional model\nFact + dimension tables\nSingle definition of truth"]
end
G --> BI["Power BI\nSemantic Models"]
G --> ML["ML / AI\nPipelines"]
The Silver layer is where teams most often cut corners — and where they pay for it six months later when five Gold tables disagree on the same metric.
Bronze-Silver-Gold has become the default answer to "how should we structure our lakehouse?" It's a good answer. But there are a few failure modes that appear with enough consistency to be worth naming explicitly.
Bronze: the historical record, full stop
Raw data goes to Bronze exactly as it arrives. No transformations, no business rules, no "fixing obvious errors." The entire point of Bronze is that it's the faithful record of what you received from the source system, when you received it. When an upstream system had a bug and you need to replay six months of history with corrected logic, Bronze is the ground truth you replay from. The moment you start cleaning data in Bronze, you lose that. Teams that skip "just obvious quality issues" in Bronze regret it the first time a data definition changes retroactively.
Silver: where trust gets built
Schema conformance, deduplication, basic quality checks, key lookups, null handling. Silver isn't analysis-ready data — it's technically clean, source-agnostic data. The consumers of Silver are Gold-layer jobs, not report users. If your Silver tables are feeding reports directly, you probably skipped building a proper Gold layer, and you're encoding business logic in your BI tool instead of the lakehouse.
Gold: expensive to change, so be deliberate
Gold tables encode business logic. Once reports and semantic models are built on Gold, changing a Gold table's definition means changing every downstream consumer. Be conservative about what business logic lives here, document it clearly, and resist the temptation to keep adding "one more business rule" until Gold becomes a place where six slightly different definitions of "active customer" coexist peacefully. They won't coexist peacefully. Six months later someone will write a seventh.
The failure mode I see most: Teams build Bronze and Gold but treat Silver as optional ("we can clean data in the semantic model or in Power Query"). Then, a year later, they have five Gold tables each with their own implementation of the same is_valid_order logic, and they disagree on the answer by about 3%. Which one is right? Nobody knows. Silver is where you make that decision once.
The Copy Storm Problem
Enterprise Fabric deployments develop a predictable failure pattern I think of as the copy storm. It goes like this: Team A needs the customer dimension. They copy it into their workspace. Team B also needs it and doesn't know about Team A's copy, so they make their own. Six months later there are four copies of the customer dimension in four workspaces, each refreshed on a different schedule, each with subtly different column names, and each with a different business owner's definition of customer_status. Reports disagree. Stakeholders are confused. Finance stops trusting the numbers.
The solution isn't a governance committee (though good luck explaining that to a large org). The practical answer is shortcuts and centralized certified semantic models: a central data team owns and publishes Gold tables in a shared lakehouse, other teams access those tables via shortcuts or build on certified semantic models rather than creating their own copies. This requires organizational buy-in that's often harder to get than the technical implementation, but the architecture should make copying the data harder than using the shared source.
Capacity, Fabric Units, and Why Your Dashboard Suddenly Froze
Fabric bills by Capacity Units (CU), and the throttling model is worth understanding before you encounter it under production load.
Fabric smooths usage across a rolling 24-hour window. Interactive operations (report loads, query execution) get a burst allowance — they can consume capacity above the provisioned rate for short periods. Background operations (pipeline runs, scheduled Spark jobs, model refreshes) consume from the same capacity pool. When the pool is exhausted, interactive queries start queuing. They wait up to 10 minutes before Fabric starts rejecting them with throttling errors.
The retry mechanism uses exponential backoff with jitter. First retry after 1 second, then 2 seconds, 4, 8 — with randomization at each step to prevent a thundering herd when capacity releases and every queued request tries to proceed simultaneously. This is sensible design, but the user experience when it kicks in is "my dashboard is just... hanging." If you see that pattern, check the Capacity Utilization report in the Fabric admin portal. A large scheduled Spark job consuming burst capacity during business hours is a common and fixable culprit.
Right-sizing capacity is more nuanced than picking a SKU. You need to understand your concurrency patterns (how many users are hitting reports simultaneously), your background job schedule (are Spark jobs competing with peak dashboard usage?), and whether you're using Direct Lake (which amortizes transcoding costs across queries) vs. Import (which front-loads all compute at refresh time). The Metrics App in Fabric gives you historical utilization data — use it before you decide whether to scale up or just reschedule your overnight pipelines.
What Makes Fabric's Architecture Actually Coherent
A lot of data platform stacks are coherent in the slide deck and fragmented in practice — you end up with Kafka here, a data warehouse there, a separate ML platform over here, and six different teams responsible for moving data between them. Fabric's architecture is coherent at a deeper level because of a few genuine design decisions rather than just branding.
Delta Parquet as the universal format means no translation layer between compute engines. Spark and SQL and Power BI Direct Lake all speak the same language. OneLake as the single storage tier means there's one place to configure governance, one place to check lineage, one billing surface for storage. V-Order as a write-time optimization means the system is optimized for the most common analytical access pattern without requiring any configuration from the developer.
The rough edges are real — the V-Order gap for external-origin data, governance complexity around shortcuts, capacity planning that requires more understanding than a spreadsheet — but they're the rough edges of a coherent design, not the fundamental incoherence of a cobbled-together platform. For teams building new data infrastructure in 2025–2026, that distinction matters.