Why We're All In on Databricks (Or the Patterns Databricks Makes Easy)

Executive Summary

Thirty years, one lesson: From a 1997 radio loyalty card to Bing scale experiments, every platform I used made us copy data to get answers.
The lakehouse pattern won: Databricks, Snowflake, Microsoft Fabric, AWS SageMaker and Google BigQuery now share one pattern. Raw data lands, gets refined and is served from governed tables.
Databricks leads on AI: Gartner's 2026 Magic Quadrant for AI Platforms for Data Science and Machine Learning places Databricks highest in Ability to Execute and furthest in Completeness of Vision.
Open data, one catalog: Open formats, commodity storage and Unity Catalog put security, privacy, quality and cost under one roof. Lakebase brings application data onto the same copy.
Why we're all in: DataMind brings these patterns to mid-market businesses with enterprise-scale experience. I earned the Databricks Certified Data Engineer Professional credential so you know you're getting verified expertise on a powerful, complex platform.

Databricks Certified Data Engineer Professional badge

We All Started Somewhere

My first database project, in 1997, turned spreadsheets and paper forms into a SQL database for the X96 Freeloader Card, a loyalty program for an alternative rock radio station in Salt Lake City. I paid a database developer $5,000 (about $10,500 today)¹ to build it, plus a small tool that uploaded scanner data whenever listeners showed their cards at promotions and concerts.

The idea was brilliant, but our tools could do little beyond picking contest winners in a spreadsheet. I did learn a lot about data normalization. When I took his database apart, I found he’d split one worksheet into seven tables to store each card entry and cardholder. I was hooked.

Learning Business Intelligence on the Job

In graduate school, I freelanced on small Excel and Access projects for consultants at Microsoft and a national bank. That work led to a full time role as statistician for consumer loan operations. A team of five kept an $80B consumer loan servicing portfolio running out of Access databases, and my new boss had no dashboards. My job was to bring real time business intelligence to origination, servicing, payoff and closing.

There I learned star schemas, OLAP cubes and dashboards by doing. These technologies were still new, especially in the corporate IT departments I depended on (and often worked around). Soon we had dashboards for all 20,000 retail banking associates and a central database for planning and managing more than 2,000 branches.

Timeline from 1970 to 2026. The top lane shows the platform eras: normalized tables, warehouse and cube, scale out, lakehouse, and one copy for AI. The bottom lane shows what we built in each era: the X96 Freeloader Card, national bank dashboards, Bing search, a Microsoft lakehouse, fraud detection on AWS, AI/BI and private customer agents on Databricks, and DataMind. — *Thirty years of data platforms, and what we built on each one.*

In 2009, Microsoft hired me to help design a business intelligence system for Bing and Online Ads. It processed terabytes of data, loaded the results into a star schema and produced metrics for experiments and executive dashboards. A few hours of Bing search traffic would take down the best relational database of its time, so we had our own version of MapReduce. Through many late nights, I learned to solve problems that had been impossible (or impossibly slow) a few years earlier.

From MapReduce to Spark

Google introduced MapReduce in 2004 to process large data sets on clusters of low cost computers, first for jobs like building its web index.² Hadoop brought the model to open source software around 2006. At Bing we used Microsoft’s parallel systems, Cosmos and SCOPE, which ran SQL style jobs with custom C# code across tens of thousands of machines.³

The model changed how we designed data systems:

Data locality: Move the computation to the machines that hold the data.
Full scans: Scan large data sets instead of relying on indexes and lookups.
Schema on read: Define the schema when the system reads the data, so new data is easier to handle.
Wide tables: Use denormalized tables for event and log data, even when values repeat.
Append only storage: Keep the original data and add records instead of updating them.
Fault tolerance: Recover from machine failures by running failed tasks again.

These platforms broke the warehouse rules. A warehouse ran under tight memory and expensive custom storage, so it normalized everything and updated in place. Big data accepted duplicate values, kept every record, and ran batch jobs over terabytes and petabytes of cheap disk.

But Hadoop had a speed problem. Ben Horowitz and Michael McGuiness of a16z, an early Databricks investor, put it plainly: “Hadoop’s critical flaw was that each round of a computation saved its results to disk before the next could begin.”⁴ Machine learning suffered most, because every training round reloaded the same data. In 2012, researchers at the Berkeley AMPLab introduced Spark, which kept data in memory and cut the slow disk trips between chained MapReduce jobs.⁵ The Spark creators founded Databricks in 2013 to hide the clusters, scaling and security behind one platform for the enterprise.⁴

As I moved from statistician to manager of statisticians to PM to data scientist, everyone around me, myself included, was winging it. We had new technology, new data and sometimes only brute force to get answers to decision makers. Over time the patterns became tools, then automation, then part of the enterprise platform.

The Enterprise Platform Emerges

By 2016 I was leading some of the best data scientists money can buy at Microsoft. Some came from top PhD programs and research institutions, and others, like me, had scraped through hard problems with yesterday’s platforms. Everyone on the team used different tools: the command line, local Jupyter notebooks on sandbox compute, early Azure tools and a few homegrown kits. We could do data science, but we couldn’t build on each other’s work without rebuilding the tooling too.

Then Microsoft did the deal with Databricks, and this magical tool popped up inside Azure. Within a year every notebook and pipeline lived in the same place. The team deployed models and pipelines from the same code we used for exploration and training. Our throughput went through the roof.

Databricks has since grown into one of the largest data and AI companies, valued at $190 billion in August 2026 on a $7 billion revenue run rate,⁶ and it puts on my favorite tech conference, the Data + AI Summit.⁷ The financial figures are Databricks’ own, and they are now in dispute. On October 28, Snowflake’s CEO asked Databricks in public to “release all the numbers” and to say how much of that growth is pass-through model inference, meaning AI tokens resold at thin margin. Databricks is private and has not broken it out.¹⁶

Big data left us with two worlds. In the a16z telling, “Engineers and data scientists lived in the lake with Python while analysts lived in the warehouse with SQL.” Every crossing meant another pipeline and another copy.⁴ Many platforms now do what Databricks did ten years ago. They build a lakehouse that takes any data and serves any consumer from one data set, with compute separate from storage. Snowflake, Microsoft Fabric, AWS SageMaker and Google BigQuery all run the same medallion pattern. Raw data lands, gets refined and is served from curated tables.⁸

Comparison of five lakehouse platforms, Databricks, AWS SageMaker, Microsoft Fabric, Snowflake and Google BigQuery, across core design, open formats, catalog, pipelines, ML and agents, real time, pricing and 2026 Gartner placement. — *Five platforms, one pattern. Sources are listed on the plate.*

What the Lakehouse Solves for AI

Each era left a gap the next one had to close.

Dimension	Legacy database design	2010s big data design	AI ready lakehouse design
Data model	Normalized tables for apps	Wide denormalized logs, star schemas	Medallion layers, governed tables plus documents and vectors
Storage	Single server disks	HDFS and cloud object storage	Open formats (Delta, Iceberg) on object storage
Compute	One engine, scale up	MapReduce, then Spark, scale out	Serverless engines per workload on one copy
Copies of data	App DB plus warehouse	App DB, warehouse, lake, serving store	One logical copy (LTAP, Lakehouse//RT)
Governance	Per database grants	Per system, often gaps	Unity Catalog across data, models, agents
Main consumer	Reports	Analysts and data scientists	People and agents together

The lakehouse closes those gaps with a short list of moves. Scale out on commodity compute instead of buying a bigger server. Keep the data in cheap object storage and size each workload on its own. Run BI, streaming, data science and machine learning on the same governed tables, so there is one copy to secure and one lineage to audit. Get ACID transactions, schema enforcement and version history on files in a bucket, which object storage never gave you on its own. Declare the result you want and let the platform handle orchestration, change capture, quality checks and retries. Put the engineers, analysts and data scientists in the same workspace so the handoffs shrink.

The lakehouse pattern is where AI starts, because an agent is only as good as the data it can reach and the rules it has to follow. “AI does not have an intelligence problem,” says Databricks CEO Ali Ghodsi. “It has a context problem.” Agents already send Databricks more queries than people do.⁴ Databricks spent 2026 pushing that pattern toward agents. Lakebase, its serverless Postgres, writes application data straight into the same open Delta and Iceberg tables, so an agent reads live state and history from one copy.⁹ Unity AI Gateway extends the same governance to the models, agents and tools that touch the data, with spend caps and tracing.¹⁰ Real time pipelines and Lakehouse//RT bring query latency down to milliseconds for the agent loops that need it.¹¹¹² Launch dates and latency figures are in the endnotes.

Leaders outside Databricks land in the same place. Gartner predicts that through 2026, organizations will abandon 60 percent of AI projects that lack AI ready data. Its survey of 1,203 data management leaders found 63 percent either lack the right data practices for AI or aren’t sure.¹³ Gartner’s advice centers on governance, active metadata and pipelines that feed both training and production. MIT’s Project NANDA reported in 2025 that 95 percent of organizations saw no measurable return from generative AI, and it blamed brittle workflows and missing context. The models were fine.¹⁴ The bottleneck is the data and the rules around it, far more often than the model.

We see this every day from the inside. AI agents and the people using them make a mess fast. An assistant (yes, ours too) will happily copy a customer list into a spreadsheet, a summary into a document and a lead file into a chat thread. Soon the same data lives in six places with six sets of permissions. Without governance, controls and one platform everyone goes to first, the copies win. Before DataMind, we built AI/BI and private customer agents on Databricks for an ed tech company. Each customer got an isolated workspace on its own private data, and agents could query that data and nothing else. The fix is one governed home with interfaces to every kind of data we touch: documents, free text, transactional records, and structured lead and customer datasets. That home is the lakehouse, and that’s why we start there.

Why DataMind Is All In on Databricks

After two master’s degrees and dozens of challenges, I founded DataMind Technology this year. My goal is to take what I learned beyond big tech and massive enterprises to the tens of thousands of small and midsize businesses that can win with better data. In 2026 a data platform matters more than ever, because we have to learn AI while we build it.

At the June summit, I was blown away that Databricks was right where I was building for clients: AI agents that write code, answer questions of data, deploy code and recommend the next best action. Databricks has the most complete vision for how data of all types now needs to be structured to serve AI agents and the people using them. Gartner’s 2026 Magic Quadrant for AI Platforms for Data Science and Machine Learning names Databricks a Leader, positioned highest in Ability to Execute and furthest in Completeness of Vision.¹⁵

The bigger reason we lead with Databricks is open data formats on commodity storage, all governed in Unity Catalog. Governance was the painful part I skipped over above. With Unity Catalog, security, privacy, data quality and cost management live under one roof instead of across a battery of bespoke tools and micro vendors. With Lakebase Postgres, application and data engineering teams work from the same data, and their workloads no longer compete.

2026 Gartner Magic Quadrant for AI Platforms for Data Science and Machine Learning, showing Databricks in the Leaders quadrant, highest in Ability to Execute and furthest in Completeness of Vision. — Source: Gartner, Magic Quadrant for AI Platforms for Data Science and Machine Learning, Yogesh Bhatt, Afraz Jaffri, Diarmuid Curran, 22 June 2026. Graphic courtesy of Databricks. Click the image for the complimentary report from Databricks. Abstract: https://www.gartner.com/en/documents/8001969 . This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Databricks.¹⁵

Not every company needs Databricks. If your team lives in Microsoft 365 and Power BI and the data fits a warehouse, Microsoft Fabric gives you the same medallion pattern with less to learn. If you need SQL reporting on a few clean tables, a plain warehouse is cheaper and simpler. Databricks has a learning curve, which is why I sat the exam instead of reading the brochure. Some of the features I like most, such as AUTO CDC for change capture, run only on serverless pipelines or the paid Lakeflow editions.¹⁷ The bill is usage based, which rewards good engineering and punishes sloppy jobs, so budget caps and cost tags go in on day one. The open formats are the hedge. Delta and Iceberg tables in your own storage account stay yours if you ever change engines.

After a week in San Francisco at the Data + AI Summit, I spent long summer nights studying for the Databricks Certified Data Engineer Professional exam. We’re data scientists at heart, but data engineering is 80 to 90 percent of the data scientist’s job. That role is changing fast. Horowitz and McGuiness see more knowledge workers moving “from performing every step themselves to specifying, delegating, and judging the work.” Their bet is that “execution and judgment are economic complements.” Cheaper execution raises the value of good judgment.⁴ That’s the future of data science we’re building toward, and the platform underneath it matters more than ever. That’s the next post.

If You Run a $5M to $100M Company

Two worries come up in almost every first conversation with an owner: what this costs, and whether you lose control of your data. Both are fair questions. Here is how I answer them.

Cost. You do not need Bing's budget to use Bing's pattern. Storage is cheap object storage. Compute is serverless and billed by the job, so a nightly pipeline for a 40-person company costs a few dollars, and a Free Edition workspace costs nothing while you learn.
Total cost of ownership. An AI-ready lakehouse, an analytics platform and a company knowledge graph, which we build together as the Lakehouse + Knowledge Graph, cost a smaller business far less than building the same thing from scratch. The hard engineering ships with the platform and you pay for the compute you use. We can show you the math for your business before you spend a dollar.
Control. Your data sits in your own low-cost, file-based storage account, in open formats (Delta and Iceberg). The platform computes on it and you keep the keys. If you change engines in five years, the tables stay where they are.
Copies. Copies cost more than platforms: the CRM export in a spreadsheet, the finance extract on a shared drive, the customer list pasted into a chat thread, each with its own permissions and its own version of the truth. One governed copy ends that, and it is the part people and AI agents both need.
Start small. One governed copy is a decision before it is a purchase. You can make it in a Postgres schema at $5M in revenue, long before you make it in Databricks, and the medallion pattern moves with you.

Book a free discovery session

Forty-five minutes. Bring a real problem.

Endnotes

Inflation adjustment: $5,000 in 1997 is about $10,497 in September 2026 dollars (CPI-U 160.5 to 336.940). BLS CPI-U series CUUR0000SA0. ↩
Dean and Ghemawat, MapReduce, OSDI 2004. ↩
Chaiken et al., SCOPE, PVLDB 2008. ↩
Ben Horowitz and Michael McGuiness, Databricks & the Future of the Firm, a16z, October 27, 2026. ↩ ↩ ↩ ↩ ↩
Zaharia et al., Resilient Distributed Datasets, NSDI 2012. ↩
Databricks press release, August 13, 2026. ↩
Databricks Data + AI Summit. ↩
Lakehouse pattern: Armbrust et al., CIDR 2021. Medallion layers: Databricks docs and Microsoft Learn. ↩
LTAP (Lake Transactional/Analytical Processing) launched June 16, 2026: Databricks press release on LTAP and Lakebase and LTAP docs. ↩
Databricks blog, Unity AI Gateway generally available, August 4, 2026 and What's new in Unity AI Gateway, June 16, 2026. ↩
Real time mode for Spark Declarative Pipelines entered Public Preview in May 2026 with latency as low as five milliseconds: Databricks release notes, May 2026 and real time mode docs. ↩
Lakehouse//RT, in Beta, answers queries on governed tables in as little as 10 milliseconds: Databricks press release on Lakehouse//RT, June 16, 2026. ↩
Gartner press release, Lack of AI Ready Data Puts AI Projects at Risk, February 26, 2025. ↩
MIT Project NANDA, The GenAI Divide: State of AI in Business 2025, July 2025. ↩
Gartner, Magic Quadrant for AI Platforms for Data Science and Machine Learning, Yogesh Bhatt, Afraz Jaffri, Diarmuid Curran, 22 June 2026. Complimentary copy from Databricks. Gartner abstract. GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally, and MAGIC QUADRANT is a registered trademark of Gartner, Inc. and/or its affiliates and are used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designations. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. ↩ ↩
Sridhar Ramaswamy's October 28, 2026 post on X and Ali Ghodsi's October 29 reply, as reported by Analytics India Magazine, October 29, 2026 and BW Businessworld. The pass-through split is a question Ramaswamy raised. Databricks has not disclosed it. ↩
Databricks docs, The AUTO CDC APIs (updated October 9, 2026): AUTO CDC requires serverless Lakeflow pipelines or the Pro or Advanced editions, and is not supported by open source Apache Spark Declarative Pipelines. ↩

Bibliography

Analytics India Magazine (2026): ‘You’re Not Doing This AI Stuff,’ Databricks CEO Tells Snowflake. October 29, 2026.
Armbrust, Ghodsi, Xin and Zaharia (2021): Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics. CIDR 2021.
Armbrust et al. (2020): Delta Lake: High Performance ACID Table Storage over Cloud Object Stores. PVLDB 13(12).
Bureau of Labor Statistics (2026): Consumer Price Index for All Urban Consumers, series CUUR0000SA0, and CPI news release.
Chaiken et al. (2008): SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets. PVLDB 1(2).
Databricks (2026a): Databricks Grows More Than 80% Year over Year, Surpasses $7B Revenue Run Rate. Press release, August 13, 2026.
Databricks (2026b): Databricks Launches LTAP, the First Lake Transactional/Analytical Processing Architecture. Press release, June 16, 2026.
Databricks (2026c): LTAP overview. Documentation, updated October 21, 2026.
Databricks (2026d): Unity AI Gateway is generally available. Blog, August 4, 2026.
Databricks (2026e): What's new in Unity Catalog at Data + AI Summit 2026. Blog, June 16, 2026.
Databricks (2026f): Real time mode in Spark Declarative Pipelines. Documentation, updated October 13, 2026.
Databricks (2026g): Databricks Launches Lakehouse//RT. Press release, June 16, 2026.
Databricks (2026h): Databricks positioned highest in execution and furthest in vision for the second consecutive year in Gartner Magic Quadrant. Blog, June 24, 2026.
Databricks (2026i): Data + AI Summit.
Databricks (2026j): The AUTO CDC APIs: Simplify change data capture with pipelines. Documentation, updated October 9, 2026.
Databricks (n.d.): What is the medallion lakehouse architecture? Documentation.
Dean and Ghemawat (2004): MapReduce: Simplified Data Processing on Large Clusters. OSDI 2004.
Horowitz and McGuiness (2026): Databricks & the Future of the Firm: How agents and new database technology will change the shape of the firm. a16z, October 27, 2026.
Gartner (2025): Lack of AI Ready Data Puts AI Projects at Risk. Press release, February 26, 2025.
Gartner (2026): Magic Quadrant for AI Platforms for Data Science and Machine Learning. Yogesh Bhatt, Afraz Jaffri, Diarmuid Curran, 22 June 2026. Abstract.
Gartner via Databricks (2026): Complimentary copy of the 2026 Magic Quadrant for AI Platforms for Data Science and Machine Learning.
Ghemawat, Gobioff and Leung (2003): The Google File System. SOSP 2003.
Microsoft (n.d.): Implement medallion lakehouse architecture in Microsoft Fabric. Microsoft Learn.
MIT Project NANDA (2025): The GenAI Divide: State of AI in Business 2025. July 2025.
Zaharia et al. (2012): Resilient Distributed Datasets: A Fault Tolerant Abstraction for In Memory Cluster Computing. NSDI 2012.