Most New Databases Are Now Built by Machines, Not People
A plain-English look at Databricks' big announcement this week, and what it means for the future of data engineering.
Here is a number that stopped me this week. Databricks, one of the largest data companies in the world, says that about 80% of the new databases created on its platform are now made by AI agents, not by people. Not by engineers typing commands. By software, spinning up a database on its own, using it for a few minutes, and throwing it away.
That one fact explains almost everything the company announced on June 16 at its Data + AI Summit in San Francisco. And it points at a real shift in what data engineers will be doing for the next ten years.
Let me back up and explain it without the jargon.
The 40-year-old split between OLTP and OLAP
For about forty years, most companies have kept two copies of their data.
The first copy runs the business. When you buy something online, log in, or send a payment, that hits a "transactional" database. It has to be fast and correct right now. Engineers call this OLTP.
The second copy is for studying the business. How were sales last quarter? Which products are slowing down? That work happens in a separate "analytical" database, sometimes called a warehouse. Engineers call this OLAP.
You keep the two apart for a good reason. If you run heavy reports on the live database, you can slow down the website everyone is trying to use. So companies copy data from the first system to the second, over and over, all day long.
Those copies are the pipelines. And pipelines are where data engineers spend a huge part of their lives. Building them. Fixing them when they break at 2 a.m. Explaining why a dashboard is an hour behind. Paying for all the extra storage and computing the copying needs.
What Databricks announced: LTAP
This week Databricks said: what if you just stopped making the second copy?
Its new design is called LTAP. The name doesn't matter. The idea does. Keep one copy of your data, in one place, and let both jobs, running the business and studying it, read and write to that same copy. No pipeline in the middle. The company's slogan for it is "one data, zero compromises, zero copies."
It is worth being clear-eyed here. LTAP is "coming soon," not something you can fully use today, and most of the bold claims come from Databricks itself. Rivals like Snowflake and Microsoft are chasing the same goal in their own way, and plenty of engineers will doubt it holds up under real pressure. The company's CEO, Ali Ghodsi, expects that doubt. He has heard it before. So treat this as a direction, not a finished product.
But the direction is the interesting part.
Why now? AI agents.
Why is this happening now, after forty years? Because of AI agents.
A human engineer might create a database and use it for months. An AI agent works differently. It might create a database, try an idea, check the result, and delete the whole thing in a couple of minutes. Then do it again, hundreds of times. Databricks says its platform now handles around 12 million database launches a day.
Agents don't want to wait an hour for a pipeline to catch up. They want the freshest data instantly, and they want to spin things up and tear them down constantly. The old two-copy world was built for humans working at human speed. It was not built for this.
So the data world is being rebuilt around its new main user: the agent.
What it means for the future of data engineering
Here is the part that matters if this is your job.
A lot of data engineering today is plumbing. Moving data from A to B. Watching pipelines. Patching them. If the pipeline starts to disappear, that work shrinks.
That sounds frightening, but it has happened before in tech. We used to hand-manage physical servers. Then the cloud took most of that away, and the job moved up, toward design, architecture, and judgment. The same thing is likely here.
The work that grows instead is the thinking work:
- Deciding what is true. When everyone and every agent reads one copy, getting that copy right, clean, governed, and trustworthy, becomes the whole game.
- Setting the rules. Who and what is allowed to touch the data? With thousands of agents acting on live systems, guardrails matter more, not less.
- Designing for machines. Building data that agents can safely read, reason over, and act on without a human checking every step.
- Watching the watchers. Someone still has to notice when an agent does something dumb at scale.
Notice that none of that is "move data from A to B." The plumbing fades. The thinking stays.
The plumbing fades. The thinking stays.
So what is the human part of the job?
The honest summary is this. This specific product might work brilliantly or land with a thud. But the reason behind it is not going away. The main user of our data systems is quietly becoming software, not people. Once you accept that, a lot of how we have built things for forty years starts to look like it was made for a world that is ending.
So here is the question I keep coming back to, and I would like to know what you think. If the pipelines go away and agents become the main users, what is the most human, least replaceable part of a data engineer's job? My bet is judgment: deciding what is true and what is allowed. What is yours?
Leave a Comment