Getting Rid of Data Debt with AI

TL;DR:

• Data debt is the valuable information you have collected and stored, but cannot draw on (or if you can, it isn't usable). This is a different problem from data quality; it is about accessing this hidden, valuable information and using it to draw out insights
• The reason it never gets solved is that most organizations define it as a catchall which turns it into a behemoth project nobody can start. If you narrow the issue to access, it becomes more manageable
• Three things have changed that make this problem solvable and very relevant: (1) The commoditization of connecting data efficiently between systems/ tools through CDC and automated backfills, (2) Tooling and AI that compensate for team gaps (especially data engineers), and (3) Effective AI genuinely requires this data to be surfaced
• None of this works without figuring out the architectural design: what data matters, where it lives, what the schema should be. Then you can hand the technical work to automated tooling

Data debt is pervasive: data piles up from multiple systems over long periods, gets put into a vault somewhere, and is never seen or used again.

This is the reality in many organizations. They describe an environment with a lot of data (albeit of questionable quality and not very accessible), but little to no real insight coming from it.

The worst part is that most people there know this; they understand they are sitting on an enormous amount of information that could drive business value. But it still goes nowhere, because they don't know how to use it (other than downloading a CSV and building a few Excel pivots).

AI is helping make sense of that data and get more value from it, but it can't be the answer on its own.

So just like technical debt (which I wrote about last week), data debt becomes a similar case study — the debt persists and accumulates because nobody has ever had the capacity, drive, or expertise to act on it.

And just like tech debt, I believe AI has opened up new opportunities to tackle data debt in all types of organizations. We just have to be a bit smarter about the issue, and break down the problem into modular bits, starting with the most important piece — getting access to the right data.

What Data Debt Actually Is (And What It Isn't)

My definition of data debt is the valuable information you have collected, paid for, stored, but cannot draw upon (or if you can, it is not usable). Similar to tech debt, it is a debt in that the data costs money to carry it, while returning no value and accumulating/ degrading if you don't do anything about it.

There are many dimensions to data debt.

If you look at the classic definition, the term ends up being a catch-all. Look it up, and you will see it reference everything from poor quality to inconsistent definitions to legacy systems to undocumented pipelines or unmanaged data.

For example, here are some definitions from reputable sources:

• Google Research (Sculley et al., NeurIPS 2015) — The original serious treatment, from the paper Hidden Technical Debt in Machine Learning Systems. Their argument focuses on the idea that data dependencies cost more than code dependencies because they're far harder to detect. You can trace a code dependency through an import statement, whereas data dependencies are only unearthed when something upstream changes and everything downstream degrades without warning
• Matillion — Splits it into four categories covering everything from schema design through to governance and consumption
• Syniti — Frames it as the accumulated cost of shortcuts and deferred decisions across the whole data lifecycle (ingestion, modeling, governance, consumption)
• Hackolade — Breaks it into structural debt (bad schema design), content debt (inaccurate or duplicate records), and operational debt (undocumented pipelines and shadow systems).

My biggest problem with these perspectives is that the issue becomes overbearing and impossible to tackle all at once. Most of these definitions are broad enough to contain almost any data problem you've ever had.

I mean, tech debt is similar, but at least with technologies you can modernize and replace, one system at a time. Data is an output of many systems and these problems originate from so many different sources that it becomes extremely hard to tackle.

But that is only if you define it in this all-encompassing way.

Unfortunately, that is how most organizations define it and try to tackle it. First, they think about data quality, whether you can trust the data or define it. Then they think of the sheer volume of data and its many sources across the organization. And soon you have a meeting room full of 10 different people describing their own individual problems, leading to a behemoth project of consolidation, data quality improvement, and analytical improvement.

And guess what? In the end nothing happens. Why? Because it is impossible to deal with all of this at once.

For example, I worked with an airline company that wanted to solve for their data debt, and to do that, they decided to build a customer data model. Their first mistake was trying to connect all 80 systems and agree on definitions across them all. Unfortunately for me, that discussion (and the scale of the task) eventually killed the project. This was despite my suggestion that we start small and deal with the most important data sources, connections and definitions first.

So returning to my definition of the term, it goes back to drawing upon the valuable information your company has access to, and using it to eventually draw out insights. The most important piece here in my mind? Accessing that data before it accumulates in a negative way and becomes unusable/ stale!

How Debt Accumulates & Doesn't Disappear

Similar to tech debt, the biggest problem with data is that it accumulates in systems, and the task to unravel it and fix the sourcing of it becomes too big to do easily.

And this problem is brought to you by technology!

Every team and every person uses technology. Now each of these technologies/ tools generates data. What is more, this data is often seen as a differentiator; the executive team signs up for these tools because they generate data you can use to derive insights and improve operations.

But unfortunately, they buy the tool and forget all the steps it takes to use the data it generates.

For most companies, they parked the data (either within the tool or on some unorganized SQL database) because storage is cheap. Say, we might need it later, and carry on with your operations because no one has time to figure out how to use that data properly. And the gap continues to widen as there is no priority to come back to that data and use it; the amount continues to grow and it becomes harder to model or wrangle it as time passes.

The consequences of this are pretty well known, so I'm not going to dive into them in too much detail:

• Data Staleness — The data still exists but the meaning around it changes, especially when definitions or metrics shift. Using that data becomes difficult because of its staleness
• New Pipelines/Dashboards — Instead of going back and updating what exists, people rebuild data access. This creates both data and tech debt, duplicating costs and creating redundancy. There is a lot of this happening with AI now (which we will talk about next week)
• Unsustainable Data Volumes — The biggest thing holding back organizational leaders from dealing with this is the sheer scale of the problem and not knowing how to even approach it
• Unconnected is the Norm — Operational tools don't get connected because it is seen as hard and difficult, so the cycle continues

Fixing these things has always fallen into the realm of data engineering. Unfortunately, most of the engineering teams I see are understaffed, lack clear direction, are ticket-based, and don't have a clear data model or strategy to work from. Therefore, the data debt problem continues to accumulate no matter how many resources you throw at it.

But this is stuff you know already. If you work in a large corporate organization, you've seen this before, and you don't need to be told why it happened. Instead, you need to understand what to do about it (because most leaders are afraid to start without that path), and that's what I'll spend most of this article on.

How To Solve for Data Debt

Now, I won't contend that solving these issues is easy — especially in large organizations or if the data team isn't well resourced. But it's not impossible, especially with the advances in technology, AI, and connected systems.

Before, this might have been a transformation project for a large organization. And for small and medium-sized organizations, this type of task has always seemed out of reach; while they were accumulating data, they never had the team to build the capability to use it. I would often hear the phrase "We don't have a Google-type budget to do this stuff." In my mind, that was an excuse. And nowadays, it is a really bad excuse.

Three things have changed:

1. Connecting Systems is Being Commoditized — Connections and migrations once required bespoke engineering and pipelines. But now, getting data from one place to another and from system to system can be as easy as configuring a connection
2. Tooling & AI Can Compensate for Team Gaps — Smaller, less specialized teams can now approach these things with the right tools, primarily because the expertise is embedded in the tools
3. Effective AI Requires This — The raison d'être! If you want AI to live up to its expectations, you need to surface this locked-away data. You need these data systems connected so that inputs go to the right places at the right time. Essentially, alleviating data debt will be a requirement for data, analytics & AI success

I'm going to dig into these three components, but before I do, I need to emphasize that there is a precursor to doing this — architectural design.

Start Solving for Data Debt with the Right Approach

The problem with most companies and executives is that they buy a tool thinking it will be the silver bullet for all their data issues. This is even more apparent with AI (many of my conversations with COOs, and even CTOs, are me trying to convince them that they need to do more than just buy Claude).

The truth is that tools are now very capable of doing the technical work (e.g., coding, monitoring, system connectivity, etc.), and companies that use them correctly are getting insane amounts of value, especially given how manual it is for some organizations to connect data/ design bespoke pipelines. For example, this article is brought to you by Estuary, which I partnered with because I have seen what a difference this kind of tooling makes for clients (instead of an expensive engineering hire to build pipelines, you can connect data that had been sitting untouched for years in an incredibly cost-efficient way).

But organizations need to realize they must properly plan for how these tools can deliver the promised value, especially with AI. This involves thinking about the right architecture and data model. You buy tools because they fill a gap; but to fill that gap, you need the right plan in place to make sure the tech/ tooling knows how it should work!

So before jumping to the tooling, there are four phases that can help set this up in the right way. And to be clear, running this in a looped, modular fashion is the best way to do it (otherwise you run into the immense scale problems I mentioned above with the airline data model):

1. Figure out what data actually matters — Most organizations have a lot of data sources and a lot of data debt. And while it seems logical, the worst thing you can do is focus on what data you have, because that is how the problem grows and it becomes harder to prioritize. Instead, start with the decisions you want to make within the business, identify which data will help make those decisions, and apply the 80/20 rule (80% of the solution delivered by 20% of the data). This is a mix of stakeholder management/ consultations and business analysis. It also connects directly to the use cases (which may help point you down the right path to get rid of data debt)
2. Identify where that data lives and what it takes to reach it — Now you map the sources. Which systems hold it, who owns them, how current it needs to be, and what condition it's actually in upon arrival. The difference now vs. five years ago is that not every source is a big engineering job. Now, a lot of sources have supported connectors, especially depending on the tools you use. Estuary, for example, connects to over 200 tools/ systems (and growing), making it an easy ask to pull certain data sources. Similar to the first step, this helps you prioritize: if you have equally important data sources (for your use cases), whichever is more easily configurable should be the first choice to solve for. Here is where you also note more complex cases/ systems and potentially reference back to any tech migration efforts or timelines
3. Design the schema and architecture — This is probably the most important step. Think about the schema for the data source as it relates to identified and potential use cases. For example, what are the entities, how do they relate, what grain do you need, what is the definition of customer or revenue, and where does the modeled data land? Ensure this is mapped out within the overall data and solutions architecture. While AI can help you speed this up, human expertise is necessary, trust me
4. Hand the technical work to AI and modern tooling — I'll speak more to this in a second. But basically, once you have mapped out what you need, where it is, and what shape it should be in, the actual movement and transformation are increasingly something you configure rather than build. This is where tools like Estuary do the heavy lifting: capturing from the source, continuous sync, and materialization wherever you're modeling it. The point of the first three phases is that you arrive at this one knowing exactly what you're asking the tooling to do

Then you go back to phase one with the next slice or use case. If you've designed the data model well from the beginning, these passes should get faster because you are adding data sources to a proven architecture.

With that being said, some sources/ systems do create problems. But this is where the three changes I identified make it worthwhile.

Why Now is the Time to Solve for Data Debt

1) Connecting Systems is Being Commoditized

First off, moving data from one place to another and from system to system can be as easy as configuring a connection.

In the past, connections and migrations required bespoke engineering and pipelines. Now, this process can be automated.

Tools (like Estuary) do this through change data capture, or CDC, which has been around for a while but has become faster and more efficient in the past few years. Basically, CDC reads the database's change log and only ingests/ maps the changes. Compared to the previous process — run a large extraction query against a source operational system overnight and wait for the delay to run into the morning — this is a godsend for accelerating system connections.

For data debt specifically, this goes beyond just connecting the systems. Because the data has likely sat unused for a long time, it's important that when you stand up a connection, you (or the tool) run a backfill — go and get the historical data sitting in that system. This used to be a pain in the ass, but it is much more straightforward now with modern DE tools, especially if you have the right schema set up. After that, switch to CDC to keep it up to date from then on.

So basically the backfill pays down the debt you have already accumulated and the CDC allows you to stop accumulating more of it. Instead of just connecting a new pipeline, you rebuild the system in the right way, with the right context, that will keep working efficiently moving forward without constant maintenance.

Speaking of efficiency, this also keeps getting faster. Backfills used to take a long, long, long time. But now tools are working off more efficient languages like Rust which takes a historical load that used to be a weekend job into something you can run anytime (Estuary recently rewrote their transaction loop and are reporting up to 8x higher throughput on a single worker, sustaining around 200GB per hour). So huge volumes of historical data is no longer a data debt blocker. Worth noting that this isn't a replacement for your data engineering function, but its capability that your data engineering team needs to be more efficient.

Doing this type of backfill properly, with CDC moving only the changes, also significantly reduces the compute and processing costs of this kind of migration.

For example, when your database saves a set of changes that belong together — say, twenty rows across five tables — it treats them as a single unit. All of it lands, or none of it does. Most pipelines don't preserve that. They deliver the pieces separately, at different times, so your warehouse briefly shows a version of reality that never existed in the source system: an order with no line items, a debit with no matching credit. Teams deal with this by running reconciliation jobs to confirm the numbers eventually agree and by writing "eventually consistent" into the documentation.

That is data debt being created by the pipeline you built to fix data debt. By writing your captured changes to append-only logs stored in your own cloud storage, a single source transaction usually spans several of them. This means that instead of staging all these shifts in memory, your source arrives as a single transaction, no matter how many machines handle it along the way. Estuary's CEO David Yaffe puts it as a trade-off teams have always been stuck with: speed or correctness, pick one. This kind of approach (especially built in Rust) allows you to get both.

Of course I also have to mention the other part of connected systems in today's embedded AI world — the abundance of MCPs and connectors. This allows your AI harness to read and work within systems it is already connected to. Estuary have their own MCP integration for exactly this, so an agent can work against your pipelines and collections directly. While a lot of your legacy tech won't have MCPs/ connectors, some tools that store unused data might, and that connection goes a long way toward helping you figure out how to source/ route it into your analytical data layer according to your existing data model!

2) Tooling & AI Can Compensate for Team Gaps

Smaller, less specialized teams can now address data sourcing issues with the right tools, primarily because the expertise is embedded in them.

Think about what this all used to require:

• First somebody who could code.
• Then for them to actually build the scripts.
• Then profiling a dataset to work out what condition it's in.
• Then tracing column-level lineage so you know what breaks if you change something.
• Oh and don't forget working through what an undocumented field from 2016 means and writing the documentation to fill the gaps.

Each one of these tasks was a ticket, and data engineers were inundated with them in a never-ending cycle. Therefore, organizations didn't have time or money to allow engineers to reduce any data debt.

Now, teams have AI agents, affordable tools, and skills to do the vast majority of those things at speeds previously unheard of. That doesn't mean you stop needing data people. But it means a much smaller team can cover ground that used to require a whole function. And extra resource can focus on these never-thought about tasks (like Data Debt).

This doesn't stop at just connecting data — the tooling to assess, align, and fix data quality is also improving significantly, allowing teams to better maintain the pipelines they spun up. So once you solve that first barrier of getting the right data into the right place, addressing its quality issues isn't as big a hurdle either.

This is also why the "we don't have a Google-type budget" line is no longer defensible — all my clients right now are installing this type of thing with 0–2 data engineers on staff and running it for hundreds of dollars a month in tooling (instead of thousands).

3) Effective AI Requires This

AI is the future and if you want to use it properly, you need to surface locked-away data.

Essentially, alleviating data debt will be a requirement for data, analytics & AI success.

But companies are nowhere close to that point yet. While they are hooking up AI tools and harnesses, only 7% of people said their organization's data was completely ready for AI adoption. Meanwhile, Gartner predicts that through 2026 organizations will abandon 60% of AI projects that aren't supported by AI-ready data. Honestly, with the embedded AI hype starting to die down, I'm seeing that in organizations that haven't built these foundations.

Just to be clear, this isn't just the "connect your data or your AI initiatives will fail" argument (which honestly gets a little bit boring). Rather it is that the data your business has generated (or has stored in the back closet) is bespoke and is the secret sauce in today's AI-driven world. Every month, models get better and cheaper, and the tooling around them gets more capable and more commoditized. Your competitor can buy the same models and the same platforms you can. But they can't buy that data: customer behaviour, operational costing, process efficiencies. That's proprietary, specific to you, and accumulates.

But you need access to it for AI to make it work.

So every year you leave that data locked away in operational systems or random tools is a year you are forgoing the true advantages of AI — using your data as a knowledge system to improve your business.

What to Take Away

The core of this article and argument is that data debt has remained an issue for years because:

1. The problem was too big to tackle all at once, and
2. It was economically unfeasible to resolve

This article is trying to change that narrative. Just like I outlined last week with tech debt, the world of AI has opened up new possibilities; couple technological advancements and highly capable tools with an architectural mindset to overcome the niggling problems that have negatively impacted your company for years!

And start with connecting the right data! The cost and ease of data access continues to fall, and the longer your proprietary information remains locked up, the less ability your company has to reap the benefits of AI.

At the same time, you still need the right people to decide what the data means, what it's for, and how to structure it. That is the new bottleneck, but it is a whole lot easier than it was 5–10 years ago!

Looking ahead to next week, we have the last of the three debt articles — AI Debt. Everybody is starting to see the debt AI is creating; with such an easy way to create stuff, things don't get documented, understood or used. And then it goes to die. So instead of debt accumulating because fixing them was too expensive, this is about cheap production creating things we don't need and that are hard to maintain.

A note on this partnership: I worked with the team at Estuary on this piece. They do real-time data movement — change data capture, streaming and batch across 200+ connectors — which is exactly the "getting your data somewhere you can use it" problem this article is about. The bit that made them the right partner for this particular piece is that a connection backfills your history and then keeps it current, so you're paying down the data debt you have and not building more of it. They've also just shipped a ground-up rewrite of their runtime in Rust, and their whitepaper is worth checking out. And there is a free tier if you just want to try it out. Thanks to them for supporting the newsletter!

Thanks for the read! Comment below and share the newsletter if you think it's relevant! Feel free to also follow me on Substack and LinkedIn, or reach out if you are looking for some top-notch freelance consulting input! See you amazing folks next week!

Originally published at The Data & AI Ecosystem. It goes out every Sunday. Subscribe here.