Post cover image

September 10, 2026

What Is a Data Product?

Motivation and implementation (based on my imagination)

By Vu Trinh

10 min read

To sustain my writing, I put this article behind a paywall; if you have a Medium membership, I really appreciate it. If you prefer to read this for FREE, please visit here. Also, I post only 1/4 of my deep-dive data engineering articles on Medium; for the full repo: https://vutr.substack.com/

Intro

On the journey to explore (new?) concepts in data engineering (first attempt with the data contract), we'll now turn our attention to data products.

We'll discuss what it is, how it's gonna be implemented (based on my imagination), and the challenges it presents.

As in the data contract article, because I don't have prior experience with the data product concept, what you're gonna read in this article is purely based on my research, observation, and a little bit of imagination :d

So, feel free to criticize me or give me feedback.

What is it?

If I were asked this question just 1 day before the time I wrote this article, I would have told the one who asked, "It's a part of the data mesh; go and check it".

By the way, I just wrote an data mesh article → link.

—

Turns out, the idea of productizing data existed nearly a decade before Data Mesh.

The first(?) documented definition came from DJ Patil in 2012, in a piece called Data Jujitsu: The Art of Turning Data into Product.

Patil proposed putting the "product" first in the design process, then the "data"; delivering the MVP first to pain out the ideas behind the solution and allow a small customer segment to test first.

None

Patil is the one who co-coined the term "data scientist" with Jeff Hammerbacher back in 2008 while at LinkedIn

This contrasts with what we usually do with our serving data dataset, where we could spend months building infrastructure, data modeling, and pipelines, only to realize that the final result does not align with customer expectations.

Patil's proposal allows data practitioners to manage expectations and develop the "product" in a more agile manner.

But at that time, the term "data product" didn't get as much attention, given that people were still immersed in the "big data" era (MapReduce still had its place).

—

Then, in 2019, Zhamak Dehghani published her data mesh work, bringing the term back into the conversation.

This time with more attention than the first

In Zhamak's framing, a data product is an independently deployable, high-cohesion component that bundles the data and metadata, the code for ingesting, processing, serving, and governing it, and the infrastructure to run all of that.

None

Compared to Patil's proposal, a data product is more specific,

It is a unit with an owner, an interface, and a place in a mesh of other data products.

Most people (including me) learned the term from Zhamak's Data Mesh proposal and assume it started there.

I believe it didn't.

I don't know whether Zhamak borrowed this from Patil, but I'm pretty sure both of them are inspired by the process of building software products.

—

And then comes Chad Sanderson.

He didn't coin the term, but he's probably done more to make "data product" mean something concrete to engineers who have no interest in adopting a full data mesh org model.

None

Sanderson doesn't associate data product with data mesh.

A software product is the sum of engineering systems organized around a business need, with an interface and a backend.

In his framing, a data product is the same shape. It is the sum of data components that are organized to serve a business need.

Thus, for example, a dashboard is a data product.

Like Patil and Zhamak, Sanderson's view is also heavily inspired by the software engineering playbook.

What problems does it solve?

Before jumping into how it would be implemented, let's slow down a bit and see which kinds of problems it (promises to) solve.

We also put the lens of Patil, Zhamak, and Chad here.

—

Back in 2012, Patil's watched data scientists (including himself) get handed a hard problem and immediately reach for the biggest technical hammer (e.g., the most advanced ML technique) available.

Because that's the part that provides the most dopamine (yeah, he's right :d)

None

His argument was that solving the problem this way is the shortcut to disaster: the problem only gets more complex the more directly you attack it, until it's scoped down and approached with the right mindset.

His fix was a discipline: start simple, ground the product in the real world, and treat the whole thing as a product with users, not a research exercise with a dataset.

—

Fast forward to 2019, Zhamak pictured the setup almost every data team builds: one central team, and every other team in the company throwing data at it and hoping something useful comes out.

It works well at first.

But it stops working once ownership is siloed within the "black-box" data platform team, which understands the technical aspects but not the business the data came from. When a source system changes upstream, that team finds out the same way everyone else does: some downstream use cases break.

None

Zhamak's observation draws on what had already happened in software: this is the same bottleneck associated with monolithic software architectures, and the fix was the same one software found: push ownership out to the people closest to the domain (who actually understand the data).

If the domain team is responsible for a (data) product, something with an owner, an interface, a defined shape, instead of just a dump of tables somewhere in the warehouse, then "who broke this and why it breaks" stops being a mystery. Under this lens, the problem data products solve is trustworthiness and reliability.

—

For Chad's lens, I found it's quite similar to Zhamak's.

Sanderson's saw that most data teams have focused on speed and removing bottlenecks and have lost track of what the whole data system was for in the first place: providing a scalable source of truth to support stakeholders' business decisions.

As I understand it, unlike Zhamak, Chad doesn't spend much time arguing that treating data as a product is the right idea; he mostly assumes it.

He believes it is the right approach, but treating data as a product alone is not enough. His position is explicit: "the data contract should be built in service of these products, not the other way around." The contract must be there to guardrail the data product.

None

His actual diagnosis isn't "why do we need data products"; it's "why do the data products keep breaking?"

His answer, built from his experience running a data platform: software teams treat data as a product of their own application, ship changes with zero visibility into who depends on it.

(To me, Chad tries to solve the problems with the data contract, not the data product)

Nothing is in between to catch the change; catalogs and monitoring only see the damage after it's done. Through this lens, the problem is that a data product with no enforced interface will eventually break.

Implementation

During my research, I found two standards for data products: the Open Data Product Specification (ODPS) and the Open Data Product Standard (ODPS).

Both are ODPS and they confused me alot :D

However, in this article, I focus only on the Open Data Product Standard.

From now on, when I write ODPS, I want to refer to the Open Data Product Standard.

Standard

Bitol, which maintains the Open Data Contract Standard (ODCS), also maintains its own standard for the data product.

The reason why I focus only on the Open Data Product Standard is that I feel I can get my head around it, as I already discussed ODCS in the last article.

The ODCS and ODPS address two distinct questions and usually go together.

ODCS describes a promise about data: its shape, the quality bar it should meet, or what's guaranteed to a consumer.

# ODPS high-level information.
# Source: https://bitol-io.github.io/open-data-product-standard/v1.0.0/#table-of-content
apiVersion: v1.0.0
kind: DataProduct
name: Customer Data Product
id: fbe8d147-28db-4f1d-bedf-a3fe9f458427
domain: seller
status: draft
tenant: RetailCorp
description:
  purpose: Enterprise view of a customer.
  limitations: No known limitations.
  usage: Check the various artefacts for their own description.
tags: ['customer']

ODCS describes something one level up. It answers the question "What is this thing as a whole, who's responsible for it, and which contracts does it stand behind?"

# Product's ports information
# Source: https://bitol-io.github.io/open-data-product-standard/v1.0.0/#table-of-content
inputPorts: # Expectations [Optional]
- name: payments
  version: 1.0.0 # [Required]
  contractId: dbb7b1eb-7628-436e-8914-2a00638ba6db # [Required]
- name: payments
  version: 2.0.0 # [Required]
  contractId: dbb7b1eb-7628-436e-8914-2a00638ba6da # [Required]
- name: onlinetransactions
  version: 1.0.0 # [Required]
  contractId: ec2a112d-5cfe-49f3-8760-f9cfb4597544 # [Required]
- name: onlinetransactions
  version: 1.1.0 # [Required]
  contractId: ec2a112d-5cfe-49f3-8760-f9cfb4597547 # [Required]
outputPorts: # Promises [Required]
- name: rawtransactions
  description: "Raw Transactions"
  type: tables
  version: v1.0.0 # [Required]
  contractId: c2798941-1b7e-4b03-9e0d-955b1a872b32 # [Required]
- name: rawtransactions
  description: "Raw Transactions"
  type: tables
  version: v2.0.0 # [Required]
  contractId: c2798941-1b7e-4b03-9e0d-955b1a872b33 # [Required]
  sbom: # The SBOM can/should be at the version level [Optional]
  - type: "external" # default
    url: "https://mysbomserver/mysbom"
  inputContracts: # or dependencies [Optional]
  - id: dbb7b1eb-7628-436e-8914-2a00638ba6db # or contractId
    version: v2.0.0
  - id: ec2a112d-5cfe-49f3-8760-f9cfb4597544
    version: v1.0.0
- name: consolidatedtransactions
  description: "Consolidated transactions"
  type: tables
  version: v1.0.0
  contractId: a44978be-1fe0-4226-b840-1b715bc25c63
- name: fulltransactionswithreturns
  description: "Full transactions with returns"
  type: tables
  version: v0.3.0
  contractId: ef769969-0cbe-4188-876f-bb00abadaee4

In this model, a single product can expose multiple "ports", each backed by its own separate ODCS contract (point to the contract ID). There are three port types:

  • Input ports (what the product consumes)
  • Output ports (what it promises downstream)
  • Management ports (observability, lineage, admin, often left out of summaries).

So the picture will look like this: the product standard handles high-level information about the product (e.g., ownership, ports), while ODCS handles the actual promise-keeping on a port-by-port basis.

None

To me, this is aligned with Chad Sanderson's vision: the data product must come with the data contract.

The validation engine

So, what happens next after you define the YAML file?

There must be a kind of validation engine here (just like the one I imagined for the data contract). Because the most important information in a data product YAML file is the product's related contract, I think validation should primarily focus on the contract's constraints.

—

First, there will be a step to validate the YAML content.

Bitol introduces the Data Product CLI and the Data Contract CLI to create, lint, and publish ODPS/ODCS files. It ensures that the fields and their values comply with the standards.

None

Source: https://github.com/datacontract/datacontract-cli

None

Source: https://github.com/datacontract/dataproduct-cli

As I imagined, one uses the Data Product CLI to validate the ODPS standard, then walks through each port to retrieve the contract ID (which points to an ODCS file). The datacontract-cli will kick in to validate each port.

—

Those ODPS/ODCS files will be committed and pushed.

None

The CI process will kick in and perform the "physical" validation here: the contracts' defined schema and data quality.

Besides the CI, I think validation can happen where the data flows: by running against the live dataset to confirm that schema, quality, and SLA commitments are being met.

Or the consumer side could run their own validation on top of everything, confirming that what they're actually receiving satisfies all the ports (contracts).

—

For YAML file validation, the datacontract/product CLI can help.

For contract schema validation, I expected it to be a lightweight process where you need to check that field names and types are correct; nothing is renamed or dropped underneath the consumer without warning. The validation is performed on schema information (stored in a central repository)

—

Contract data quality validation is where the heavy lifting happens.

Based on my imagination of the data quality validation engine's conceptual architecture. It must have two components: the rule engine and the compute engine.

None

Although some rules can be checked using metadata (e.g., the update time to verify freshness), most quality rules require data scanning; compute engine resources must be considered.

None

Also, the data quality validation should be flexible enough to allow the user to add or adjust rules. This is where the rule engine comes into play: it stores all the rules and exposes a method for the user to interact with them.

Regarding the engine's tech stack, it will depend on your company's context and requirements.

Ideally, this compute engine is also the one your company uses for data transformation to avoid syntax mismatch.

However, this does not rule out using different compute engines for different validation use cases. For example, near-real-time validation could use Spark Streaming or Flink.

For the rule engine, it could be Great Expectations, dbt tests, or a self-developed solution.

—

The crucial thing to note is that there is no single standard solution called the "validation engine", its components depend on the organization's requirements and use cases.

Thoughts

Like the data contract, a data product (associated with a bunch of contracts) will also face the cultural obstacle.

The enforcement gives people more work to do and adds more time to the overall raw-to-insight process.

Both data contract and product usually don't give people the "tangible feeling" that things such as "data lakehouse", "data warehouse," or "ETL" do. It can easily become something people feel like they have to do rather than something they want to do for the benefits it brings.

So, implementing those concepts requires extensive education and alignment and involves many people. And whenever something involves a lot of people, it's always a hard mission.

—

If your company has implemented or is currently implementing data products, feel free to leave a comment and share your experience. I'd love to hear about it!

Outro

In this article, I lay out my research on data products, the problems they solve (through different lenses), and then move on to discuss data product standards and validation mechanisms.

Thank you for reading this far.

See you in my next article.

Reference

[1] Pawel Rzeszucinski, PhD, Data Jujitsu on a plane: DJ Patil's playbook on building robust data products (2024)

[2] Zhamak Dehghani, Data Mesh Principles and Logical Architecture (2020)

[3] Chad Anderson, The Consumer-Defined Data Contract (2023)