5 Top Open-Source Data Science Tools for the AI Era

AI tools have become part of how many data scientists work nowadays. We can use ChatGPT, Claude, Gemini, Codex, and other AI tools to write code, analyze data, create SQL queries, or help us understand our results.

However, using AI doesn't mean we only need the AI model itself.

There are still many things around the data science workflow that AI tools can't handle well on their own. The AI needs to understand our data, the changes it makes need to be controlled, and we still need to check whether the result is correct.

Luckily, many open-source tools can work alongside AI for these purposes, and I want to explore them as they become more interesting now that we work with AI assistants and agents.

What are these tools, and when should we use them? Let's get into it.

Microsoft Data Formulator

As data scientists, one activity we perform frequently is data exploration.

We explore the data to understand the distribution, compare variables, find unusual observations, and create visualizations. AI tools can help with this process because we can describe what we want using natural language.

Microsoft Data Formulator takes this approach further.

Data Formulator is an open-source AI-powered data analysis tool developed by Microsoft Research. Instead of relying solely on a chat interface, the tool combines natural-language interaction with visual data exploration.

For example, imagine that we have customer transaction data.

We could ask:

Show the monthly revenue for each customer segment.

Data Formulator can create the required transformation and visualization. After seeing the result, we can continue exploring it, change the fields, or ask another question. For me, this is where Data Formulator becomes useful.

I would use it when I want AI to help with exploratory data analysis or visualization, but still want to interact with the results rather than receive only an AI-generated answer.

However, the tool is still developing quickly, and I wouldn't use it to replace my production analysis pipeline. I see it more as a tool to help us explore the data faster.

lakeFS

AI agents can do more than answer questions.

They can now execute code, modify files, run data transformations, and perform multiple tasks without us having to manually run every step.

This raises another issue: what if the AI edits incorrect data? lakeFS can be helpful in this situation.

The lakeFS is an open-source data version control tool that provides Git-like concepts for data lakes. We can create branches, commit changes, merge them, and revert to previous versions of the data. It is available under the Apache 2.0 license.

Let's take an example.

Imagine we want an AI agent to clean several million customer records.

Normally, we could let the agent run the transformation directly on the data. However, if something in the generated code is wrong, the original data might also be affected.

With lakeFS, we can first create another branch.

customer-data

main
|
+--- ai-cleaning-test

customer-data

main
|
+--- ai-cleaning-test

The AI agent works with ai-cleaning-test while the main version remains unchanged.

lakeFS uses zero-copy branching, so creating a branch doesn't require copying the entire dataset. The branch initially points to the existing data, and only changed objects require additional storage.

This feature was already useful for normal data engineering, and AI agents make the use case more important. The current lakeFS documentation even describes giving each AI agent an isolated branch, so its work can be reviewed before the changes reach production.

Of course, lakeFS isn't necessary for every project. If I am analyzing a few CSV files locally, I wouldn't add another infrastructure component only for version control.

I would consider lakeFS when data is stored in a data lake, and AI agents automatically modify datasets. The more freedom we give the agent, the more useful it becomes to have a way to check and reverse its changes.

OpenMetadata

AI models are already good at generating SQL.

However, generating valid SQL and understanding the business data are two different things.

For example, imagine our database contains these tables:

customer
customer_master
customer_profile
customer_profile_v2

customer
customer_master
customer_profile
customer_profile_v2

Which table should the AI use? The schema alone might not provide enough information.

There might also be business definitions that the AI doesn't know. A column calledactive_customer, for example, could mean a customer who purchased during the last 30 days or something completely different.

OpenMetadata is an open-source metadata platform that collects information such as data descriptions, ownership, lineage, glossary terms, quality results, and other context around organizational data. The project uses the Apache 2.0 license.

What makes OpenMetadata interesting in the AI era is its MCP server. MCP, or Model Context Protocol, allows AI tools to interact with external systems in a standardized way.

OpenMetadata can expose its metadata through MCP so AI assistants such as Claude or other compatible clients can search the organization's data context directly.

For example, instead of expecting the AI to guess which customer table is correct, we could ask:

Which table contains the current customer information?

The AI assistant can search OpenMetadata and retrieve the available data. For me, this addresses one of the bigger problems when companies start connecting AI with internal data. The model might know SQL very well, but it doesn't automatically know our company.

There is still a limitation. OpenMetadata only becomes useful when the metadata itself is maintained. If no one writes the table descriptions or keeps the business glossary up to date, the AI will receive incomplete information.

I would use OpenMetadata when the company has many data sources, and we want AI assistants to understand more than the raw database schema. For a small project with several obvious tables, it could be unnecessary.

WrenAI

OpenMetadata helps an AI assistant find information about the data.

WrenAI focuses specifically on helping an AI agent understand business data for SQL and analytics generation.

WrenAI is an open-source context layer for AI agents. It allows us to describe entities, relationships, calculations, and other business information that isn't always obvious from the database schema.

Let's use a simple example. Imagine we have a revenue column.

The technical definition looks obvious enough, but the business definition might be:

Revenue =
Completed transaction amount
minus refunds
excluding tax

Revenue =
Completed transaction amount
minus refunds
excluding tax

An AI model looking only at the table schema wouldn't necessarily know this.

We could include all that information in the prompt, but it becomes difficult when we have hundreds of tables and many business rules.

WrenAI stores this kind of context in a model that AI agents can use when generating queries. It can include relationships, business calculations, instructions, previous examples, and information about data values.

The project is also designed to work with AI coding agents. Its current setup provides skills usable by tools such as Claude Code, Cursor, Codex, and others.

For me, WrenAI becomes interesting when we are building text-to-SQL or AI analytics systems. The problem with text-to-SQL isn't always that the model can't write SQL. The model often doesn't have enough information to know what the correct SQL should mean.

I would use WrenAI when an AI agent frequently queries business databases and the business logic is too complicated to explain repeatedly in prompts. If we only have a few simple tables, normal schema context might already be enough.

OpenLIT

The last tool is related to AI evaluation and monitoring.

When we build a normal data application, an error can often be quite obvious. The application crashes; however, AI applications are different.

The application might run successfully and return an answer that looks completely reasonable, but the answer could still be wrong.

OpenLIT is an open-source AI engineering platform built around OpenTelemetry. It provides observability, evaluations, prompt management, guardrails, and monitoring for AI systems.

For example, imagine we build a RAG application for company documents. The user asks:

How many annual leave days do I have?

The application returns 15 days. But why did it return 15?

Maybe the retrieval system selected an outdated document. Maybe the prompt caused the model to ignore the correct policy, or perhaps the model itself generated information that wasn't in the retrieved data.

With observability, we can inspect the steps that happened before the final answer. The project also supports more than just LLM tracing. It includes evaluations, guardrails, prompt management, and even GPU monitoring.

I would use OpenLIT when an AI application starts becoming difficult to debug.

A basic chatbot with one model call probably doesn't require this much monitoring. However, once we add retrieval, tools, agents, or several model calls, knowing what happened at each step becomes useful.

The limitation is that observability itself creates more infrastructure and data to manage.

We also need to be careful with sensitive information, as traces may contain user prompts, retrieved documents, or model responses.

Conclusion

AI tools are changing how we work as data scientists, but I don't think that means our existing data problems disappear. AI still needs to understand what the data means. We still need to control the changes it makes and check whether its output is correct.

In this article, we have explored five of them:

1. Microsoft Data Formulator for AI-assisted data exploration,
2. lakeFS for data version control,
3. OpenMetadata for organizational data context,
4. WrenAI for business-aware AI analytics,
5. and OpenLIT for AI observability.

I hope it has helped!