Your AI agent is only as valuable as the data and context it can reach.

Ask an agent a question when it has access to all the data and business context needed to reason properly,  and you’re likely to get an appropriate response. That’s because the agent has access to the two pillars it needs, data and context. 

But what if your data or context exist on-premises? That can present access issues, and that’s a big problem. If your agent can’t reach the data or context it needs, then in the best case, it might fail to return an answer. In the worst case, it will make a reasoned guess. 

The issue is how to get your agents access to data and context when, for security and compliance reasons, that data has to stay within the borders of a nation or a network. 

It’s a problem that requires a particular approach and technology to solve it. In this article, we’ll look at what makes on-premises data so challenging to access when it comes to AI infrastructure, and why a federated data architecture with a strong context layer is the solution.

Why on-premises data is challenging for AI agents

For AI, context is everything. It’s the new bottleneck for AI applications that makes or breaks the difference between AI results that can be trusted and those that can’t. The solution requires access, enabling your agents to access context from your business alongside the data from multiple sources from across your enterprise.

But most AI agents and tooling are not built around universal data access. And some data can’t be stored in the cloud for a variety of reasons anyway. For example: 

• The data might be highly sensitive, personally identifiable information (PII) that requires rigorous safeguards and handling. Healthcare and financial information fit into this bucket.
• Regulatory rules might require certain data to remain in a highly controlled, on-premises environment. Tax data, defense and export control data, and national critical infrastructure data are a few examples.
• Data sovereignty laws often restrict moving customer data or other highly sensitive information outside of a particular national border.

But AI agents that can’t access that data, can’t draw on it or any of the context surrounding it, and that impacts their ability to function effectively in an enterprise setting. For example, think of an agent that summarizes the contents of a confidential call in a regulated industry like finance, or an AI chatbot that helps employees select a new company insurance plan that fits their unique needs based on privileged individual data that can’t be stored in the cloud for compliance reasons.

This means you need a flexible AI data architecture, one that enables AI agents to access on-premises data in a safe, highly governed manner.

An architecture for serving AI agents in a hybrid environment

To solve this, you have one of two choices.

The first is to give your agents in the cloud access to your on-premises data. This is usually done via a secure private link between your network and your cloud provider.

For the most sensitive workloads and cases involving data sovereignty, the second is to run some large language models (LLMs) and the AI agents that use them in-house, using an air gapped technology. This approach co-locates your agents within an enclosed environment and your data so that no sensitive customer information ever needs to traverse an external network boundary.

But both of these options require a different architecture that meets the needs of AI agents.

Access to data across your data estate

Nearly all of us work in a highly distributed data environment, but our data architecture doesn’t always match this. Matillion and IDG estimate that the average company is drawing data from 400 different sources. More than 20% of the companies surveyed draw from 1,000 or more.

A hybrid data environment makes this even more complicated. Applications and AI agents need to access sensitive data where it lives, not in a centralized data warehouse. This access needs to be secure and governed to prevent data leaks and unauthorized access.

The solution to this is data federation. Instead of centralizing data in a single location (which always worked better in theory than in practice), you can use connectors to fetch it from its primary source. A security and governance layer uses role-based access control to ensure users and applications can access only the data they’re allowed.

In the past, data federation was slow and impractical. But technology has come a long way. Improvements in removing performance bottlenecks in distributed queries and in fine-tuning incremental update algorithms mean you can access the data you need today where it lives, centralizing only the subset of data that requires lightning-fast performance.

A comprehensive approach to context engineering

Then there’s the matter of context, and access to it. Data doesn’t exist in isolation from other data, or from the business context that gives it meaning. AI agents need more than raw data; they need data with meaning. This is context, and it makes the difference between agents that return hallucinatory nonsense and those that return accurate answers.

Creating context requires context engineering to build a rich context layer from data across your enterprise. This context layer is composed of data products, which turn raw data into packaged, curated datasets that are versioned, discoverable, and secure.

For AI workloads, data products are the best way to engage in context engineering. They also give data the attributes required for strict data governance of AI-driven solutions: efficiency, quality, interoperability, and transparency.

How Starburst turns on-premises data into context

Whether you operate in a cloud, on-premises, or within a hybrid environment, your AI data architecture needs to support three things:

• Federated access to data across your enterprise
• Access to data in a high-speed storage format for optimal performance when it counts
• A way to turn raw data into high-quality, governable datasets

Starburst is the enterprise intelligence platform built for the AI age. It can run in and connect to data in any environment. To Starburst, your on-premises data isn’t fundamentally different from your cloud data; it’s just another data source you can use to curate context for your AI agents.

With Starburst, you can get started with context engineering for AI today by connecting to data across your hybrid enterprise. For the most demanding yet sensitive workloads, you can store data on premises using Apache Iceberg, the modern, open table format built for performance. Data consumers can easily discover on-premises Iceberg tables and join their data with distributed datastores, such as your cloud-based data warehouse.

Starburst supports context engineering with data products, enabling you to combine distributed data sources into production-ready datasets that are discoverable, high-quality, and well-governed. This creates a single layer for accessing AI context via multiple consumption models:

• AI agents can use the Model Context Protocol (MCP) to negotiate data access without hard-coded APIs.
• Data consumers and business stakeholders can access the same context with AIDA, Starburst’s AI Data Assistant, asking questions directly of data with natural language queries.

Because data access runs through one layer, data governance does, too. Use access controls, data lineage, and audit logging to support compliance whether your data lives in the cloud or on premises. Each data product carries its access and governance rules as part of the product itself, which means data and governance are always tightly coupled.

Build sovereign AI systems without compromising control

Making data available for AI doesn’t mean compromising security and compliance. With the right data architecture, you can build a single context layer that makes data accessible while adhering to data sovereignty requirements.

To learn more about how to create a production-scalable data architecture that meets your strict compliance needs, download our ebook on building sovereign AI systems.