Agents of Data: Preparing Organizations for Agentic AI

Agentic AI has gone from curiosity to core strategy in what feels like a matter of months. But while the technology is racing ahead, most organizations are still figuring out what it really means to put agents into the heart of everyday work.

In this episode of the Agents of Data Podcast, Matillion CTO and Co-Founder Ed Thompson is joined by:

• Julian Wiffen, Chief of AI & Data Science at Matillion
• Sam Perrin, Senior Staff Software Engineer, AI
• Sarah Schlobohm, senior AI leader and Chief AI Officer

They cover everything from “goat zooming” and runaway interns to enterprise governance, evaluation gyms, and how Matillion’s Maia, Matillion's agentic data team, fits into this new operating model.

You can watch the full podcast here. 

Organizational Readiness For Agentic AI

Sarah’s view is blunt: most organizations are not ready for agentic AI.

For her, readiness comes down to two simple questions:

• Is the data ready?
• Are the people ready?

In many companies, the honest answer to both is still “no.”

On the people side, teams are only just starting to understand what’s possible with agents – and what isn’t. Change management, training, and basic familiarity are still early.

On the data side, issues like fragmented ownership, poor documentation, and unclear governance make it hard for agents to safely take on real work. Even simple things like “which table is the real one?” or “which revenue definition do we use?” can block automation.

Agentic AI doesn’t magically remove those problems. It amplifies them.

Building Mastery With Agentic Tools In Engineering

Inside Matillion’s own engineering teams, Sam describes a very wide spectrum of adoption:

• Some engineers use LLMs as a “second brain” – bouncing ideas, drafting snippets, exploring designs.
• Others have gone much further, using coding agents to implement entire features, generate tests, and automate repetitive development tasks.

The key lesson: mastery is not a one-and-done rollout. It’s a journey.

Matillion deliberately:

• Encourages everyone to try multiple tools (Claude Code, Windsurf, Cursor, etc.)
• Creates space for experimentation rather than prescribing a single “official” way
• Identifies super-users and turns them into internal champions who help colleagues level up

Measurement is still hard, but even simple metrics help. For example, tracking whether a Jira ticket used AI or not gives a surprisingly useful signal. Over time, Matillion has also learned to look at tool usage patterns (tokens, sessions, active users) as a proxy for value.

Extending The Agentic Journey Beyond Engineering

Engineers are often early adopters, but Sarah points out that in many companies, they’re not the first to fully embrace AI.

To bring agentic workflows to the whole business, she advocates:

• Baseline AI training for everyone
  • Think “AI fundamentals” alongside annual GDPR or security training.
  • The goal is to remove fear and demystify what these systems can and cannot do.
• Team-specific problem-finding sessions
  • Ask each team: “My job would be great apart from…”
  • The answers (“password resets”, “misfilled forms”, “manual report tweaks”) become ideal agentic use cases.
• Start with boring work
  • Anything that feels like autopilot – repetitive, low-judgment tasks – is prime territory for agents.
  • As Ed puts it, hunt for the dull work where people don’t feel they add much value.

Sam and Julian often describe a good agent as an “infinite intern”:

If you can explain the task with a few clear bullet points and leave an eager intern to churn through it, it’s probably a great job for an agent.

Designing Safe Agentic Systems: Thinking Inside The Box

A recurring theme is constraint. Contrary to the “think outside the box” cliché, Julian wants agents to work inside very clearly drawn boxes:

• Tight, well-defined problem spaces
• Limited, explicit tools
• Structured inputs and outputs (e.g., JSON, yes/no, classification labels)

Early in their work on Maia, the team learned that even simple guardrails – like forcing yes/no answers or mandating JSON output – dramatically improved reliability and reduced token waste.

Sam emphasizes the importance of human-in-the-loop for destructive or high-cost actions:

• In a development warehouse, you may accept more autonomy.
• In production, or for actions like booking travel or committing spend, you want a plan + approval pattern:
  • The agent proposes a plan
  • A human reviews and approves
  • Only then does the agent execute

Over time, those approvals can be adjusted based on risk, environment, and confidence – but the default is caution.

Evaluation Gyms And “Fuzzy” Testing For Agents

Traditional software testing is binary: tests pass or fail. Agentic systems don’t fit that model.

To deal with this, Matillion has built what Julian calls “the gym” – a framework for repeatedly exercising agents under different conditions and scoring their behavior.

In their gym:

• A headless version of the agent interacts with simulated scenarios, much like a human tester would.
• An AI judge evaluates the resulting logs and outcomes.
• Telemetry flows into tools like Langfuse to track tool usage, decisions, and performance.

Sometimes they even teach agents to sit multiple-choice exams (for example, Matillion’s professional certifications) so they can quickly and objectively measure improvements when prompts, models, or retrieval strategies change.

This approach shifts quality from pass/fail to metrics such as:

• Accuracy scores
• Success rates across scenarios
• Coverage of desired behaviors
• Regression detection when models or prompts change

Sarah notes that this probabilistic mindset is more familiar to data scientists than to traditional software engineers. Getting both groups working closely together is crucial.

Using Agents To Tame Messy, Unstructured Data

On the data side, Sarah sees two major challenges:

PII and sensitive information

• HR, finance, and customer data all carry different levels of sensitivity.
• Guardrails, classification, and access control matter more than ever when agents can “see” across tables and systems.

The unstructured data iceberg

• Many organizations have invested heavily in structured data (the “fits in Excel” world).
• The real untapped potential is in unstructured data (the “lives in Word or PDF” world): contracts, forms, policies, emails, reports.

Agentic AI is particularly powerful for turning this unstructured content into structured signals. Examples include:

• Extracting effective dates, parties, and obligations from contracts
• Classifying documents by topic, risk, product, or clause
• Flagging the 5 issues that cause 70% of form rework, then auto-correcting them

These are highly auditable use cases where humans can easily sample and validate the agent’s work – again mirroring how you’d supervise a new employee.

Governance, Auditability, And Regulated Industries

For regulated sectors like financial services and insurance, auditability is non-negotiable.

Sarah draws a parallel with MLOps best practices:

• Detailed logs of decisions and actions
• Full traceability of which model, prompt, and context led to which outcome
• Clear ownership and review processes

Julian adds that, when agents operate over time, they generate a huge amount of golden telemetry:

• Why certain decisions were made
• Which tools were called
• What data was read or written

That data is invaluable not just for regulators, but for improving the agent itself. It allows teams to replay scenarios, tune prompts, and refine boundaries without guessing.

Just as importantly, organizations need realistic expectations. Human processes are not 100% accurate either. If a human only gets a particular judgment call right 90% of the time, holding agents to a 100% standard is unrealistic.

Security, Shadow AI, And The Tooling Dilemma

One of the more pragmatic points Sarah raises is the risk of shadow AI.

If organizations don’t provide safe, sanctioned tools, people will reach for whatever works: using consumer chat apps on their phones, pasting screenshots of potentially sensitive data into public models.

A better approach is to:

• Provide secure, enterprise-grade AI access with SSO, logging, and clear policies.
• Allow different tools for different roles – coding agents for engineers, creative tools for marketing, analysis agents for data teams.
• Balance the desire for standardization with the reality that different people are productive with different tools.

It’s similar to the old password problem: make security so strict that nobody can use it, and you drive users to sticky notes on their monitors.

Learning From Feedback: Making Agentic Systems Better Over Time

Everyone agrees that feedback loops are at the heart of successful agentic systems.

There are two main types:

Explicit feedback

• Thumbs up / thumbs down
• “Was this helpful?” ratings
• Direct corrections from users

Implicit feedback

• Did the user come back and ask a human for help afterwards?
• Did they continue using the agent for similar tasks?
• Did they run, schedule, or deploy what the agent built?

For Maia, the agentic data team inside Matillion’s Data Productivity Cloud, this second category is especially important.

A key question becomes:

If Maia helped build a pipeline, did that pipeline then get run, scheduled, and trusted in production?

Over time, the team sees:

• The share of pipelines touched by Maia
• How often Maia’s work is kept vs. discarded
• Which types of tasks and datasets lead to successful sessions

Parallel to that, Maia is constantly learning about the data landscape itself: which columns are empty, where values look strange, which tables are effectively deprecated. That knowledge can be turned into documentation, semantic layers, and better future behavior – without retraining frontier models.

Looking Ahead: Agents As Everyday Collaborators

Toward the end of the conversation, Ed and Sarah look to the future.

Rather than a sudden jump to some sci-fi version of AGI, they expect:

• Progressive augmentation, where agents take on more of the tedious work and humans move up the value chain.
• More asynchronous, longer-running tasks, where you hand a bundle of work to an agent (“these five Jira tickets”) and it comes back with a ready-to-review pull request.
• Richer multimodal experiences, like AR glasses that quietly overlay names, context, and reminders about the person you’re talking to.

Sarah uses the analogy of smartphones: they started as mobile phones and turned into pocket supercomputers. AI feels similar – we’re still early in discovering entirely new things we can do, not just “the old things faster.”

The goal she hopes for? Something closer to a “Jetsons” future: helpful, sometimes sassy assistant systems that remove drudgery and expand human creativity – not replace it.

How Maia Helps Data Teams Get Agentic-Ready

At Matillion, these ideas aren’t theoretical. They’re being put into practice every day with Maia, the agentic data team that:

Automates low-level data engineering work inside the Data Productivity Cloud

Operates with structured constraints and human-in-the-loop controls

Is continuously evaluated in Matillion’s “gym” so it gets more capable and reliable over time

The result is a new way of working: data engineers, analysts, and business users collaborate with Maia to build, test, and run pipelines faster – while staying firmly in control of governance, quality, and outcomes.

If you’re exploring how to make your own organization ready for agentic AI, seeing Maia in action is one of the fastest ways to turn these ideas into something concrete.

Want to see what Maia, the agentic data team, could do with your data?

Book a Maia session with our team and experience agentic data engineering first-hand.