Data Management Trends 2025: Experts Insight

In case you missed it, Matillion recently hosted the Fifth Annual Data Management Trends Webinar that brought together some of the industry's experts to discuss the latest advancements in the world of data integration and management. This year’s webinar featured Ciaran Dynes, Chief Product Officer, Mark Balkendende, VP of Product Marketing, and Joe Reis, Author of Fundamentals of Data Engineering. Before we dive into their 2025 predictions, let’s go back to see how their 2024 predictions went.

2024 Report Card 

Unstructured Data in Analytics & AI Projects

The prediction that unstructured data would become mainstream in analytics and AI projects didn’t fully materialize. While it’s widely used in data science, it hasn’t yet become standard in everyday ETL workflows for data engineers. The team acknowledged progress but felt the adoption wasn’t as widespread as expected.

Takeaway: Unstructured data is still emerging in everyday ETL processes, with notable success stories but limited mainstream use.

Vector Embedding & Storage

Vector embedding and databases were expected to gain widespread traction, but it didn’t become a core part of data engineering in 2024. While many major databases now support vector storage, its adoption by data engineers has been slower than anticipated. The score was upgraded from a D to a C+ after discussion, reflecting progress but not the transformative shift hoped for.

Takeaway: Vector embedding is gaining traction, but it hasn’t yet become essential for most data engineers.

Prompt Engineering as a Core Skill for Data Engineers 

Prompt engineering was expected to be a core data engineering skill. In fact, 2024 saw the beginnings of prompt engineering adoption in data engineering, but it still represents substantial opportunities for growth in data engineering workflows particularly those handling unstructured data The team debated its future impact, with some feeling it should score higher, but others felt it hasn’t had the expected transformative effect yet.

Takeaway: Prompt engineering is growing but is still not a core skill for most data engineers.

Data Lineage 

Data lineage saw a significant resurgence, driven by increased focus on data governance, transparency, and AI integration. Major vendors like Snowflake and Databricks have emphasized data catalogs, which are integral to lineage. The team gave this a solid B, recognizing its growing importance in data management.

Takeaway: Data lineage is critical for AI integration and will continue to increase in importance.

AI Regulations (C+)

AI regulations didn’t unfold as expected in 2024. While there were some notable developments, such as the EU AI Act, the regulatory landscape didn’t expand as anticipated. In the US, state-level regulations created a patchwork approach, making the overall picture unclear. The team gave themselves a C+, recognizing progress but not the widespread regulatory movement they anticipated.

Takeaway: AI regulations are still in their early stages, with varying adoption across regions. The regulatory environment will continue to evolve.

Looking Ahead: 2025 Predictions 

Ciaran's Prediction: 

The year that agentic agents go from demo-ware to hype circle

2025 will see agentic agents move from demo-ware to the spotlight. While the concept gained traction in 2024, it's set to become a major topic this year. We're optimistic about agentic agents, which combine various tools and capabilities to solve complex problems. Unlike single-task Copilots, agentic agents can tackle multi-step tasks, like building an end-to-end data pipeline that predicts customer purchase propensity. This approach is particularly powerful in the data solution space.

Beyond the demos, agentic is poised to be a key way to interact with large language models in 2025. Notable vendors like Salesforce have already made strides, leveraging their productivity assets like Slack. We're excited about agentic’s potential and are working on something exciting at Matillion in this area in the coming weeks.

Joe’s Prediction:

Boring is back! 

2025 will see the return of the "boring" essentials. As AI and agents take center stage, the focus will shift to the foundational elements that make them work effectively—like data governance, data lineage, and data modeling. After years of excitement over new technologies, companies are realizing the importance of ensuring their data is in good shape, especially when layering AI tools like LLMs and agents on top.

We’re also seeing a resurgence in knowledge graphs, graph databases, and semantic technologies, which had been sidelined. Additionally, LLMs are now being used for tasks like code and data migration, effectively reducing human toil in these complex processes. While these may seem mundane compared to the flashy trends, they’re critical for the real-world application of AI, especially when it comes to building sustainable, functional systems.

Ciaran agrees with Joe—these fundamentals are crucial. Technologies like RAG, vector databases, and graph databases are essential for making AI systems work. They ensure that models can self-validate and improve responses. While human oversight will always be necessary, these foundational tools will enhance engineers' productivity, not replace them. The idea of "digital labor" is exciting because it complements existing teams, empowering them rather than replacing them.

Mark's Prediction:

Less Integration… of data platforms

The era of best-of-breed systems for data movement, transformation, and orchestration is over. Companies are shifting focus toward platforms that can handle a broader range of needs, including data integration, AI, and generative AI. Rather than stitching together multiple systems, businesses are seeking comprehensive platforms that streamline processes, allowing data teams to spend more time working with data and less time managing disparate integration tools. In 2025, platforms will be essential for data teams, making integration simpler and more efficient.

Ciaran's Prediction:

Data Governance Comes Back into Vogue

Data governance is making a strong return due to the increasing demand for AI projects, which require robust governance to ensure the integrity and traceability of results. Security concerns, non-repudiation of results, and accountability are central to this shift. As companies treat data more like a product, with defined SLAs and measurable quality metrics, data governance becomes crucial for maintaining high standards. AI can help streamline governance processes by automating tasks such as rule creation, documentation, and enforcement. AI agents can even suggest governance rules based on best practices, making the process less burdensome.

Governance had previously fallen out of vogue during budget cuts when organizations focused on essential functions like security and analytics. However, with advancements in AI and better data governance tools, platforms with strong APIs will become indispensable, making automated data governance a key enabler of efficient data operations.

Ultimately, data governance should be a company-wide initiative. It needs to span beyond engineering departments to ensure value is derived from all data assets. Viewing data governance through the lens of "data as a product" fosters a data-sharing culture within organizations, where governance serves a practical, value-driven purpose.

Joe’s Prediction: 

The convergence of skills 

Joe's second prediction highlights the increasing "convergence of skills" within the data and tech industry. He points out that, historically, professionals like software engineers, machine learning (ML) engineers, and data scientists have worked in silos, with limited overlap between their expertise. However, the rise of AI, particularly generative models like ChatGPT, has created a shift toward a more integrated skill set.

Joe observes that professionals from diverse fields—software engineering, analytics, and even machine learning—are now eager to learn about AI, machine learning, and related technologies. This change is driven by the explosion of interest in AI tools and platforms, where even non-experts are talking about advanced concepts like vector databases and fine-tuning. This blending of skills, or the "polyglot data practitioner," is becoming a theme for the year.

The convergence of skills also impacts the tools used in the industry, with platforms like Matillion needing to evolve beyond traditional ETL or ELT workloads to accommodate a wider range of use cases, including unstructured data and machine learning tasks. Joe's work-in-progress book, "Mixed Model Arts" (MMA), reflects this shift, emphasizing the importance of understanding data modeling across different domains—applications, analytics, and AI.

Joe and the other speaker also discuss the evolution of the data engineer role, which has expanded from being just an SQL or Python expert to someone who needs to understand the math behind models, vector chunking, and large language model operations. The idea is that professionals in this space are now juggling a wide range of tech and business challenges, making the role more exciting and dynamic than ever.

The conversation also touches on the value of having a broad skill set in analytics, not just for data engineers, but for anyone in tech roles like sales or marketing. The foundation of analytics can make professionals more well-rounded and capable of tackling various business challenges.

Mark's Prediction:

Data as the product is the methodology

Mark's prediction focuses on data as a product becoming the key methodology for organizations. He argues that while data mesh may be fading, treating data as a product is what remains central. Companies will increasingly have data product leaders managing specific data products (like customer or product data). This shift aligns data management with product management principles. While some, like Carson Bunch, see data mesh still growing, Mark believes data as a product will dominate as the primary approach going forward.

Ciaran's Prediction:

New Term Alert: Service-as-a-software

Ciaran introduces the concept of "Service-as-a-Software", which reflects a shift in how AI automates services, eliminating the need for human intervention. The idea is that AI can aggregate and orchestrate multiple APIs to deliver a complete service, making software more self-realizing and self-aware. This represents a fundamental change in IT architecture, where systems become increasingly autonomous.

However, this autonomy brings challenges, particularly around ensuring these AI agents stay within guardrails. Without proper control - hence the previous emphasis on Data Governance -  they could continue to solve problems endlessly, potentially leading to runaway effects. The discussion also touches on enterprise-grade agents, which must be guaranteed to operate within specific constraints to ensure performance and security.

Ciaran speculates that as this concept evolves, we could see new models like generative software as a service, where AI agents handle tasks autonomously. But whether this becomes a mainstream practice will depend on how well it can be controlled and integrated into business environments. This service-as-a-software idea challenges traditional software development paradigms, and while it shows promise, its long-term viability remains uncertain.

Joe’s Prediction: 

The Inversion of Roles

Joe predicts the Inversion of Roles, where the role of engineers shifts from low-level coding to becoming orchestrators and managers of workflows. Instead of focusing on technical details, engineers will spend more time working with the business side to understand business problems and translate those into solutions using emerging workflows. This shift will require engineers to adopt more business-oriented skills and communicate effectively in business terms. While this represents a significant change, there's a potential risk that engineers may still need to clean up after automation tools or bots cause issues.

ChatGPT 2025 predictions 

1. AI-powered automation will dominate Data Management 
2. Rise of Unified Data Ecosystems 
3. Generative AI for Unstructured Data Utilization 

The Fifth Annual Data Management Trends Webinar offered key insights into the constantly changing world of data integration and management. While 2024 saw some progress in areas like unstructured data, vector embedding, and data lineage, certain predictions, such as the rise of prompt engineering and widespread AI regulations, fell short. In 2025, experts predict major shifts, including the rise of agentic agents, a focus on data governance, and the convergence of skills across tech roles. As businesses prioritize AI-powered automation and integrated platforms, data professionals will increasingly shift towards orchestration and business collaboration. The year ahead promises to shape the future of data management.

Watch the full webinar here!