What's new at Matillion: Innovations driving modern data integration

As we know, the pressure on data teams to deliver faster, business-ready data for analytics is higher than ever. At Matillion, we’re seeing trends that highlight the evolving demands of modern data integration. This blog dives into Matillions Chief Product Officer, Ciaran Dynes, and VP of Product Marketing, Mark Balkenende's discussion on the latest market shifts, explores customer needs, and introduces how Matillion’s innovations help reduce time to value, enable AI-driven innovation, and support enterprise-ready solutions. 

How the market is evolving 

Data friction and skill diversity 

As data demands increase, so does data friction - teams are faced with more complexity in managing and integrating vast data sets. Modern data teams need a mix of skill sets, for example, engineers, site reliability engineers, security expertise, and hybrid coding capabilities. At Matillion, we aim to empower all members of the data team, whether they are seasoned coders or prefer low-code//no-code solutions. Our focus includes leveraging AI technology to support diverse personas, making collaboration seamless across roles. 

Infrastructure 

For two decades, businesses have started migrating applications to the cloud. Today, many are asking, “How can I move away from managing infrastructure altogether?” Matillion users are increasingly interested in shifting from traditional SQL and Python workloads to adopting AI and cloud-native operations for greater scalability and reduced overhead. 

Data Apps and Data Products

The conversation around data has evolved beyond integration into creating data apps and accelerating data products. These applications help businesses derive actionable insights faster. Concepts like the semantic layer - organizing metadata to enhance discoverability - are becoming essential for building scalable data solutions. 

Co-Pilots

Matillion’s version of “co-pilots” emphasizes solving operational challenges. While the first iterations enable intuitive drag-and-drop functionality, the ultimate goal is more profound, addressing operational bottlenecks for data engineers and broader personas alike. Co-pilots are not singular but plural - designed to adapt and grow alongside business needs. 

Upcoming innovations from Matillion

At Matillion, we’re always innovating to meet the demands of our customers. Here’s a glimpse of what’s coming:

End-to-end Lineage

Ciaran's standout feature is the newly enhanced end-to-end data lineage. This visual, no-code/low-code solution allows teams to trace data from its source through transformation and into its final use cases. This capability boosts productivity, scalability, and maintainability, offering a complete view of your data ecosystem. 

Iceberg and Kafka Support

Mark mentions that Matillion's new Iceberg support promises to simplify catalog management for those working with Snowflake or AWS. We’re also expanding capabilities in streaming data integration, including Kafka support! More details about this connector can be found here. 

Enterprise-ready architecture 

Matillion’s full SaaS, pushdown-based architecture sets the stage for robust enterprise security. And while we’re not revealing all the details just yet, the term embedded hints at exciting enhancements designed to take ELT to new levels. 

AI integration 

For businesses ready to embrace AI but unsure where to begin, we ask, “What are you doing with your documents now?” Matillion provides solutions to unlock the potential of unstructured data, guiding businesses from exploration to implementation. 

Let’s dive deeper into Matillion's end-to-end lineage feature! 

Matillion's end-to-end data lineage feature provides unmatched visibility into data pipelines, enabling quick troubleshooting and impact analysis. In a real-world scenario, if a dashboard appears empty, lineage allows users to trace data from its source through transformation and orchestration pipelines, identifying issues like misconfigured filters or missing data. The feature simplifies root cause analysis and impact analysis and provides access to operational and technical metadata. By offering these capabilities, Matillion helps data engineers optimize workflows, debug faster, and maintain reliable pipelines at scale while seamlessly integrating with partner tools for broader data management needs. To learn more about this feature, check out the blog on it here! 

Exploring Matillion's high-code capabilities 

Matillion's new high-code capabilities in the Data Productivity Cloud (DPC) empower data engineers with greater flexibility by integrating programmatic workflows with visual pipelines. Users can directly write and import SQL and Python, collaborate through Git for version control, and maintain a structured DevOps process. A recent demo highlighted key features, such as creating Git branches, integrating code into pipelines, and automating execution with scheduled runs. Looking ahead, DBT integration will further enhance these capabilities, enabling seamless programmatic workflows. These innovations help data engineers balance code and visual workflows, improving efficiency and enabling high-quality pipeline delivery.

See how SQL and ETL innovation is shaping data integration workflows.

The AI advantage 

Matillion's AI-driven features, including Co-Pilot and Auto-Documentation, simplify complex data tasks, allowing users to interact with data using natural language. These tools eliminate the need for specialized infrastructure, making tasks like building data tables in a warehouse more accessible. For example, a customer in the optical industry used Matillion’s AI and Snowflake integration to reduce training time by 50%, improving operational efficiency and reducing errors. In demos, Matillion’s tools automate processes like transcribing sales calls and processing support manuals, turning voice or PDF data into actionable insights without requiring coding. This AI-powered, no-code approach transforms how businesses handle data, enhancing efficiency and accuracy.

To conclude

At Matillion, we’re focused on empowering all users, from data engineers to business analysts, with tools that boost productivity. One key challenge is ensuring data quality, which we’re addressing with AI-powered Copilots. Initially assisting with pipeline building, our Copilots will soon expand to data quality, analyzing sample data to identify issues like inconsistent capitalization or missing values. Looking ahead, they’ll also suggest performance improvements and optimize pipeline logic, making data workflows more reliable and efficient. Copilots are evolving from tools to partners, helping teams innovate and focus on what truly matters.

Want to see the demos in action? Catch up here!