Skip to main content

Secure, scalable, and serverless: Matillion’s Unlimited Scale

Published Date
Author Ashley Lozito

Orchestration of data jobs is crucial and can’t be addressed as an afterthought.  Data owners need confidence in not only the first run of the data but also the hundredth and thousandth runs as well.  The challenge with this is scale; data volumes continue to rise and what worked on a small batch starts to break down as data expands to gigabytes or petabytes.

When we ask our customers to identify pain points, we learn that provisioning, sizing, and replicating VMs are always at the top of the list.  These tasks can be complex and time-consuming — robbing valuable engineering resources that could be focused on tasks that add value to the data journey.

Expanding compute without complexity

Elasticity is the biggest benefit to moving data into the cloud – not only to accommodate for volume but for compute as well.  So we thought, “why don’t integration platforms work the same way?” and this gave rise to Matillion’s new Unlimited Scale capability.  

This year we are bringing the full compute elasticity of the cloud to our customers, allowing them to scale data processing indefinitely, seamlessly, and putting an end to worries about resource allocation. 

 We’re delivering a processing platform based on serverless remote runners; agents that run at optimal compute levels by self-adjusting in real-time.  Any size workload simply runs with the click of a button, or on schedule, and the processing resources automatically scale up and down as needed. Matillion’s Unlimited Scale architecture takes care of the details.  Our flexible serverless architecture is the first of its kind.

 

Addressing the concerns of the entire business

The mission of the Data Productivity Cloud is to ensure the success of data teams and deliver business-ready data, faster – and Unlimited Scale does just that while addressing the following concerns:  

  • Administrative: Unlimited Scale removes administrative overhead by abstracting the complexity of resource allocation, allowing our customers to move faster and stop worrying about increasing compute demands from job loads or data volumes.  
  • Financial: Unlimited Scale employs responsible financial governance to ensure cloud spend is optimized and FinOps teams stay happy. 
  • Security: We take customer compliance and data sovereignty requirements into account with every new feature we introduce.  The Unlimited Scale agents are hybrid; they can be run in your cloud or ours so customers retain full control of the flow of data, just as they always have with Matillion.

Built for the future, available now

This technology is built to pair with our SaaS platform coming next year, but you don’t have to wait.  It’s available today and some of our largest workloads are being orchestrated to our Unlimited Scale agents in our preview program.  To learn more about this groundbreaking functionality and our ongoing preview, we welcome you to check out Matillion Product Director Tom Riding’s session on Unlimited Scale at Data Unlocked on the Data Productivity Cloud Success Center.