Databricks Serverless SQL: Boosting Productivity and Lowering Costs

Data managers are always looking for ways to get the most from their investments, and Databricks Serverless SQL offers a compelling solution to improve productivity while keeping costs to a minimum. This article will explain how switching to serverless SQL warehouses in preference to the "classic" Databricks compute option can help you and your team achieve this, with a focus on the bottom line.

What is Databricks SQL?

Databricks SQL is an intelligent data warehouse solution built on a lakehouse architecture, designed to simplify data management and analytics. It uses DatabricksIQ to automatically optimize queries and management, enhancing performance and efficiency. The platform provides unified governance via Unity Catalog, ensuring consistent security. It is serverless and cost-effective, integrating with tools like Matillion, Power BI, and Tableau. Predictive optimisations, powered by machine learning, improve infrastructure and workload performance. Databricks SQL streamlines analytics by removing data silos, enabling swift transitions from BI to ML.

Databricks Serverless SQL vs Classic Compute

The key difference between these two architecture options is about how resources are managed and paid for.

The Classic Compute option is simple, and requires that you set up and manage fixed-size clusters. You have the choice of leaving them running continually, or else stopping and starting them manually – potentially leaving your users exposed to some latency. These Classic Compute, always-on clusters are available in two options:

Classic: the cheapest option, suitable for basic interactive SQL queries on a self-managed warehouse, providing entry-level performance.

Pro: offers medium-priced performance, extending the SQL experience for exploratory SQL, SQL ETL/ELT, data science, and ML on a self-managed warehouse. Pro supports Photon and Predictive IO, but not Intelligent Workload Management.

In contrast, the Serverless option gives true on-demand access to resources that automatically scale up or down as needed. You only pay for what you actually use, and the infrastructure management is handled by Databricks.

Serverless: the most expensive option, but it's a fully managed service that only runs on demand. Serverless offers the best performance for high-concurrency workloads. Serverless SQL warehouses support all performance features, including Intelligent Workload Management, and autoscaling. Startup time is almost undetectable: between 2 and 6 seconds.

Challenges with Traditional Compute

Before diving into the advantages of Serverless, it’s important to understand the limitations of classic Databricks compute that can occur in practice. In the classic approach:

• Teams can lose time waiting for clusters to start - typically around 4 minutes each time - which slows down work for analysts and data scientists, impacting productivity.
• Maintaining always-on clusters can be wasteful, because you're still paying even when the system isn't fully used.
• Managing traditional compute clusters adds to your infrastructure overhead.
• Fixed size clusters are not always optimal for fluctuating data loads.

The Databricks Serverless SQL Serverless Advantage: Cost Savings and Increased Output

Databricks Serverless SQL offers a more efficient and cost-effective way to manage data analytics, and offers these key benefits:

• Instant Compute: Resources are available immediately, so there's no waiting time, making work faster and more productive.
• Cost Efficiency: Managed infrastructure with automatic scaling means your team can focus on analysis rather than managing clusters. You only pay for what you use, avoiding costs for unused resources or extra capacity.
• Built-in Guardrails: You choose an appropriate cluster size, expressed as a base cost per hour, and then set limits on how much the serverless infrastructure can scale up as demand increases.
• Better Performance: Features such as query caching and indexing are applied automatically to speed up data processing.

Making the Switch

Transitioning to Databricks SQL Serverless is straightforward. For existing Databricks customers, it's as simple as selecting the 'serverless' option when defining a new SQL warehouse.

Is it right for you?

Databricks SQL Serverless provides the necessary performance and scalability for enterprise lakehouses. Many companies have already seen significant gains in performance and cost savings by using it in preference to traditional compute.

To use Databricks SQL Serverless, your account should not be on a free trial, and your workspace must be on the Premium plan or above. Additionally, your workspace needs to be located in a region that supports Databricks SQL Serverless.

Conclusion

By embracing Databricks SQL Serverless, data managers can achieve significant improvements in total cost of ownership and productivity. The combination of instant compute, reduced management overhead, and cost-efficient scaling allows data teams to deliver more value to the business, focusing on the strategic rather than the mundane.

Fully supporting Databricks SQL Serverless in this endeavor is the Matillion Data Productivity Cloud - a game-changer when it comes to building and managing data pipelines. Matillion is the data integration platform that accelerates data engineering processes, offering a seamless blend of productivity, collaboration, speed, and flexibility. With a code-optional approach, Matillion democratizes access to AI and analytical insights, enabling data teams to operate at scale with unparalleled efficiency. Whether through its no-code components for Generative AI or its robust Hybrid SaaS deployment, Matillion stands as a powerful ally for data managers intent on enhancing pipeline orchestration while integrating seamlessly with Databricks SQL Serverless.