October 17, 2024

Landkreditt modernizes its data operations for improved performance and growth

Schedule a demo

4x

faster data loading times with 9 years of additional historical data added

30x

performance improvement in data processing

50%

reduction in job management overhead through streamlined ETL workflows

Challenge

Landkreditt’s legacy data infrastructure struggled to keep up with the group's expanding demands and data aspirations. As customer growth surged alongside the introduction of a wider array of financial products and services, the outdated infrastructure became unable to handle the increasing data volumes and performance scales. Consequently, this led to performance issues, inefficiencies, limited access to historical data, and a reliance on manual, time-consuming development processes that impeded timely insights.

Solution

Landkreditt sought to establish a modern data platform by implementing Matillion ETL alongside Snowflake and Power BI. This setup enabled the company to efficiently load, transform, and orchestrate data from various sources, significantly reducing loading times as well as simplifying and streamlining its previous ETL processes into more efficient ELT processes with unique push-down capabilities in its DAIRY-coined data platform.

Landkreditt went further and also adopted a streamlined modern data development practice and culture with Azure DevOps, automated, test, build, and release pipelines, and Git repositories, which helped standardize its data operations and improved speed, consistency, and overall quality.

Results

Data loading became four times faster, with a 30x performance improvement in overall data processing. Historical data expanded by nine years, and development workflows were standardized, resulting in more efficient data operations and optimized performance. The streamlining of ETL workflows led to a 50% reduction in job management overhead and significantly reduced maintenance efforts. Matillion's intuitive interface and comprehensive documentation made onboarding easy, while its robust features ensured data and ETL job quality, enabling Landkreditt to meet business needs accurately and efficiently.

The DAIRY project, and what has been delivered now, has elevated the business’s demand for insights even higher than before. As a result, the push for data and deriving insights has grown significantly. With our modern DAIRY data platform and our streamlined ELT processes, we can now meet these demands and deliver insights efficiently.

Gudbrand Olimb
CTO and Enterprise Architect
Landkreditt

Challenge: Legacy infrastructure limits, performance strains, and inefficient data processes

Landkreditt, a financial services group founded and still owned by a cooperative of Norwegian farmers who offer wide-ranging private and corporate products and services, relies heavily on data to drive business decisions, ensure compliance, and support its growth ambitions. Before implementing Matillion, the company encountered challenges with its legacy data infrastructure, which started to reach its limits.

The existing data infrastructure and data warehouse had been in use for over a decade, and the company struggled to keep up with the vast amounts of data it processed daily. As its data needs grew and business evolved—especially with the introduction of a new insurance company a few years earlier—Landkreditt began to struggle with its increasing amounts of data, processing it timely and efficiently and meeting new business and performance requirements.

The development process was a constant balancing act, as the company’s team had to minimize the amount of data loaded into the warehouse to ensure its performance. This meant they needed to maintain limited historical data, leaving them with incomplete insights and complicating their efforts to achieve a holistic view. With the legacy data warehouse, the team had to make tough decisions about which data points to keep and which to ignore, potentially sacrificing opportunities for more in-depth analytics.

Additionally, the development process itself was far from efficient. With long lead times, manual intervention, and scattered data sources, Landkreditt’s team faced a lack of standardization. Jobs that should have been simple often required separate workflows, and managing data across various systems made troubleshooting and quality control more difficult.

All these inefficiencies were not only time-consuming but also hindered the ability to attract and onboard new talent, prolonged the time to insights, and resulted in hours spent on maintenance instead of new deliveries and innovative ideas.

The organization was at a crossroads: It needed a modern, efficient, and scalable data architecture to support its growth and provide timely and accurate data and insights across its operations —all without the risks and high upfront costs of a major system overhaul.

We sort of hit the roof a bit when it came to performance, affecting development, affecting storage, affecting multiple things, and we also had an expiring infrastructure.

Bjørn Anders Gislerud
Project Manager & Data Platform Architect
Landkreditt

Solution: Implemented a unified data platform to improve efficiency and scale development

To address its challenges, Landkreditt implemented a comprehensive solution by combining the capabilities of Matillion, Snowflake, and Power BI to create its DAIRY platform — a Data platform for Analysis, Insights, Reporting, and Ymse (Norwegian for all that is unknown, undefined, or future requirements).

The transition began with a careful evaluation of various options, and Matillion stood out as the perfect partner to complement Landkreditt’s move to Snowflake's AI data cloud. A key decision driver for choosing Matillion was its distinctive and flexible pay-as-you-go pricing model, which minimized risk and allowed Landkreditt to test the waters without a huge upfront investment.

Another key decision driver for choosing Matillion as its ETL/ELT tool was that Landkreditt could do more than just load data efficiently; the company could also transform and orchestrate it in one unified place. This meant it no longer needed multiple vendors for these tasks, which was preferred and aligned well with the group's aims to keep compliance work manageable and simplify the number of vendors to oversee.

The implementation allowed the team to shift to a modern data development practice and culture, leveraging Azure DevOps, automated, test, build, and release pipelines and Git repositories to maintain multiple development branches. This brought standardization, increased quality, and agility to their data operations, as well as streamlined the development process.

One of the standout improvements came in the way Landkreditt loaded data. Matillion enabled the company to load data from large and complex financial core source systems into Snowflake more efficiently, allowing it to ingest all columns from every table rather than just the essentials. This provided a richer data set to work with and reduced the time required to complete loading by fourfold.

The team also switched from manually maintaining individual ETL jobs for each table to using control tables and generic Matillion jobs with variables and Python scripts to automate and streamline entire data loading and transformation processes. This shift dramatically cut down development time and made onboarding new team members easier since there were now clear and standardized processes.

The combination of Matillion's efficient ETL processes, Snowflake’s highly performant and robust data cloud architecture, and Power BI’s flexible reporting gave Landkreditt the modern data infrastructure it needed. It streamlined workflows, ensured data consistency, and set the stage for future growth, which helped the company access and use data in ways it hadn't been able to before.

Every day, Matillion enables us to run high-quality, high-performance ETL/ELT workflows, and now also provides a streamlined approach to scheduled and on-demand AI and machine learning model inference.

Gudbrand Olimb
CTO and Enterprise Architect
Landkreditt

Benefits: Boosted performance, streamlined data operations, and enhanced business agility

After implementing Matillion ETL, Landkreditt experienced a transformation in its data operations that rippled across its entire business.

The group’s data infrastructure performance saw a significant boost, with data loading times reduced by fourfold. This improvement meant teams could now handle more data than ever before, including nine additional years of historical data, all without compromising performance. The ability to load full tables and every column from source systems gave teams a richer and more complete view of Landkreditt’s data assets and landscape, which enabled deeper insights and analytics that simply weren’t possible before.

Development efficiency also soared. Matillion’s user-friendly interface, coupled with Landkreditt's modern data development approach using Azure DevOps and Git repositories, allowed Landkreditt’s small team of data engineers to create, test, and deploy data workflows much quicker than they could with their previous system. The standardization brought by Matillion’s approach—using control tables, variables, and a single set of generic jobs—meant fewer chances for error, easier troubleshooting, and a more straightforward path for any new team members to come in and hit the ground running. In fact, through the use of the Matillions approach, Landkreditt saw a fifty percent reduction in job management overhead through streamlined ETL workflows compared to their legacy data warehouse.

Landkreditt also saw a significant improvement in developer productivity. Matillion’s intuitive design meant that developers could quickly build and maintain jobs without extensive training. The built-in search features, which allow users to easily trace data lineage or understand how different components interact, provided further efficiency and transparency in development.

Landkreditt’s transformation didn’t stop there. The benefits extend beyond just development. Across the organization, departments could now access data through a unified reporting platform in Power BI, allowing them to make informed business decisions faster. The structure and consistency Matillion brought allowed end users to self-serve their reporting needs, with all data housed in one central place. This shift not only streamlined business operations but also improved data governance, ensuring that every part of the organization was working from the same, accurate set of data.

In addition, Landkreditt was able to build a proactive and new data culture. With their new DAIRY platform, the company could quickly identify and resolve data discrepancies, and Matillion’s pushdown capabilities enabled it to maintain data quality across the board. This was a huge shift from its previous approach, where accessing and distributing data involved multiple systems, manual workarounds, and long lead times. Now, the company can deliver changes rapidly, react to business needs quickly, and ensure data accuracy.

Overall, the partnership with Matillion ultimately empowered Landkreditt to be more agile, more innovative, and more data-driven in every aspect of its operations.

Now, I develop new jobs and workflows in Matillion a lot quicker than I did in the old system with just a year and a half of experience; it’s a lot of fun.

Elisabeth Carson
Senior Data Engineer
Landkreditt

What's next?

With Matillion ETL firmly in place and the legacy infrastructure nearly sunsetted, Landkreditt is exploring new possibilities to maximize the value of its modern data platform.

Now that data operations are more efficient and accessible, Landkreditt is looking to dive deeper into areas that have yet to be fully explored—especially regarding enhancing operational processes and leveraging additional data points from both its internal datasets and relevant third-party data providers.

The company is also considering Matillion's Data Productivity Cloud to further streamline its development processes and data delivery. Moving toward this cloud-based approach could help it take full advantage of new AI capabilities that enhance development productivity. While the internal data requirements and network access will be part of the company’s evaluation process, it sees this direction as a promising path for future scalability and agility.

As it moves forward, one of Landkreditt’s biggest goals is to expand the breadth of data sources they can pull into Matillion and Snowflake. Since it already successfully migrated its core financial services data, including some partner data, Landkreditt plans to explore new sources like CRM data and mutual fund market information. With the technical barriers now removed, Landkreditt is also open to enriching its data warehouse with any data source that could bring business value, whether it’s a database, API integration, or file-based systems.

As it continues to enhance its data operations, Landkreditt is particularly focused on making data more accessible and usable for the entire organization. With its foundations now set, Landkreditt sees Matillion as a tool not just for keeping up with current business needs but as a partner that will help the company grow, innovate, and respond quickly to whatever the future holds.

About Landkreditt

Landkreditt is a financial services group offering a wide range of banking, insurance, mutual fund, and real estate products and services to private and corporate customers. Founded in 1915 and still owned by a cooperative of Norwegian farmers, Landkreditt manages kr38 billion in assets for several thousand customers across Norway. With a strong focus on growth and customer service, it leverages data to drive smarter business decisions and ensure seamless operations across all departments.

By staying at the forefront of technology and data management, Landkreditt continues to provide reliable financial products and services while meeting customers' needs and the industry's regulatory demands.

About Matillion

Matillion is a top provider of data transformation solutions designed for cloud data warehouses with AI capabilities. The platform helps companies efficiently extract, transform, and load their data, which enables them to gain valuable insights and make informed, data-driven decisions.

Equipped with a wide range of pre-built connectors and powerful integration features, Matillion empowers organizations to harness the full potential of their data, boosting efficiency, fostering innovation, and supporting growth. By simplifying data integration, the company has become a trusted partner for businesses aiming to enhance their data processes.

Our latest resources

View all resources

Ready to get moving?

See how quickly your team can start delivering business-ready data, with Matillion.

Join Matillion

We need driven, curious people who think big and move fast to join our team.