Future-proofing data platforms to support business growth at St James’s Place

SJP’s legacy data warehouse struggled to support multiple applications, failed to meet SLAs and lacked scalability.

Working with Matillion partner, Snap Analytics, SJP implemented Snowflake on AWS and adopted Matillion for data transformation.

With Matillion, SJP's daily loads now run within the expected timeframe, consistently hitting SLAs over 98% of the time. 

St. James Place (SJP) is the largest advice-led wealth management firm in the UK, managing over £168 billion in assets. SJP operates through a vast network of partners, specializing in ISAs, pensions, and bonds, and employs over 2,000 staff at its main office in Cirencester, UK. Their data engineering team, led by Barney Eccleson, plays a pivotal role in ensuring smooth data platform delivery.

The SJP legacy SSIS solution struggled to support multiple applications and failed to meet the necessary SLAs, hitting them approximately 7% of the time. The existing system lacked scalability, and SJP needed to move to a more robust, flexible solution that could fulfil the company's needs through 2030.

As Barney explained, "With daily loads taking longer than a day, our old system simply couldn't keep up with our growing data demands." The company's initial attempts to switch to other platforms revealed the risks of a "lift and shift" approach, leading them to evaluate more efficient solutions.

Working with Matillion consultancy partner, Snap Analytics, SJP implemented Snowflake on AWS and adopted Matillion as their data transformation tool. This decision was driven by several factors, including Matillion's ease of use and mature product set. Matillion's REST API and integration with other AWS components, like S3 and SNS, played a critical role in orchestrating complex data workflows.

SJP's data engineering team needed to integrate and transform data from various sources, including large XML files, a CRM system on Aurora, and hundreds of smaller data sources. The XML files, often larger than 50 MB, required unpacking and multiple levels of flattening, a task Matillion handled with ease. 

The data transformation followed a three-tiered, medallion structure:

• Bronze: Extract and load into Snowflake, with extensive data quality checks and metadata-driven processes, all orchestrated by Matillion.
• Silver: Transform the data into SJP's canonical model, focusing on consistency and shared jobs for Change Data Capture (CDC).
• Gold: Deliver to different personas, ensuring each delivery zone has its unique model.

This approach allowed SJP to create a solid new data platform to replace their legacy data warehouse and provide the flexibility to meet changing business requirements.

With Matillion, SJP's daily loads now run within the expected timeframe, consistently hitting SLAs over 98% of the time. 

This change has positively impacted both customer and partner satisfaction. The SJP app, built on top of the new data platform, provides customers with near real-time information, a feature that would have been impossible with the old system.

SJP's financial actuaries already benefit from the new data platform directly. This shift has increased productivity and allowed SJP to deliver better services to its clients. According to Barney, "The actuaries used to get frustrated with limited data access, but now they can get answers to complex questions almost instantly. It's a complete transformation."

The team continues to migrate functionality onto the new platform, with the long term goal of replacing the legacy system entirely. SJP’s scaled agile methodology determines the order of the elements to move next and allows constant refinement of migration priorities according to customer demand.

The team is focused on expanding the use of the new data platform, with plans to develop a data marketplace and integrate additional data sources. Matillion's REST API will help to streamline processes as they explore more automation and DevOps practices.

The journey from legacy ETL to a modern, cloud-based data platform has transformed the way SJP operates, enabling them to provide better services to their clients and stay ahead in the competitive financial services industry.

St. James's Place plc, formerly St. James's Place Capital plc, is a British financial advice company. The head office is in Cirencester, in Gloucestershire, and there are over twenty other offices in the United Kingdom. It is a combined adviser, fund manager and life insurance business.

Snap Analytics partners with enterprises to help them connect their data, technology and teams to deliver exceptional business outcomes.