Matillion's AI-Driven Test Plan Generator

Quality engineers working on SAAS architecture face constant challenges in deploying software smoothly. Despite employing quality-driven test strategies, ensuring clear visibility into changes bound for production remains a hurdle. Exploratory testing sessions emerge as a crucial component, complementing automated testing by validating core functionalities and new features. However, challenges, like efficiently gathering tickets from Scrum teams and tracking service versions, persist.

To streamline this process, the Quality Engineering team embarked on an innovative journey, leveraging our AI-driven Data Productivity Cloud (DPC) to create a unified pipeline for generating comprehensive test plans. Our mission was to compile test plans encompassing changes across all services, linked to respective JIRA tickets, identify test cases, and catalogue versions across environments for stack visibility.

By empowering quality engineers with rapid access to crucial information, we aimed to enhance productivity and enable informed decision-making prior to testing.

The problem at hand

Our quality engineers faced a cumbersome process of manually generating test plans by extracting data from Jira and deployment systems. Recognizing the inefficiency, we envisioned an AI-driven solution to streamline exploratory testing practices: 

1. Identifying the Problem:
   • Manual test plan creation from Jira and deployment data was time-consuming.
2. Recognizing the need:
   • Understanding the inefficiency, we sought an AI-driven solution for streamlined testing using our very own Data Productivity Cloud (DPC)
3. Developing the AIpPipeline:
   • Utilizing Matillion's AI-powered Data Productivity Cloud (DPC) to automate test plan generation. We identified we had the necessary features and capabilities to make this a reality.
4. Streamlining tracking:
   • We knew our deployment history was stored in DynamoDB tables for efficient deployment tracking across environments.
5. Automating comparison:
   • The next step was to create transformation pipelines for automated version comparison and change tracking.
6. Optimizing deployment identification:
   • We were then able to refine methods to identify preprod and production deployments through intelligent filtering.
7. Harnessing AI for analysis:
   • Utilising our AI features to analyse deployment and Jira data, we wanted to generate insightful summaries and test suggestions.
8. Seamless Jira integration:
   • Integrating the AI insights with the Jira API would allow us to create a test plan automatically and fill it with everything we needed, from service versions, changes, test recommendations and suggestions.
9. Embracing DPC's potential:
   • We felt leveraging DPC's capabilities would enhance efficiency and decision-making in test planning.

Streamlining Deployment Tracking

Every deployment within each environment (dev, preprod, and prod) of the Data Productivity Cloud is meticulously tracked in a DynamoDB table. This table captures crucial details: artifact names, versions, deployment environments, timestamps, and associated Jira ticket IDs. Leveraging Matillion's DynamoDB connector, we seamlessly extract this raw data into a staging table, laying the foundation for further processing.

Automating Version Comparison

To facilitate environment comparison, we employ a transformation pipeline to split version numbers into Major, Minor, and Patch components. This then allows us to perform ranking and comparisons against version numbers allowing us to track every deployment since our last production release to obtain the changes in each and every deployment. This is easily manageable with the DPCs native transformation components.

This compares the current pre-prod deployments vs prod and gets a list of Jira tickets associated with the deployments. These Jira tickets are all of the changes currently in preprod ready to be tested and released.

Optimising Production Deployment Identification

Identifying current preproduction and production deployments is streamlined through a series of transformations. By intelligently partitioning and filtering records based on deployment time and artifact type, we isolate and store our current preproduction and production deployments in a separate table.

This gets all successful pre-prod and production deployments from the source table and ranks them within an environment and artifactHash partition in order of deployment datetime in descending order. 

This gives us the latest preprod and production deployments for each service equaling 1. We then split preprod and prod and join them into one table before writing the data into the `DEPLOYMENTS_CHANGELOG` table. 

This is our cleaned change log with the latest versions we will test and release:

Harnessing AI for Insightful Analysis

Here's where the magic of AI comes into play. Utilizing Matillion's AI prompt components, we aggregate deployment data, Jira ticket information, and service versions preparing it for analysis. Through contextual prompts, our AI generates comprehensive summaries of changes and suggests relevant test cases, enabling the teams to take a data-driven test approach to quality validation. Moreover, all supporting component notes are also generated seamlessly through our new AI note generator.

Using the power of our AI prompt components we pass our previously generated lists of Jira ticket information and deployment information and ask the AI to generate a change summary in Atlassian Bullet format as well as a change log of the service versions in Atlassian Table format.

Seamlessly Integrating with Jira

Finally, we integrate our AI-generated insights to post a request to the Jira API and streamline test plan creation. Leveraging Matillions own Custom Connectors and tailored queries, we effortlessly translate our data into actionable test plans within the Jira ecosystem in particular using the Jira Atlassian Document Format(ADF). The output gives us everything we need within seconds, from service versions, changelogs, desired document formats, and even identifying test cases based on changes!

This is where we create the test plan. We take all of the generated AI results and query them to variables which are then used in the payload we send to Jira to create our test plan using a Custom Connector. 

Embracing the Future of Data Management with DPC

Matillion's AI-driven pipelines, powered by the Data Productivity Cloud, make end users more efficient; whether you’re a data engineer or, in our case, a quality engineer, we are able to move faster with AI-driven pipelines. By providing a unified platform to address the most complex use cases, Matillion empowers organizations to unlock the full potential of their data. Whether it's accelerating time-to-insights, ensuring regulatory compliance, or driving operational efficiency, 

Transforming Data Management with Innovation

Through the magic of AI and our Data Productivity Cloud, Matillion empowers organizations to easily tackle even the toughest data challenges. We've made strides in efficiency, slashing manual efforts by over two hours while seamlessly gathering data from various sources in just two minutes. This streamlines processes and enables more relevant end-to-end testing, driving faster insights and better outcomes.