Transforming Sales: How Data Integration and Advanced Text Analytics Propel Business Growth

During every non-trivial sales cycle, data is likely to end up scattered around many different operational systems. From CRM platforms and email correspondences to social media interactions and support tickets, much of it in the form of unstructured text, voice transcripts or video call recordings.

Each piece of data holds a critical puzzle piece to understanding and engaging prospects effectively. But in order to obtain a fully comprehensive view of each prospect there are two main challenges to overcome:

  1. Dispersion of the data among many systems
  2. Interpreting the sentiment and concerns expressed in natural language

 

To enhance the sales cycle and make it as efficient as possible, it's important to solve the two challenges independently. Data integration first …

Data Integration - dealing with dispersed data

First of all, it is vital to harness all available data sources. Prospects can give direct feedback, express sentiment, and voice their concerns anytime and through any medium.

From a data engineering perspective, this text data – along with a hundred other nuanced factors that could potentially block or facilitate a sale – ends up dispersed among many systems. It is impossible to gain a holistic view of each prospect without integrating all the data sources. This is how to ensure that no critical information is overlooked.

One effective methodology for this kind of integration is to use techniques borrowed from Data Vault as a data modeling technique. It doesn't have to be complex: just a set of tables comprising:

  • A reference table for the prospects, with one record per prospect. This is a "hub" table
  • One table per source system, linking back to the hub, containing every fragment of text that the prospect has used in that source system. These are known as "satellite" tables because they surround the hub

They would be placed in the silver layer if you're following a medallion data architecture.

This style of data modeling presents a scalable and flexible approach, well-suited for handling multiple and varying data sources. Its simple structure, consisting of hub and satellite tables, effectively captures the different aspects of data and relationships inherent in business operations.

Importantly, it accommodates the continuous addition of new data sources without disrupting the existing architecture, which is common in dynamic sales environments.

It's also easy to extend in the future to a larger data vault model by adding link tables that express a relationship between hubs.

Unstructured textual analytics

Now that the data dispersion problem has been solved, textual, natural language interaction data is still impossible to process using traditional analytical methods. It's all there: all the feedback from prospects, their feelings, concerns, and all the other subjective factors that could either help or hinder a sale… but it's buried in the text.

This is where Large Language Models (LLMs) come into play. LLMs are advanced artificial intelligence systems capable of understanding and generating human-like text based on the data they are trained on. By deploying LLMs, businesses can unlock valuable insights from unstructured text, transforming it into actionable intelligence.

Now that all the pertinent data has been transformed into data vault structures, they serve as a centralized repository where data is methodically organized. This unstructured data is easily aggregated by prospects and is ready for processing and analysis through an LLM.

LLMs can discern sentiment, categorize feedback, and identify key terms and phrases that indicate buying propensity or hesitations in customer dialogues.

To bring this to life, I have a representative data set for one single prospect.

Sales Sentiment Analysis Example

Here's what the data might look like for a prospect (identified as #20) who has communicated with us using three different sales channels.

Salesforce Opportunity data for prospect #20

This is just plain text extracted using a Salesforce Query component.

What makes your financial advisory team different from others?
Why should I choose your company over your competitors?
Salesloft data for prospect #20

This is the result of processing through Amazon Transcribe after downloading a voice call

{
  "accountId": "000000000000",
  "jobName": "Amazon Transcribe Job",
  "results": {
    "items": [],
    "transcripts": 
[
      {
        "transcript": "I'm interested in your financial services. 
Can you provide any references from satisfied  customers? 
Can you provide some realworld examples of how your financial services have made a  difference?."
      }
  ]
  },
  "status": "COMPLETED"
}
Gong data for prospect #20

This data is from the Gong API:

{
  "callTranscripts": [
    {
      "callId": "7782342274025937895",
      "transcript": [
        {
          "sentences": [
            {
              "end": 462343,
              "start": 460230,
              "text": "I'm interested in learning more about personal experiences with your company. Can you share any success stories from people who've used your financial services?"
            }
          ],
          "speakerId": "Prospect",
          "topic": "Conversation"
        },
        {
          "sentences": [
            {
              "end": 462343,
              "start": 460230,
              "text": "Do you have any recommendations from customers in the manufacturing industry?"
            }
          ],
          "speakerId": "Prospect",
          "topic": "Conversation"
        },
        {
          "sentences": [
            {
              "end": 462343,
              "start": 460230,
              "text": "What kind of results have people seen in the past?"
            }
          ],
          "speakerId": "Prospect",
          "topic": "Conversation"
        }
      ]
    }
  ],
  "records": {
    "currentPageNumber": 0,
    "currentPageSize": 100,
    "cursor": "eyJhbGciOiJIUzI1NiJ9.eyJjYWxsSWQiM1M30.6qKwpOcvnuweTZmFRzYdtjs_YwJphJU4QIwWFM",
    "totalRecords": 263
  },
  "requestId": "4al018gzaztcr8nbukw"
}

Conclusion

If you study the example above and focus especially on the text rather than all the semi-structured wrappers, you'll probably work out fairly quickly that this prospect is interested in hearing testimonials from prior customers in a similar situation. Although - interestingly - without ever actually using the word itself.

It's fine to read one or two manually! But to make this kind of solution operational, you're going to have to deal with thousands of cases, each probably far more complex than the simple example I put together.

The key to making this work at scale and in a maintainable way is to introduce an augmented data engineering platform that can:

  • Transform and integrate data from diverse sources, using modeling techniques such as data vault in a medallion data architecture
  • Manage the intricacies and coordination involved in transferring data between a database and a Large Language Model

The Matillion Data Productivity Cloud allows you to easily develop and deploy this combination of an integrated data model - such as Data Vault - and an LLM for unstructured data analysis. Together the solution presents a formidable approach to optimizing sales processes. Using this approach, businesses can ensure that they are not merely collecting data, but are also effectively interpreting and utilizing it to drive sales and improve customer relationships.

For a more technical view, follow this link to see how this sales sentiment analysis example looks when deployed in the Matillion Data Productivity Cloud.

Ian Funnell
Ian Funnell

Data Alchemist

Ian Funnell, Data Alchemist at Matillion, curates The Data Geek weekly newsletter and manages the Matillion Exchange.