- Blog
- 06.24.2024
- Data Fundamentals, Product
Sentiment Analysis in Databricks with Titan Text Express using Amazon Bedrock
Sentiment analysis is a powerful technique that allows organizations to gain valuable insights from unstructured data sources, such as product reviews, social media posts, and customer feedback. This article will guide you through various approaches to performing sentiment analysis in Databricks, starting with Python-based methods.
We will explore Titan Text Express, a cutting-edge natural language processing (NLP) library offered through Amazon Bedrock. Before diving into the implementation details, we will provide an overview of sentiment analysis, explaining its significance and the underlying principles that drive this analytical approach.
What is Sentiment Analysis?
Sentiment Analysis is a technique that extracts a numeric sentiment score, typically on a scale from negative to positive, from unstructured text data. This process allows organizations to quantify emotions and opinions in text, such as reviews or social media posts, to drive actionable insights.
Large Language Models (LLMs) play a pivotal role in performing Sentiment Analysis. These models, trained on vast amounts of text data, understand context, idioms, and subtleties in human language, enabling them to provide accurate sentiment scores. LLMs like Titan Text Express use transformer architectures to capture semantic nuances, making them effective at discerning sentiment from complex and varied text inputs.
Data engineers are crucial in this process as they ensure reliable data preparation. This involves cleaning, normalizing, and structuring raw text data before feeding it into the LLM. They must interface seamlessly between the database and the LLM, ensuring efficient data pipeline flows and maintaining data integrity to facilitate precise sentiment evaluations. This rigorous data management underpins the success of sentiment analysis endeavors.
Business examples of Sentiment Analysis:
- Brand monitoring: Analyzing customer reviews, social media mentions, and feedback to gauge brand perception.
- Customer support: Prioritizing and routing support inquiries based on sentiment scores.
- Market research: Evaluating public opinion on products, services, or industry trends.
What is Titan Text Express?
Amazon's Titan Text Express is a machine learning-based natural language generation (NLG) tool designed to create human-like text from input data. Leveraging advanced algorithms, it can generate coherent and contextually relevant content, making it useful for tasks like drafting emails, summarizing documents, and composing articles.
Pros:
- High-quality text generation.
- Scalable and adaptable to various domains.
- Reduces manual content creation time.
Cons:
- May generate contextually inaccurate content.
- Requires post-editing for high-stakes materials.
- Dependency on quality of input data.
Ideal Use Cases:
- Automating customer service interactions.
- Generating digital content for social media.
- Summarizing research papers or lengthy documents.
- Assisting in creative writing tasks.
How to perform Sentiment Analysis in Databricks with Titan Text Express using Python with the Amazon Bedrock SDK
Prerequisites for the boto3 Amazon Bedrock Python SDK
Start by installing the prerequisite libraries (note the Databricks SDK for Python is in beta, version 0.28.0, at the time of writing):
python3 -m pip install databricks-sdk boto3
Then load your source data into Databricks.
Python boto3 for Titan Text Express and Databricks SDK
The example below involves product reviews, and assumes that the source data has been loaded into a table named "stg_sample_reviews" with four columns: id (the primary key), stars, product and review.
The Python script is shown below. Note it is good practice to handle credentials more securely than shown in this simple example. You might choose to use a secret management service instead of environment variables or hardcoding.
import os
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.sql import StatementParameterListItem
import logging
import json
import boto3
import botocore
from botocore.exceptions import ClientError
logger = logging.getLogger("demo")
# Use the Amazon Bedrock InvokeModel API
def analyze_sentiment(text):
abc = boto3.client(service_name="bedrock-runtime", region_name="us-east-1")
model_id = "amazon.titan-text-express-v1"
prompt = f"""Your job is to analyze online product reviews.
Provide a numeric rating that reflects the overall sentiment of the review.
The rating should be a single number between 1 and 5, where 1 represents the most negative sentiment and 5 represents the most positive sentiment.
Respond with only your numeric rating. Do not include any justification of the rating. Do not use the word 'rating'.
Review: {text}
"""
body = json.dumps({"inputText": prompt, "textGenerationConfig" : {"topP":0.9, "temperature":0.5}})
response = abc.invoke_model(body=body, modelId=model_id, accept='application/json', contentType='application/json')
response_body = json.loads(response.get('body').read())
return response_body.get('results')[0].get('outputText')
sqlWhId = os.environ["DBKS_SQL_WHID"]
vCatalog = os.environ["DBKS_CATALOG"]
vSchema = os.environ["DBKS_SCHEMA"]
# Connect to workspace
wsclient = WorkspaceClient(
host = os.environ["DBKS_HOSTURL"],
token = os.environ["DBKS_TOKEN"]
)
ddl = wsclient.statement_execution.execute_statement(warehouse_id=sqlWhId,
catalog=vCatalog,
schema=vSchema,
statement=f"CREATE OR REPLACE TABLE `stg_sample_reviews_genai` (`id` INT NOT NULL, `ai_score` VARCHAR(1024) NOT NULL)")
# Fetch rows from the table
xsr = wsclient.statement_execution.execute_statement(warehouse_id=sqlWhId,
catalog=vCatalog,
schema=vSchema,
statement="SELECT `id`, `review` FROM `stg_sample_reviews`")
for r in xsr.result.data_array:
ai_score = analyze_sentiment(r[1])
dml = wsclient.statement_execution.execute_statement(warehouse_id=sqlWhId,
catalog=vCatalog,
schema=vSchema,
statement=f"INSERT INTO `stg_sample_reviews_genai` (`id`, `ai_score`) VALUES (:id, :ai_score)",
parameters=[ StatementParameterListItem.from_dict({"name": "id", "type":"INT", "value": r[0]}),
StatementParameterListItem.from_dict({"name": "ai_score", "type":"INT", "value": ai_score}) ])
After running the above script, you should find a new table has been created, which contains the AI-generated review score for every input record. Join this table to the original on the common id column to compare the AI-generated sentiment scores against the original star review.
The LLM was asked to score between 1 and 5, so you may choose to classify the scores more broadly as follows:
- 4 or 5 - Positive
- 3 - Neutral
- 1 or 2 - Negative
Sentiment Analysis in Databricks using Matillion to run Titan Text Express via Amazon Bedrock
In the Matillion Data Productivity Cloud, orchestration pipelines like the one shown in the screenshot below can:
- Directly extract and load data, or call other pipelines to do so (as shown)
- Invoke Titan Text Express, with a nominated prompt, against all rows from a nominated table
Sentiment Analysis in Databricks using Matillion
Data pipelines such as this manage all the connectivity and plumbing between the Databricks source and target tables, and the LLM.
This allows you to focus on the overall design and architecture, and the data analysis. To compare the AI-generated sentiment scores against the original star review, use a transformation pipeline like the one in the next screenshot.
Checking the results of Sentiment Analysis in Databricks using Matillion
The data sample shows two of the records. In one case the LLM's decision matches the original sentiment identically, but in the other record the ratings differ slightly. This is an example of the subjective nature of sentiment analysis.
Summary
Matillion is a data pipeline platform that enables data teams to build and manage pipelines efficiently for AI and analytics applications at scale. It offers a code-optional UI with pre-built components and supports coding in SQL, Python, and DBT. Matillion integrates with cloud platforms, CDPs, LLMs, and provides Git integration for asynchronous collaboration.
Features include AI-generated documentation, no-code connectors, REST API connectivity, parameterization with variables, data lineage, pushdown ELT, and hybrid SaaS deployment. Matillion incorporates AI components for generative AI prompting, vector store connectivity, reverse ETL of insights, and a natural language copilot for pipeline building. It facilitates augmented data engineering with AI-enhanced workflows and scalable pipeline orchestration.
For more examples of Matillion's AI components in action, check out our library of AI Videos and Demos.
To try Matillion yourself, using your own data, sign up for a free trial.
If you are already a Matillion user or trial customer, you can download the sentiment analysis example shown in the screenshots earlier, and run it on your own platform.
Featured Resources
Big Data London 2025: Key Takeaways and Maia Highlights
There’s no doubt about it – Maia dominated at Big Data London. Over the two-day event, word spread quickly about Maia’s ...
BlogSay Hello to Ask Matillion, Your New AI Assistant for Product Answers
We’re excited to introduce a powerful new addition to the Matillion experience: Ask Matillion.
BlogRethinking Data Pipeline Pricing
Discover how value-based data pipeline pricing improves ROI, controls costs, and scales data processing without billing surprises.
Share: