- Blog
- 09.11.2024
- Data Fundamentals, Product, Leveraging AI
Sentiment Analysis in Databricks with OpenAI GPT-3.5 Turbo
In this article, we will explore how to perform Sentiment Analysis using Databricks with OpenAI's GPT-3.5 Turbo. We will demonstrate several techniques, including implementing these processes with Python, to help you efficiently analyze textual data.
Additionally, the article will provide insights into OpenAI GPT-3.5 Turbo's capabilities, beginning with a thorough explanation of what Sentiment Analysis is and how it can be applied to various datasets.
What is Sentiment Analysis?
Sentiment Analysis, also known as Opinion Mining, is a subfield of Natural Language Processing (NLP) that involves determining and extracting subjective information from text data. It typically employs large language models to classify text into positive, negative, or neutral sentiments by analyzing features such as words, phrases, and context.
This analysis helps understanding user opinions and feedback, making it essential for product reviews, social media monitoring, and customer service. The architecture for sentiment analysis can be enhanced by integrating cloud databases and AI models to scale efficiently and handle real-time data streams.
Examples of Sentiment Analysis applications include:
- Customer Feedback Analysis: Aggregating and interpreting customer reviews from multiple platforms to assess product or service satisfaction.
- Social Media Monitoring: Analyzing tweets, comments, and posts to gauge public sentiment about a brand, event, or policy.
- Financial Market Prediction: Evaluating news articles and financial reports to predict market trends based on public sentiment towards specific stocks or sectors.
What is OpenAI GPT-3.5 Turbo?
When it was launched, OpenAI's GPT-3.5 Turbo represented a significant leap in generative AI, offering improved performance and more efficient resource utilization compared to its predecessors. This model excels at natural language understanding and generation, benefiting from extensive pre-training on diverse datasets. For data engineers and data architects, GPT-3.5 Turbo's capabilities can enhance ETL pipeline documentation, automate data quality insights, and facilitate conversational interfaces for querying databases. Its advanced token efficiency reduces computational overhead, making integration with existing cloud architecture more scalable.
GPT-3.5 Turbo offers significant benefits such as high-quality text generation, fast processing speed, and cost-efficiency, making it a powerful tool for various applications like content creation, customer service, and data analysis. Its ability to understand and generate human-like text based on diverse prompts allows for nuanced and contextually relevant outputs. However, some drawbacks include potential biases in the generated content, occasional inaccuracies, and the risk of over-reliance on AI for tasks requiring nuanced human judgment. GPT-3.5 Turbo is best used in scenarios where rapid, coherent text generation is needed, such as drafting emails, generating code snippets, or brainstorming ideas, but it should be complemented with human oversight to ensure the quality and appropriateness of the final output.
How to perform Sentiment Analysis in Databricks with OpenAI GPT-3.5 Turbo using Python
Start by installing the prerequisite libraries (note the Databricks SDK for Python is in beta, version 0.28.0, at the time of writing):
python3 -m pip install databricks-sdk openai
Then load your source data into Databricks. The example below involves product reviews, and the data has already been loaded into a managed table named "stg_sample_reviews" with four columns: id (the primary key), stars, product and review.
Here is the Python script. Note it is good practice to handle your host and token credentials more securely than shown in this simple example. You might choose to use environment variables or a secret management service instead of hardcoding them.
Set the following values to those in your own Databricks workspace:
- sqlWhId - The ID of your SQL Warehouse
- vCatalog - Your Unity Catalog name
- vSchema - Your Unity Catalog Schema name (a.k.a. Database name)
To authenticate with OpenAI you will need to set up an environment variable named OPENAI_API_KEY containing your API key.
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.sql import StatementParameterListItem
from openai import OpenAI
sqlWhId = "xxxxxxxxxxxxxxxx"
vCatalog = "your-catalog-name"
vSchema = "your-schema"
modelname = "gpt-3.5-turbo"
tablename = "gpt_3_5_turbo"
systemPrompt = "Your job is to analyze online product reviews"
# Function that will be called for every input row
def process_row(resp):
userPrompt = f"""Provide a numeric rating that reflects the overall sentiment of the review.
The rating should be a single number between 1 and 5, where 1 represents the most negative sentiment and 5 represents the most positive sentiment.
Respond with the numeric rating only. Do not include any justification of the rating.
review: {resp}"""
ccresp = oaisdk.chat.completions.create(
model=modelname,
temperature=1,
messages=[ {"role": "system", "content": systemPrompt},
{"role": "user", "content": userPrompt} ])
msg = ccresp.choices[0].message.content.strip()
return(msg)
# Establish an OpenAI connection using environment variable OPENAI_API_KEY
oaisdk = OpenAI()
# Connect to workspace
wsclient = WorkspaceClient(
host = 'https://xxxxxxxxx.cloud.databricks.com',
token = 'dapixxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
)
ddl = wsclient.statement_execution.execute_statement(warehouse_id=sqlWhId,
catalog=vCatalog,
schema=vSchema,
statement=f"CREATE OR REPLACE TABLE `stg_sample_reviews_{tablename}` (`id` INT NOT NULL, `ai_score` VARCHAR(1024) NOT NULL)")
print(ddl.status)
# Fetch rows from the table
xsr = wsclient.statement_execution.execute_statement(warehouse_id=sqlWhId,
catalog=vCatalog,
schema=vSchema,
statement="SELECT `id`, `review` FROM `stg_sample_reviews`")
print(xsr.status)
for r in xsr.result.data_array:
ai_score = process_row(r[1])
print(f"ID {r[0]}: Score: {ai_score}")
dml = wsclient.statement_execution.execute_statement(warehouse_id=sqlWhId,
catalog=vCatalog,
schema=vSchema,
statement=f"INSERT INTO `stg_sample_reviews_{tablename}` (`id`, `ai_score`) VALUES (:id, :ai_score)",
parameters=[ StatementParameterListItem.from_dict({"name": "id", "type":"INT", "value": r[0]}),
StatementParameterListItem.from_dict({"name": "ai_score", "type":"INT", "value": ai_score}) ])
After running the above script, you should find a new table has been created, which contains the AI-generated review score for every input record. Join this table to the original on the common "id" column to compare the AI-generated sentiment scores against the original star review.
The LLM was asked to score between 1 and 5, so you may choose to classify the scores more broadly as follows:
- 4 or 5 - Positive
- 3 - Neutral
- 1 or 2 - Negative
Sentiment Analysis in Databricks using Matillion
In the Matillion Data Productivity Cloud, orchestration pipelines like the one shown in the screenshot below can:
- Directly extract and load data, or call other pipelines to do so (as shown)
- Invoke OpenAI GPT-3.5 Turbo, with a nominated prompt, against all rows from a nominated table
Performing Sentiment Analysis in Databricks using Matillion
Data pipelines such as this manage all the connectivity and plumbing between the Databricks source and target tables, and the LLM.
This allows you to focus on the overall design and architecture, and the data analysis. To compare the AI-generated sentiment scores against the original star review, use a transformation pipeline like the one in the next screenshot.
Checking the results of Sentiment Analysis in Databricks using Matillion
The data sample shows two of the records. In one case the LLM's decision matches the original sentiment identically, but in the other record the ratings differ slightly. This is an example of the subjective nature of sentiment analysis.
Summary
Matillion is a sophisticated data pipeline platform that empowers data teams to efficiently build and manage pipelines for AI and analytics, handling tasks at scale. The platform boosts productivity with code-optional features and fosters collaboration with first-class Git integration, enabling asynchronous work. Matillion supports hyperscalers, CDPs, LLMs, and vector stores, democratizing AI access with no-code connectors and AI-generated documentation. Its UI offers pre-built components, with the flexibility to code in SQL, Python, or DBT. Matillion excels in pushdown ELT and hybrid SaaS deployments, facilitating data lineage and reverse ETL. Matillion's copilot uses natural language to streamline pipeline creation, making it ideal for augmented data engineering.
For more examples of Matillion's AI components in action, check out our library of AI Videos and Demos.
To try Matillion yourself, using your own data, sign up for a free trial.
If you are already a Matillion user or trial customer, you can download the sentiment analysis example shown in the screenshots earlier, and run it on your own platform.
Ian Funnell
Data Alchemist
Ian Funnell, Data Alchemist at Matillion, curates The Data Geek weekly newsletter and manages the Matillion Exchange.
Follow Ian on LinkedIn: https://www.linkedin.com/in/ianfunnell
Featured Resources
Why 2026 Changes Everything for Enterprise Data and AI
This year marks a turning point. Enterprises are moving from AI experimentation to practical deployment – get the exec ...
BlogThe Future of Data Belongs to the Bold: Why Being a Challenger Matters When Choosing a Data and AI Partner
Matillion has been named a Challenger in the 2025 Gartner® Magic Quadrant™ for Data Integration Tools – recognition that we ...
Data SheetsReady to lead your team into an AI-first future?
95% of generative AI pilots at companies are failing, according to ...
Share: