- Blog
- 09.03.2024
- Leveraging AI, Product, Data Fundamentals
Sentiment Analysis in Redshift with OpenAI GPT-4o
This article aims to explore effective techniques for conducting Sentiment Analysis on Amazon Redshift data using OpenAI's GPT-4o. It will demonstrate a variety of methods, including how to implement sentiment analysis using Python, making it accessible for both novice and experienced data engineers.
Additionally, the article will provide a comprehensive overview of OpenAI GPT-4o, discussing its capabilities and integration. The journey will begin with a foundational explanation of Sentiment Analysis, setting the stage for deeper technical insights and practical applications within a Redshift environment.
What is Sentiment Analysis?
Sentiment Analysis, leveraging large language models (LLMs), has become a cornerstone for deriving nuanced insights from textual data. These advanced models, trained on vast datasets, can discern complex emotional undertones in human-generated content, offering precise sentiment classification beyond rudimentary positive, negative, or neutral labels. LLMs can dynamically contextualize phrases, accounting for subtleties like sarcasm or idiomatic expressions. This results in a robust analysis pipeline that is invaluable for various applications within a data ecosystem, such as:
- Social Media Monitoring: Analyzing real-time Twitter feeds to gauge public sentiment about a brand or product, including trends in consumer satisfaction or emerging complaints.
- Customer Feedback Evaluation: Processing extensive user reviews from e-commerce platforms to extract sentiments concerning specific product features, enabling focused improvements.
- Financial Market Prediction: Assessing sentiment from news articles and financial reports to predict stock market movements or economic indicators based on public and expert opinions.
What is OpenAI GPT-4o?
OpenAI GPT-4o offers data engineers and data architects a powerful tool for natural language processing tasks, capable of parsing and generating human-like text with high accuracy. Leveraging a transformer-based architecture, GPT-4o excels in contextual understanding and response generation, making it valuable for automating ETL documentation, generating complex SQL queries, and even designing schema recommendations based on input requirements. Its integration capabilities with existing data pipelines facilitate real-time data analysis and insights extraction, optimizing the overall data workflow.
GPT-4o excels in generating human-like text, making it a powerful tool for a wide range of applications including customer service, content creation, and personalized marketing. Its advanced language modeling capabilities can streamline workflows, enhance user engagement, and automate repetitive tasks with high accuracy. However, GPT-4o is not without its drawbacks: it can sometimes produce contextually irrelevant or biased responses, and its reliance on vast amounts of training data raises privacy and ethical concerns. Additionally, the computational resources required to run GPT-4o can be significant, posing a challenge for small-scale implementations. The best use cases for GPT-4o are scenarios where natural language understanding and generation are paramount, but the potential risks and costs are manageable, such as virtual assistants, automated report generation, and interactive educational tools.
How to perform Sentiment Analysis in Redshift with OpenAI GPT-4o using Python
Start by installing the prerequisite libraries:
python3 -m pip install psycopg2-binary openai
Then load your source data into Redshift. The example below involves product reviews, and the data has been loaded into a table named "stg_sample_reviews" with four columns: id (the primary key), stars, product and review.
Here is the Python script. Note it is good practice to handle credentials more securely than shown in this simple example. You might choose to use environment variables or a secret management service instead of hardcoding them.
Also please note that handling large amounts of data using fetchall() can be inefficient, and may result in memory issues. For large datasets, you should use a cursor to fetch rows incrementally using the fetchmany() method instead.
If you are working in a schema other than "public" you will need the -c connection option to specify the object search path. It is named yourSchemaName in the code sample. Newly created objects will be added to this named schema.
To authenticate with OpenAI you will need to set up an environment variable named OPENAI_API_KEY containing your API key.
import psycopg2
from openai import OpenAI
modelname = "gpt-4o"
systemPrompt = "Your job is to analyze online product reviews"
# Function that will be called for every input row
def process_row(row):
print(f"Processing id: {row[0]}")
userPrompt = f"""Provide a numeric rating that reflects the overall sentiment of the review.
The rating should be a single number between 1 and 5, where 1 represents the most negative sentiment and 5 represents the most positive sentiment.
Respond with the numeric rating only. Do not include any justification of the rating.
review: {row[1]}"""
ccresp = oaisdk.chat.completions.create(
model=modelname,
temperature=1,
messages=[ {"role": "system", "content": systemPrompt},
{"role": "user", "content": userPrompt} ])
msg = ccresp.choices[0].message.content.strip()
return(msg)
# Database connection parameters
db_params = {
'dbname': ...,
'user': ...,
'password': ...,
'host': ...,
'port': '5439',
'options' : '-c search_path=<<yourSchemaName>>,public'
}
# Create a connection to the Redshift database
try:
conn = psycopg2.connect(**db_params)
except Exception as e:
print(f"Unable to connect to the database: {e}")
exit(1)
conn.autocommit = True
# Establish an OpenAI connection using environment variable OPENAI_API_KEY
oaisdk = OpenAI()
# Create a cursor object
cur = conn.cursor()
# Your query to fetch rows from the table
query = f'SELECT "id", "review" FROM "stg_sample_reviews"'
try:
# Create the table to hold the results, if it does not already exist
cur.execute(f'CREATE TABLE IF NOT EXISTS "stg_sample_reviews_{modelname}" ("id" INT NOT NULL, "ai_score" VARCHAR(1024) NOT NULL)')
cur.execute(f'DELETE FROM "stg_sample_reviews_{modelname}"')
# Execute the query
cur.execute(query)
# Fetch and process each row
for row in cur.fetchall():
ai_score = process_row(row)
cur.execute(f'INSERT INTO "stg_sample_reviews_{modelname}" ("id", "ai_score") VALUES ({row[0]}, {ai_score})')
except Exception as e:
print(f"SQL error: {e}")
finally:
cur.close()
conn.close()
After running the above script, you should find a new table has been created, which contains the AI-generated review score for every input record. Join this table to the original on the common "id" column to compare the AI-generated sentiment scores against the original star review.
The LLM was asked to score between 1 and 5, so you may choose to classify the scores more broadly as follows:
- 4 or 5 - Positive
- 3 - Neutral
- 1 or 2 - Negative
Sentiment Analysis in Redshift using Matillion
In the Matillion Data Productivity Cloud, orchestration pipelines like the one shown in the screenshot below can:
- Directly extract and load data, or call other pipelines to do so (as shown)
- Invoke OpenAI GPT-4o, with a nominated prompt, against all rows from a nominated table
Performing Sentiment Analysis in Redshift using Matillion
Data pipelines such as this manage all the connectivity and plumbing between the Databricks source and target tables, and the LLM.
This allows you to focus on the overall design and architecture, and the data analysis. To compare the AI-generated sentiment scores against the original star review, use a transformation pipeline like the one in the next screenshot.
Checking the results of Sentiment Analysis in Redshift using Matillion
The data sample shows two of the records. In one case the LLM's decision matches the original sentiment identically, but in the other record the ratings differ slightly. This is an example of the subjective nature of sentiment analysis.
Summary
Matillion is a data pipeline platform designed to expedite pipeline construction for AI and analytics at scale, fostering productivity, collaboration, and speed with its code-optional interface. The platform integrates seamlessly with hyperscalers, CDPs, and LLMs, democratizing AI access. It offers a UI with pre-built no-code components, alongside coding options in SQL, Python, and DBT. Matillion features robust Git integration for asynchronous collaboration, AI-generated documentation, and versatile connector options, including REST APIs. Its pushdown ELT architecture, hybrid SaaS deployment, and data lineage support cater to complex pipeline orchestration. The platform also includes no-code Generative AI and vector store connectivity, enabling easy reverse ETL of AI insights.
For more examples of Matillion's AI components in action, check out our library of AI Videos and Demos.
To try Matillion yourself, using your own data, sign up for a free trial.
If you are already a Matillion user or trial customer, you can download the sentiment analysis example shown in the screenshots earlier, and run it on your own platform.
Ian Funnell
Data Alchemist
Ian Funnell, Data Alchemist at Matillion, curates The Data Geek weekly newsletter and manages the Matillion Exchange.
Follow Ian on LinkedIn: https://www.linkedin.com/in/ianfunnell
Featured Resources
ETL and SQL: How They Work Together in Modern Data Integration
Explore how SQL and ETL power modern data workflows, when to use SQL scripts vs ETL tools, and how Matillion blends automation ...
WhitepapersUnlocking Data Productivity: A DataOps Guide for High-performance Data Teams
Download the DataOps White Paper today and start building data pipelines that are scalable, reliable, and built for success.
BlogWebhooks and Pushdown Python: Building Interactive and Efficient Data Applications
Part 5 of our blog series demonstrating the art of the possible, using Matillion products and features to build the MatiHelper ...
Share: