No time to walk through the tutorial? Test this application here.
During one of Mantium’s Spring, 2022 NLP Lab meetings, we read The Unreliability of Explanations in Few-Shot In-Context Learning by Xi Ye and Greg Durrett. This paper provides a critical investigation of language model prompting methods that use step-by-step explanations (Nye et al., 2021) or so-called chain of thought reasoning (Wei et al., 2022) to improve performance. While standard prompts simply provide examples of desired output, given an input, chain of thought prompts provide additional, step-by-step information that documents how the model should arrive at the expected output (for example, see Figure 1). Interestingly, in some contexts, this kind of prompting does indeed improve model performance. However, Xi and Durrett (2022) found that it doesn’t generalize well across domains. Specifically, they found that explanations generated by GPT-3 were at risk of hallucination when applied to logical reasoning tasks, even relatively simple ones.
However, while GPT-3 might not always generate reliable explanations for some logic based questions, is it possible that it could perform better on other domains of explanation? To answer this question, we’ve designed an explanation-based prompt that focuses on affective reasoning. That is, we ask the model to determine the sentiment of a document and explain how it arrived at that conclusion. And, of course, we used the Mantium platform to engineer our prompt and deploy a web application where people can interact with it!
Conventional sentiment analysis is a natural language processing (NLP) technique that involves determining the overall sentiment that a document expresses. For example, determining the sentiment of customers’ opinions about a particular product or service is a popular application of sentiment analysis. Often, sentiment analysis considers two possible sentiment categories, positive and negative. However, we wanted a more psychologically rich representation of sentiment, so we also included “neutral” and “mixed” in our set of sentiment categories, where “mixed” indicates that a document expresses both positive and negative sentiment.
However, our goal wasn’t just to build another sentiment classifier — instead, we wanted to try to build a sentiment classifier that could explain it’s own predictions! This tutorial will explore how to build a Sentiment Analysis app with in-context explanations without writing a single line of code.
Until recently, getting a model to behave in a certain way required training the model on data, often a lot of it. In-context learning changed everything — rather than training a model and then using it to make predictions, users can simply add instructions or examples to the model at inference time. Providing context in this manner allows users to control the behavior of the model without any training. In-context learning is also often referred to as Prompting, which usually follows one of two styles: Zero-Shot and Few-Shot. Zero-Shot refers to prompt with no examples, while the few-shot prompting style requires n number of examples (n-shot).
Given an input document, the model returns a sentiment classification. In addition, it provides an explanation that aims to provide a high-level summary of why a particular sentiment classification was assigned to the document.
AI Providers, then paste your API key into the OpenAI form.
Click AI Manager > Prompts > Add New Prompt, and fill out the following:
Before filling out the Prompt Text, select the AI Provider settings as shown in the list below.
We have prepared a prompt text sample (see below) for you to paste into the Prompt Text field. So what is a prompt? A prompt is string containing descriptions or examples and patterns of a language task. The prompt is a collection of examples describing the sentiment analysis task for this use case, consisting of original content, an explanation of the content, and sentiment classification. Providing a few of these to the model to get output prediction of input text is called Few-shot learning.
The piece contains a few examples of the sentiment analysis task, including a particular text, explanations, and sentiment classification. This collection of examples constitutes our prompt.
We use the Explain-then-Predict prompting style, which precedes the sentiment label with an explanation that influences the predicted label. This is because we want the prompt to generate explanations and sentiment classification.
Let’s examine one of the examples in our sentiment analysis prompt to understand this better. After the original content, you will notice that the explanation comes before the sentiment label or classification, after which you then have the classification.
I had a terrible day. My dog died, my truck broke down, and I found out my girlfriend is cheating on me.
The author wrote that they had a "terrible day", which explicitly expresses negative sentiment. They also wrote that their dog died, their truck broke down, and that their girlfriend is cheating on them, which are all events that are likely to cause negative sentiment in humans.
Therefore, this text expresses negative sentiment.
This is an explanation-based sentiment classification system. Given a text, it provides an explanation of the sentiment expressed in the text. Then it determines whether the text, overall, expresses positive, negative, neutral, or mixed sentiment. Here are some examples: ### Example 1. I had a terrible day. My dog died, my truck broke down, and I found out my girlfriend is cheating on me. The author wrote that they had a "terrible day", which explicitly expresses negative sentiment. They also wrote that their dog died, their truck broke down, and that their girlfriend is cheating on them, which are all events that are likely to cause negative sentiment in humans. Therefore, this text expresses negative sentiment. ### Example 2. We ate at Alinea last night and...honestly, it left me feeling a bit cold. Sure, it was showy -- technically very impressive, tbh -- and the service was extraordinary. But, something was missing. It was like the food had no soul. Probably won't go back. The author wrote that eating at Alinea left them feeling "cold". However, they also indicated that their experience was "technically very impressive" and that the service was "extraordinary". In the end, though, they asserted that the food "had no soul" and that they would not go back to the restaurant, which strongly indicates that, overall, they did not enjoy their experience.\n\nTherefore, this text expresses negative sentiment. ### Example 3. I saw the film Shoplifters last night with my boyfriend. We ate popcorn and candy while we watched it. The author reports watching a movie with their boyfriend and eating popcorn and candy. They do not provide any indication of whether or not these experiences were positive or negative. Therefore, this text expresses neutral sentiment. ### Example 4. "Shining Girls" is undeniably kooky, but the characters, situations and the city itself are so vividly brought to life that you'll be dying to figure out what happens next. The author claims that while "Shining Girls" is a bit odd, it is so "vividly brought to life" that you'll be "dying to figure out what happens next." In general, when a person is "dying" to do something, it means that it is highly anticipated and positively evaluated. Therefore, this text expresses positive sentiment. ### Example 5. The food was delicious, but the service was terrible. The author wrote that the "food was delicious", but also that the "service was terrible". The former expresses positive sentiment and the latter expresses negative sentiment. It is not clear which of these sentiment expressions takes precedence. Therefore, this text expresses mixed sentiment. ### Example 5. You are an asshole, but I love you. The author claims that the reader is "an asshole", but they also indicate that they love the reader. This implies that even though the reader is an "asshole", they still feel positively toward the reader. Therefore, this text is positive.
Add Stop Sequence to add one)
You can ignore the
Here’s how the settings for your application should look.
Response Length controls the length of an output response. The response length is set to 100 tokens.
Top P is another way to control “creativity” using a different probability method.
Stop sequences are another method of controlling output – they allow you to define any text sequences that force the model to stop. If you don’t have a stop sequence, the model might generate a stream of the requested response length, or stop in the middle of a sentence.
You can test the output of the prompt (see image below). We suggest clicking Test Run multiple times and tweaking the Temperature and Top P values to test output consistency as needed. Once you are satisfied with the results, click Save in the top right corner of the application.
I bought this for my husband who plays the piano. He is having a wonderful time playing these old hymns. The music is at times hard to read because we think the book was published for singing from more than playing from. Great purchase though!
Mantium enables sharing your Prompts and Intelets using the One-Click Deploy feature. From the prompt creation interface we can deploy the application by simply clicking on the Deploy button in the top right corner.
Another way to deploy a prompt is to navigate to the Prompt drawer view. From your list of prompts, click on the Sentiment Analysis + Explanation prompt, and then click Deploy , at the bottom right corner.
After deploying your application, you will get a unique URL that you can share so that others can interact with your prompt.
At Mantium, we are building a community where anyone can easily build AI applications irrespective of their technical background!
We’d love to have you in our growing community on Discord. Click Here to Join
Most recent posts