Go back

Building sentiment analysis in Customer Service

To kick off the new year at Cleverly, we decided to set aside one day per month to explore new things. It could be anything from developing internal tools or researching novel technology, to letting our craziest ideas run loose. So, on the 22nd of January, we held our first Lab Days session, during which we worked in dedicated groups looking into different topics. One of those topics was sentiment analysis, and it’s one we’ve been dwelling on ever since, because it poses a lot of questions, for example:

  • Are there only positive and negative sentiments towards a business or product, or do they need to be more granular like “frustrated”, “angry”, “tired”, in order to be properly classified?
  • What about straight-to-the-point messages (eg. “Thanks for checking”); are they neutral or does gratitude count as positive?
  • If a customer feels the need to reach out to support, doesn’t that imply they are having a problem?
  • And if so, then how can their message be anything other than negative?

While we still don’t have all the answers here, we do have some thoughts on sentiment analysis we’d like to share with you. But first things first.

What is sentiment analysis and why should you care about it?

Simply put, sentiment analysis is the field within artificial intelligence that, using algorithms, determines if a written message has a positive, a negative, or a neutral tone. Applied to customer service, it allows support teams to assess how customers are feeling based on their emails, reviews, social media posts, and more. 

Businesses can use sentiment analysis to evaluate customers’ reviews, which in turn allows the support team to proactively reach out to customers who’ve had a bad experience and try to turn the situation around. Not all customers reach out to the company with a bad review, leaving it instead on a reviews website or on social media. With sentiment analysis in place, a company can much more effectively scan all that text available online, sort it and have support agents act accordingly. 

Sentiment analysis can also work as a display of customers’ emotions in an email thread, direct message, or chat. Displaying the sentiment of a message can help support agents in determining how to more accurately reply if the tone of a message is positive or negative. It can also be used to sort incoming requests by urgency based on the tone of the customer’s message, so that agents know first-hand which customers’ are angry and handle their requests first, without the need to manually going through them to assess their priority. 

As for managers, sentiment analysis can be a powerful tool to measure customers’ satisfaction with the support team. Since it can look into data all over the internet, not just e-mails and requests interaction feedback, sentiment analysis can help paint a clearer picture of how customers really feel towards a team, company, or brand. 

Sentiment analysis challenges

But as useful as sentiment analysis might be, it is a challenging field to explore and apply. It is, after all, based on language, which is not as straightforward as a list of positive and negative words. In fact, it can be so ambiguous to the extent that a word can be used to give off the exact opposite of what it usually means. 

Language ambiguity

If we think about customer service, after being informed about a delayed delivery or denied a refund, a customer could say “That’s great!” Great is usually, well, great, but in this case, it has a negative connotation as it is being used as sarcasm. 

Some companies are already training their algorithms to recognize this literary device, but that still isn’t enough to bypass language’s ambiguous traits. There are other factors like context, slang words, or even certain languages that are so complex that make it difficult for machines to correctly assess the tone of what is being said. 

At Cleverly, we’re in a phase where we are annotating and labeling samples of customer messages to then teach our models. First, we defined five levels of sentiment: very negative, negative, neutral, positive, and very positive. Then, we established the differences between each of these sentiments. In addition to the obvious distinction between negative, positive and neutral, we established that a very negative message implied the use of, for example, very strong negative words, including swear words, and capitalized text, while a negative one used negative words and expressed dissatisfaction, but in a more constrained manner. 

However, as we began annotating our sample, we quickly noticed that, even after coming up with these parameters, the same message could sound very negative to some of us and just negative for others. There were even messages that we labeled as negative, positive, AND neutral. It isn’t just the fact that language is ambiguous on its own, it’s also that we can interpret it in very different ways. And if that is the case, how can we train algorithms in a way that is unbiased.

Sentiment gray areas

As we were discussing how positive or negative certain samples were, another topic came up: how do we label the ones that have a mixed sentiment? That is to say, messages that begin with a negative tone and end with a positive one, and vice versa. For example: “Hi. I ordered a book from your website and was really happy with how fast it was delivered! However, it came damaged and I’m a bit disappointed that you didn’t pack it more carefully.” 

In this case, the customer was first happy about the book — positive — but then showed disappointment in the damage and the carelessness of the company — negative. Looking at just this message it is hard to decide if it should be labeled as positive or negative, since both sentiments are present. Or do they cancel each other out and the message becomes neutral? Since we work with machine learning-based sentiment analysis — training a model to recognize the tone, as opposed to other methods like rule-based sentiment analysis — we need to find the most accurate label for these sorts of messages (this is a real example we labeled), so that the algorithm can learn how to handle them properly. 

We found that, when faced with mixed sentiment messages, it’s important to look at the context of the whole conversation, and not just at the specific message. Let’s look at another example we struggled with while labeling: “Hi there, I understand there are delays at the minute, but it has been almost a week over the estimated delivery now, just wondering if you have any info on my order.” The customer doesn’t use any explicitly negative words, but the beginning of the message could indicate that this isn’t the first time they are contacting support to get information about the item they purchased. By pulling up the e-mails that came before, and eventually after, we could get a better idea of the sentiment of the customer in regards to this delay and having to contact support. 

All aboard the sentiment train

Despite its complexity, sentiment analysis is a useful and important tool for monitoring, as well as improving, customer experience. It is not for nothing that we are investing in it, after all. 

Although this technology still needs a lot of improvement, it’s one that plays in a constantly innovative field and sees new applications to it as time goes by. More and more businesses — for example, e-commerce ones that have seen a surge in growth recently — are becoming more aware of the value in tracking customers’ sentiments to identify pain points and improve customer service. 

Being able to assess customers’ sentiments has never felt more important than today, and recognizing emotions early on in the support process — especially negative ones — can really boost support efficiency and provide a competitive advantage. Not just that, but with technology and AI being ever so present in nearly all aspects of our lives, addressing feelings is a way to act more human in customer service. 

At Cleverly, we will be rolling out sentiment analysis as an added layer to our product very soon. If you are interested in being part of our early beta, please sign up here