Chatbot Testing: Features to Check & Quality Tips to Keep In Mind

Chatbots have become a popular way to enhance customer support. There’s no need to go deep into the benefits. Customers don’t have to wait until a human assistant is available to get the answers they need. Meanwhile, companies can increase satisfaction rates, simultaneously reducing costs required for a customer support unit.

But like it always happens with technologies, there’s one “but.” You need to make sure that a chatbot will be a decent alternative for a human assistant – an equivalent for one.

The truth is, a bot can communicate with lots of customers simultaneously, but the value of this communication will depend on a number of tech aspects. In other words, you will need to build a smart chatbot that covers major user requests. And for this, you’ll need a well-thought-out business logic, accurate implementation, and thorough testing.

In this article, I’ll focus on the latter – chatbot testing and quality assurance activities that will bring you closer to releasing a smart and functional chatbot.

Why Testing On Your Own Is Not Enough

The main purpose of a chatbot is to deliver the best user experience and value through conversations. Does it mean that an average user can cover the testing? Unfortunately, no.

On the one hand, you don’t need to have any specific skills to estimate whether a chatbot shows clear answers and offers enough queries. Try out several scenarios and see if the information you get meets your expectations.

On the other hand, the communication flow is one of many aspects that require testing. User interface, response speed, and error handling are among the other things that need a QA inspection. And unlike conversational flow, an average user won’t share insights that would be valuable for quality enhancement.

If you have doubts about the necessity of involving professional QA resources, consider the common chatbot failures. I’ve analyzed some data on this subject to come up with the list of common issues, and it consists of the following points:

• Broken scripts that result in malfunctioning.
• Long pauses before answers, no instant replies.
• No connection to other business channels.
• Too many intents or business tasks for one bot.
• Lack of utterance and/or accuracy.
• No margin for error.
• Poor navigation.
• Poor conversation design.

If a chatbot is not helpful, it just frustrates customers, affecting your brand’s image and business efficiency. Meanwhile, software testing makes it possible to prevent all the mistakes listed above.

Testing a Chatbot: What to Check?

A QA team needs to create a custom testing strategy for a chatbot, just like for any other software product. This article isn’t a definitive guide on how to create such a strategy, but you’ll certainly find some useful hints and tips on what to test and how to improve the quality of an end-product.

Slot Filling – Functional Testing

A chatbot can be effective only in case it understands the context where it operates. To make this happen, developers need to ‘teach’ an application different word categories and specific terms.

Let’s say, we have a coffee shop chatbot that lets users place orders. You can list options and make them clickable or teach the bot to recognize orders. In the first case, it may be easier for developers to program the functionality, but it may be less convenient for users. Take a look at an example of such flow:

This flow can vary, and you may need to add other stages – hot, cold & iced; with tea, alcohol, or flavor; and so on. The logic will get more complicated as you add an opportunity to order coffee beans, deserts, or other items a customer can normally find in a coffee shop.

The other scenario is more complicated. It deals with direct user requests and extracts the necessary information. For example, a user sees a “Hi! What would you like to order?” message and types the reply. Let it be “A cappuccino with one sugar and a no sugar cold brew for Ben.”

There are several entities here a chatbot should understand:

• Type of coffee – cappuccino, cold brew.
• Sugar options – yes, no.
• A number of sugars – one, zero.
• Customer’s name – Ben.

Developers insert these data at the early stages and build the chatbot logic around it. Reading and extracting the information by a functioning bot is also known as slot filling.

Testing

To find out if the bot is familiar with core entities, a QA engineer will need to run domain-specific tests. It is essential to verify that the chatbot understands all the coffee-related lingo customers may use.

Quality Tips

Make sure you’ve added a sufficient number of entities and values for each so that the chatbot can understand a request correctly. In a perfect case, you will need to cover both scenarios – one for those who already know and one for those who are choosing what to order.

Validating Inputs – Some More Functional Testing

If a chatbot accepts inputs like email addresses, phone numbers, and zip codes, it is critical for the system to recognize a correct data format.

Testing

Feed the chatbot email addresses, phone numbers, and zip codes in different incorrect formats, for example:

• an email address without @ symbol;
• an email address with spaces;
• incorrect number of digits in a zip code;
• letters in a mobile number;
• and other similar options.

Thus, you will learn how the system accepts invalid data and suggest improvements if the response is not user friendly. Here’s an example of such a conversation:

Quality Tips

Data validation should be immediate and followed by a corresponding message. The message can show an example of a correct email address, mobile number, etc., or just notify about the mistake. If the input is valid, a user should see a ‘success message’ – proof that the system has received your data and recognized it as valid.

Unknown Inputs – Error Handling & Limit Value Testing

A team cannot foresee all the possible user inputs, especially if there is an input option that allows typing a request. Therefore, it is necessary to investigate the boundaries of the programmed logic. The other task is to make sure the bot has decent responses to the inputs it doesn’t understand.

Testing

Run exploratory tests to learn how a chatbot handles non-standard inputs. Such inputs can include:

• Casual conversations that are unrelated to the chatbot’s purpose.
• Meaningless sentences or expressions that aren’t commonly used.
• Grammar errors, misspelled words, slang, different variants of English.
• Confusing and offensive messages.

Quality Tips

Customer creativity has no boundaries, especially when they want to mess with a chatbot. You cannot cover all possible scenarios and messages. There are several things to do:

• Add variants of common keywords with typos to the chatbot’s lingo. If the bot isn’t able to understand slight query alterings, developers need to teach it to do. For example:



• Work on small talk to support the basic casual conversation. It will give a human touch, making a conversation casual and more fun. These queries catch users’ attention and have a positive effect on user experience. Here are some examples:





• Come up with replies to offensive scenarios. You will need to create some extra slots for strong language for a chatbot to provide an ingenious response.



• Write an emergency reply for unmatched queries. While the points above are rather recommendations, this one is a must. You can use the “Sorry, I didn’t get that” response for any query that doesn’t match logic completely.
• Avoid cyclic loops. When a chatbot doesn’t receive an appropriate query or answer to a question, it should ask to specify the request or suggest more options. Make sure there is an end to the conversation, and the program doesn’t show the same response forever on, like here:

  

  Instead, program a chatbot to finish a conversation or connect a human expert after three or five “I didn’t get that” requests in a row.

A chatbot is a business tool, not a toy users can test for smart responses. And nevertheless, humanization features have a positive effect on brand image and user experience.

When it comes to swearing, however, it would be reasonable to keep up with the brand’s tone of voice. Make some jokes and respond to offensive queries if you can. Otherwise, just quit strategically. The only thing that is common regardless of the communication strategy is to never swear back. It isn’t very polite.

Interface & Graphical Content – UI Testing

Chatbots often show buttons with response options, provide links to a brand’s website or other resources, have cards with embedded information or other images. All these elements should be functional – links lead to the correct sources, images optimized and displayed correctly, all the chains of responses connected so that a conversation doesn’t stop abruptly.

Testing

It is necessary to check the entire flow looking for broken links, graphic elements, and gaps in logic.

Quality Tips

Provide an opportunity for users to leave feedback. Thus, they will report if they find some defects you might have missed. 100% test coverage is a myth, and leaving a couple of little things unnoticed is totally okay.

Cross-Device Support – Compatibility Testing

One more essential thing is to ensure that a chatbot looks properly on different devices. It is a common practice to integrate a bot into a messenger, but that’s not the only option. While messenger design has been already customized for a variety of devices, connecting a chatbot to a website may not be as seamless.

Testing

Check how a chatbot window looks on different devices, browsers, and OS versions. Resize a browser window on the computer screen to make sure that the graphics appear undistorted and the layout isn’t broken. Texts should remain readable and not overflow out of the boundaries.

Quality Tips

Check the chatbot on widely used devices, browsers, and OS versions before the release. Adjust the chatbot window style and parameters whenever necessary.

Conversational Flow – UX Testing

It is one of the most obvious aspects of a chatbot to check. UX testing will be based on both business requirements and users’ personal experience. It is essential to make sure that a chatbot fulfills its core purpose, generating correct and accurate responses.

Testing

Based on the business requirements, create the scenarios for user queries. The main idea is to check these queries from a point of view of a user who sees your chatbot for the first time. So, start by answering the following questions:

• Does the chatbot understand the questions?
• Does it provide instant responses to those queries?
• Is the information relevant, accurate, and useful to a user?
• Does a user journey require more or fewer steps to finish?
• Is it engaging for users? Do they tend to abandon a chatbot?
• How can we make the conversation more engaging?

Quality Tips

The main rule is to keep a conversation as normal as possible. So here are some basic rules that will help you to achieve this purpose.

#1. A conversation with a chatbot should have a logical flow, much like a real-life conversation with a human. For example, start with a greeting and follow up with a question to start the conversation:

• How can I help you?
• What are you looking for?
• Are you interested in…?
• … or something of a kind.

#2. Pay attention to input variations. If you have a text field where users can type their queries, make sure the chatbot understands different requests. Let’s say, you want to order an item from a stationery shop. There are several options to start a conversation:

• I would like to order a notebook.
• I need a notebook.
• What notebooks do you have?
• May I have a dotted lavender 5×3.5-inch notebook?

It may be reasonable to provide a standard set of answers, it depends on the purpose of your chatbot and the variety of items/services available.

#3. Every response should be logically connected to the previous query or question. It is critical that the chatbot provides only relevant and required information. Notify a customer about errors in their queries instantly. Confirm the success message in case the chatbot requests to enter contact data.

#4. There should always be a final message signaling that the conversation is over. It can provide links to some helpful resources, guidelines for further communication, or a goodbye phrase.

#5. Consider an opportunity to connect a user to a human assistant. Whether you want it or not, there are some questions that bots cannot handle. And just like that, provide an opportunity to come back to the beginning or retrieve to the previous step. A chatbot requires clear navigation, just like any other online service.

Response Speed – Performance Testing

People using chatbots expect to get accurate responses to their requests and to get them instantly. It means that chatbot performance testing is essential. Therefore, latency – the time a chatbot needs to retrieve the response and display it – should be minimum.

Testing

A QA engineer should check different queries that cover small and large dialogues and feature different types of content. It should take the same time for each response to appear on the screen, regardless of its place in a user journey and featured content.

Quality Tips

The delay that lasts more than a couple of seconds causes frustration, and a user is likely to abandon a chatbot. The delay may depend on the response’s content, but the recommended response time is up to two seconds.

Connecting a Chatbot – Integration/API Testing

As a rule, a chatbot doesn’t function on its own – it becomes an integral part of the company’s digital ecosystem. That is, you’ll eventually need to connect it to the official website, a social network account, messenger, application, or another digital medium.

Testing

If you use bot building software, you are likely to find an API that allows connecting a chatbot to your system and run API tests. If you use a custom solution, it is essential to run integration tests to check how a bot and other system features behave when connected.

Quality Tips

Pay attention to both tech and design aspects of your chatbot. It is a part of your digital system, and it should integrate seamlessly. All the functional and performance aspects mentioned above should meet the quality standards after a chatbot is connected to the website or other platform. The design should be consistent with other media and brand image in general. The latter aspect falls under UX testing, but the inconsistency may become apparent at this stage.

Literacy and Multilingual Answers – Localization Testing

Last but not least, avoid errors in responses – spelling, grammatical, lexical, any errors! In most cases, they won’t have a catastrophic effect, but attentive users can get skeptical about the level of services you deliver. Long story short, little things matter, and that’s why texts should be error-free.

If a website is multilingual, it makes sense to create a chatbot that is multilingual as well. Keep in mind that localization encompasses other aspects as well – time zones, currencies, metric systems, and more.

Testing

Entrust localization testing to someone good at editing texts. If a QA specialist isn’t confident in their proofreading skills, assign this task to another person on the team. Besides, a fresh perspective is always helpful.

So, check the responses for spelling, lexical, and grammatical errors, punctuation, and cohesion. Make sure the formatting is correct, too. Whenever you have things listed, use bullets or numbers. Keep sentences short and divide information into paragraphs in case a message is long.

Quality Tips

It would be great for a chatbot to recognize the user’s time zone and greet them accordingly. As for the language, the system should be able to access the corresponding device settings or start the conversation flow by specifying these details. If currency and other localization details are important, the chatbot should specify those via questions, too.

To Sum Up

Testing a chatbot is not an easy task, and it requires the participation of a professional QA company. Besides, the work on a software solution doesn’t end with its release – a chatbot will require maintenance and, most likely, improvements. After the launch, a company starts to receive user feedback that always comes with useful insights for quality enhancement.

Don’t forget that a chatbot can learn from interactions with users. A detailed analysis of user requests, bot responses, missing queries, etc. helps to gather data you can use for enhancing user experience. And each new bot version will require a set of tests to verify it works just fine 😉