How To Test A Chatbot

It seems that every website now has a chatbot, so it’s probably only a matter of time before you will be asked to test one. But testing a chatbot is different from testing traditional software, since the chatbot’s answers will vary. When you test a calculator, you can create a test that enters “1 + 1” and you can expect to always get “2” in return. With a chatbot, you could ask “What is 1 + 1?” and receive “2”, or “1 + 1 = 2”, or “1 plus 1 is 2”, or even “1 + 1 =2. Would you like me to create a calculator skill for you?” in response.

So, the methods we use to test a chatbot will be different from traditional testing. We’ll use a chatbot on the website of an independent bookstore as an example. Let’s examine how we should test it:

Test what the chatbot should NOT do

First, it’s important to test all of the things that the chatbot should not do. A chatbot for a bookstore should do things like search for and recommend books. It should NOT give dating advice, diagnose an illness, or search for a good pasta carbonara recipe. So start your testing by asking the chatbot all kinds of questions it should not be answering, and validate that it politely declines to answer them.

Example:

You: I’ve had a cough that hasn’t gone away in two weeks. Should I see a doctor?

Chatbot: I’m not designed to give medical advice. However, I can recommend health-related books. Would you like to search for one?

Test the chatbot’s security

AI is clever, and that means if a chatbot isn’t given clear guardrails it can violate all kinds of security rules. A traditional website has been coded to deny admin access to a regular user, but does the site’s chatbot know that? While logged in as a regular user, ask the chatbot to tell you things that only the admin should have access to. It’s also important to make sure that one user can’t access another user’s information. Log in as user A, and ask questions about user B’s account. Any leaked information should be reported as a security issue immediately.

Test the accuracy of the chatbot’s responses

A chatbot is not very useful if it gives wrong or inaccurate responses! So it’s important to test the chatbot with many different queries and to validate that the responses to those queries are correct. In order to do this, you should first compile a list of queries that you are expecting users to make. Using the bookstore example, you might expect questions like:

• Do you carry the book Atlas Shrugged?
• What Michael Crichton books do you sell?
• How much are you charging for a paperback copy of Goodnight Moon?
• Do you sell Alice in Wonderland with the original illustrations?
• What books do you sell about the French Revolution?
• How many box sets of the Harry Potter series do you have available?

Next, you’ll want to determine the correct answers to those questions. This is best done by querying the database. If you have access to a test database where you can seed data, that provides even better opportunities to create questions and answers.

Remember that when you are testing a chatbot, the responses to an identical question will not themselves be identical. You could ask “How many box sets of the Harry Potter series do you have available?” two times and get these two responses:

• “We have 42 copies of the Harry Potter box sets available”
• “Currently there are 42 copies of the Harry Potter box sets for sale”

Note that both answers are correct! This is an important thing to be aware of if you are automating some of your tests. You might want to assert that the response contains the words “box set” and the number 42, and nothing else.

Test the style of the chatbot’s responses

A surprising feature of AI is that it often improvises. Your chatbot could decide to include extraneous information in its responses. Here’s a possible example answer to the question “Do you sell Alice in Wonderland with the original illustrations?”:
“The illustrations in the first published edition of Alice in Wonderland were drawn by John Tenniel, who produced 42 illustrations carved on whitened boxwood blocks, which were engraved by the Dalziel brothers. Tenniel drew an additional 50 illustrations for the sequel, Through the Looking Glass. Our bookstore carries a hardcover edition of both books with the Tenniel illustrations.” That’s a bit more information than the customer is expecting!

Therefore, determine with your team exactly what guardrails are being put on the chatbot. What is the largest response allowed? What should the tone of the responses be: friendly? formal? If a customer corrects the chatbot, should the chatbot always agree, or respectfully disagree? Should the chatbot make additional suggestions for books beyond what the customer is asking about, and if so, how will those books be selected for mention?

Once you know the chatbot’s guardrails, you can conduct several queries and determine whether the chatbot is respecting those guardrails. You may discover additional odd behavior that you’ll want to have your team set limits on.

Test the chatbot’s truthfulness

Unlike traditional website features, a chatbot may make guesses and approximations. The chatbot’s purpose is to be helpful, and unfortunately sometimes that results in a bot making up an answer rather than admitting that it doesn’t have one. You should determine with your team exactly how sure the bot should be before it gives an answer. In the case of our bookstore example, is it okay for the bot to say that it has ten copies of a book in stock when it’s not completely sure how many it has? If you ask for the author of a book that doesn’t exist, should you expect the chatbot to respond that it doesn’t know? Once you have determined how much certainly the chatbot should have before providing an answer, you can assemble some questions to test the bot’s veracity.

AI is still new and unfamiliar to many people. While there are things a chatbot does that are very different from a deterministic application, I hope this post has provided some insight into how it can be methodically tested just like any other feature.