What Should a QA Engineer Test in a Chatbot?
A chatbot should be tested from multiple perspectives.
A good chatbot testing strategy can include:
- Functional Testing
- Conversation Testing
- Intent Testing
- Context Testing
- Accuracy Testing
- Negative Testing
- Boundary Testing
- Security Testing
- Performance Testing
- UI Testing
- Integration Testing
- Regression Testing
- Accessibility Testing
- Human-like behavior testing
Let’s understand each one.
1. Functional Testing
First, test whether the chatbot’s basic functionality works correctly.
For example, imagine an e-commerce chatbot.
It should help users with:
- Product information
- Order status
- Returns
- Refunds
- Delivery information
- Cancellation
Example Test Case
Test Scenario: Check whether the chatbot provides order status.
Input:
Where is my order?
Expected behavior:
The chatbot should ask for the required order information.
For example:
Please provide your order ID so I can check the status.
What should QA verify?
Check whether:
- Chatbot opens correctly
- User can send messages
- Chatbot responds
- Response appears correctly
- Required information is requested
- Links work
- Buttons work
- Conversation continues correctly
2. Conversation Testing
This is one of the most important areas of chatbot testing.
A chatbot should understand a conversation rather than treating every message as a completely new request.
Consider this conversation:
User:
I want to book a flight to Delhi.
Chatbot:
Sure. When would you like to travel?
User:
Tomorrow.
The chatbot should understand that “tomorrow” refers to the flight booking.
It should not respond:
Tomorrow for what?
QA Should Test:
- Conversation flow
- Follow-up questions
- Previous messages
- Topic changes
- Conversation history
- Multi-turn conversations
3. Intent Testing
Intent means understanding what the user actually wants.
Consider these three messages:
I want to cancel my order.
Please cancel my order.
How can I cancel my order?
The wording is different, but the intent is similar.
A good chatbot should understand the intent correctly.
Example
User:
I don’t want this product anymore.
The chatbot should ideally understand that the user may want to cancel or return the product.
QA Testing Approach
Create multiple variations of the same request.
For example:
| User Input | Expected Intent |
|---|---|
| I want to cancel my order | Cancel Order |
| Cancel my purchase | Cancel Order |
| I don’t want this order | Cancel Order |
| Can I stop my order? | Cancel Order |
| I placed the order by mistake | Cancel Order |
This is extremely useful for chatbot testing.
4. Context Testing
Context is one of the biggest challenges in chatbot testing.
Let’s look at an example.
User:
What is the price of iPhone 17?
Chatbot:
The price is ₹79,999.
User:
Does it come with a charger?
The chatbot should understand that “it” refers to the iPhone.
If the chatbot responds:
What product are you asking about?
then context handling may be poor.
Test Cases for Context
Try:
- Pronouns such as “it”, “that”, “this”
- Follow-up questions
- Previous conversation references
- Multiple products
- Changing topics
- Returning to previous topics
5. Accuracy Testing
Accuracy is extremely important.
A chatbot can give a grammatically perfect answer that is completely wrong.
For example:
User:
What is the refund period?
Suppose the actual company policy says:
Refunds are processed within 7 business days.
But the chatbot says:
Refunds are processed within 30 days.
The response may look professional, but it is incorrect.
This is a serious defect.
QA Should Verify:
- Factual correctness
- Business rules
- Product information
- Pricing information
- Policy information
- Calculations
- Dates
- URLs
- Contact information
For business-critical chatbots, accuracy testing should be taken very seriously.
6. Negative Testing
Don’t test only normal questions.
Try to break the chatbot.
This is where QA Engineers can add significant value.
Examples
Enter:
asdfghjkl
or:
???
or:
I don’t understand anything
The chatbot should handle unexpected input gracefully.
Try:
- Random characters
- Emojis
- Very short messages
- Very long messages
- Misspelled words
- Multiple questions
- Empty messages
- Repeated messages
- Unsupported requests
Example
User:
Helo i wnt to chek my ordr
The chatbot should ideally understand the message despite spelling mistakes.