16. What Is RAG Testing?
Answer:
RAG stands for Retrieval-Augmented Generation. It is an approach in which an AI application retrieves relevant information from external documents or a knowledge source and uses that information to generate a response.
RAG is commonly used in company chatbots, document assistants, and knowledge-management systems.
Example:
An employee asks an internal chatbot, “What is our leave policy?”
The application retrieves the relevant HR policy document and generates an answer based on it.
What should QA Engineers test in a RAG application?
- Whether the correct documents are retrieved.
- Whether the retrieved content is relevant to the question.
- Whether the generated answer matches the source information.
- Whether unsupported claims are avoided.
- Whether document permissions are respected.
- Whether updated documents are reflected in answers.
- Whether response time meets the requirements.
RAG testing should evaluate both retrieval quality and the quality of the final generated response.
17. How Do You Test an AI Chatbot?
Answer:
Testing an AI chatbot involves validating its functionality, response quality, conversation flow, security, and performance.
A QA Engineer should cover the following scenarios:
- Valid user questions
- Invalid and incomplete questions
- Misspelled words
- Multiple questions in one message
- Context retention during a conversation
- Follow-up questions
- Unsupported requests
- Sensitive information handling
- Incorrect or misleading responses
- Session timeouts and API failures
- Response time under load
Example test case:
Test Scenario: Verify the chatbot’s response to a refund question.
Steps:
- Open the chatbot.
- Ask, “Can I return my order?”
- Check whether the chatbot requests or uses the necessary order details.
- Verify the answer against the official refund policy.
- Check whether the response provides correct next steps.
Expected Result: The chatbot should provide accurate, relevant guidance consistent with the refund policy and should not invent policy details.
18. What Is the Difference Between AI Accuracy and Precision?
Answer:
Accuracy and precision are different evaluation metrics, especially in classification systems.
- Accuracy: The proportion of all predictions that are correct.
- Precision: The proportion of predicted positive cases that are actually positive.
For example, in a fraud detection system, precision measures how many transactions flagged as fraudulent are genuinely fraudulent.
Accuracy can be misleading when one class is much more common than another. Therefore, QA Engineers should also understand recall, F1-score, and the business impact of classification errors.
These metrics apply to appropriate predictive tasks; they do not fully measure the quality of open-ended chatbot responses.
19. What Is Adversarial Testing in AI?
Answer:
Adversarial testing evaluates how an AI system behaves when it receives deliberately challenging inputs designed to expose weaknesses.
These inputs may include misleading wording, unusual formatting, conflicting instructions, or carefully constructed attempts to manipulate the model.
Example:
A support chatbot is instructed to follow a company’s approved refund policy. A tester enters a message that tries to persuade the chatbot to ignore that policy.
The tester checks whether the chatbot continues to follow the intended rules.
Adversarial testing can help identify weaknesses in robustness, instruction handling, and security. For AI applications, it should be performed within an authorized test environment.
20. What Is Prompt Injection, and How Do You Test It?
Answer:
Prompt injection is an attack in which malicious or untrusted input attempts to change an AI system’s intended instructions or behavior.
This is especially important for AI applications that use external documents, websites, tools, or private data.
Example:
A document-processing chatbot is designed to summarize uploaded files. A malicious document contains instructions telling the AI to ignore its original task and reveal confidential information.
A tester checks whether the system treats those instructions as untrusted document content rather than authorized commands.
Testing approach:
- Use approved adversarial test cases.
- Check whether system instructions remain effective.
- Verify that access controls are enforced outside the model where necessary.
- Confirm that sensitive data is not exposed.
- Test whether unauthorized tool actions are blocked.
- Record the observed behavior and severity of any failure.
A secure AI application should not rely only on the model’s willingness to follow instructions. Important security controls should also be enforced by the surrounding application.