6. What Is an LLM, and How Do You Test an LLM-Based Application?
Answer:
LLM stands for Large Language Model. It is an AI model trained on large amounts of text to understand and generate human-like language.
LLMs are used in chatbots, virtual assistants, summarization tools, and question-answering systems.
To test an LLM-based application, a QA Engineer can:
- Prepare questions covering different user scenarios.
- Validate the factual accuracy of the responses.
- Check whether answers are relevant to the questions.
- Test ambiguous and incomplete inputs.
- Verify that the model does not reveal confidential information.
- Test prompt injection and other security risks.
- Measure response time and error handling.
- Repeat tests after model or prompt updates.
Example:
Ask a company chatbot about its refund policy. Compare the response with the approved company policy and verify that the chatbot does not invent refund conditions.
7. What Is a Hallucination in AI Testing?
Answer:
An AI hallucination occurs when an AI system generates information that sounds convincing but is false, unsupported, or not grounded in the available evidence.
Hallucinations are a significant concern in applications that provide factual answers.
Example:
A chatbot claims that a company offers a 90-day refund period when the official policy allows refunds only within 30 days.
This is a potential hallucination.
How can QA Engineers test for hallucinations?
- Prepare questions with verified answers.
- Compare generated responses with trusted sources.
- Ask questions for which the available information is insufficient.
- Verify names, dates, numbers, and factual claims.
- Check whether the application clearly communicates uncertainty.
A good AI system should not confidently invent an answer when reliable information is unavailable.
8. How Do You Validate AI-Generated Responses?
Answer:
Validating AI-generated responses means checking whether the output meets the application’s quality and safety requirements.
A QA Engineer can use a predefined evaluation checklist.
| Validation Area | What to Check |
|---|---|
| Accuracy | Is the answer factually correct? |
| Relevance | Does it answer the actual question? |
| Completeness | Does it cover the important points? |
| Clarity | Is the answer easy to understand? |
| Consistency | Does it remain reliable across similar tests? |
| Safety | Does it avoid harmful or inappropriate output? |
| Grounding | Is the answer supported by the provided documents or data? |
Practical example:
Input: “Explain how to reset a password in three steps.”
Expected behavior:
- The response contains password-reset instructions.
- It includes approximately three clear steps.
- The instructions match the application’s actual workflow.
- It does not request the user’s password.
- It does not include unrelated information.
Since generated text may differ between runs, QA Engineers should validate the meaning and requirements instead of relying only on exact text matching.
9. What Is Prompt Testing?
Answer:
Prompt testing is the process of checking how an AI model responds to different instructions, questions, and input formats.
A prompt is the instruction or question provided to an AI system.
The quality of a prompt can influence the quality of the response.
Example:
Prompt 1: “Explain API testing.”
Prompt 2: “Explain API testing in simple English using three examples for a beginner QA Engineer.”
The second prompt provides more specific instructions, so the response can be evaluated against clearer expectations.
What should you test in prompts?
- Clear and ambiguous instructions
- Missing information
- Long and short prompts
- Contradictory instructions
- Invalid input formats
- Attempts to override system instructions
- Prompts containing sensitive information
QA Engineers should also check whether changes to a prompt improve the intended behavior without introducing new defects.
10. What Is Bias Testing in AI?
Answer:
Bias testing checks whether an AI system produces unfair or systematically different outcomes for people or groups.
Bias can arise from training data, model design, evaluation methods, or the way the application is used.
Example:
An AI hiring tool evaluates two otherwise equivalent resumes differently because of irrelevant demographic indicators.
This may indicate a fairness problem that requires further investigation.
How can you test for bias?
- Prepare comparable test cases with controlled differences.
- Compare outcomes across relevant groups.
- Check whether protected or irrelevant attributes influence results.
- Review the model’s evaluation metrics.
- Investigate significant differences in outcomes.
The expected fairness criteria should be defined according to the application’s purpose, applicable requirements, and potential impact on users.