11. What Is the Difference Between False Positive and False Negative in AI?
Answer:
These terms describe two types of incorrect classification.
False Positive: The model identifies something as positive when it is actually negative.
False Negative: The model identifies something as negative when it is actually positive.
Example: Fraud Detection
- False Positive: A legitimate transaction is flagged as fraudulent.
- False Negative: A fraudulent transaction is classified as legitimate.
Both errors matter, but their business impact may differ.
For example, false negatives can allow fraud to go undetected, while false positives can block genuine customers from completing transactions.
A QA Engineer should evaluate both errors against the business requirements and acceptable risk levels.
12. What Is Model Drift?
Answer:
Model drift refers to changes that cause an AI model’s performance or behavior to become less reliable over time.
This can happen when real-world data changes, user behavior evolves, or the relationship between input data and expected outcomes changes.
Example:
A model trained to detect fraudulent transactions may become less effective when fraud patterns change.
How can QA Engineers help detect model drift?
- Monitor prediction quality over time.
- Compare current data with historical data.
- Track important evaluation metrics.
- Review production errors and user feedback.
- Run regression tests when a model is updated.
Monitoring is particularly important for AI systems that operate continuously in changing environments.
13. How Do You Perform Regression Testing for AI Applications?
Answer:
AI regression testing verifies that changes to the model, prompts, data, or application code have not negatively affected existing functionality.
Unlike traditional regression testing, AI regression testing may need to evaluate both functional behavior and response quality.
Example:
A company updates its customer support chatbot to improve answers about delivery tracking.
After the update, QA Engineers should check delivery tracking as well as existing features such as refunds, cancellations, and account support.
Steps to perform AI regression testing:
- Maintain a collection of representative test questions.
- Store reference answers or evaluation criteria.
- Run tests against the updated application.
- Compare the new outputs with the baseline.
- Investigate significant quality changes.
- Report defects and repeat the tests after fixes.
A change in wording alone does not necessarily indicate a failure. The evaluation should focus on correctness, relevance, and other defined acceptance criteria.
14. Can AI Testing Be Automated?
Answer:
Yes. Many AI testing activities can be automated, although some scenarios still require human review.
Automation can help execute large test datasets, validate response formats, compare outputs against reference information, and measure quality metrics.
Common automation approaches include:
- API testing of AI endpoints
- Python-based test scripts
- Automated response assertions
- Dataset-driven testing
- Evaluation frameworks
- Performance testing
- Security test suites
- Regression testing pipelines
Example:
A QA Engineer can write an automated test that sends a question to an AI API, verifies the HTTP status code, checks that the response contains required fields, and records the response time.
However, a successful HTTP response does not prove that the AI answer is factually correct. Additional semantic or human evaluation may be necessary.
15. Which Tools Are Used for AI Testing?
Answer:
The tools depend on the type of AI application and the testing requirements.
| Tool | Common Testing Use |
|---|---|
| Postman | Testing AI APIs and validating response structures |
| Python | Building custom test scripts and evaluation pipelines |
| Pytest | Organizing and executing automated tests |
| Playwright | Testing AI-enabled web application workflows |
| Jupyter Notebook | Exploring datasets and analyzing model outputs |
| Ragas | Evaluating RAG applications using suitable metrics |
| DeepEval | Evaluating LLM application outputs |
| Promptfoo | Testing prompts, comparing model outputs, and evaluating security scenarios |
| JMeter | Load and performance testing of supported AI endpoints |
Tool selection should be based on the application’s architecture, evaluation goals, and technical requirements. Not every tool is required for every AI testing project.
You can also explore the official Playwright documentation to learn how to automate web application testing.