AI systems can appear convincing in a demo and still fail on real user requests. This role owns the evaluation discipline that answers whether an AI feature is useful, accurate, safe, consistent, and worth its cost — before and after release. You will translate product requirements and failure risks into test cases, datasets, metrics, automated evaluations, and release criteria that teams can trust.
Mô tả công việc
- Define evaluation plans for LLM applications, RAG systems, AI agents, and other AI-powered product features
- Translate user needs, product requirements, and risk scenarios into measurable quality dimensions and acceptance criteria
- Design representative test sets covering common cases, edge cases, adversarial inputs, multilingual requests, and known failure modes
- Choose appropriate evaluation methods: deterministic tests, human review, model-based judging, and task-specific metrics
- Set release thresholds and escalation rules with Product, Engineering, and Risk stakeholders
- Build and maintain versioned evaluation datasets, benchmark suites, test fixtures, and labelled examples
- Automate evaluation runs so teams can compare models, prompts, retrieval configurations, and application changes consistently
- Integrate evaluation into development and release workflows, including regression checks where appropriate
- Ensure evaluation results are reproducible and tied to the exact model, prompt, code, configuration, and dataset versions tested
- Analyse failures such as hallucinations, retrieval misses, tool-use errors, instruction-following failures, and inconsistent outputs
- Distinguish model limitations from problems in data, retrieval, prompts, application logic, or evaluation design
- Produce actionable error taxonomies and recommend targeted fixes to the responsible teams
- Verify whether proposed fixes improve the target problem without causing regressions elsewhere
- Create clear annotation guidelines and reviewer workflows for cases requiring human judgement
- Calibrate reviewers and measure agreement where subjective labels are used
- Audit automated and model-based judges for bias, instability, and blind spots
- Use human review to validate important automated evaluation results, especially for high-impact or ambiguous cases
- Develop evaluation coverage for privacy leakage, prompt injection, unsafe outputs, policy compliance, and other relevant risks
- Monitor production feedback and incidents to identify new failure patterns and refresh test suites
- Establish regression practices for model or provider changes and maintain a record of material evaluation results
- Help teams understand the limits of current benchmarks and where additional evidence is required
Yêu cầu
- 4+ years in software engineering, QA automation, data science, ML engineering, or applied AI, with hands-on testing or evaluation experience
- Strong Python skills and experience building automated test or data-processing workflows
- Practical understanding of LLM applications, including prompting, RAG, tool use, and common failure modes
- Experience designing test cases, datasets, metrics, and repeatable evaluation processes
- Ability to analyse structured and unstructured outputs and turn failures into actionable engineering findings
- Understanding of statistical reasoning, sampling, and the limits of aggregate metrics
- Working proficiency in English; ability to evaluate Vietnamese-language outputs is an advantage
Ưu tiên
- Experience with LLM-as-a-judge, pairwise comparisons, human annotation, and evaluator calibration
- Experience with RAG evaluation, retrieval metrics, grounding, citation correctness, or agent/tool-use benchmarks
- Experience with red-teaming, prompt-injection testing, privacy evaluation, or AI safety testing
- Experience integrating evaluation into CI/CD and experiment tracking
- Experience with multilingual AI products or high-impact decision-support applications
Quyền lợi
- Competitive salary commensurate with experience
- Performance-based bonus
- Full statutory insurance coverage (BHXH, BHYT, BHTN) per Vietnamese labor law
- Direct ownership of a critical AI quality function in a fast-growing company
- Clear career path in AI Engineering / Applied AI