Note: The job is a remote job and is open to candidates in USA. OVA.Work is seeking a Language Model Evaluator to assess and improve the quality of Large Language Models and Generative AI applications. The role involves evaluating model performance, collaborating with various teams, and developing evaluation frameworks to enhance model behavior and user experience.
Responsibilities
• Design and execute evaluation frameworks for Large Language Models (LLMs) and Generative AI applications
• Assess model outputs for accuracy, relevance, factual consistency, reasoning quality, coherence, fluency, and completeness
• Evaluate model performance across diverse domains, languages, and real-world use cases
• Develop benchmark datasets, evaluation prompts, and test scenarios for model validation
• Measure and monitor hallucinations, bias, toxicity, harmful content, and safety risks
• Perform prompt testing, regression testing, and comparative evaluations across different models and versions
• Analyze model performance using quantitative metrics and human evaluation methodologies
• Collaborate with AI Engineers and Prompt Engineers to improve prompts, retrieval pipelines, and model responses
• Evaluate Retrieval-Augmented Generation (RAG) systems, AI agents, and tool-calling workflows
• Document evaluation findings, recommendations, and quality improvement plans
• Support model release decisions through structured quality assurance processes
• Stay current with emerging LLM evaluation techniques, benchmarks, and Responsible AI practices
Skills
• Bachelor's or Master's degree in Computer Science, Artificial Intelligence, Data Science, Computational Linguistics, Statistics, or a related field
• 2–5 years of experience in AI evaluation, NLP, machine learning, quality assurance, or data analysis
• Strong understanding of Large Language Models (LLMs), Generative AI, and Natural Language Processing (NLP)
• Experience creating evaluation datasets, benchmarks, and test cases
• Strong analytical skills with the ability to identify quality issues and failure patterns
• Programming experience in Python and SQL
• Familiarity with data analysis libraries such as Pandas and NumPy
• Experience using spreadsheets, visualization tools, or notebooks for analysis
• Excellent written communication and attention to detail
• Experience with LLM evaluation frameworks such as DeepEval, Ragas, LangSmith, Promptfoo, OpenAI Evals, or similar tools
• Experience evaluating Retrieval-Augmented Generation (RAG), AI agents, and function-calling workflows
• Knowledge of prompt engineering, model fine-tuning, and reinforcement learning from human feedback (RLHF) concepts
• Familiarity with AI safety, red teaming, adversarial testing, bias detection, and Responsible AI principles
• Experience with multilingual model evaluation
• Knowledge of cloud AI platforms and MLOps workflows
• Experience with annotation tools and human-in-the-loop evaluation processes
Company Overview
• OVA is the most advanced Automated, Intelligent, intuitive On-boarding platform for Staffing Firms of all sizes. It was founded in 2018, and is headquartered in Alpharetta, Georgia, USA, with a workforce of 51-200 employees. Its website is https://www.ova.work/.