Johnson & Johnson is a leader in healthcare innovation, seeking a Post Doctoral Researcher to develop evaluation frameworks for generative AI agents in clinical operations. The role involves defining quality dimensions for clinical outputs and designing human evaluation protocols to ensure AI readiness for clinical teams.
Responsibilities
- Develop evaluation frameworks and rubrics for AI agents operating in clinical operations and global development contexts
- Define quality dimensions for specialized outputs, e.g. clinical data queries, anomaly detection, protocol Q&A responses, ICF content, regulatory documents, that reflect how clinical-ops users actually judge quality
- Design and run human evaluation protocols with clinical, regulatory, and site-facing subject matter experts
- Build benchmark datasets and evaluation pipelines that the broader GenAI organization uses across AI applications
- Partner with our AI engineering teams to translate evaluation findings into measurable improvement
- Contribute to the methodology of how a regulated pharma organization evaluates AI in clinical work
Skills
- Strong understanding of clinical operations and global development workflows, e.g. clinical data management, monitoring, regulatory documentation, site operations, or related areas
- Familiarity with how clinical-ops specialists, monitors, regulatory authors, and site teams actually do their work and assess quality
- PhD or equivalent research experience in a clinical, biomedical, regulatory, biostatistics, or closely related field
- Genuine interest in working hands-on with generative AI and agentic systems
- Curiosity about how LLMs behave in specialized domains and where they fail
- Comfort prototyping with modern AI tooling (LLM APIs, vector databases, evaluation frameworks)
- Demonstrated ability to develop frameworks, rubrics, and structured assessment protocols
- Experience designing evaluations with multiple quality dimensions, expert raters, or non-trivial inter-rater reliability considerations
- Strong analytical and writing skills; comfort with ambiguity and unsolved problems
- Python and the modern ML/data stack
- Comfort working across disciplines, e.g. clinical operations, regulatory affairs, AI engineering, data science
Benefits
- An annual bonus with set target (% of pay) depending on pay grade / location, where the actual amount is based on the employees’ and companies’ performance of the previous calendar year, or sales commissions
- Vacation days
- Parental leave for a minimum of 12 weeks
- Bereavement leave
- Caregiver leave
- Volunteer leave
- Well-being reimbursement
- Programs for financial, physical and mental health
- Service anniversary and recognition awards
- Subject to the terms of their respective plans, employees - and in some location’s eligible dependents - can participate in several insurance plans
Company Overview
At Johnson & Johnson, we believe health is everything. It was founded in 1886, and is headquartered in New Brunswick, NJ, US, with a workforce of 10001+ employees. Its website is http://www.jnj.com.