Note: The job is a remote job and is open to candidates in USA. CentralReach is a leading provider of autism and IDD care software for Applied Behavior Analysis (ABA) and related therapies. As a Clinical AI Evaluation Specialist, you will define quality standards for AI outputs and enhance evaluation systems to ensure compliance with ABA practices and governance policies.
Responsibilities
- Contribute to evaluation frameworks for clinical AI products, including defining acceptance criteria, test plans, clinically relevant rubrics, and performance benchmarks
- Conduct structured sampling reviews of AI-generated outputs, assessing across criteria alignment with ABA principles and BACB ethical standards
- Design and implement monitoring automation in collaboration with engineering, including automated quality checks, alerting thresholds, and drift detection for AI systems
- Contribute to the creation and maintenance of reusable governance artifacts including system cards, monitoring playbooks, evaluation of SOP templates, and risk assessment documentation
- Assist with pre-deployment validation testing for new AI products and features, executing defined testing and results of documentation
- Participate in risk tiering assessments for new and evolving AI products, providing clinical and evaluative perspectives
- Support incident investigation for AI-related issues if/as needed and in alignment with clinical best practice and CR Development Policy
- Collaborate with product management and engineering teams to provide input during product planning and development, ensuring governance considerations are integrated early
- Contribute to the development of evaluation methodology and standards for clinical AI use cases, documenting approaches and refining processes based on findings and emerging best practices
- Contribute to AI literacy and training content for internal teams and customers, drawing on hands-on clinical evaluation experience
- Stay current with developments in AI evaluation methodology, LLM behavior, ABA best practices, and relevant regulatory and ethical standards
- Support CentralReach's thought leadership in responsible AI through contributions to conference presentations, professional publications, or applied research as opportunities arise
Skills
- Board Certified Behavior Analyst (BCBA) in good standing
- 2+ years of demonstrated experience in quality assurance, evaluation, data analysis, or technical assessment roles, preferably within healthcare or behavioral health settings
- Demonstrated experience with large language models, generative AI, and their applications and limitations in clinical or professional contexts
- Experience with prompt/context engineering, AI output evaluation, or AI workflow design
- Experience with statistical sampling methodologies or healthcare quality assurance frameworks
- Familiarity with AI governance frameworks (NIST AI RMF, EU AI Act risk categories, AMA governance guidance)
- Experience developing evaluation rubrics, scoring frameworks, or quality measurement tools
- Background in clinical informatics, research methodology, or systematic quality improvement
- Demonstrated proficiency with data analysis tools and comfort working with quantitative and qualitative evaluation data
- Strong analytical skills with the ability to develop systematic, evidence-based evaluation approaches
- Familiarity with HIPAA compliance, healthcare data standards, and ethical considerations in AI
- Excellent written and verbal communication skills with the ability to document methodology clearly and convey findings to clinical, technical, and leadership audiences
- Demonstrated ability to work collaboratively across clinical, technical, and product teams
- Experience in a remote work environment with strong self-direction and accountability
Benefits
- Competitive compensation
- Comprehensive health benefits
- Generous PTO
- 401(k) matching
- Paid parental leave to our full-time employees
- Hybrid work schedules
- Career development support
- Wellness programs
- Opportunities to give back through CR Cares™, our community engagement initiative
Company Overview
Company H1B Sponsorship