Applied AI Researcher at Distyl AI

Setup: Hybrid
Location: San Francisco, California
Type: Full-time
Posted: Oct 16, 2025

Description

Distyl AI is seeking an Applied AI Researcher to join their Benchmarking team. This role involves designing and implementing evaluation frameworks for intelligent systems, exploring new paradigms for assessment, and driving internal research priorities and industry standards. The ideal candidate will have a strong research background, experience in designing and running evaluations, strong statistical and analytical rigor, expertise in building with AI models, a proven track record of research results, daily AI tool usage, and strong programming and data analysis skills. The company offers a competitive salary, comprehensive benefits including 100% covered medical, dental, and vision for employees and dependents, equity options, a 401K plan, and a hybrid work environment with offices in San Francisco and New York City.

Required Skills

AI
Machine Learning
Data Analysis
Programming
Statistical Analysis
Benchmarking
Experimental Design
Compound AI Systems
Agentic Collaboration

Additional Information

Required Previous Experiences: Experience Designing and Running Evaluations: You’ve built or maintained benchmarks, test suites, or experimental frameworks to measure model or system performance.
Proven Track Record of Research Results: Whether you’ve published in top journals, posted amazing work on twitter, or somewhere else we want to see what you've done.
Preferred Qualifications: Strong research track records
operate in an AI-native way
Statistical and Analytical Rigor: You design fair, reproducible experiments and can extract signal from noisy empirical results.
Experience Building with Models, Not Just Building Models: We develop intelligent systems using models rather than training or fine-tuning them. Ideal candidates have expertise in compound AI systems, agentic collaboration, and associated techniques (ensembling, ReAct, graph-of-thoughts, etc.).
Uses AI Every Day: Before you can revolutionize someone else’s workflow, you need to revolutionize yours. You should be using tools like ChatGPT, Cursor, and Perplexity to accelerate your workflow.
Strong Programming and Data Analysis Skills: While you might not consider yourself a software engineer you need to be able to build prototypes of your ideas and then perform the experiments to prove the effectiveness to a F500 Head of AI.
Biases Towards Showing vs Telling: Our customers want to see the power of AI today vs discuss the most elegant idea that will take 5 years to realize.