Data Scientist, Evaluation at Sunset

Department: Engineering

Location: New York, New York
Level: mid
Posted: Aug 13, 2026

Description

Replay is hiring its first Data Scientist focused on evaluation to establish decision-grade measures for the quality and safety of de-identified enterprise data. The role builds evaluation corpora, experiments, quality metrics, pipelines, and feedback loops; analyzes failure patterns across models, prompts, judges, thresholds, and workflows; and connects offline measures to escaped sensitive information, utility loss, review burden, and delivery decisions. It requires hands-on Python and SQL, statistical evaluation knowledge, startup ownership, and the ability to independently verify AI-generated evidence.

Required Skills

Python
SQL
Sampling
Uncertainty
F1
Calibration
Agreement
Class imbalance
Distribution shift
Adversarial corpora
Programmatic verifiers

Additional Information

Required Previous Experiences: At least three years of professional experience in applied science, data science, machine learning, quantitative research, or a closely related role
Experience designing evaluations or experiments that changed a product, model, release, or operational decision
Startup experience with broad ownership and building measurement foundations while decisions are moving quickly
Preferred Qualifications: Experience evaluating NER, entity resolution, information extraction, document understanding, multimodal, retrieval, or LLM systems
Experience with privacy, de-identification, data quality, model risk, safety, or other high-trust decision systems
Experience designing human-review, adjudication, weak-supervision, or active-learning systems
Experience building adversarial corpora, replay systems, simulation environments, programmatic verifiers, or model-judge evaluations
Experience connecting offline measures to escaped defects, customer outcomes, review effort, or preserved data utility
Experience measuring quality across multi-stage batch or data pipelines