Skip to main content

AI Coding Agent Evaluation Task Designer at toloka-ai

Setup
Remote
Location
New York, New York
Type
Contract
Level
senior
Posted

Description

Mindrift is recruiting experienced software developers for a project-based opportunity to create and evaluate tasks used to benchmark AI coding agents. The work involves building realistic simulated developer environments, designing intermediate-state coding tasks and acceptance criteria, writing functional and integration tests that distinguish valid from incorrect agent solutions, and iterating based on QA feedback. Candidates should have 8+ years of software development experience, proficiency with Python/FastAPI, JavaScript or TypeScript/React, Docker, Postgres, Kafka, Redis, testing, and English at B2+ level. The engagement is not permanent employment; contributors apply, qualify, join a project, complete accepted tasks, and are paid per task, with an advertised effective rate of up to $200 per hour.

For job seekers

Ready to find a role that actually fits?

Upload your résumé, start a Job Search Thread, and let Metaintro rank real openings against your experience — then guide you from search to offer.

Match

Compare live roles against your current evidence.

Position

Turn proof projects into role-specific applications.

Improve

Use market feedback to keep the skill plan current.

Return to navigation