Performance Engineer, Inference at Sarvam

Department: Infrastructure

Setup: On-site
Location: Bengaluru, Karnataka
Level: senior
Posted: Aug 10, 2026

Description

Sarvam is hiring a senior Performance Engineer, Inference to own the production serving path for large distributed models. The role integrates model and kernel artifacts into a multi-node, multi-tenant serving stack; extends distributed prefill-decode, KV/cache transfer, routing, and scheduling; builds and trains speculative decoders; and owns latency, throughput, GPU utilization, and cost metrics. Candidates need 5+ years in ML systems, including 2+ years at production scale, experience with 100B+ parameter models, source-level knowledge of SGLang, vLLM, Dynamo, or TensorRT-LLM, distributed serving expertise, speculative-decoding competency, and C++/CUDA proficiency.

Required Skills

SGLang
vLLM
NVIDIA Dynamo
TensorRT-LLM
C++
CUDA
Speculative decoding
Distributed serving
KV cache
Nsight Systems
py-spy

Additional Information

Required Previous Experiences: 5+ years of experience in machine learning systems
2+ years of experience serving models at production scale
Experience serving 100B+ parameter models in production across multi-node tensor, pipeline, or expert parallelism
Experience operating and extending distributed serving stacks, including disaggregated prefill-decode, distributed KV/cache transfer, and cross-node routing and scheduling
Experience building and training speculative decoders, including draft models, distillation, acceptance-rate tuning, and stack composition
Experience reading and modifying C++ and CUDA code
Preferred Qualifications: Upstream contributions to SGLang, vLLM, Dynamo, llm-d, TensorRT-LLM, or LMDeploy
Direct production experience with Dynamo or llm-d at scale
Experience operating a forked runtime in production
Published or shipped speculative-decoding work
Experience with MoE serving, long-context, multi-model, Indic, or multilingual workloads