Machine Learning Engineer, Inference & Serving (Speech LLM) - San Francisco at Plaud

Department: Global Product R&D Center

Compensation

$195,000 – $365,000/yr

Setup: Hybrid
Location: San Francisco, California
Type: Full-time
Level: senior
Posted: May 8, 2026

Description

Plaud is hiring a SpeechLLM Research Engineer to build and optimize high-throughput, low-latency inference systems for large language models and foundational speech models. The role focuses on real-time streaming, continuous batching, KV cache management, GPU architecture, distributed multi-GPU inference, and model compression or quantization. Candidates should have hands-on experience with LLM or speech-model serving, GPU systems, and real-time conversational AI. The position offers a $195,000–$365,000 base salary plus bonus and equity, healthcare, retirement matching, paid time off, and a hybrid office arrangement requiring at least three days in the office per week.

Required Skills

Inference engines
Continuous batching
KV cache
GPU architecture
Tensor parallelism
Kubernetes
Model quantization
WebSockets

Additional Information

Required Previous Experiences: Hands-on experience building and deploying high-throughput, ultra-low-latency inference engines for large language models or foundational speech models
Practical experience with continuous batching, KV cache management, and stateful connections for real-time conversational AI
Deep understanding of GPU architectures and memory hierarchies
Experience collaborating between machine learning training and backend infrastructure teams
Preferred Qualifications: Familiarity with vLLM, TensorRT-LLM, SGLang, or NVIDIA Triton Inference Server
Experience with real-time audio streaming, WebSockets, WebRTC, and neural audio codecs
Experience with speculative decoding, lookahead decoding, or chunked prefill
Experience with post-training quantization and model deployment in FP8, INT8, AWQ, or GPTQ
Experience deploying multi-GPU and multi-node inference pipelines with Kubernetes