Senior AI Infrastructure Engineer, Observability at firmus

Language

Location: Singapore
Type: Full-time
Level: senior
Posted: Sep 4, 2026

Description

Firmus Technologies is hiring a Senior AI Infrastructure Engineer, Observability, to establish GPU and host health standards, service-readiness criteria, dashboards, alerts, queries, diagnostic checks, and operational runbooks for customer and internal AI workloads. The role covers GPU fault diagnosis, telemetry analysis, failure detection, incident response, commissioning support, and knowledge publication, with a Singapore location, full-time employment, and occasional overseas travel.

Required Skills

GPU fault diagnosis
Linux
Python
PromQL
LogQL
Grafana
DCGM
NCCL

Additional Information

Minimum Education: Bachelor's degree in computer science or a related technical field, or equivalent practical experience
Required Previous Experiences: 7+ years in GPU, HPC, AI infrastructure, or closely related large-scale systems engineering environments, including ownership of health monitoring or diagnostics
Production GPU fault diagnosis experience, including identifying faults through XID events, ECC/memory errors, NVLink issues, power or thermal limits, and catching at least one before job failure
Hands-on GPU diagnostics and validation using DCGM, NCCL/collective tests, and stress testing, with checks developed for other teams
Experience analyzing infrastructure telemetry, building trusted dashboards and alerts, and using PromQL, LogQL, Grafana, or comparable tooling
Preferred Qualifications: Production experience on large GPU systems with tight GPU-to-GPU interconnect
Experience across multiple sites or GPU generations, especially preparing health monitoring before a new GPU class entered production
Background at an AI cloud, GPU cloud, or HPC centre running customer workloads
Experience working with data or ML engineers on GPU/host failure prediction from health signals
Structured operational knowledge for AI-assisted incident recovery