Note: The job is a remote job and is open to candidates in USA. Nscale is the GPU cloud engineered for AI, providing high-performance infrastructure for AI start-ups and enterprise customers. As a Staff Observability Platform Engineer, you will build and evolve the observability platform to enable visibility into GPU clusters and AI workloads while partnering with various teams to enhance operational efficiency.
Responsibilities
- Design, build, and evolve observability platforms across metrics, logs, traces, alerting, and telemetry pipelines
- Lead the implementation of scalable observability solutions that support Nscale's growing GPU and AI infrastructure
- Partner with SRE, infrastructure, platform, and AI/ML teams to ensure observability is embedded throughout the software and infrastructure lifecycle
- Drive improvements in monitoring coverage, alert quality, service health visibility, and incident response effectiveness
- Develop standards, frameworks, and reusable patterns that simplify observability adoption across engineering teams
- Identify reliability risks and operational blind spots, helping teams proactively address them before they impact customers
- Contribute to architectural decisions around telemetry collection, storage, retention, cardinality management, and performance optimization
- Lead technical initiatives and projects that improve platform scalability, reliability, and operational efficiency
- Mentor engineers and provide technical guidance through design reviews, code reviews, and knowledge sharing
- Participate in incident investigations and postmortems, translating operational learnings into durable platform improvements
- Evaluate new observability technologies and practices, balancing innovation with operational simplicity and long-term maintainability
Skills
- 6+ years of experience in SRE, platform engineering, infrastructure engineering, observability engineering, or related disciplines
- Strong experience building and operating observability platforms in cloud-native, distributed environments
- Deep hands-on experience with several of the following technologies: Prometheus, Thanos, VictoriaMetrics, Grafana, Loki, Tempo, OpenTelemetry, ClickHouse, Elastic, or similar platforms
- Strong software engineering skills with proficiency in Go, Python, or equivalent languages
- Experience operating and troubleshooting Kubernetes-based platforms at scale
- Strong understanding of monitoring, logging, tracing, telemetry pipelines, and modern observability practices
- Experience designing systems with scalability, reliability, performance, and operational simplicity in mind
- Proficiency with Infrastructure-as-Code tools such as Terraform, Ansible, or equivalent
- Ability to lead technical initiatives and influence engineering decisions across multiple teams
- Excellent communication skills with the ability to explain technical tradeoffs and align stakeholders around pragmatic solutions
- Experience operating observability systems in GPU, AI/ML, HPC, or large-scale compute environments
- Familiarity with Slurm, Kubernetes GPU scheduling, or AI infrastructure platforms
- Experience with high-volume telemetry pipelines and streaming technologies such as Kafka, Vector, or Fluent Bit
- Knowledge of observability challenges related to model training, inference workloads, GPU utilization, and distributed AI systems
- Experience mentoring engineers and helping grow technical capability across teams
Company Overview