Senior AI Infrastructure O&M Engineer

About the Role

We are seeking a Senior AI Infrastructure O&M Engineer to lead end-to-end operational governance, infrastructure reliability, and incident response for our large-scale Intelligent Computing (AI/HPC) Data Center. In this role, you will take full ownership of cluster management, lead high-severity troubleshooting (such as mass GPU dropouts or cluster-wide outages), optimize AI monitoring stacks, and mentor engineering teams to ensure world-class operational excellence.

Key Responsibilities

• End-to-End Operational Governance: Oversee the operational quality and lifecycle management of the entire AI computing infrastructure; lead safety, compliance, and strict security/confidentiality protocols.
• Major Incident & Cluster Emergency Response: Lead root-cause analysis (RCA) and emergency response for critical failures; handle major cluster disruptions including mass GPU/NPU node dropouts, high-speed interconnect failures, and hardware-level replacements.
• Observability & Asset Management: Drive the optimization of AI cluster monitoring and telemetry systems; standardize hardware asset tracking and deliver data-driven optimization reports.
• Engineering Oversight & Technical Quality: Serve as the technical gatekeeper during facility/cluster expansion and construction; conduct full-dimensional operational data analytics to produce high-value operational insights.
• Team Leadership & Resource Management: Mentor junior and mid-level engineers; build technical training curricula; manage cleanroom environment compliance and spare parts inventory.

Qualifications & Requirements

(1) Master’s degree or higher in Computer Science, Software Engineering, Network Engineering, Electronic Information, IoT, or a related STEM discipline.

(2) Working Experience:

• 3+ years of hands-on experience in AI/HPC data center operations and maintenance; OR
• 2+ years in software development, automated O&M (DevOps/SRE), or IoT platform engineering.

(3) Skills & Expertise:

• AI Infrastructure Architecture: Strong understanding of end-to-end AI/HPC infrastructure operations across GPU/NPU servers, high-speed fabrics (InfiniBand/RoCE), and storage networks; proven capability to diagnose and resolve complex outages (e.g., node drops, fabric degradation).
• Systems & Databases: Proficient in Linux administration and basic operations/maintenance for enterprise databases (MySQL, Oracle, etc.).
• Observability & Analytics: Hands-on experience optimizing monitoring platforms (Prometheus/Grafana/custom AI telemetry stacks); ability to extract high-value insights from telemetry data and translate them into actionable engineering solutions.
• Technical Gatekeeping & Incident Management: Strong background in engineering oversight for data center expansion, emergency management, and cross-functional coordination.
• Industry Certifications (Preferred): Professional certifications from leading vendors (e.g., Red Hat RHCE/RHCA, Cisco CCNP/CCIE, Huawei HCIP/HCIE, or equivalent).
• Management & Leadership: Ability to lead and train engineering teams (10+ engineers) and drive team capability growth.