Platform Reliability Engineer

Firmus Technologies

Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific.  

Founded in Australia in 2019, our mission is to create the most efficient AI infrastructure by combining cutting-edge technology with a steadfast commitment to sustainability. 

At Firmus, we are unique in our approach. We design, build, and operate a new class of digital infrastructure – the AI Factory. Through our model-to-grid technology approach, we have pushed the boundaries of multi-generational liquid cooling systems, energy management, AI software orchestration, and construction. For our customers, this approach allows us to make every watt count and deliver low-cost AI tokens globally. 

 

Firmus AI Cloud

Our large-scale GPU cloud platform, Firmus AI Cloud, is purpose-built to deliver energy-efficient AI compute at scale to customers. 

It empowers developers, enterprises, educational institutions, and government users to train and deploy AI models with unmatched efficiency and cost savings. With an ever-growing suite of services and applications, we are committed to delivering a cloud experience that is market-leading, proprietary, and built to scale. 

 

AI FactoryOS Operations 

AI FactoryOS is Firmus' proprietary operating system for the AI Factory. It governs GPU telemetry, cooling, power and grid interaction as one integrated layer, so that every Firmus site can be optimised and monitored as a single system. 

AI FactoryOS Operations runs that platform in production and owns the 24/7 reliability of AI FactoryOS, Firmus AI Cloud and the platforms built on them, together with the service levels the estate is measured against. 

The remit is an engineering one. The function builds the guarded automation, remediation and operational tooling that turn manual response into a software-defined capability, and builds and operates the shared services the estate's own operation depends on. The function works closely with the engineering teams that build the platform, supplying the production evidence that shapes what they fix and what they build next. 

 

Role Summary  

Firmus runs large-scale, state-of-the-art AI infrastructure built on the latest generation of GPU rack-scale systems and operated as one estate to power the next generation of AI innovation. The Platform Reliability Engineer is part of the team that operates the platform services Firmus AI Cloud runs on in production: the GPU compute fleet, exabyte-scale storage, the multi-tenant control and management plane, and the virtualisation and shared services that host it. Together, these are the foundation every customer workload depends on.

This is a hands-on site reliability engineering role that works across the stack, from Linux and Kubernetes to storage, virtualisation and the compute fleet. The team runs each service to a declared service level, measures reliability against it, and treats operations as a software problem, engineering repetitive manual work out of the system rather than absorbing it. The role picks up incidents escalated from first response, owns each one through to resolution, and works across teams and vendors to restore service, bringing in specialist engineers when a fault needs deeper investigation. Automation is a first-class part of the role: faults are diagnosed using the observability stack, post-incident reviews feed fixes back into the platform, and every resolved fault is a candidate for a runbook or an automated fix.

 

Key Responsibilities

• Triage, diagnose and resolve incidents across the stack, including Kubernetes clusters and workloads, storage, virtualisation, network connectivity and the underlying Linux hosts, owning each incident from escalation through to resolution.
• Operate the multi-tenant control and management plane that Firmus' AI and infrastructure services depend on, including the tenancy, quota and access controls that support separation between tenant workloads and data.
• Operate the virtualisation platform that hosts the management plane, and the shared services that run on it, to their declared service levels.
• Operate Firmus' exabyte-scale distributed and high-performance filesystems and S3-compatible object storage to their declared service levels, handling capacity, performance and availability incidents and escalating to specialist engineers and vendors (for example VAST or WEKA) with clear evidence when further investigation is needed.
• Operate the GPU compute fleet in production: node health and readiness, GPU and node fault detection and handling, firmware and driver currency to the supported baselines, remediation and return to service, and support for cluster provisioning and validation activities.
• Reduce toil by converting repeat fixes into runbooks and automated remediation delivered as controlled code, contributing to the team's automation and remediation tooling.
• Diagnose performance problems across the data path between compute and storage, gathering the evidence needed to resolve them and working with network and storage specialists where required.
• Deliver change as code through the agreed change process, with peer review, testing, staged rollout where applicable, and a rollback plan.
• Use the observability stack (for example Grafana, Prometheus, Loki and Tempo) to track service level indicators against their objectives, query metrics, logs and traces, correlate signals across services and diagnose faults, and improve dashboards and alerting so that each alert is actionable and tied to a runbook.
• Execute the patching cycle and urgent vulnerability remediation across the services the team operates.
• Manage vendor support cases across the hardware and software portfolio, raising and tracking cases under the relevant support agreements, supplying logs and diagnostic evidence, and following cases through to agreed response and resolution times.
• Take part in technical recovery during major incidents and in blameless post-incident reviews, feed recurring issues back to the engineering teams as clear requirements, share the follow-the-sun on-call roster, and keep incident tickets, operational procedures and runbooks accurate and current so that others can execute them successfully.

 

Skills & Experience 

Required Skills

• Solid experience operating production infrastructure in a 24/7 environment using site reliability engineering practices, including service level objectives, toil reduction, owning incidents from escalation to resolution, oncall participation, post-incident review and writing runbooks that others can execute successfully.
• Strong working knowledge of Linux systems administration and troubleshooting, including storage, host networking, processes, memory and I/O, and a systematic approach to isolating faults.
• Working experience with scale-out, parallel or enterprise storage (for example VAST, WEKA, Ceph, Lustre, GPFS or NetApp), including diagnosing performance and capacity problems using evidence rather than assumption.
• Familiarity with a virtualisation platform (for example Proxmox, VMware or KVM).
• Working experience using observability tooling to diagnose faults (for example Grafana, Prometheus, Loki, Tempo or Elasticsearch), including writing queries across metrics and logs (such as PromQL and LogQL) and correlating signals across services.
• Working experience with infrastructure-as-code and GitOps practices (for example OpenTofu or Terraform, Ansible, Argo CD), and writing scripts or small tools for operational automation in a language such as Python or Go.
• Familiarity with server hardware operation and support, including working with hardware and software vendors to raise support cases and supply diagnostic evidence.Platform Reliability Engineer
• Familiarity with cyber security principles and how they apply to operating production infrastructure.
• Clear communication skills, with the ability to write incident updates, ticket notes and escalations that others can act on, and a working style suited to a team that shares accountability for a broad portfolio across regions.

Preferred Experience

• Experience in infrastructure operations for GPU or HPC environments and the demands of large-scale distributed training and inference, including GPU health and fault signals (for example DCGM health checks, NCCL tests and XID errors).
• Experience in multi-tenant service provider, cloud or colocation environments with contractual response commitments and customer-facing reporting.
• Working knowledge of networking fundamentals, including TCP/IP, DNS, routing, VLANs, load balancing and firewalls, and the ability to trace a connectivity fault across hosts, switches and services.
• Knowledge of high-performance fabrics and data-path technologies, such as InfiniBand, RoCE, RDMA and GPUDirect Storage, and DPU or SmartNIC-based networking.
• Familiarity with bare-metal lifecycle and provisioning automation (for example NVIDIA Infra Controller, MAAS, Ironic or Metal3), and standard operating environments with known good-state enforcement.
• Hands-on experience operating and troubleshooting Kubernetes, including workloads, networking, storage and cluster health, and multi-tenant and GPU-enabled Kubernetes patterns.
• Experience with ITSM and incident management tooling (for example Jira Service Management, ServiceNow or PagerDuty).
• Experience working under ISO 27001 or SOC 2 controls.
• Relevant certification, such as Certified Kubernetes Administrator (CKA), a Linux certification (for example RHCSA or LFCS), or a vendor certification.
• A Bachelor's degree in computer science, engineering or a related discipline, or an equivalent combination of relevant experience and training.

 

Location & Reporting 

Location: Based in Australia or Singapore, with travel to Australian AI Factory sites as required. 

On-call: The function runs 24/7. First line monitoring and first response sit with th