Production Engineer

About the role

Recursive is building AI that improves AI, starting with automated AI research. We’re hiring a Production Engineer to build and operate the systems our researchers and engineers rely on: cloud infrastructure, GPU clusters, deployment tooling, and secure access.

You’ll work across the stack, from Linux hosts and Slurm jobs to Kubernetes services and infrastructure automation. This is a hands-on role with substantial operational ownership. You’ll help teams get unstuck, diagnose failures, and turn recurring problems into reliable tools and processes.

Where you can make an impact

• Reliable compute: Keep GPU and CPU environments healthy across cloud providers so research workloads can run consistently.

• Repeatable infrastructure: Make provisioning, configuration, deployment, and access predictable, reviewable, and easy to operate.

• Researcher productivity: Give teams clear documentation, useful diagnostics, and practical support as their workloads change.

What you’ll do

• Provision and manage infrastructure across AWS, GCP, and neo cloud providers using infrastructure as code and configuration management.

• Operate Slurm clusters and container platforms; diagnose GPU, network, storage, scheduling, and job startup and cleanup failures.

• Build Python and Bash automation and secure CI/CD pipelines, making changes testable, repeatable, and easy to roll back.

• Manage IAM, secrets, and zero-trust access across providers using tools like Tailscale, keeping permissions deliberate and systems secure.

• Improve logs, metrics, alerts, and recovery procedures; work with providers to resolve faulty nodes and verify systems before returning them to service.

• Partner with researchers and engineers to support agentic workloads, understand bottlenecks, and prioritize fixes that remove recurring friction.

What we’re looking for

• Clear communication, curiosity, and ownership. You can guide researchers through unclear requirements, manage changing priorities, and revise your approach as new evidence appears.

• Deep Linux systems experience and strong practical troubleshooting across hosts, networking, storage, and GPU environments.

• You can read unfamiliar automation, reason about failures, and verify changes before running them with elevated privileges.

• Hands-on infrastructure as code and configuration management experience with tools such as Terraform, Pulumi, Puppet, or Ansible.

• Practical Kubernetes and container deployment experience, plus a working understanding of CI/CD, IAM policies, and secure access.

• Comfort using AI coding agents for diagnosis and automation, with the judgment to review generated code, test assumptions, and understand its operational risks.

Especially relevant experience

• Slurm ecosystem tools such as Slinky, AWS ParallelCluster, Pyxis, and prolog/epilog diagnostics.

• Container delivery with EKS, ECS, Helm, Kustomize, Argo CD, or GitHub Actions.

• Observability and hardware management with Prometheus, Grafana, Datadog, or Redfish.

• Experience operating GPU clusters, including NVIDIA or AMD diagnostics and collective communication tooling such as NCCL or RCCL.

• Rust programming, SGLang gateways, or Modal applications.

• Cloud security posture tools such as Wiz, security data lakes, or hands-on support for security and compliance audits.

What success looks like

Researchers can get the compute and access they need, understand failures, and keep work moving. Infrastructure changes are reproducible and reviewed, recurring incidents become less frequent, and recovery is documented and dependable. You leave the platform easier for colleagues to operate and extend.