Software Engineer, ML Training Performance
Foster City, CA, United States5d ago
Free account needed to apply or save. It takes a minute, and we'll bring you right back to this job.
Skills
About the role
In this role, you will:
-
Design, implement, and operate cutting-edge ML training performance optimization techniques — distributed training (data/tensor/pipeline parallelism), mixed-precision, gradient checkpointing, and GPU kernels — to scale our VLM, VLA, and Foundational models training efficiency.
-
Work on RL framework and RL performance optimization for driving foundation model
-
Profile and debug training throughput bottlenecks across the compute, compiler, framework, and cluster-networking stack.
-
Collaborate closely with ML researchers, data engineers, and hardware engineers to define requirements and align on architectural decisions for large-scale training infrastructure.
-
Implement custom kernels and build benchmarks to track and validate training-performance optimizations across the stack.
Qualifications
-
4+ years of total experience, including 2+ years working on large-scale model training platforms.
-
Experience with distributed training frameworks (Megatron-LM, TorchTitan), efficiently utilizing GPU/accelerator clusters at scale.
-
Experience with RL framework such as Verl/MILES and RL algorithms(GRPO, PPO)
-
Experience with collective communication libraries (e.g., NCCL) and optimizing communication/compute overlap in multi-node training.
-
Experience using profiling tools (NVIDIA Nsight, PyTorch Profiler, or XLA/JAX profiling) to identify and resolve training bottlenecks.
-
Proficient in Python or C++.
-
Experience with ML compilers, performance tooling, or hardware/software co-design (e.g., GPU/TPU/accelerator architectures).
Note: You do not have to meet all the requirements below to be considered for this position: