Research Scientist

About the Role

Join a research and engineering team developing benchmarks and evaluation methods for advanced language models. Your work will help AI labs and enterprise teams assess model capabilities and make better-informed decisions about production use.

What You'll Do

• Evaluate newly released AI models across a range of use cases.

• Build benchmarks from the ground up, including dataset development and labeler coordination.

• Develop scalable and accurate automated evaluation methods for generated text.

• Work with engineering teams to implement and scale evaluation systems.

• Collaborate with AI labs and enterprise teams to translate evaluation needs into research projects.

What We're Looking For

• At least 3 years of full-time AI/ML research experience, particularly in benchmarking, evaluation, or language models.

• A track record of published research, ideally in NLP or benchmarking, and hands-on experience creating benchmarks or evaluation methods.

• Strong Python skills and experience building clean, maintainable software; Django, Flask, or similar Python HTTP server experience is useful.

• Experience with PyTorch or TensorFlow, language modeling, LLM infrastructure, and relevant deep learning techniques.

• Strong computer science fundamentals and the ability to turn research findings into practical systems.

• Experience collaborating through team development workflows, Git, code review, and feedback.

• A degree in Computer Science, Machine Learning, or a related field; a Master's or PhD is preferred. Candidates with a Bachelor's degree and 1 to 3 years of applied experience may also be considered.

Compensation & Benefits

Base salary: $150,000 to $200,000 annually. Visa sponsorship is available.

Location

On-site in San Francisco, California.