Software Engineer, LLM Evaluation - English

Our client’s mission is to empower people, build community, and bring the world closer together. Through their apps and services, they are building a different kind of company that connects people worldwide and provides meaningful ways to share what matters most.

We are looking for a Software Engineer, LLM Evaluation, to join their team in the Netherlands remotely.

The successful candidate will work closely with researchers and engineers to evaluate model performance, conduct data-driven analyses, and contribute to research initiatives related to LLM pretraining and evaluation.

Job Profile for Software Engineer, LLM Evaluation
Responsibilities will include, but not be limited to:

• Analyse and evaluate large language models and their performance across various tasks and benchmarks
• Execute benchmark evaluations and generate performance metrics and insights
• Conduct quantitative and qualitative data analysis to support research objectives
• Contribute to the design, implementation, and validation of new evaluation methodologies
• Support research initiatives related to LLM pretraining and model evaluation
• Develop and maintain machine learning and deep learning systems and tooling
• Build research workflows and experimental frameworks using Python and PyTorch
• Enable rapid experimentation and support the execution of research initiatives
• Collaborate with researchers and engineers within a multidisciplinary team environment

Candidate Profile for Software Engineer, LLM Evaluation

• Must be fluent in English, both written and spoken
• Bachelor's degree in Computer Science, Artificial Intelligence, Machine Learning, or a related technical discipline. Master's degree in Computer Science, Machine Learning, AI, or a related discipline is desirable; a PhD in a relevant field would be considered a strong advantage
• 3–5 years of experience working with large language models, including pretraining and evaluation
• Strong hands-on programming experience in Python
• Proven experience building and maintaining ML/DL systems and research infrastructure
• Experience developing machine learning and deep learning solutions using PyTorch
• Publications in machine learning, natural language processing, artificial intelligence, or related fields are a plus
• Experience working with transformer architectures, large language models, and/or multimodal models is an advantage
• Ability to write clean, efficient, and production-quality code
• Strong interest in model evaluation, experimentation, and data analysis
• Demonstrated scientific curiosity and problem-solving skills

What Our Client Offers

• 25 holidays per annum
• Pension plan
• Opportunity to work alongside experienced researchers and engineers on cutting-edge AI initiatives
• Exposure to state-of-the-art methodologies and technologies within the LLM ecosystem
• Collaborative and research-driven environment in a technologically advanced office