Software Engineer, Machine Learning Infrastructure

Job Overview

As a Machine Learning Software Engineer within the Dojo team at Tesla, you will play a pivotal role in enhancing the capabilities of our cutting-edge Dojo training accelerator. This position involves close collaboration with top-tier ML Researchers, Compiler, and Hardware Engineers to address unique challenges at the intersection of AI and ML training accelerators. Your expertise will be crucial in optimizing and scaling our neural network training infrastructure.

Key Responsibilities

• Collaborate with machine learning researchers and engineers to run Full Self-Driving (FSD) models on our in-house ML training accelerator.
• Profile performance of training workloads in our cluster, identify bottlenecks in and between CPU/Dojo code execution, and work on optimizing throughput and scalability within and across nodes to reduce convergence time.
• Coordinate with the team managing the hardware cluster to maintain high availability and job throughput for machine learning tasks.
• Integrate the training software into our continuous integration cluster to support metrics persistence across experiments, weekly/nightly neural network builds, and other unit/throughput tests.

Required Skills and Experience

• Degree in Engineering, Computer Science, or equivalent experience with evidence of exceptional ability.
• Practical experience programming in Python and/or C++.
• Experience working with training frameworks, ideally PyTorch.
• Proficient in system-level software, particularly hardware-software interactions and resource utilization.
• Understanding of modern machine learning concepts and state-of-the-art deep learning.
• Experience in profiling and optimizing CPU-accelerator interactions (pipelining compute/transfers, etc.).
• DevOps experience, particularly in managing clusters of training nodes and filesystems for large amounts of training data.

Benefits

As a full-time Tesla employee, you are eligible for a comprehensive benefits package starting from day one, including:

• Aetna PPO and HSA plans with $0 payroll deduction options.
• Family-building, fertility, adoption, and surrogacy benefits.
• Dental and vision plans with options for $0 paycheck contribution.
• Company-paid HSA contributions, life, AD&D, short-term, and long-term disability insurance.
• 401(k) with employer match, Employee Stock Purchase Plans, and other financial benefits.
• Employee Assistance Program, sick and vacation time, and paid holidays.
• Back-up childcare and parenting support resources.
• Voluntary benefits including critical illness, hospital indemnity, accident insurance, theft & legal services, and pet insurance.
• Weight Loss and Tobacco Cessation Programs, Tesla Babies program, commuter benefits, and employee discounts and perks.

Compensation

The expected annual salary for this position ranges from €104,000 to €360,000, depending on experience and level, plus cash and stock awards and benefits. The total compensation package may vary based on market location, job-related knowledge, skills, and experience.

Join us at Tesla and contribute to building a sustainable future by enhancing our machine learning infrastructure and accelerating the world's transition to sustainable energy.