Software Engineer - Pretraining Data

About Magic

Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone. Our approach combines frontier-scale pre-training, domain-specific RL, ultra-long context, and test-time compute to achieve this goal.

About the Role

As a Software Engineer working on our pretraining data, you will write efficient and robust pipelines for giant, multimodal datasets. You will develop and optimize web scraping techniques to harvest and maintain data at internet-scale.

Responsibilities

Design and implement multimodal (video, audio, text, etc.) web crawlers for scraping and indexing petabytes of data.
Create large-scale data processing pipelines using tools like Ray, Apache Spark, Apache Flink, Google BigQuery, etc.
Implement and scale deduplication techniques across modalities and apply heuristic and model-based techniques for parsing and filtering crawled data.
Identify new data sources for inclusion in pre/post-training datasets.

What We’re Looking For

Strong proficiency in distributed computing and parallel processing techniques.
Obsession with details, reliability, and good testing to ensure data quality and integrity.
Experience with designing and maintaining high-performance, scalable data architectures.
Ability to design, develop, and operate an LLM data pipeline from web scraping to data loading.

Our Culture

Integrity: Words and actions should be aligned.
Hands-on: At Magic, everyone is building.
Teamwork: We move as one team, not N individuals.
Focus: Safely deploy AGI. Everything else is noise.
Quality: Magic should feel like magic.

Compensation, Benefits, and Perks (US)

Annual salary range: $100K - $900K
Equity is a significant part of total compensation, in addition to salary.
401(k) plan with 6% salary matching.
Generous health, dental, and vision insurance for you and your dependents.
Unlimited paid time off.
Option to work in-person in SF or remotely.
Visa sponsorship and relocation stipend to bring you to SF.
A small, fast-paced, highly focused team.

Benefits
Extracted with AI

401(k)
Vision insurance
Generous health, dental and vision insurance
Unlimited paid time off
Visa sponsorship
Relocation stipend

Similar jobs

Last update: 23 minutes ago

yourfirm GmbH

Senior Fullstack Developer for AI-Driven Mission Technologies

Seeking a Senior Fullstack Developer for AI-driven mission technologies, focusing on Java, JavaScript, Python, and C++. Remote work available.

2 months ago

Senior C++ Computer Vision Engineer

Join a cutting-edge AI-DeepTech startup in Berlin as a Senior C++ Computer Vision Engineer. Work on world-class on-device AI technology.

2 months ago

Senior Backend Engineer C++

Join DeepL as a Senior Backend Engineer C++ to design and maintain scalable backend services using C++ and AI technologies.

2 months ago

Senior Full-Stack Engineer ReactJS/NodeJS

Join Gorgias as a Senior Full-Stack Engineer specializing in ReactJS and NodeJS, enhancing AI-powered ecommerce solutions.

2 months ago

Cloud Data Engineer

Seeking a Cloud Data Engineer with expertise in AWS, Python, and CI/CD for a hybrid role in Hannover. Join our dynamic team!

2 months ago

Salesforce Software Engineer

Join AnyDesk as a Salesforce Software Engineer to develop and maintain internal business systems in a dynamic, remote-friendly environment.

2 months ago

Expert Machine Learning Engineer

Join Dataroots as an Expert Machine Learning Engineer to design and deliver AI-powered solutions, focusing on machine learning models.

2 months ago

Staff Software Engineer

Join Aiven as a Staff Software Engineer to develop cloud operations platforms using open-source technologies. Hybrid work in Berlin.

2 months ago

Staff Software Engineer, Fullstack, Capacity & Efficiency Engineering

Join Uber as a Staff Software Engineer in Amsterdam, focusing on fullstack development and capacity efficiency engineering.

2 months ago

Senior Backend/Data Engineer

Join Zalando as a Senior Backend/Data Engineer in Berlin to enhance our audience-building platform using AWS, Java, Scala, and SQL.

2 months ago

Staff Software Engineer, Data Platform

Join Personio as a Staff Software Engineer in Berlin to build scalable data platforms using Kafka, Kubernetes, and AWS. Drive innovation and excellence.

2 months ago