Senior Data Platform Engineer

About the Role

Stability AI is seeking a Senior Data Platform Engineer to enhance and expand our data platform using AWS and GCP services. This role involves understanding our needs, selecting the appropriate tech stack and tools, gaining buy-in from stakeholders, and choosing the best long-term solutions. As a technical thought leader, you will help hire and mentor other members of the team.

Responsibilities

Architect, implement, and monitor data pipelines.
Conduct knowledge-sharing sessions with peers to ensure effective data use.
Work with team members and leaders to further develop our data strategy.
Develop and maintain a data catalog for a comprehensive understanding of the data and events within our platform.
Maintain infrastructure-as-code (AWS CDK) provisioning, spin up new and/or revised data platform resources and environments to enhance data collection and processing from our applications and data sources.
Act as the primary technical decision-maker for the data platform and help hire and support analytical data personnel (e.g., Data Scientist, Data Analyst, BI Developer).

Technical Experience

Containerized compute: Kubernetes (EKS, GKS), ECS, Docker
Data ingestion: Airbyte, Mage AI, AWS Glue
Pipeline orchestration: Airflow, Mage AI
Data manipulation: Pandas, Numpy, Polars
Data lake: S3, Iceberg, Delta Lake
Data warehouse: BigQuery, Redshift
Database: PostgreSQL
Data visualization: Looker, Superset/Preset
Data quality (optional): Great Expectations, Monte Carlo
Data catalog (optional): Amundsen, AWS Glue Catalog
Event streaming (optional): Kafka, Flink, Redis, Pub/Sub
Infrastructure-as-code: AWS CDK, Terraform
CI/CD: GitHub Actions

Qualifications

Must have 7+ years of hands-on experience working with data platforms.
Experience being involved in the design and implementation of at least one data platform.
Experience in streaming and/or batch analytics.
Experience with scalable distributed data processing, management, and visualization tools.
Experience using business intelligence tools and data frameworks.
Strong working knowledge of Python for both provisioning cloud infrastructure and working with data.

Compensation

The salary range for this role is between $130,000 and $190,000. Individual pay within the range is based on factors like job-related skills and experience. Total compensation also includes stock options and benefits.

Equal Employment Opportunity

We are an equal opportunity employer and do not discriminate on the basis of race, religion, national origin, gender, sexual orientation, age, veteran status, disability or other legally protected statuses.

Benefits
Extracted with AI

Remote work flexibility
Stock options

Similar jobs

Last update: 23 minutes ago

Stability AI

Senior Data Engineer

Join Stability AI as a Senior Data Engineer to build scalable data infrastructure for AI models. Remote work from Germany.

About the Role

Responsibilities

Technical Experience

Qualifications

Compensation

Equal Employment Opportunity

Benefits Extracted with AI

Similar jobs

Senior Data Engineer

Remote Data Engineer - Research

Senior Backend Engineer (AI)

Lead Architect - Gen AI API Platform