Senior Data Engineer


At the heart of the CircleCI product offerings is the flow of data. You will be the key person that creates robust data pipelines that are high volume, sometimes extremely sensitive to completeness and often require integration with sources that require programming skills. Comfortable with databases, AWS, Python, cmd line, product management, engineering, and marketing, this is a role that will require cross-functional skills and cross-department communication. Expect to spend time designing and architecting next-generation solutions to solve engaging challenges as CircleCI usership grows and products evolve at a terrific pace.

About CircleCI

Velocity is critical for software teams in today's competitive landscape, but maintaining speed can be difficult as apps and systems grow larger and more complex. CircleCI’s platform allows developers to rapidly release code (for web and mobile apps) they trust by automating the build, test, and deploy process. CircleCI enables developers to detect and fix bugs before they even reach customers. Thousands of leading companies including Facebook, Kickstarter, Shyp and Spotify rely on CircleCI to accelerate delivery of their code and enable developers to focus on creating business value fast.

CircleCI is a Bay Area Best Places to Work 2016 award winner. Founded in 2011 and headquartered in beautiful downtown San Francisco with a global remote workforce, CircleCI is venture backed by Scale Venture Partners, DFJ, Baseline Ventures and Harrison Metal Capital.

About Data at CircleCI

The Data team is growing from analysis, projections and research to data as a product. Today, the Data team works closely with Product Management, Engineering, Marketing, and User Experience to create actionable insights from the millions of builds performed every week on CircleCI. The Data team is critical to CircleCI success. As a result, the team supports highly cross-functional initiatives that rely heavily on data and user-research to drive decision making and deliver features or improvements as a continuous flow. We are looking for a smart, passionate person to help create the backbone of our long term data strategy.

Joining the Team

After introductory discussions with recruiting and a member of the technical staff, you will work through a brief data/programming assignment that we use during an on-site interview to get to know more about each other. This helps you and CircleCI ensure that our approach to problems and skills are a good fit.


  • Establish, build, and maintain a data management platform and ETL pipeline
  • Automate big data flow and processing
  • Maintain the data warehouse and data architecture
  • Connect to new APIs, new data sources and build new data pipelines
  • Design and deploy visualization/insights sharing applications for internal customers.
  • Work within engineering teams to integrate new capability into analytics infrastructure

Essential Qualifications:

  • 5+ years experience in Data Engineering role
  • Prefers automation to being a hero
  • Ability to design and implement robust data pipelines and work in a team environment on a production codebase using version control system
  • Strong familiarity with data warehousing concepts - dimensional data warehousing (e.g. Kimball Group), star and snowflake schema, etc.
  • Comfortable with data infrastructure concepts (e.g. MPP databases, CAP theorem) and platforms/providers (e.g. AWS, DigitalOcean, GCP)
  • Experience building pipelines that run within cloud-based environments
  • Proficiency in Python (or similar programming language) for transforming data
  • Intermediate understanding of Linux/DockerDemonstrated database schema and design experience
  • Experience with machine learning including natural language processing and behavioral analysis

Preferred Qualifications:

  • Experience with the following technologies: PostgreSQL, MongoDB, Docker, Spark, Hive
  • Advanced knowledge and experience with AWS
  • Advanced knowledge of Linux/Docker in deployments
  • Motivated to learn and empower others, and when meeting obstacles, not afraid to test new concepts or ask for guidance

If you’re interested in joining the team at CircleCI, please send a resumé and cover letter describing what inspires you and why you think you are a good fit for the team. If you contribute to an open source project, write a blog, or have a presence on the web (Twitter, GitHub, LinkedIn, etc.) we would love to hear about it.

We care deeply about diversity and inclusivity. We’re hiring at all experience levels, and seek talented teammates from a wide variety of backgrounds and experiences who are equally committed to cultivating a work environment of respect and kindness. We carefully consider every applicant that takes the time to apply.

View this job on