Lead Data Engineer

Type: Full-time, Direct Hire

Location: 100% Remote

Salary: $120k - $145k 10% bonus

Applicants must be authorized to work for any employer in the US. We're unable to sponsor or take over sponsorship of employment visas at this time. Please note that we cannot work C2C or with 3rd Party Vendors.

Are you a hands-on expert in designing, developing, and optimizing scalable data pipelines and systems with Databricks and Spark? Do you thrive in a leadership role while staying actively involved in technical problem-solving and development? If so, we'd love to hear from you!

We're seeking a Lead Data Engineer to join our machine-learning engineering team and contribute both technical expertise and leadership skills. In this role, you'll lead a team of data engineers, mentor team members, and drive projects, while remaining deeply involved in hands-on development to build scalable, secure, and efficient data solutions. You'll collaborate closely with machine-learning engineers, software engineers, data scientists, and data analysts to deliver high-impact data products for our clients and enhance our internal data capabilities.

What you'll do

• Lead and mentor a team of data engineers, fostering collaboration, innovation, and adherence to best practices in data engineering and development.


• Stay actively engaged in hands-on development, designing, building, and maintaining data pipelines, systems, and infrastructure using technologies like Databricks, Spark, AWS, Python, and SQL databases (e.g., PostgreSQL).


• Develop and implement scalable ETL/ELT processes, ensuring the efficient ingestion, transformation, and delivery of data to support business and analytics needs.


• Provide technical leadership to your team, guiding architecture and design decisions, and ensuring high-quality code through reviews and mentorship.


• Collaborate with cross-functional teams to translate business problems into data-driven technical solutions.


• Monitor and troubleshoot data pipelines and systems to ensure optimal performance, data quality, and availability.


• Research emerging data technologies and trends to evaluate tools and enhance engineering practices.


• Manage the software development lifecycle (SDLC) and DevOps workflows, including CI/CD pipelines, version control (Git), and containerization tools like Docker.


• Define and enforce data governance, security, and privacy policies.


• Build and maintain data lakes and storage solutions to support advanced analytics and business intelligence

What will make you a good fit

• Bachelor's degree in computer science, information technology, or a related field (Master's degree preferred).


• 6 years of experience in data engineering, including at least 2 years in a leadership role.


• Proven hands-on expertise with Databricks and Spark for building and optimizing data pipelines and processing large datasets.


• Extensive experience with cloud-based data engineering (preferably AWS), Python, and SQL.


• Strong ability to balance hands-on development and leadership responsibilities, staying involved in technical tasks while guiding the team.


• Demonstrated experience in team mentorship and collaboration, including managing technical projects and resolving complex challenges.


• Strong knowledge of data governance, security, and retention best practices.


• Familiarity with agile development methodologies and tools like Jira, and experience implementing DevOps practices (e.g., CI/CD, Jenkins, Docker).


• Exceptional problem-solving and communication skills

Preferred qualifications:

• Databricks certification (e.g., Databricks Certified Data Engineer Associate or Professional).


• Expertise in optimizing ETL processes and designing efficient workflows in Databricks.


• Knowledge of data lineage and data quality frameworks.


• Familiarity with data science and machine-learning workflows and their data requirements.


• Industry experience in the energy or utility sector.


• Master's degree in a STEM field.

All qualified applicants will receive consideration for employment without regard to race, color, national origin, age, ancestry, religion, sex, sexual orientation, gender identity, gender expression, marital status, disability, medical condition, genetic information, pregnancy, or military or veteran status. We consider all qualified applicants, including those with criminal histories, in a manner consistent with state and local laws, including the California Fair Chance Act, City of Los Angeles' Fair Chance Initiative for Hiring Ordinance, and Los Angeles County Fair Chance Ordinance. For unincorporated Los Angeles county, to the extent our customers require a background check for certain positions, the Company faces a significant risk to its business operations and business reputation unless a review of criminal history is conducted for those specific job positions.