Demo

Databricks Lead Data Engineer

Aegis Security Insurance
San Diego, CA Full Time
POSTED ON 2/15/2025
AVAILABLE BEFORE 5/7/2025

Databricks Lead Data Engineer

Full Time

Remote - San Diego, CA, US

Salary Range : $200,000 To $260,000 Annually

Databricks Lead Data Engineer - K2 Insurance Services

K2 Insurance Services, LLC seeks a full-time Lead Databricks Data Engineer to spearhead our data engineering initiatives, focusing on designing, building, and optimizing scalable data solutions using Databricks. As a leader, you will supervise a team of data engineers, collaborate with cross-functional stakeholders, and define best practices to unlock the full potential of our data.

K2 Insurance Services offers the opportunity to join an established company in growth mode. Our compensation program includes competitive pay; bonus plan; medical, dental, and vision insurance; paid time-off in year of hire; and 401(k) with employer match.

Salary Range : $200,000 To $260,000 USD per year

Key Responsibilities :

  • Leadership & Strategy :

Supervise and mentor a team of data engineers in implementing Databricks-based ETL / ELT solutions.

  • Define and drive the data engineering strategy, ensuring alignment with business goals.
  • IT Product Owner role for organizational data warehouse project.
  • Databricks Expertise :
  • Design and develop scalable data pipelines and ETL / ELT processes using Databricks.

  • Optimize and tune Spark jobs for performance and efficiency.
  • Develop and enforce best practices for Databricks cluster management and data security.
  • Data Architecture :
  • Build and maintain robust data models to support analytics and reporting needs.

  • Integrate Databricks with various data sources (cloud storage, databases, APIs).
  • Implement Delta Lake for reliable, scalable, and performant data lakes.
  • Collaboration :
  • Work closely with data scientists, analysts, and stakeholders to deliver actionable insights.

  • Act as the technical liaison between the data engineering team and other stakeholders.
  • Innovation & Optimization :
  • Stay updated on the latest Databricks and Azure features to drive continuous improvements.

  • Automate repetitive processes and streamline data workflows.
  • Qualifications :

  • Proven experience (10 years) in data engineering, with 2 years as a team supervisor or equivalent role.
  • Expertise in Databricks, including Spark, Delta Lake, Delta Live Tables, and MLFlow.
  • Proficiency in Python, SQL, and at least one cloud platform (Azure, AWS, or GCP).
  • Hands-on experience with data lakes, data warehouses, and big data technologies.
  • Strong understanding of CI / CD pipelines and infrastructure as code (IaC).
  • Excellent problem-solving and communication skills.
  • Prior insurance industry experience is a strong plus.
  • Preferred Skills :

  • Familiarity with Databricks Unity Catalog and Lakehouse architecture.
  • Experience with workflow orchestration tools.
  • Knowledge of machine learning pipelines and integrations.
  • Certifications in Databricks, Azure, AWS, or GCP.
  • Previous IT product owner role.
  • Salary : $200,000 - $260,000

    If your compensation planning software is too rigid to deploy winning incentive strategies, it’s time to find an adaptable solution. Compensation Planning
    Enhance your organization's compensation strategy with salary data sets that HR and team managers can use to pay your staff right. Surveys & Data Sets

    What is the career path for a Databricks Lead Data Engineer?

    Sign up to receive alerts about other jobs on the Databricks Lead Data Engineer career path by checking the boxes next to the positions that interest you.
    Income Estimation: 
    $92,929 - $122,443
    Income Estimation: 
    $122,257 - $154,284
    Income Estimation: 
    $92,929 - $122,443
    Income Estimation: 
    $122,257 - $154,284
    Income Estimation: 
    $71,122 - $96,652
    Income Estimation: 
    $92,929 - $122,443
    Income Estimation: 
    $122,257 - $154,284
    Income Estimation: 
    $143,391 - $179,890
    View Core, Job Family, and Industry Job Skills and Competency Data for more than 15,000 Job Titles Skills Library

    Job openings at Aegis Security Insurance

    Aegis Security Insurance
    Hired Organization Address San Diego, CA Full Time
    Software Engineer I Founded in 1965, Vikco Insurance Services LLC ("Vikco"), a subsidiary of K2 Insurance Services, LLC,...
    Aegis Security Insurance
    Hired Organization Address Harrisburg, PA Full Time
    Property Claims Examiner K2 Claims Services, LLC (K2 Claims) is seeking Property Claims Examiner to join its Claims Team...

    Not the job you're looking for? Here are some other Databricks Lead Data Engineer jobs in the San Diego, CA area that may be a better fit.

    Databricks Lead Data Engineer

    K2 Insurance Services, LLC, Poway, CA

    Databricks Lead Data Engineer

    K2 Insurance Services, LLC, Bonita, CA

    AI Assistant is available now!

    Feel free to start your new journey!