What are the responsibilities and job description for the Lead Data Engineer position at Incedo Inc.?
Lead Data Engineer- data vault
Job Location: Atlanta, GA/Encino, CA (Hybrid)
Role Overview:
We are seeking a Lead Data Engineer with expertise in data modeling, Data Vault architecture, and cloud data warehouse (CDW) architecture utilizing the Kimball model. The ideal candidate will have hands-on experience with data fabric technologies (e.g., Databricks), Spark, and Python on Azure Cloud. This role involves designing, implementing, and optimizing data pipelines and data platforms to support business intelligence and advanced analytics.
Key Responsibilities:
- Design and implement scalable and efficient data models, including Data Vault and Kimball dimensional modeling for cloud-based data warehouses.
- Architect and manage Azure Cloud-based data infrastructure, ensuring scalability, security, and performance.
- Develop and optimize data pipelines using Databricks, Spark, and other data fabric tools to process large volumes of data.
- Collaborate with data scientists, analysts, and business stakeholders to understand data needs and deliver high-quality solutions.
- Implement best practices for data engineering, including data quality, governance, and lifecycle management.
- Build and maintain infrastructure-as-code using Terraform or other DevOps tools.
- Create reusable data engineering components for ingestion, transformation, and orchestration.
- Troubleshoot, optimize, and maintain existing data systems to meet evolving business needs.
Required Skills and Experience:
- 10 years of experience in Data Engineering or a similar role, with a proven track record of leading data projects.
- Expertise in Data Vault 2.0 architecture and Kimball modeling for data warehouse design.
- Proficiency with Azure Data Services (e.g., Azure Data Factory, Azure Synapse, Azure Databricks).
- Hands-on experience with Databricks and Spark for distributed data processing.
- Strong programming skills in Python for ETL/ELT development and automation.
- Deep understanding of CDW architecture and data fabric frameworks.
- Experience with CI/CD pipelines and infrastructure-as-code tools like Terraform.
- Solid knowledge of data governance, security, and compliance best practices.
Educational Qualifications:
- Bachelor’s degree in Computer Science, Data Engineering, or related fields. Master’s degree preferred.
- Certifications in Data Engineering, Azure Cloud, or Databricks are highly desirable.
