What are the responsibilities and job description for the Lead Data Engineer position at Confidential Careers?
This role is for candidates who can work on W2
About the Company:
A large-scale project that would process 100s TB of ERP data and other relevant data. Auditors would use this processed information to opine on the client engagements and provide value-added services to their clients. The project currently uses Azure data technologies and Databricks.
About the Role:
We are looking for an experienced professional with at least 6 years of hands-on experience with Databricks technology.
Responsibilities:
- Design, build, and manage scalable data architectures using Databricks and Azure data technologies.
- System Design and implement modern data lakehouse solutions, ensuring seamless data integration, storage, and processing.
- Collaborate with cross-functional teams to gather and translate business requirements into effective technical designs.
- Ensure data quality and governance practices are implemented throughout the data lifecycle.
- Optimize data workflows for performance, reliability, and security using Azure one lake, Databricks Workflow and Databricks Compute, Unity catalog, Azure Synapse and Powerbi.
- Develop and enforce best practices in data modeling, pipeline design, and storage architecture.
- Conduct regular assessments of data systems to identify and address performance bottlenecks or areas for improvement.
Qualifications:
10 Years of total IT experience in Data engineering and Data warehouse project development.
Required Skills:
- 6 years of hands-on experience with Azure Databricks and expertise in PySpark & Python development.
- Proven expertise in designing and managing scalable data architectures. Experience in Databricks Serverless is plus.
- 6 years’ experience with Azure Synapse, Data Factory, and other Azure data technologies.
- 8 years’ experience in data modeling, 6 Years experience with Data pipelines design and implementation, and cloud storage architecture.
- Deep understanding of data quality and governance practices, and hands-on with Data quality and governance using Unity catalog or azure purview.
- Ability to collaborate with cross-functional teams and translate business requirements into technical designs.
Preferred Skills:
- Familiarity with other modern data processing tools and technologies.
- Previous experience in a similar role working with large-scale data systems.
