Data Quality Lead Engineer

Role: Data Quality Lead Engineer

DQ lead with strong Healthcare background and Databricks experience,
Data Quality Lead will oversee and improve the quality of our data assets. The ideal candidate will have a strong background in data engineering, data lakes, and data quality implementation. They will also be proficient in monitoring and improving the overall health of data and have experience with Databricks and Azure. This role involves creating and reporting data quality progress, working with business teams, and ensuring data quality at the source.
Key Responsibilities:
1. Data Quality Management:
- Develop and implement data quality strategies, policies, and procedures.
- Monitor data quality metrics and establish processes to improve data integrity, accuracy, and completeness.
Implement data quality tools and solutions to automate data quality checks and validation processes.
2. Data Engineering & Data Lake:
Collaborate with data engineering teams to design, implement, and maintain data quality pipelines within data lakes.
- Ensure data governance and compliance standards are adhered to across all data engineering projects.
Optimize data storage and retrieval processes for efficiency and performance.
3. Data Quality Implementation:
- Lead data quality initiatives and projects, ensuring timely and successful delivery.
- Conduct data quality assessments and audits to identify and resolve data quality issues.
Work closely with business and IT stakeholders to understand data quality requirements and develop solutions to meet their needs.
4. Monitoring & Improvement:
- Continuously monitor data quality metrics and implement corrective actions as needed.
Develop and maintain data quality dashboards and reports to track progress and communicate findings to stakeholders.
- Conduct root cause analysis of data quality issues and implement preventive measures.
5. Experience with Databricks & Azure:
- Utilize Databricks for data processing, analysis, and quality assurance.
- Leverage Azure services for data storage, management, and integration.
- Implement and manage data quality solutions using Databricks and Azure tools.

6. Collaboration & Mitigation:
Work with business teams and upstream application teams to mitigate data quality issues at the source. Collaborate with stakeholders to define and implement new data quality (DQ) business rules.
Build exception handling and inline data quality processes within Databricks for the entire data lifecycle.
7. Data Profiling & Rule Definition:
Perform data profiling to identify data quality issues and opportunities for improvement.
Work with business stakeholders to define and implement data quality rules and standards.
Ensure data quality rules are integrated into data processing workflows and monitored for compliance.
8. Issue Management:
Implement a robust issue management process to track and remediate data quality-related issues.
Identify data quality issues proactively by understanding different consumption patterns and implementing checks and balances. Monitor data quality and publish regular reports on data health.
9. Leadership & Collaboration:
Lead analysts and the daily monitoring team to identify data anomalies.
Work closely with domain leads and business data stewards in the remediation process.
Foster a culture of data quality awareness and continuous improvement.
10. Reporting & Communication:
Create and present data quality reports to senior management and stakeholders.
Communicate data quality progress, challenges, and solutions effectively to both technical and non-technical audiences. Collaborate with cross-functional teams to ensure data quality standards are met.

Qualifications:
Minimum of 5 years of experience in data engineering, data quality management, and data lake implementation. Proven experience with Databricks and Azure.
Strong analytical and problem-solving skills.
Excellent communication and presentation skills.
Ability to work effectively in a fast-paced, dynamic environment.
Knowledge of data governance and compliance standards.

Preferred Skills:
Experience with other data quality tools and technologies. Certification in data management or data quality. Familiarity with machine learning and AI techniques.