DataHub Developer

Position Overview

We are looking for an experienced DataHub Developer with Committer Experience to join our team and contribute to the design, development, and optimization of enterprise metadata management and data lineage solutions. The ideal candidate will have strong expertise in data cataloging , data lineage , data governance , and hands-on experience with DataHub , Spark-based frameworks , and machine learning for anomaly detection. This role demands a mix of open-source contribution, technical problem-solving, and metadata management expertise.

Key Responsibilities

• DataHub Development and Integration
• Lead projects involving metadata cataloging using the DataHub open-source framework.
• Design and develop custom APIs to integrate ETL pipelines and enable real-time metadata ingestion.
• Ingest metadata from multiple systems, including data lakes, upstream, and downstream systems, to provide a holistic metadata ecosystem.
• Customize and extend DataHub to enrich impact analysis by identifying pipelines reading / writing to data assets.
• Data Lineage and Governance Implementation
• Provide end-to-end data lineage solutions for PII identification, governance, and compliance reporting.
• Develop and implement processes to enhance impact analysis and ensure seamless data governance practices.
• Spark-Based Framework Development
• Design, develop, and maintain Spark-based custom frameworks for config-as-code mechanisms to facilitate data enrichment and transfer.
• Improve the performance and scalability of Spark applications to ensure seamless data processing.
• Provide recommendations and guidance on the design and development of ETL pipelines using Spark.
• Machine Learning Integration for Anomaly Detection
• Collaborate with ML engineers to create features from profiled batch data.
• Develop and integrate machine learning models for anomaly detection in data patterns.
• AWS Cost Optimization and Platform Efficiency
• Lead AWS cost optimization initiatives to enhance platform-wide efficiency.
• Successfully support Spark version upgrades and ensure the platform's scalability and performance.
• Community Engagement and Contributions
• Act as a committer to the DataHub open-source community by contributing new features, fixing issues, and enhancing documentation.
• Participate in open-source discussions, propose architectural improvements, and represent the organization in community events.

Required Qualifications

Preferred Qualifications