Datahub Consultant

Role: Datahub Consultant

Location: Austin, TX (Onsite)

Job Description:

• Directed projects involving data cataloging using the Datahub open-source framework, anomaly detection through machine learning models, and Spark based framework.
• Ingested metadata from multiple systems to pull metadata information of assets from data lake, upstream and downstream systems.
• Developed custom API solutions that can bring data of ETL pipelines as a push mechanism to Datahub. This enriched the impact analysis to identify the data pipelines reading/writing to a data asset.
• Provided a holistic picture of end-to-end lineage that helped with PII identification, Governance, Impact analysis.
• Improved the performance of Spark-based applications, ensuring seamless functionality.
• Provided recommendations on design and development of ETL pipelines using Spark. Developed and maintained Spark based client custom framework used for config-as-code mechanism of data enrichment and transfer.
• Successfully supported Spark version upgrades and executed AWS cost optimization initiatives for platform-wide efficiency.
• Worked with ML engineers to create features from profiled batch data; and identify anomalies in data patterns.