What are the responsibilities and job description for the Azure Data Engineer with Databricks Expertise position at Group Nine LLC?
Title : Azure Data Engineer with Databricks Expertise
Location : NYC, NY / Iselin, NJ [Hybrid]
Type : Long term contract
Job Description :
We are seeking highly skilled Azure Data Engineer with strong expertise in Databricks to join our data team. The ideal candidate will design, implement, and optimize large-scale data pipeline, ensuring scalability, reliability, and performance. This role involves working closely with multiple teams and business stakeholders to deliver cutting-edge data solutions.
Key Responsibilities :
Data Pipeline Development :
Build and maintain scalable ETL / ELT pipelines using Databricks.
Leverage PySpark / Spark and SQL to transform and process large datasets.
Integrate data from multiple sources including Azure Blob Storage, ADLS and other relational / non-relational systems.
Collaboration & Analysis :
Work Closely with multiple teams to prepare data for dashboard and BI Tools.
Collaborate with cross-functional teams to understand business requirements and deliver tailored data solutions.
Performance & Optimization :
Optimize Databricks workloads for cost efficiency and performance.
Monitor and troubleshoot data pipelines to ensure reliability and accuracy.
Governance & Security :
Implement and manage data security, access controls and governance standards using Unity Catalog.
Ensure compliance with organizational and regulatory data policies.
Deployment :
Leverage Databricks Asset Bundles for seamless deployment of Databricks jobs, notebooks, and configurations across environments.
Manage version control for Databricks artifacts and collaborate with team to maintain development best practices.
