Databricks Engineer with Azure Experience

Location: New York City, NY or Jersey City, NJ (Local candidates only)

No H1

Position Type: Contract (Multiple Openings)

Interviews: Web and onsite (Face to Face)

Job Description

We are looking for experienced Databricks Engineers with extensive Azure cloud experience to join our dynamic team. This role will be instrumental in helping drive data initiatives within our organization by leveraging the powerful capabilities of Databricks on the Azure platform. If you’re passionate about data, analytics, and cloud solutions, this is the perfect opportunity to work with cutting-edge technologies in a fast-paced environment.

Key Responsibilities

• Design, develop, and maintain Databricks solutions on the Azure cloud platform to enable advanced data processing, analytics, and insights.
• Collaborate with data scientists, data engineers, and business teams to ensure that Databricks-based solutions meet both business and technical needs.
• Work with large-scale data lakes and data warehouses to design optimal data pipelines and workflows in Databricks.
• Implement ETL/ELT pipelines using Databricks, Azure Data Factory (ADF), and other related tools to transform raw data into actionable insights.
• Optimize the performance and scalability of data solutions, ensuring high availability and fault tolerance for mission-critical applications.
• Integrate Databricks with various Azure services, such as Azure Synapse Analytics, Azure SQL, Azure Data Lake Storage, and Azure Blob Storage to deliver robust data solutions.
• Design and implement security and governance measures to ensure data privacy, compliance, and protection across the Databricks and Azure ecosystem.
• Provide support for data-related issues, debug production issues, and ensure smooth day-to-day operations of the Databricks environment.
• Work with teams to implement CI/CD pipelines for Databricks and Azure resources, facilitating continuous integration and continuous deployment of code and data solutions.
• Stay up to date with the latest Databricks and Azure advancements, tools, and best practices, incorporating them into projects where applicable.
  
  

Required Qualifications

• 5 years of experience in data engineering, with a strong focus on Databricks and Azure cloud services.
• Expertise in building and managing ETL/ELT pipelines in Databricks, with deep knowledge of Apache Spark and its underlying architecture.
• Proficiency in Azure Data Factory (ADF), Azure Data Lake, and other Azure data services.
• Strong programming skills in Python, Scala, or SQL within the Databricks environment.
• Experience with data modeling and designing large-scale data architectures that include data lakes and warehouses.
• Knowledge of data governance, security, and privacy practices within Azure and Databricks environments.
• Strong problem-solving skills, with the ability to troubleshoot data and performance issues within Databricks and Azure.
• Experience with CI/CD tools such as Azure DevOps, Git, or other related platforms to automate the deployment of data solutions.
• Excellent communication skills and the ability to work in a fast-paced, collaborative environment.
• Experience working with streaming data solutions using Azure Stream Analytics, Event Hubs, or Databricks Structured Streaming.
• Familiarity with Power BI or other visualization tools to build real-time dashboards on top of Databricks solutions.
• Certifications in Azure or Databricks technologies.
• Experience working in financial services or highly regulated industries.