Summary

Looking for a skilled Senior Azure Databricks Engineer. Data Lead at customer site, this role requires a combination of technical expertise in data processing in Databricks on Azure ecosystem and consulting attitude for delivering successful data projects. If you are passionate about turning data into actionable insights and have a knack for optimizing data systems, read on for the qualifications and responsibilities.

Responsibilities

• Lead, design and develop solutions in team setting consisting of Business users, data owners, data engineers, BI and ML engineers
• Develop and maintain scalable data pipelines within Azure Databricks
• Design and implement data modeling, data warehousing, and data lakes using Azure services
• Maintain and evolve schema fulfill BI reporting and ML needs
• Write SQL queries and scripts for data manipulation and retrieval
• Ensure data quality and consistency across Ziosk and other data sources
• Optimize data processing performance within Azure Databricks
• Collaborate with cross-functional teams to gather and understand data requirements
• Experience in PowerBI for creating insightful dashboards and interactive reports will be a plus.

Qualifications

• Proficient in Databricks Notebook, PySpark, Python, SQL
• Experience with Medallion Architecture, Delta Lakes, Databricks Unity Catalog, Data Transformation, Streaming data processing and Data Warehouse.
• Knowledge of Apache Spark and its core components
• Experience with Azure services like ADF
• Experience with Spark DataFrame and Spark Streaming
• Ability to write and optimize Spark SQL queries
• In-depth knowledge of SQL, including Data Query Language (DQL), Data Manipulation Language (DML), and Data Definition Language (DDL)
• Experience with SQL Server database management
• Excellent Communication and Consulting Skills. Must be comfortable working with Business and technical holders. Should have demonstrable experience consulting with stakeholders and understand and develop requirement

Good To Have (Huge )

• Experience with Azure DevOps
• Azure Bicep experience is a plus
• Expertise in Python with C#, R or Scala experience as an added advantage
• Proficiency in PowerBI
• Experience in creating robust dashboards and reports
• Proficiency in DAX (Data Analysis Expressions)
• Skilled in PowerQuery for data transformation
• Background in data visualization techniques and tools

Education

• Bachelor s degree in Computer Science, Engineering, or a related field