Job Title: Senior Azure Databricks Developer

Location: Chicago, IL - onsite from day one

Type: Full Time Position

Job Description:

Relevant Experience (in Yrs): 10 years

Experience required:

• Overall 10 years of experience as developer in Big data technologies like Databricks/Spark and Hadoop ecosystems
• Experience in developing Spark applications using spark-SQL in Databricks for data extraction, transformation and aggregation from multiple file formats for analyzing and transforming the data to uncover insights into the customer usage patterns
• Extract, transform and load from sources system to Azure Data Storage services using a combination of Azure Data factory, T-SQL, Pyspark and U-SQL Axure Data Lake Analytics. Data ingestion to one or more Azure services (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks
• Hands on experience on developing SQL scripts for automation
• Extensive hands-on experience implementing Lake house architecture using Data bricks Data Engineering platform, SQL Analytics, Delta Lake, and Unity Catalog
• Good understanding of spark architecture with Databricks structured streaming, setting up Azure with Databricks, managing clusters in Databricks
• Excellent interpersonal, verbal, and written communication skills
• Bachelor s degree or equivalent work experience

Roles and responsibilities:

• Responsible for estimating the cluster size, monitoring and troubleshooting of spark databrick cluster
• Drive analysis, architecture, design, governance and development of data warehouse, data lake, and business intelligence solutions
• Research, synthesize, recommend, and select technical approaches for solving difficult migration, development, and integration problems
• Conduct and support workshops, design sessions, and project meetings as needed, playing a key role in client relations
• Develop and optimize ETL processes using Databricks and related tools like Apache Spark
• Collaborate applications teams/Business users to develop new pipe lines with Cloud data migration methodologies and processes including tools like Azure Data Factory, Event Hub, etc
• Experience working in agile development projects and sprint deliver