Azure Databricks Lead/Architect

Title: Azure Databricks Lead/Architect

Location: Chicago, IL - Day 1 onsite & Hybrid Model)

Duration: Long Term

Mandatory skills

Required Skills & Experience:

• 12 years of professional experience in Data Engineering, Architecture, or Data Platforms.
• 5 years of hands-on experience with Azure Databricks, Apache Spark, and PySpark.
• Deep expertise in ETL design, data modelling (Data Vault, Star Schema, Medallion architecture), and query optimization.
• Strong experience with SQL, Python, and Data Pipeline Orchestration.
• Hands-on experience in CI/CD practices, GitHub Actions, and version control tools.
• Expertise in Azure ecosystem (Azure Data Lake Storage, Azure Data Factory, Synapse Analytics, Event Hub, Kafka).
• Knowledge of structured streaming and real-time data processing.
• Experience implementing security and governance best practices for cloud-based data platforms.

Responsibilities:

• Architecture & Strategy:
• Design and develop scalable Lakehouse architectures that enable high-performance data ingestion, processing, and analytics.
• Define and enforce best practices for ETL, data modeling, and data governance within the Azure ecosystem.
• Architect and optimize Medallion Architecture (Bronze, Silver, Gold) for efficient data transformation and analytics.
• Ensure high availability, security, and performance of Azure Databricks environments.
• Define data infrastructure strategies, ensuring alignment with business objectives and scalability requirements.
• Technical Leadership & Development:
• Provide architectural guidance and hands-on development support for ETL pipelines, batch & streaming solutions in Azure Databricks using PySpark and SQL.
• Drive the implementation of Data Vault, Star Schema, and Data Warehouse modeling techniques.
• Implement CI/CD pipelines, version control, and automation to streamline deployment and testing.
• Lead the integration of Azure Data Services such as Azure Data Factory, Azure Synapse, Delta Lake, and Delta Live Tables.
• Apply Chaos Engineering practices to ensure system resilience and fault tolerance.
• Collaboration & Mentorship:
• Work closely with Product Owners, Data Engineers, Data Scientists, and Business Analysts to deliver high-quality data solutions.
• Provide technical mentorship to engineering teams, enabling them to follow best practices in Databricks development, ETL, and data pipeline management.
• Collaborate across departments to design functional and scalable solutions that balance business needs and technical feasibility.
• Analytics & Optimization:
• Conduct performance tuning and optimization of Databricks workloads, Spark jobs, and SQL queries.
• Define and implement monitoring frameworks to track data quality, lineage, and processing performance.
• Design efficient data governance, access control, and security policies for Databricks and Azure environments.
• Lead root-cause analysis and problem-solving for complex data engineering challenges.

Preferred Skills (Bonus Points):

• Experience with Snowflake and BI Tools (Power BI, Tableau).
• Exposure to Graph Databases and Knowledge Graphs.
• Familiarity with Machine Learning or AI-based data solutions in Databricks.
• Experience in Python library development for reusable data processing functions.