What are the responsibilities and job description for the Sr Azure Data Engineer position at Zapcom?
Job Description
- Design and implement data ingestion, transformation, and movement using Azure Data Factory (ADF), Azure Synapse, and Data Lake.
- Collaborate with business stakeholders, data scientists, and engineers to build robust data solutions.
- Develop and manage high-volume, real-time, and batch data ingestion pipelines using Azure Data Factory (ADF).
- Implement event-driven architectures for real-time data movement and processing.
- Develop large-scale data processing solutions using Azure Databricks and Apache Spark with PySpark, Scala, or Python.
- Optimize data partitioning, caching, and indexing for efficient performance.
- Manage complex transformations and aggregations for structured and unstructured datasets.
- Design and implement high-performance data models in Azure Synapse Analytics using dedicated SQL Pools and Spark Pools.
- Optimize query performance, workload management, and cost efficiency in Synapse Analytics.
- Implement column store indexes, partitioning strategies, and data caching to enhance performance.
- Design and manage secure, scalable data lakes using Azure Data Lake Storage Gen2 (ADLS Gen2).
- Implement RBAC (Role-Based Access Control), encryption, and data masking to ensure security.
- Implement Azure Monitor, Log Analytics, and Application Insights for data pipeline monitoring and troubleshooting.
- Optimize cost management, auto-scaling, and performance tuning across Azure services.
Requirements
Requirements
8 years of experience in data engineering, ETL development, and cloud-based data integration, Strong experience with Azure Data Factory (ADF) : ETL / ELT pipeline orchestration and data movement. Azure Databricks : Large-scale data transformation and big data processing with Apache Spark, PySpark, Scala, or Python. Azure Synapse Analytics : Data warehousing, SQL Pools, Spark Pools, and performance tuning. Azure Data Lake Storage Gen2 (ADLS Gen2) : Secure, scalable data lake architecture. SQL Server : Advanced T-SQL, stored procedures, indexing, and query optimization. Power BI, Tableau, or other BI tools for data visualization. Financial / Banking experience.
