Senior Data Engineer (W2 Only)

Position: Senior Data Engineer

Location: New York City, NY (Hybrid)

Duration: 12 Months Possible Extension

Job Description:

• We are seeking a highly skilled Data Engineer / Data Architect with expertise in designing and implementing scalable data solutions on Azure Cloud.
• The ideal candidate will have a strong background in Azure Data Factory, Azure Databricks, Delta Lake, and Azure Blob Storage for large-scale data processing.
• This role requires proficiency in ETL/ELT pipeline development, data lake architecture, and machine learning model deployment using MLflow and Kubernetes.
• Additionally, the candidate should have experience in Power BI for data visualization and business insights, along with a proven track record of delivering high-impact data solutions for banking, financial services, and supply chain management.

Key Responsibilities:

1. Data Architecture & Engineering:
1. Design and implement scalable, high-performance data architectures on Azure Cloud.
2. Develop and maintain ETL/ELT pipelines using Azure Data Factory and Azure Databricks.
3. Architect and manage data lakes and Delta Lake storage for efficient data processing and analytics.
4. Optimize data storage solutions using Azure Blob Storage, Azure SQL, and Synapse Analytics.
2. Data Processing & Machine Learning Deployment:
1. Develop real-time and batch data processing solutions using Apache Spark on Azure Databricks.
2. Deploy and manage machine learning models using MLflow, Kubernetes, and Azure Machine Learning.
3. Implement data governance, security, and compliance best practices for data solutions.
3. Data Visualization & Business Insights:
1. Create interactive Power BI dashboards to provide actionable business insights.
2. Integrate data models, reports, and analytics for decision-making in banking, financial services, and supply chain management.
3. Collaborate with business stakeholders to define KPIs and data-driven strategies.
4. Performance Optimization & Automation:
1. Optimize query performance, data indexing, and partitioning strategies for large-scale data sets.
2. Automate data workflows and monitoring using Azure DevOps and CI/CD pipelines.
3. Implement best practices for data modeling, storage, and retrieval to improve system efficiency.

Required Skills & Qualifications:

Cloud & Data Technologies:

• Expertise in Azure Data Factory (ADF), Azure Databricks, Data Lake, and Azure Blob Storage.
• Strong knowledge of Azure Synapse Analytics, Azure SQL Database, and Cosmos DB.
• Hands-on experience with Apache Spark, Python, Scala, and SQL for data processing.

ETL/ELT & Data Pipelines:

• Proven experience in building end-to-end ETL/ELT data pipelines.
• Strong understanding of data lake architecture, structured & unstructured data management.

Data Visualization & BI:

• Proficiency in Power BI for creating dashboards, data models, and reports.
• Ability to integrate multiple data sources for in-depth analytics.

Thanks & Regards

Krishna Chalasani

Account Manager

E-mail: krishna(@)icsglobalsoft.com