What are the responsibilities and job description for the Senior Data Engineer-New York City, NY W2 position at ICS Global Soft, Inc.?
Job Details
Role: Senior Data Engineer
Location New York City, NY 10007 USA Day 1st onsite
Years of experience 8 years minimum 13 years maximum
Number of Internal interviews 2
Client Interview (Y/N): Most probably NO
Mode of client interview: In-person for local candidates, virtual for outer state
Tentative start date ASAP
Key Responsibilities:
- Data Architecture & Engineering: Design and implement scalable, high-performance data architectures on Azure Cloud. Develop and maintain ETL/ELT pipelines using Azure Data Factory and Azure Databricks. Architect and manage data lakes and Delta Lake storage for efficient data processing and analytics. Optimize data storage solutions using Azure Blob Storage, Azure SQL, and Synapse Analytics.
- Data Processing & Machine Learning Deployment: Develop real-time and batch data processing solutions using Apache Spark on Azure Databricks. Deploy and manage machine learning models using MLflow, Kubernetes, and Azure Machine Learning. Implement data governance, security, and compliance best practices for data solutions.
- Data Visualization & Business Insights: Create interactive Power BI dashboards to provide actionable business insights. Integrate data models, reports, and analytics for decision-making in banking, financial services, and supply chain management. Collaborate with business stakeholders to define KPIs and data-driven strategies.
- Performance Optimization & Automation: Optimize query performance, data indexing, and partitioning strategies for large-scale data sets. Automate data workflows and monitoring using Azure DevOps and CI/CD pipelines. Implement best practices for data modeling, storage, and retrieval to improve system efficiency.
Required Skills & Qualifications:
Cloud & Data Technologies:
- Expertise in Azure Data Factory (ADF), Azure Databricks, Data Lake, and Azure Blob Storage.
- Strong knowledge of Azure Synapse Analytics, Azure SQL Database, and Cosmos DB.
- Hands-on experience with Apache Spark, Python, Scala, and SQL for data processing.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
