What are the responsibilities and job description for the Azure Data Engineer - Hybrid , Chicago, IL position at Next Gen IT Inc?
Job Details
Azure Data Engineer
Hybrid - Chicago, Illinois, 3x days (No Relocation)
Exp - 12 yrs
Top Skills
- kafka
- sql
- azure
- Databricks
- MLOps (must have)
Top Skills' Details
3-5 years of experience in data engineering or related field
Strong UNIX/Linux administration skills and command-line proficiency
Experience with high-performance computing environments and workload management
Proficiency in SQL and experience with major database systems (PostgreSQL, SQL Server, Oracle, MySQL)
Hands-on experience with Azure cloud data services (Azure Data Factory, Synapse Analytics, Azure SQL)
Knowledge of modern data flow architectures (Lambda, Kappa, Delta) and implementation patterns
Experience with ML flow frameworks and ML lifecycle management tools
Programming skills in Python and shell scripting for automation and data processing
Familiarity with containerization (Docker) and orchestration (Kubernetes)
Experience with data processing frameworks (Spark, Databricks, Azure Batch)
Understanding of networking concepts for data movement between on-premises and cloud
Job Description
Data Engineer to support their Vegetation Management Team
A mid- Sr level Data Engineer is responsible for designing, building, and maintaining the data infrastructure and pipelines that enable efficient data processing, storage, and analysis within a hybrid computing environment. This role bridges the gap between data collection and data consumption across both high-performance computing (HPC) servers and Azure cloud platforms, ensuring that data is accessible, reliable, and optimized for analytics, machine learning, and business intelligence purposes.
Additional Skills & Qualifications
Experience with real-time data processing and streaming technologies (Kafka, Event Hubs)
Knowledge of infrastructure-as-code tools (Terraform, ARM templates)
Experience with distributed file systems (HDFS, Lustre) and object storage
Familiarity with graph databases and specialized analytics datastores
Understanding of data governance, security, and compliance requirements across hybrid environments
Experience with optimization techniques for large-scale data transfers between systems
Knowledge of GPU acceleration for data processing pipelines
Background in implementing CI/CD for data solutions
