Senior Cloud Data Engineer

Position: Senior Cloud Data Engineer – Python, Spark, Synapse, and Azure for AI/ML Data Processing

Location: Minneapolis, MN (Onsite – Minimum 3 days per week)

Experience Required: 10 years

Employment Type: Full-Time/Contract

Prerequisites:

Candidates must have a fully functional development setup, including VSCode (or equivalent), Python, Spark, and connectivity to Azure Synapse and ADLS2.

We are hiring an experienced Senior Cloud Data Engineer to lead the development of ETL pipelines and data workflows that support advanced AI/ML initiatives. This role requires deep expertise in Azure-based data solutions, including Synapse Analytics, Data Factory, Spark, and SQL Pools. The ideal candidate will bring robust experience in cloud engineering, data warehousing, and API development.

Collaboration with cross-functional teams such as Data Scientists and ML Engineers is key, as is the ability to lead the design and implementation of scalable, secure, and efficient data solutions.

• Design and implement robust ETL pipelines using Python, Apache Spark, and Azure Synapse.
• Develop Azure Data Factory workflows to orchestrate complex data operations.
• Optimize Synapse Analytics for batch and real-time data processing.
• Build scalable data warehousing solutions to support analytical and AI/ML requirements.

• Create and manage CI/CD pipelines using Azure DevOps for seamless deployment.
• Leverage Infrastructure-as-Code (IaC) tools like Terraform or ARM templates for environment automation.
• Optimize Azure Data Lake Storage (ADLS Gen2) for efficient data storage and access controls.

• Work closely with Data Scientists, ML Engineers, and Business Analysts to design tailored solutions.
• Coordinate with DevOps and Security teams to ensure smooth application deployment.
• Mentor junior developers and provide technical leadership during project execution.

• Develop RESTful APIs for secure data access and exchange.
• Integrate data pipelines with external systems and internal APIs.
• Containerize and deploy data solutions using Kubernetes.

• Participate in Agile practices, including sprint planning and retrospectives.
• Drive iterative development with a focus on quality and timely delivery.
• Adapt to evolving business priorities in a dynamic environment.

• Proficiency in Python and Apache Spark for large-scale data engineering.
• Extensive experience with Azure Synapse Analytics, Data Factory, SQL Pools, and Spark Pools.
• Expertise in building data warehouses on Azure.
• Strong SQL skills, including performance optimization and complex queries.
• Hands-on experience with Azure DevOps and CI/CD pipelines.
• Familiarity with Java for API integration and microservices.
• Knowledge of Kubernetes for container orchestration.
• Experience working with Azure Data Lake Storage (ADLS Gen2).

• Proven ability to lead Agile development cycles and mentor team members.
• Strong interpersonal skills to collaborate with cross-functional teams.

• Analytical mindset with a focus on problem-solving.
• Clear communication skills for technical and non-technical stakeholders.
• Ability to manage priorities in a fast-paced setting.

• Azure certifications in data engineering or cloud architecture.
• Experience deploying AI/ML models in cloud environments.
• Familiarity with data privacy and governance best practices.