What are the responsibilities and job description for the Data Engineer with Python and Azure position at Avant-Garde Solutions, Ltd.?
Data Engineer
Our Boston based client is looking for a Data En gineer for a full time Hybrid role . The Data Engineer will engage closely with business & IT colleagues to improve, streamline and automate business processes; will design, build and manage applications/workflows in a cloud environment;will leverage tools which automate processes, enabling DevOps capabilities to manage all aspects of application development.
Essential Duties & Responsibilities:
- Design, develop and maintain APIs, data pipelines & automation tools using Python, FastAPI and Azure Services.
- Write clean, efficient, and maintainable code that follows best practices. Exercise the full software development lifecycle in developing proper Python packages.
- Optimize application performance and scalability for meeting business needs.
- Identify opportunities for reusable code and create microservices and reusable components to increase team’s ability to deliver quality solutions quickly.
- Work directly with analysts to gain insight into existing data sets. Evaluate use cases for new data turning requests into actionable designs.
- Incorporate new data sources from external vendors using streams, flat files, APIs, and databases.
- Maintain and provide support for the existing data pipelines using Python, ADF, and SQL
- Plan & execute using agile methodologies, developing & delivering within predefined sprints
- Identify & deploy appropriate file formats for data ingestion into various storage and/or compute services via ADF for multiple use cases
- Develop and implement tests to ensure data quality across all integrated data sources.
Required Experience and Skills:
- Bachelor’s or higher degree in Computer Science, Engineering or related field of study.
- 4 years of professional programming experience.
- Strong experience in Python development, testing frameworks, workflow automation and cloud-based technologies.
- Solid understanding of OOP and algorithms, data structures, and design patterns.
- Familiarity with Snowflake or any other OLTP/OLAP databases.
- Familiarity with Azure services including Logic Apps, Azure Functions, Azure Storage, SQL Database, SQL Managed Instance.
- Familiarity with DevOps tools and CI/CD pipelines, dependency management, database systems, and cloud solutions.
- Passionate about technology and excel in cloud platforms, implementation, and troubleshooting.
- Excellent communication and collaboration skills, and willing to learn new skills.
- Supportive of a culture of continuous process and organizational improvement.
