What are the responsibilities and job description for the Data Engineer position at Q Consulting?
As the Data Engineer - Azure, you will be part of a shared service team and responsible for the design and development of Azure Data Services. You will be responsible for delivering high quality solutions on time and within budget as part of an agile engineering team. In addition, you will contribute to the development and maintenance of the Cloud Data Services architecture blueprint for the organization.
- Partner with external vendor platforms to support ETL/ELT integrations and support the migration of existing internal on-premises data/ETL implementations to Azure cloud native solutions.
- Partner with project managers, business analysts, end users and third-party contacts to design, implement and test software applications.
- Partner with other engineers, analysts, and designers in conceptualizing and developing new software programs and applications.
- Plan phases of the software development life cycle (SDLC) for a variety of projects that are specific to Azure.
- Design, build, and maintain ETL/ELT data pipelines using Azure Data Services.
- Work collaboratively as part of a team to develop Azure Cloud Data ETL/ELT solutions using ADLS Gen 2, Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Stream Analytics, Blob Storage, Azure SQL MI, Non-relational data stores.
- Participate in driving best practices around data engineering and Data Services software development process.
- Design and build highly scalable data pipelines and frameworks for near real-time and batch data ingestion, processing and integration.
- Follow approved DevOps process and coordinate with deployment teams to deploy changes in higher environments.
- Design, develop, optimize, and maintain data architecture and pipelines that adhere to organization's Enterprise ETL principles and business goals.
- Acquire and maintain expertise in data elements and how the data elements relate to specific organization product lines.
- Provide mentoring, guidance, and training to team members on how to use the Cloud Data Services Platforms most effectively.
- Other projects and duties as assigned.
- At least seven years of Azure Data Engineering experience. Experience must include requirement gathering, development, testing and maintenance.
- Data analysis, data modeling, and data integration using azure technologies like Azure Data Factory (ADF).
- At least three years of hands-on experience with Azure Synapse, Azure Stream Analytics, Azure Cosmos, Azure SQL, Azure Data Factory, Azure Event Hubs, Azure Event Grid, Databricks, ADLS Gen 2 preferred.
- Experience working with event-based/streaming technologies to ingest and process data; building ETL / data warehouse transformation processes; software troubleshooting and agile software development methodologies.
- Expert or professional-level Azure certifications are a plus.
- Experience in extracting and transforming data from multiple sources and integrating in centralized platform.
- Hands-on experience with Data Lake/Big Data Projects implementation on Azure Cloud platforms.
- Experience writing complex SQL queries in DWH/RDMS/Big Data environments.
- Experience in ADF Pipeline Creation with multiple integration runtime.
- Experience in developing and managing Azure SQL DB, Azure Cosmos, and Azure SQL DW.
- Experience in Azure Storage and Azure Data Lake Storage Gen 2.
- Strong data engineering background.
- Microsoft Certified: Azure Data Engineer Associate.
Network Engineer III - 3rd Shift
Atomic Data -
Minneapolis, MN
Professional Services Engineer II - 2nd Shift
Atomic Data -
Minneapolis, MN
Professional Services Engineer II - 3rd Shift
Atomic Data -
Minneapolis, MN