What are the responsibilities and job description for the Azure Data Engineer- Hybrid, Pittsburgh, PA position at A.C. Coy?
Overview
Location: Hybrid- Pittsburgh, PA
Job Type: Full Time/Permanent
Work Authorization: No Sponsorship
The A.C.Coy has an immediate need for an Azure Data Engineer for a full time opportunity.
Responsibilities
- Automate data pipeline monitoring, error handling, and performance tuning to ensure data integrity and availability
- Collaborating with the Digital Data Solutions Team or external consultants as needed
- Design, develop, and implement scalable and high-performance data solutions on the Azure platform, specifically architecting data pipelines and ETL/ELT processes utilizing Azure Data Factory and Azure Databricks to extract, transform, and load data from various sources into Azure data storage solutions
- Optimize data storage and retrieval mechanisms to ensure efficient processing of large datasets, particularly focusing on the capabilities of Databricks
- Develop and maintain data models and schemas to support data analysis and reporting requirements
- Implement data security and privacy measures to protect sensitive data
- Develop and maintain data ingestion processes to acquire data from various sources, including APIs, databases, and streaming platforms
- Implement data quality checks and validation procedures to ensure the accuracy and integrity of data
- Design and implement data transformation and enrichment processes using Azure Databricks, Azure Data Factory, and other relevant tools
- Document technical design, deployment processes, and best practices for data engineering solutions
- Implement data partitioning, indexing, and optimization techniques to improve query performance
- Collaborate with data scientists and analysts to operationalize and deploy machine learning models on Azure, utilizing Databricks for model training and deployment
- Configure and manage data storage options, including Azure Blob Storage, Azure Data Lake Storage, and Databricks LakeHouse
- Collaborate with DevOps teams to deploy and manage data engineering solutions in Azure
Qualifications
Education: Bachelor’s degree in Computer Science, Computer Engineering, or other technical field
- Hands-on experience with Azure data services, including Azure Data Lake Storage, Azure SQL Database, Azure Data Factory, and Azure Databricks. Understanding of Azure Synapse Data Warehouse preferred
- Strong experience in data engineering and architecture, with a focus on Azure cloud technologies
- Proficiency in data-centric programming languages, including SQL, Python, and PowerShell
- Knowledge of data integration techniques and tools, such as ETL/ELT processes, data pipelines, and data warehousing concepts
- Familiarity with data modeling and schema design principles
- Understanding of data security and privacy regulations, including GDPR and CCPA
- Excellent problem-solving and analytical skills
- Strong communication and collaboration abilities
- Azure certifications, such as Microsoft Certified: Enterprise Data Engineer Associate, are a plus
