What are the responsibilities and job description for the Azure Data Engineer position at Marchon Partners?
JOB OVERVIEW :
Design, develop & maintain scalable data pipelines for an Azure Data Platform from multi-cloud and on-premise data sources to ensure completeness, accuracy, security and performance.
Identify and integrate external data sources from various source formats to meet project requirements.
Implement API-based integrations to support continuing increases in data volume and complexity.
Collaborate on the building and maintenance of an Azure data platform required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Azure modules and other programming languages such as Python and SQL / T-SQL.
Assist in the design of processes and systems to monitor data quality, ensuring production data is always complete, accurate and available for key stakeholders and business processes.
Troubleshoot data pipeline related issues and oversee their resolution.
Identify ways to improve data pipeline reliability, efficiency, quality and security (at rest / in transit).
Collaborate with business units and development teams to advance strategies for long-term data pipeline sustainability.
Collaborate with report developers / analysts and business teams to improve data models and query strategies that feed BI tools to increase data usability, accessibility and reduce usage cost.
Actively contributes to both data & application governance initiatives by implementing policies, procedures, and best practices.
KEY RESPONSIBILITIES :
ETL / ELT Development : Build, maintain, and improve ETL workflows that support data integration from multiple sources, handling various data types and frequencies. This includes event-driven solutions to sequence and automate Azure-centric data pipelines.
Data Modeling & Transformation : Collaborate with a Data Management team to design data models that align with business needs and perform data transformations to create usable data sets for analysis, reporting and query efficiency.
Pipeline Management : Manage and refine data pipelines, ensuring data is collected, transformed, and stored accurately across all systems. This includes monitoring and troubleshooting pipelines to prevent data loss, downtime and improve efficiency
SQL Query Optimization : Write, test, and optimize SQL queries to ensure efficient data retrieval and processing in support of Business Intelligence (BI) Developers / Users and downstream data consumers.
Collaboration with Analytics Teams : Work closely with data scientists, analysts, and stakeholders to support data requests, visualization needs, and other analytics functions.
Cloud Infrastructure Management : Collaborate with an Azure Cloud Services team to monitor and manage the cloud data platform to maintain performance, scalability and ensure secure data access. Collaborate management of other cloud data platforms (Oracle Cloud, AWS).
Data Quality Assurance : Conduct regular quality checks, cleanse data sets, and implement data governance practices to maintain data completeness, accuracy and reliability.
Documentation & Reporting : Keep thorough documentation of data processes, configurations, and workflows. Provide regular updates on system performance and data availability to downstream users and program management.
Innovation & Continuous Improvement : Stay up to date on industry trends, experiment with new data tools and techniques, and apply best practices to drive continuous improvement.
AI / ML Integration : Support AI / ML initiatives to enhance and support data processes, predictive modeling, and drive business value for business units .
EDUCATION / EXPERIENCE :
Bachelor's degree in Computer Science, Information Technology, or a related field.
Minimum of 5 years of experience in ETL (Extract, Transform, Load) and Data Modeling developing workflows that support efficient data movement and processing in on-prem cloud environments.
Minimum 3 years of experience with data processing systems in Azure (Data Factory, Data Lake storage, Data Bricks, Synapse Analytics)
Minimum 3 years of experience working in an Agile development environment pairing DevOps with CI / CD pipelines.
Experienced in working with DevOps tools such as Git, Jenkins, CI / CD, Jira
Experience with sourcing data from leading SaaS providers and their API solutions such as Oracle Fusion Cloud ERP, Salesforce CRM and CSM, Workday HCM and relational databases like Oracle, MS SQL Server and Postgres.
Proven experience developing efficient data models such as dimensional data models paired with data mesh or data fabric data management approaches that support large-scale analytics.
Proficiency in Analytics and visualization tools like Power BI, Tableau and Oracle Analytics Cloud.
Familiarity with Laboratory Information Management Systems (LIMS) such as Antrim, Atlas, MOLIS and HL7 healthcare data messaging (Preferred).
REQUIRED SKILLS AND ABILITIES :
Data Transformation Skills : Deep understanding of data transformation techniques to ensure completeness, accuracy and performance. Proficient in converting raw data into valuable datasets
Cloud Infrastructure Knowledge : Proficiency with Azure cloud infrastructure management and best practices for data security, scalability, and cost efficiency.
Analytical Problem-Solving : Ability to troubleshoot data issues and proactively improve processes to enhance data reliability.
Code Optimization : Proficient skills in writing and optimizing complex Python, Java, JSON, Spark and SQL / T-SQL queries for better performance, throughput and cost optimization.
Data Governance : Strong knowledge of data governance practices, including data quality, lineage, data at rest and in-transit security and role-based access control protocols.
Collaborative Communication : Ability to work cross-functionally with data scientists, analysts, visualization teams and business users to deliver comprehensive insights.
Excellent problem solving, critical thinking and analytical skills
Outstanding verbal and written communication skills.
WORK ENVIRONMENT :
The employee will primarily work in a typical office environment including use of cubicles, computers and overhead lighting. Temperature extremes will be minimal to nonexistent.
The noise level in the work environment is usually moderate. The employee will be required to use a computer, spreadsheets, database management, email, and the Internet. The employee is frequently required to use a calculator; fax, copy machine, and phone system.
Keep a pulse on the job market with advanced job matching technology.
If your compensation planning software is too rigid to deploy winning incentive strategies, it’s time to find an adaptable solution.
Compensation Planning
Enhance your organization's compensation strategy with salary data sets that HR and team managers can use to pay your staff right.
Surveys & Data Sets
What is the career path for a Azure Data Engineer?
Sign up to receive alerts about other jobs on the Azure Data Engineer career path by checking the boxes next to the positions that interest you.
