Senior Data Engineer
JOB SUMMARY
The Senior Data Engineer is an interdisciplinary individual on the Data Analytics team who collaborates closely with multiple stakeholders, across the enterprise and Information Technology (IT), to ensure the most important data is accessible and well-understood.
A Senior Data Engineer designs, develops, implements, and supports new and existing highly efficient ELT/ETL processes and data sets. Other responsibilities include
•Working closely with data consumers, solution architects, security and governance teams to implement solutions to answer complex questions and drive business decisions.
•Applying your proven communication skills, problem-solving skills, and knowledge of best practices in designing, developing, and deploying data and analytic solutions.
Senior Data Engineers need to be adept in several technical and business skills. These include working with diverse datasets, parsing and understanding data, working with domain experts, data scientists and analysts in framing the business problems and provisioning integrated data quickly across multiple environments.
Senior Data Engineers need to be inquisitive and motivated to learn modern technologies and capabilities which could benefit the organization and lead the effort in evaluating the technology for acceptance at our company. Senior Data Engineers also assist the business data science efforts with source data, building data sets, helping evaluate models and integrating analytics and data science model outputs into business processes.
RESPONSIBILITIES
•Hybrid cloud environment: the Senior Data Engineer works in a hybrid cloud ecosystem, composed of Azure, Oracle and on-premises technologies, building and supporting data and analytics solutions. The Senior Data Engineer will need to learn the data, tools and capabilities resident in this hybrid ecosystem, such as Synapse, Data Lakes, Dedicated Pool, Azure ML, SQL Server, SSIS and SSAS.
•Build data pipelines: Managed data pipelines consist of a series of stages through which data flows. Designing, building and maintaining data pipelines, in Azure and the on-premises ecosystems, will be the primary responsibility of the data engineer.
•Drive data centric decision making. Assists with enhancing the data and metadata management infrastructure to ensure data quality, accessibility and security.
•Collaborate across departments: Collaborates with business data consumers, of various skill levels, in refining their requirements for various data and analytics initiatives. This collaboration can lead to building enterprise data products, enabling data-driven decision making.
•Lead, educate and train: Be curious and knowledgeable about innovative technologies and data initiatives. Research and propose data ingestion, preparation, integration and operationalization tools or techniques to aid these initiatives. Train team members, data consumers, data scientists and data analysts in these technologies and preparation techniques.
•Participate in ensuring compliance and governance during data use: Data engineers work with data governance teams (and information stewards within these teams) in building, vetting and promoting content, which adheres to data governance and compliance initiatives.
•Become a data and analytics evangelist: The Senior Data Engineer a blend of data and analytics "evangelist," "guru" and "fixer." This role promotes the available data and analytics capabilities and expertise to business unit leaders educating them in leveraging these capabilities in achieving their business goals.
QUALIFICATIONS
Education and Training
•A bachelor's in computer science, statistics, applied mathematics, data management, information systems, information science or a related quantitative field [or equivalent work experience] is required.
•An advanced degree in computer science (MS), statistics, information science (MIS), data management, information systems, information science (post-graduation diploma or related) or a related quantitative field [or equivalent work experience] is preferred.
•The ideal candidate will have a combination of IT, data governance, analytics, and communication skills.
Previous Experience
•At least 5 years or more of work experience in data management disciplines including data integration, modeling, optimization, data quality and/or other areas directly relevant to data engineering responsibilities and tasks.
•At least 5 years of experience working in cross-functional teams and collaborating with business stakeholders in support of a departmental and/or multi-departmental data management and analytics initiative.
Technical [and Business] Knowledge/Skills
•Strong experience with various languages and advanced analytics tools such as SQL, Python, PowerBI, Microsoft SSIS/SSAS, Azure Synapse and others.
•Strong ability to design, build and manage data structures and pipelines for encompassing data transformation, data models, schemas, metadata and workload management. Work with both IT and business in integrating analytics and data science output into business processes and workflows.
•Strong experience in working with large, heterogeneous datasets in building and optimizing data pipelines, pipeline architectures and integrated datasets using data integration technologies, including ETL/ELT, data replication/CDC, message-oriented data movement, API design and development.