What are the responsibilities and job description for the Data Engineer IV position at Talent Grub USA inc.?
Description :
Lead Data Engineer- iGrid
Core Responsibilities
Provides technical direction, guides the team on key technical aspects and responsible for product tech delivery
o Lead the Design, Build, Test and Deployment of components
Where applicable in collaboration with Lead Developers (Data Engineer, Software Engineer, Data Scientist, Technical Test Lead)
o Understand requirements / use case to outline technical scope and lead delivery of technical solution
o Confirm required developers and skillsets specific to product
o Provides leadership, direction, peer review and accountability to developers on the product (key responsibility)
o Works closely with the Product Owner to align on delivery goals and timing
o Assists Product Owner with prioritizing and managing team backlog
o Collaborates with Data and Solution architects on key technical decisions
The architecture and design to deliver the requirements and functionality
Core Experience and Abilities
Power Systems engineering experience
DER Dispatch experience
Automated fault analysis systems experience
Ability to perform hands on development and peer review for certain components / tech stack on the product
o Standing up of development instances and migration path (with required security, access / roles)
o Develop components and related processes (e.g. data pipelines and associated ETL processes, workflows)
o Lead implementation of integrated data quality framework
o Ensures optimal framework design and load testing scope to optimize performance (specifically for Big Data)
o Supports data scientist with test and validation of models
o Performs impact analysis and identifies risk to design changes
o Ability to build new data pipelines, identify existing data gaps and provide automated solutions to deliver analytical capabilities and enriched data to applications
o Ensures Test Driven development
Experience leading teams to deliver complex products
Strong technical skills and communication skills
Strong skills with business stakeholder interactions
Strong solutioning and architecture skills
Experience building real time data ingestion streams (event driven)
Ensure data security and permissions solutions, including data encryption, user access controls and logging
Core Technical Skills
Experience with native AWS technologies for data and analytics such as Athena, S3, Lambda, Glue, EMR, Kinesis, SNS, CloudWatch, etc.
Tools and Languages such as Django, Python, Java, Scala, Pandas
Infrastructure as Code technology such as Terraform
AWS services such as S3, EMR, Glue, Lambda, Athena, Kinesis, EC2, SNS, SQS, Cloudwatch
Experience with databases such as Redshift, Document DB, DynamoDB and Mongo DB
Experience transitioning on premise big data platforms into cloud based platforms such as AWS
Additional Technical Skills (nice to have, but not required for the role)
Hadoop platform (Hive; HBase; Druid)
Spark
PySpark
Workflow Automation
DevOps pipeline (CI / CD); Bitbucket; Concourse
API frameworks
