Data Engineer

Focus on SQL Knowledge and workflow management (airflow tool) and Machine Learning Knowledge : Specifically Feature. Understanding data inputs into the model and feature analysis. Feature Engineering. Training data set composition. Experience with Pyspark to perform data transformations, feature engineering, data analysis, and ML training data generation.

This role supports the feature generation, data workflows, and maintenance of the RealTIME personalization models for the Campaigns fleet.

• Streaming industry experience is a huge plus!

EXTERNAL JOB DESCRIPTION :

Department Overview :

Applied Machine Learning Engineers, Data Engineers, and Data Scientists on the Disney Streaming Machine Learning and Innovation team develop and maintain recommendation and personalization algorithms for Streaming’s suite of streaming video apps. As a member of this team you will collaborate across Engineering, Product, and Data teams to apply machine learning methods to meet strategic product personalization goals, explore innovative, cutting edge techniques that can be applied to recommendations, and constantly seek ways to optimize operational processes.

Our team is…

• A group of engineers and data scientists with diverse expertise delivering solutions together.
• Collaborative and dynamic.
• Embracing agile practices.
• Using continuous integration / automated testing.
• Led by startup veterans.

Basic Qualifications

Required Qualifications :

• 2 years of data engineering experience
• Deep knowledge of the Python data ecosystem
• Great coding and problem-solving skills
• Experience in building large datasets and scalable services
• Experience deploying and running services in AWS and in engineering big-data solutions using technologies like Databricks, EMR, S3, Spark, and Docker
• Experience with Pyspark to perform data transformations, feature engineering, data analysis, and ML training data generation
• Excellent communication and people engagement skills

Preferred Qualifications

• Experience building streaming pipelines using Kafka, Spark, or Flink
• ML algorithmic and systems knowledge / experience

Responsibilities :

• Partner with technical and non-technical colleagues to understand ML algorithm feature, data, and workflow requirements.
• Implement necessary feature analysis, new features, training datasets and related workflows and monitoring metrics and services
• Work with Engineering teams to collect required data from internal and external systems
• Develop and maintain ETL routines using orchestration tools such as Airflow
• Collaborate with machine learning practitioners to design and build model and data forward

solutions

• Deploy scalable streaming and batch data pipelines to support petabyte scale datasets
• Enforce common data design patterns to increase code maintainability
• Create ETL architecture designs and conduct reviews
• Perform ad hoc data analysis and maintenance as necessary
• Partner with team leads to identify, design, and implement internal process

improvements

• Drive and maintain a culture of quality, innovation, and experimentation
• Work in an Agile environment that focuses on collaboration and teamwork

Required Education Bachelor’s degree or relevant years of work experience