What are the responsibilities and job description for the Data Engineer position at Datavail?
Job Description
Requirements :
- 9 years of experience in implementing large data and analytics platforms
- 6 years of experience engineering and building data platforms on Azure or AWS
- 4 years of experience engineering and building data platforms on Databricks from scratch
- Experience using Azure Synapse as source or target for the data pipelines in at least 1 project
- Hands on experience using Microsoft Fabric in at least 1 project
- Pro in all Databricks engineering services Unity Catalog, Spark Jobs, Delta Live Table (DLT), DLT Meta, Databricks Workflow, Auto Loader. (You need to ask the candidate)
- Hands on in Databricks SQL, PySpark
- Understanding of data modelling, data mappings, understand the data model changes and propose the best practices and guidelines
- Creating technical design based on end-to-end data architecture and frameworks for meta data driven data ingestion, transformation, logging and monitoring
- Deploying data pipelines / applications to higher environments using CI-CD pipelines
- Experience in writing technical user stories.
- Experience in Agile methodology and scrums.
