What are the responsibilities and job description for the Lead Data Engineer position at SRM Digital LLC?
Key Responsibilities:
- Design, develop, and maintain scalable and reliable ETL/ELT pipelines on Azure.
- Work closely with data scientists, analysts, and other stakeholders to deliver end-to-end data solutions.
- Build and optimize big data pipelines using PySpark and Azure Databricks.
- Orchestrate data workflows using Apache Airflow or Azure Data Factory.
- Implement and support MLOps frameworks and deployment pipelines.
- Monitor, troubleshoot, and ensure data quality and reliability across systems.
- Drive architectural decisions for data platform components, ensuring best practices are followed.
- Mentor junior data engineers and contribute to knowledge sharing within the team.
Key Skills and Technologies:
- Cloud Platforms: Microsoft Azure (Azure Data Lake, Azure Databricks, Azure Synapse, Azure Blob Storage)
- Programming Languages: Python (strong), SQL
- Big Data Technologies: PySpark, Spark
- ETL Tools: Azure Data Factory, custom ETL development
- Workflow Orchestration: Apache Airflow
- DevOps/MLOps: Azure ML, MLflow, CI/CD pipelines for ML models
- Version Control: Git, GitHub/Azure DevOps
- Other Tools (Preferred): Terraform, Docker, Kubernetes
