What are the responsibilities and job description for the Sr. Data Engineer position at VeriiPro?
Responsibilities
- Design, develop, and optimize ETL pipelines for large-scale data processing and transformation.
- Leverage Databricks tools and technologies, including Delta Lake and Databricks SQL, to manage and process data effectively.
- Implement real-time data processing solutions using Databricks Spark Streaming and Structured Streaming frameworks.
- Build scalable, distributed data workflows using PySpark and Spark SQL.
- Develop reliable and automated pipelines using Delta Live Tables.
- Utilize Autoloader for efficient incremental data ingestion.
- Troubleshoot and optimize performance in distributed computing environments.
- Collaborate with cross-functional teams to ensure data solutions align with business requirements.
- Maintain expertise in Azure data services and related technologies.
Qualifications
- Minimum of 12 of hands-on experience in data engineering.
- Expertise in Databricks, including Delta Lake and Databricks SQL.
- Proficiency in ETL development, PySpark, and large-scale data workflows.
- Strong knowledge of streaming data pipelines and frameworks like Spark Structured Streaming.
- Familiarity with the Azure platform and its data services.
- Exceptional troubleshooting and performance optimization skills in distributed environments.
