What are the responsibilities and job description for the Senior ML Data Engineer - Remote position at TriOptus LLC?
Job Description :
Senior ML Data Engineer Feature Engineering ETL
Qualifications
7 years in data engineering and at least 4 years focusing on ML feature engineering ETL pipeline development and data preparation for ML
Proven experience managing pipelines on Databricks using Apache Spark with a strong understanding of the medallion architecture
Familiarity with ML lifecycle management with MLflow experience as a strong plus and advanced skills in Apache Spark PySpark for big data processing and analytics
Proficient in Python for data manipulation and SQL for query optimization
Experience building pipelines for real-time and batch model serving in production environments and knowledge of CICD practices for ETLELT pipeline development
Expertise in metadata and master data management within technical data catalogues
Understanding of data security and compliance especially with sensitive data like PII
Required Skills : Data Warehouse
Basic Qualification :
Additional Skills : Data Engineer
Background Check : No
Drug Screen : No
