What are the responsibilities and job description for the Machine Learning Engineer - Unity Catalog (Databricks) position at Sryven?
Job Details
Title: Machine Learning Engineer - Unity Catalog (Databricks) Location: Seattle, WA (Remote for strong candidates, Hybrid preferable with 3 days in-office)
Job Type: Contract
Visa Types: H1B (on our payroll), (candidate's company payroll, no 1099), USC
Overview:
We are looking for an experienced Machine Learning (ML) Engineer with expertise in Unity Catalog and Feature Store in Databricks to help build and maintain a robust foundation for our data and machine learning workflows. You will work on organizing data, managing access, and ensuring ML models operate efficiently in production environments.
Key Responsibilities:
- Set up and manage Unity Catalog in Databricks to organize and secure data access across teams.
- Design, build, and operationalize Feature Stores to support machine learning models in production.
- Build efficient data pipelines to process and serve features to ML workflows.
- Collaborate with teams using Databricks, Azure Cosmos DB, and other Azure tools to integrate data solutions.
- Monitor and optimize the performance of data pipelines and feature stores.
Required Skills & Experience:
- Strong experience with Unity Catalog in Databricks for managing data assets and access control.
- Hands-on experience working with Databricks Feature Store or similar solutions.
- Proven knowledge of building and maintaining scalable ETL pipelines in Databricks.
- Familiarity with Azure tools like Azure Cosmos DB and Azure Container Registry (ACR).
- Understanding of machine learning workflows and the integration of feature stores into pipelines.
- Strong problem-solving skills and a collaborative mindset.
- Proficiency in Python and Spark for data engineering tasks.
- Experience with monitoring tools such as Splunk or Datadog to ensure system reliability.
- Familiarity with Azure Kubernetes Service (AKS) for deploying and managing containers.
Desirable Skills:
- Experience with other Azure tools or services related to machine learning and data engineering.
- Understanding of cloud-native architectures and best practices for data storage and retrieval.
- Knowledge of containerization and deployment practices for machine learning models.
Skill Matrix
Skill | Required | Desired | Proficiency Level |
Unity Catalog in Databricks | Yes | No | Expert |
Databricks Feature Store | Yes | No | Expert |
Python and Spark | Yes | No | Advanced |
Azure Cosmos DB | Yes | Yes | Intermediate |
Azure Container Registry (ACR) | Yes | Yes | Intermediate |
ETL Pipeline Development | Yes | No | Advanced |
Monitoring Tools (Splunk, Datadog) | Yes | No | Intermediate |
Azure Kubernetes Service (AKS) | Yes | Yes | Intermediate |
Machine Learning Workflows | Yes | Yes | Intermediate |
Collaboration and Problem Solving | Yes | No | Expert |
