Cloud Architect (Azure Data Bricks)

We are looking for an experienced professional with expertise in Azure and Databricks to assist with a critical project. This is an urgent requirement, and we need someone who can quickly contribute to setting up and managing data processes. Will be responsible for integrating data from a production plant into Azure, managing storage, and implementing MLOps for training.

Key responsibilities include

• Determine the best approach to ingest data from the production environment to Azure, which could involve setting up data pipelines or utilizing Azure Data Factory.
• Will also be tasked with selecting and configuring the appropriate Azure storage solution, such as Azure Blob Storage or Azure Data Lake, based on the data's nature and access needs.
• Data Management: Establish processes for data organization, versioning, and governance to ensure data quality and compliance.
• Define and implement an MLOps framework to support model development, deployment, monitoring, and retraining.
• Identify suitable machine learning models for training and configure the necessary resources within Azure, such as compute instances and libraries.
• Clear documentation and collaboration among team members will be essential, as the candidate will need to facilitate communication regarding workflows, roles, and best practices.
• Strong attention to security, with the candidate responsible for implementing measures to safeguard data and manage access permissions.
• Monitoring data pipelines and model performance, along with setting up an ongoing maintenance plan for smooth operations, will be part of the responsibilities.

Requirements:

• Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related field. A master's degree in a relevant discipline is a plus but not required. Additionally, certifications in Azure, Databricks, or related technologies (such as Microsoft Certified: Azure Data Engineer or Microsoft Certified: Azure AI Engineer) would be highly beneficial.
• Minimum of 3-5 years of experience in data engineering, cloud computing, and working with Azure services. Experience with Azure Data Factory, Azure Databricks, and machine learning operations (MLOps) is essential. Practical experience setting up and managing data pipelines, working with cloud-based storage solutions (Azure Blob Storage, Azure Data Lake), and integrating data from diverse sources into the cloud will be crucial.
• Hands-on experience with programming languages such as Python, SQL, or Spark, as well as knowledge of data modeling, ETL processes, and security best practices, is required.
• Familiarity with machine learning frameworks, model deployment, and automated workflows will also be important. Strong problem-solving abilities and an ability to work in a collaborative, fast-paced environment will be key to success in this role.
• Experience in ensuring compliance with data governance and security standards will be highly valued.