What are the responsibilities and job description for the Databricks Platform Administrator (w2) position at Raag Solutions?
Databricks Platform Administration
- Deploy, configure, and maintain Azure Databricks workspaces
- Manage clusters, pools, and jobs for optimal performance.
- Handle Databricks Runtime upgrades and compatibility testing.
- Configure Databricks notebooks, Repos, and Delta Lake for efficient workflows.
Security & Access Control:
- Implement Identity and Access Management (IAM) using Azure AD / Azure IAM / SCIM.
- Set up and enforce Role-Based Access Control (RBAC) and network security.
- Ensure compliance with data governance policies (Unity Catalog, creation of Catalogs, schema etc.).
- Applying grants of catalogs and schemas
- Manage personal access tokens (PATs) and OAuth-based authentication.
Performance Monitoring & Optimization:
- Monitor cluster performance, auto-scaling, and cost optimization.
- Analyze and fine-tune Spark jobs to improve execution efficiency.
- Use Databricks SQL for querying and performance diagnostics.
Integration & Automation:
- Integrate Databricks with Azure Data Factory, Synapse, Snowflake, Kafka, and other data sources.
- Automate cluster management using Terraform.
- Implement CI/CD pipelines for Databricks notebooks using GitHub or Azure DevOps.
Troubleshooting & Support:
- Investigate and resolve job failures, cluster crashes, and query performance issues.
- Provide L2/L3 support for Databricks infrastructure issues.
- Collaborate with Databricks support for bug fixes and escalations.
