What are the responsibilities and job description for the Databricks Architect position at Galent?
Top Skills
- Delta Lake
- Medallion Architecture
Responsibilities
- Candidate to build a Delta Lake. An understanding of Medallion architecture is important.
- Designed Batch and Real-time data ingestion end to end.
- Data Lineage using Data Catalog - maintained that and used the tool to discover data sources and targets.
- Designed Data Quality Strategies – identify CDE (critical data elements), assigning data stewards from the business side, and coded the DQ metrics generation in the curated zone/silver layer.
- Designed CDC/incremental loads, full load strategies – using hash keys, key columns, and merge designs.
- Designed Data Security – PII/NPI, RBAC (role-based access control).
- Azure Data Factory pipeline creation Azure DevOps.
- Designed using Azure Data Factory pipeline and Databricks – writing out to Azure Blob and then finally to Snowflake as consumption from Power BI/Snowflake.
- Initially worked with the client team and did extensive data analysis to create the model, data delivery mechanism, cloud storage, retention requirement, security, RBAC (role-based access control), etc.
- Streaming datasets/files were captured using AUTOLOADER – Delta live tables into raw and designed to be updated into silver, gold.
- Provide guidance on Azure cost management strategies, including resource tagging, budgeting, and monitoring.
- Conduct Azure security assessments, vulnerability scanning, and threat modeling to enhance cloud security.
- Discussing potential projects and their parameters with clients, executives, and software developers.
- Planning out the blueprints for software projects, including defining the scope, allocating resources, setting deadlines, laying out communication strategies, and indicating tests and maintenance.
- Assembling and leading the project team.
- Participating in and supervising each stage of the project.
- Ensuring each project stays on schedule and adheres to the deadlines.
- Determining and overseeing consistent testing, evaluation, and troubleshooting of all products in all stages of completion.
- Tracking milestones, deliverables, and change requests.
- Serving as a liaison to communicate information regarding changes, milestones reached, and other pertinent information.
- Preparing project proposals and discussing potential projects with clients and stakeholders.
- Facilitating project initiation by defining project scope and requirements and preparing the necessary documents and requirements.
- Developing project plans and timelines to ensure the timely submission of project deliverables.
- Managing project budgets and resources to ensure the timely completion of milestones.
- Identifying and managing project risks.
- Facilitating team meetings and collaboration.
- Liaising for changes and negotiations with relevant stakeholders.
- Ensuring software quality standards are met and requirements are submitted within budget and on time.
- Closing the project and ensuring proper documentation.
