What are the responsibilities and job description for the Databricks Architect position at Digitive LLC?
Job Details
Role: Databricks Architect
Location: Minneapolis, MN ( only Onsite )
Experience: 15 Years
Candidate to build a Delta Lake. An understanding on Medallion architecture is important.
Designed Batch and Real time data ingestion end to end.
Data Lineage using Data Catalog maintained that and used the tool to discover data sources and targets;
Designed Data Quality Strategies identify CDE -critical data elements, assigning data stewards from business side and coded the DQ metrics generation in curated zone/silver layer.
Designed CDC/incremental loads, full load strategies using hash keys, key columns and merge designs.
Designed Data Security PII/NPI, RBAC- role based access control.
Azure Data factory pipeline creation Azure DevOps
Designed using Azure Data factory pipeline and Databricks writing out to azure blob and then finally to snowflake as consumption were from Power BI /snowflake.
Initially worked with client team and did extensive data analysis to create the model, data delivery mechanism, cloud storage, retention requirement, security, RBAC [role based access control] etc
Streaming datasets/file- were captured using AUTOLOADER delta live tables into raw and designed to be updated into silver, gold.
Provide guidance on Azure cost management strategies, including resource tagging, budgeting, and monitoring.
Conduct Azure security assessments, vulnerability scanning, and threat modeling to enhance cloud security.
Discussing potential projects and their parameters with clients, executives, and software developers.
Planning out the blueprints for software projects, including defining the scope, allocating resources, setting deadlines, laying out communication strategies, and indicating tests and maintenance.
Assembling and leading the project team.
Participating in and supervising each stage of the project.
Ensuring each project stays on schedule and adheres to the deadlines.
Determining and overseeing consistent testing, evaluation, and troubleshooting of all products in all stages of completion.
Tracking milestones, deliverables, and change requests.
Serving as a liaison to communicate information regarding changes, milestones reached, and other pertinent information.
Preparing project proposals and discussing potential projects with clients and stakeholders
Facilitating project initiation by defining project scope and requirements, and preparing the necessary documents and requirements
Developing project plans and timelines to ensure the timely submission of project deliverables
Managing project budgets and resources to ensure the timely completion of milestones
Identifying and managing project risks
Facilitating team meetings and collaboration
Liaising for changes and negotiations with relevant stakeholders
Ensuring software quality standards are met and requirements are submitted within budget and on time
Closing the project and ensuring proper documentation
