Sr Databricks Architect

We are currently seeking an experienced Azure Databricks Architect for a long-term contract in Austin, TX which can be worked remotely within the state of Texas, however you may be required to attend onsite meetings at the client's discretion (Relocation and travel assistance not provided). The primary role of this position will be to lead the design, implementation, and optimization of a enterprise-wide data platform. This key role will involve creating a scalable, secure, and high-performance data ecosystem that empowers advanced analytics, machine learning, and business intelligence capabilities. The successful candidate will align solutions with our client's operational procedures and architecture principles, collaborating with internal technical teams. Additionally, the Architect will oversee proof-of-concept evaluations, ensure compliance with organization standards, and provide technical mentorship to operational teams.

Responsibilities:

• Design and architect comprehensive data solutions utilizing Azure Databricks, Azure Data Lake, Azure Event Hub, and other Azure data services.
• Implement data Lakehouse architecture that supports both batch and real-time data processing.
• Develop robust data ingestion pipelines using Azure Databricks, Azure Data Factory, and other ETL tools.
• Establish best practices for data governance, security, and compliance across the data platform.
• Design and implement advanced analytics and machine learning workflows within Databricks.
• Develop and execute performance optimization strategies for large-scale data processing.
• Create and maintain infrastructure-as-code (IaC) templates for scalable, reproducible data platform deployments.
• Provide architectural guidance and mentorship to data engineering and analytics teams.
• Ensure alignment of data platform architecture with enterprise objectives and overall architecture.
• Develop prototypes to demonstrate design and delivery patterns.
• Collaborate with cross-functional teams, including Development, Operations, Compliance, and Security.
• Apply principles and theories relevant to data architecture, aligning with the client's culture.
• Mentor and guide team members to enhance their technical proficiency and career growth.

Additional Responsibilities:

• Communicate team goals effectively through clear, concise written and verbal communication.
• Execute end-to-end testing for non-functional requirements.
• Resolve design and development challenges creatively and effectively.
• Stay current with emerging technologies and incorporate them into platform operations.
• Provide valuable advice and recommendations to management based on expertise.
• Expand technical proficiency in key areas of the data architecture stack.

Mandatory Required Qualifications / Experience:

• Minimum of 5 years of relevant work experience, exceeding degree requirements.

Desired Skills & Expertise:

• Advanced proficiency with Azure Databricks and a strong ability to implement best practices in large-scale organizations.
• Expert knowledge of:
• Python
• SQL
• Spark (PySpark) and Delta Lake
• Azure Data Lake Storage
• Azure Databricks

• Strong understanding of data Lakehouse architecture, cloud-native data processing, and analytics solutions.
• Experience with infrastructure-as-code tools (Terraform, Github Actions) and integrating with Github Enterprise for Code and Infrastructure deployments.
• Expertise in data modeling, performance tuning, and optimization.
• Experience securing Azure Databricks in regulated industries.
• Expertise in creating workspaces, catalogs, schemas, workflows, and designing ETL pipelines in Azure Databricks.
• Knowledge of observability and cost management strategies for the platform.
• Experience with batch and real-time streaming data pipelines.
• Proficiency in designing and implementing MLOps patterns for machine learning workflows.
• Familiarity with integrating Databricks with external governance tools like Purview.
• Experience migrating data and code to new architectures and operationalizing the platform.
• Experience designing and implementing disaster recovery strategies for Databricks across multiple regions.

  Education:

  • Bachelor's Degree in Computer Science or related field (required), or equivalent combination of education and experience.

  What we Offer:

  • Competitive compensation and benefits including health, dental, vision, voluntary life, accident, and short- term disability insurance options.
  • Hybrid working arrangement

  
  Luna Data Solutions, Inc. provides equal employment opportunities to all employees. All applicants will be considered for employment and prohibits discrimination and harassment of any type without regard to age, race, color, religion, sexual orientation, gender identity, sex, national origin, genetics, protected veteran status, and/or disability status.