Data Architect (Databricks)

Job Description (JD) for a Data Architect with Databricks (Remote, 15 Years Experience, Mandatory Databricks Expertise) :

Job Title : Data Architect (Databricks) – Remote

Location : Remote

Experience : 15 Years

Employment Type : Contract

Industry : IT / Data Engineering / Cloud

Rate - 80 to 85 on c2c

Job Summary :

We are seeking a highly experienced Data Architect with mandatory expertise in Databricks to lead the design, implementation, and optimization of modern data architectures. The ideal candidate will have 15 years of experience in data engineering, architecture, and cloud-based data solutions.

This role requires a strong foundation in data modeling, governance, ETL pipelines, and lakehouse architectures, with a focus on leveraging Databricks for scalable and efficient data processing.

Key Responsibilities :

1. Data Architecture & Design

Architect and implement scalable, high-performance data platforms using Databricks.

Define data modeling strategies (conceptual, logical, physical) for structured, semi-structured, and unstructured data.

Develop data lakehouse architectures with Delta Lake to enable real-time analytics.

Ensure data security, governance, and compliance using Unity Catalog and RBAC policies.

2. Databricks & Cloud Platform Expertise

Design and optimize Databricks clusters, notebooks, and job workflows.

Leverage Databricks Delta Engine (Photon), Auto-Scaling, and Databricks SQL for performance tuning.

Work with Databricks MLflow for model tracking and AI / ML workflows.

Integrate Databricks with cloud platforms (AWS, Azure, GCP) and storage solutions.

3. Data Pipeline Development & Optimization

Build scalable ETL / ELT pipelines using Apache Spark, Databricks Jobs, and Workflows.

Optimize big data processing using Spark tuning techniques.

Implement real-time streaming solutions using Structured Streaming and Kafka.

4. Governance, Security & Compliance

Establish data governance frameworks using Unity Catalog and lineage tracking.

Implement role-based access control (RBAC) and encryption best practices.

Ensure regulatory compliance (GDPR, HIPAA, CCPA, etc.) for sensitive data.

5. Collaboration & Leadership

Collaborate with business stakeholders, data engineers, and data scientists to define data strategies.

Provide technical leadership and mentorship to data teams.

Lead proof-of-concept (PoC) projects to evaluate new technologies.

Required Skills & Experience :

15 years of experience in Data Architecture, Data Engineering, and Big Data Solutions.

5 years of hands-on experience in Databricks (Mandatory).

Strong knowledge of Databricks Delta Lake, Unity Catalog, and MLflow.

Expertise in Apache Spark, PySpark, and Databricks SQL.

Experience designing ETL / ELT workflows and data pipelines using cloud-based solutions.

Deep understanding of data warehousing, data lakes, and lakehouse architectures.

Strong cloud experience in AWS, Azure, or Google Cloud (GCP).

Knowledge of data security, governance, compliance (GDPR, HIPAA, etc.).

Proficiency in Python, SQL, Scala, and Spark optimizations.

Experience with real-time streaming (Kafka, Kinesis, Spark Streaming).

Preferred Skills (Nice to Have) :

Experience with AI / ML model deployment in Databricks.

Hands-on experience with DataOps, CI / CD for data pipelines.

Familiarity with Snowflake, Redshift, or BigQuery.

Knowledge of Graph Databases, NoSQL (MongoDB, CosmosDB, etc.).

Why Join Us?

Fully Remote Work Opportunity.

Work on cutting-edge Databricks & AI-driven data solutions.

Competitive salary & benefits.

Opportunity to lead and mentor top-tier data teams.

Collaborative and innovative work culture.