Data Platform Engineer

Software Guidance & Assistance, Inc., (SGA), is searching for a Data Platform Engineer for a Contract assignment with one of our premier SaaS clients in San Jose, CA .

Responsibilities :

• Setup and maintain production scale Databricks environment on public cloud such as Microsoft Azure and AWS (Amazon Web Services)
• Setup and maintain production scale data storage such as ADLS (Azure Data Lake Storage) and AWS S3 for multiple tenant teams using our Data Platform
• Setup and maintain production scale micro services to support the daily operation of our data platform. Services include job scheduling, security, financial, and administrative services, etc.
• Provide triage and guidance to the team on various support issues raised by our tenants
• Develop tools and automation solutions for configuration management, service deployments, monitoring, and alerting to assist with daily RTB (Running the Business) operations
• Budget and monitor cloud spend, always think of ways to avoid cloud resource wastage, utilize 3rd party tools, or develop your own tools to help the team with cost optimization
• Assure security and privacy compliance and implement Security & Compliance solutions to lock down data stored in our data lake
• Explore GenAI technologies and find opportunity to integrate them with our data platform, providing platform enhancement or improving platform user experience in the end
• Work with various 3rd party vendors for troubleshooting, proof of concept, and other collaborative projects to enhance our product.

Required Skills

• Cloud Infrastructure Administration and Automation:
  • AWS
  • Azure
• Proficient with following storage technologies:
  • ADLS Gen2
  • AWS S3
  • Hive or MySQL
  • MongoDB
  • Vector Databases
• Setup, troubleshoot and maintain following technologies:
  • Databricks Workspace, includes but not limited to - Unity Catalog, Vector Search, SQL Warehouse, Serverless Compute
  • Spark workloads
  • Airflow and DAGs
  • Azure Kubernetes Service or Elastic Kubernetes Service
  • Collibra
  • Neo4J
  • Metric Insights
• Ability to setup monitoring and alerting with:
  • Databricks System Tables
  • Prometheus
  • Splunk
  • ELK
  • PowerBI
• Familiar with how to troubleshoot and maintain:
  • Servers with Linux system
  • Kubernetes environment
• Knowledge to operate with:
  • Jira
  • Service Now

SGA is a technology and resource solutions provider driven to stand out. We are a women-owned business. Our mission: to solve big IT problems with a more personal, boutique approach. Each year, we match consultants like you to more than 1,000 engagements. When we say let's work better together, we mean it. You'll join a diverse team built on these core values: customer service, employee development, and quality and integrity in everything we do. Be yourself, love what you do and find your passion at work. Please find us at https://sgainc.com/ .

SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. Please visit our company EEO page to request an accommodation or assistance regarding our policy.