Demo

Observability Software Engineer, AI Infrastructure

Tesla
Fremont, CA Full Time
POSTED ON 2/22/2025
AVAILABLE BEFORE 5/17/2025

As a member of Tesla's "Insane Visibility" team, you will design, implement & maintain end-to-end observability across our AI Infrastructure stack and develop the framework to benchmark performance & processing of pipelines. You'll be responsible for building dashboards, alerts & monitoring necessary for Autopilot & AI teams to address observability issues in our FSD, Robotaxi & Tesla Bot applications, ensuring these programs run smoothly throughout the full infrastructure stack.

Responsibilities

  • Design, develop and maintain observability solutions & tools, including monitoring, logging, and alerting systems, to improve system visibility and performance
  • Create dashboards & automated alerts using tools such as Grafana, Prometheus, Splunk, Catchpoint to enhance monitoring frameworks & ensure proactive issue detection and resolution
  • Analyze system metrics and logs to identify bottlenecks, optimize application performance, and ensure system reliability end-to-end while scaling
  • Partner with developers, DevOps engineers, and AI Infra teams to integrate observability best practices into the development and deployment lifecycle
  • Assist in troubleshooting and resolving production issues by leveraging observability data to identify root causes and implement preventative measures
  • Develop scripts or workflows to automate routine tasks and improve observability tool integrations
  • Create and maintain documentation for observability tools, processes, and workflows to ensure knowledge sharing and accessibility

Requirements

  • 3 years of experience in software engineering, DevOps, or SRE roles with a focus on observability or monitoring
  • Proficiency in monitoring and visualization tools (e.g., Prometheus, Grafana, Splunk, Catchpoint)
  • Strong analytical and troubleshooting skills with a focus on system performance and reliability
  • Working knowledge for High performance computing, Slurm, GPU architecture & Networking
  • Working knowledge of logging systems and distributed tracing frameworks such as OpenTelemetry
  • Expertise in scripting languages (e.g., Python, Bash) and familiarity with configuration management tools (e.g., Terraform, Ansible)
  • Experience with containerized environments (e.g., Docker, Kubernetes) and cloud platforms (e.g., AWS, Azure)
  • Strong analytical and troubleshooting skills with a focus on system performance and reliability
  • Excellent verbal and written communication skills, with the ability to collaborate effectively across teams
  • Bachelor's Degree in Computer Science, Software Engineering, or a related field, or equivalent experience
  • Compensation and Benefits

    Benefits

    Along with competitive pay, as a full-time Tesla employee, you are eligible for the following benefits at day 1 of hire :

  • Aetna PPO and HSA plans >
  • 2 medical plan options with $0 payroll deduction

  • Family-building, fertility, adoption and surrogacy benefits
  • Dental (including orthodontic coverage) and vision plans, both have options with a $0 paycheck contribution
  • Company Paid (Health Savings Account) HSA Contribution when enrolled in the High Deductible Aetna medical plan with HSA
  • Healthcare and Dependent Care Flexible Spending Accounts (FSA)
  • 401(k) with employer match, Employee Stock Purchase Plans, and other financial benefits
  • Company paid Basic Life, AD&D, short-term and long-term disability insurance
  • Employee Assistance Program
  • Sick and Vacation time (Flex time for salary positions), and Paid Holidays
  • Back-up childcare and parenting support resources
  • Voluntary benefits to include : critical illness, hospital indemnity, accident insurance, theft & legal services, and pet insurance
  • Weight Loss and Tobacco Cessation Programs
  • Tesla Babies program
  • Commuter benefits
  • Employee discounts and perks program
  • Expected Compensation

    133,440 - $292,800 / annual salary cash and stock awards benefits

    Pay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, and experience. The total compensation package for this position may also include other elements dependent on the position offered. Details of participation in these benefit plans will be provided if an employee receives an offer of employment.

    Salary : $133,440 - $292,800

    If your compensation planning software is too rigid to deploy winning incentive strategies, it’s time to find an adaptable solution. Compensation Planning
    Enhance your organization's compensation strategy with salary data sets that HR and team managers can use to pay your staff right. Surveys & Data Sets

    What is the career path for a Observability Software Engineer, AI Infrastructure?

    Sign up to receive alerts about other jobs on the Observability Software Engineer, AI Infrastructure career path by checking the boxes next to the positions that interest you.
    Income Estimation: 
    $123,167 - $152,295
    Income Estimation: 
    $146,673 - $180,130
    Income Estimation: 
    $97,257 - $120,701
    Income Estimation: 
    $123,167 - $152,295
    Income Estimation: 
    $92,369 - $122,605
    Income Estimation: 
    $117,024 - $149,811
    Income Estimation: 
    $117,024 - $149,811
    Income Estimation: 
    $137,568 - $176,908
    Income Estimation: 
    $123,167 - $152,295
    Income Estimation: 
    $146,673 - $180,130
    View Core, Job Family, and Industry Job Skills and Competency Data for more than 15,000 Job Titles Skills Library

    Job openings at Tesla

    Tesla
    Hired Organization Address Sparks, NV Full Time
    What To Expect The team is responsible for designing, developing, and installing new manufacturing lines for stamping. T...
    Tesla
    Hired Organization Address Golden Valley, MN Full Time
    What To Expect Our Associate Sales Managers are seasoned retail leaders with natural charisma. In this role, you will co...
    Tesla
    Hired Organization Address Buffalo, NY Full Time
    What To Expect Tesla is hiring Process Engineering Technician at Gigafactory New York in Buffalo, NY to support Battery ...
    Tesla
    Hired Organization Address Buffalo, NY Full Time
    What To Expect Tesla has an immediate opening for a technical people leader to join the Production Engineering team in B...

    Not the job you're looking for? Here are some other Observability Software Engineer, AI Infrastructure jobs in the Fremont, CA area that may be a better fit.

    Observability Software Engineer, AI Infrastructure

    Tesla Motors, Inc., Fremont, CA

    AI Assistant is available now!

    Feel free to start your new journey!