Infrastructure Engineer

About Arc Institute

The Arc Institute is a new scientific institution that conducts curiosity-driven basic science and technology development to understand and treat complex human diseases. Headquartered in Palo Alto, California, Arc is an independent research organization founded on the belief that many important research programs will be enabled by new institutional models. Arc operates in partnership with Stanford University, UCSF, and UC Berkeley.

While the prevailing university research model has yielded many tremendous successes, we believe in the importance of institutional experimentation as a way to make progress. These include:

• Funding: Arc will fully fund Core Investigator's (PI's) research groups, liberating scientists from the typical constraints of project-based external grants.
• Technology: Biomedical research has become increasingly dependent on complex tooling. Arc Technology Centers develop, optimize and deploy rapidly advancing experimental and computational technologies in collaboration with Core Investigators.
• Support: Arc aims to provide first-class support—operationally, financially and scientifically—that will enable scientists to pursue long-term high risk, high reward research that can meaningfully advance progress in disease cures, including neurodegeneration, cancer, and immune dysfunction.
• Culture: We believe that culture matters enormously in science and that excellence is difficult to sustain. We aim to create a culture that is focused on scientific curiosity, a deep commitment to truth, broad ambition, and selfless collaboration.

Arc has scaled to nearly 200 people to date. With $650M in committed funding and a state of the art new lab facility in Palo Alto, Arc will continue to grow quickly in the coming years.

About the position

We are seeking an Infrastructure Engineer to join our team. In this role, you will be responsible for designing and optimizing our Hybrid Cloud Infrastructure Platform across public, private, and on-premise datacenters. You will work closely with researchers, developers, and IT professionals to ensure the availability, reliability, and performance of our compute, networking, and storage. Your work will fuel the development of AI biological foundation models (i.e. Evo; Arc's recently expanded DNA foundation model), the Virtual Cell Initiative, and other cutting-edge bioinformatic projects in the context of Institute-wide efforts.

About you

• You lead with empathy. You know that successful systems are more about the user than the tool. You enjoy building relationships and credibility with your colleagues.
• You enjoy solving problems. Any new project is an interesting puzzle. So is a tricky troubleshooting issue. You get satisfaction from helping someone get to resolution.
• You're curious. You like to keep track of the latest developments in your field, and to learn about the substance behind your employer's mission.

In this position you will

• Oversee the operation and optimization of our private cloud GPU cluster, focusing on enhancing availability, performance, and user experience.
• Design and execute strategies for automating system configurations efficiently and safely, ensuring minimal disruption to production.
• Develop a unified compute capacity platform with fixed and autoscaling resources across private and public cloud resources.
• Facilitate efficient, high-throughput, and seamless data transfer between instruments and compute environments.
• Enable the continuous integration and deployment of long-running services and databases across our hybrid platform.
• Elevate system reliability by achieving additional "nines" of availability.
• Develop and maintain comprehensive security protocols, including network security measures, access controls, vulnerability assessments, and continuous monitoring, to protect infrastructure and data from potential threats and breaches.
• Collaborate with scientists to assess their computational requirements and deliver tailored resources and support.
• Create and maintain comprehensive documentation for system configurations, operational procedures, security policies, and end-user guidance through a well-organized Wiki.

Requirements

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Extensive experience with distributed systems, including cloud platforms (AWS, GCP, or Azure) and/or HPC environments (Slurm, Kubernetes, Grid Engine, Torque, etc.).
• Advanced Linux system administration skills, including performance tuning and troubleshooting
• Proficiency in bare-metal system provisioning (Ansible, Puppet, Chef, Virtualization and/or Containerization).
• Proven ability in scripting languages like Python, Bash, or Perl.
• Familiarity with network protocols, storage systems, and high-speed interconnects (InfiniBand, RoCE).
• Working knowledge with monitoring tools like Nagios, Prometheus/Grafana, or New Relic.
• Experience developing and maintaining software that interacts with Nvidia GPUs, including drivers and diagnostic tools (CUDA, nvcc, nccl, etc.).
• Strong understanding of security best practices with hands-on experience in implementing and maintaining security measures.
• Excellent problem-solving skills and the ability to work under pressure.
• Strong communication and collaboration skills.

The base salary range for this position is $149,500-$184,000. These amounts reflect the range of base salary that the Institute reasonably would expect to pay a new hire or internal candidate for this position. The actual base compensation paid to any individual for this position may vary depending on factors such as experience, market conditions, education/training, skill level, and whether the compensation is internally equitable, and does not include bonuses, commissions, differential pay, other forms of compensation, or benefits. This position is also eligible to receive an annual discretionary bonus, with the amount dependent on individual and institute performance factors.