What are the responsibilities and job description for the Engineering Manager, Data Acquisition (Foundations) position at OpenAI?
About the team :
The Data Acquisition team within the Foundations organization at OpenAI is responsible for all aspects of data collection to support all training and scaling across the research organization. Our team manages web crawling and GPTBot services and works closely with the Data Processing, Architecture, and Scaling teams. We are seeking a Senior Engineering Manager to lead our Data Acquisition team, overseeing engineering initiatives and ensuring effective collaboration across sub-teams.
What you will work on :
Lead and manage a team, driving engineering projects related to web crawling, data ingestion, and search capabilities.
Oversee the architecture, development, and deployment of scalable distributed systems capable of handling petabytes of data.
Collaborate closely with cross-functional teams, including Data Processing, Architecture, Scaling, and Legal, ensuring seamless integration, data flow, and system compliance with data privacy regulations.
Provide technical leadership, mentoring, and career development to engineers within the team, fostering an inclusive and high-performing environment.
Develop and maintain long-term strategic planning for data acquisition needs, identifying new opportunities for data collection and processing improvements.
Partner with senior leadership to align team priorities with broader organizational goals and ensure successful execution of team initiatives.
Ensure high reliability, scalability, and performance of data acquisition services, optimizing existing processes and introducing new technologies where appropriate.
Advocate for best practices in software engineering, including robust testing, system monitoring, and continuous improvement of team processes.
Oversee routine system checks and monitoring for backend services deployed in a Kubernetes Infrastructure-as-Code environment.
Qualifications :
10 years of industry experience in software engineering, with a focus on large-scale distributed systems and data processing.
4 years of experience in managing and leading engineering teams, with a proven ability to mentor and grow engineering talent.
Strong expertise in building and scaling large web crawlers, data pipelines, and ingestion systems.
Experience working with Kubernetes and Infrastructure-as-Code concepts.
Demonstrated ability to drive complex, cross-team engineering projects and lead collaboration across different functions.
Strong technical and strategic vision, with the ability to translate broad objectives into actionable engineering plans.
Excellent communication skills, with the ability to effectively interact with technical and non-technical stakeholders.
A passion for innovation and the ability to foster a culture of learning and continuous improvement.
