Data Engineering Lead Full-time San Francisco

We are seeking a Data Engineering Lead for FutureAI to drive our data infrastructure initiatives. This critical role will orchestrate our data pipeline operations within Google Cloud Platform (GCP), develop robust ETL processes, and manage diverse data ingestion workflows. As FutureAI's Data Engineering Lead you will focus on pulling data into (GCP) using Google APIs, ingesting flat files, and Kaggle datasets with text, images, attachments and video as the sources of input data. The ideal candidate will have deep expertise in data processing architectures, demonstrate experience with GCP's data services, and the ability to transform complex, multi-modal data sources into structured analysis-ready formats that power our analytics and machine learning systemsKey Responsibilities Build data pipelines for user data and structured files for developer data.Build data pipelines to ingest Google Drive data for users using Google API.Build data pipelines to ingest developer data as flat files.Set up data storage options that can address security, embedding, and latency requirements.Perform setup of DLP Filtering.Implement Bigtable schema design to store the serving data.Implement post-processing requirements including filters and DLP.Qualifications Solid education with 6-8 years experience in data engineering.Strong programming skills in SQL and Python.Proficiency in database administration.Experience with Google Cloud.Strong problem-solving skills and ability to troubleshoot complex data issues.Excellent communication skills and ability to work collaboratively in a team environment.A proactive mindset with a willingness to take ownership and drive projects to completion.Strong organisational skills along with a desire to continually be challenged.Experience with monitoring, CI / CD.Preferred Skills Has utilized Google Cloud Platform on a production grade project for at least 2 years.Self-reliant, reliable and understands the fast paced nature of startups.Capable of doing everything from initiating the project to deployment environmentWhat We Offer Opportunity to shape the future of generative application development.Substantial resources and support to pursue state-of-the-art AI research and development.Competitive compensation package, including equity.Autonomy to build and lead a world-class data engineering team.Collaboration with top talent across various disciplines.Platform to make a significant impact on products used by billions globally.Support for continued research, publication, and participation in the academic AI community.We're looking for a talented Data Engineering Lead to architect and own FutureAI's data infrastructure and pipelines. If you're passionate about building scalable data systems and want to process diverse data streams spanning text, images, and video within GCP, we want to hear from you#J-18808-Ljbffr