Demo

Lead Data engineer - c2c - candidate must be local to San Francisco, CA (posted by Sid)

Global Force USA
San Francisco, CA Full Time
POSTED ON 4/3/2025 CLOSED ON 4/25/2025

What are the responsibilities and job description for the Lead Data engineer - c2c - candidate must be local to San Francisco, CA (posted by Sid) position at Global Force USA?

Job Details

Role: Lead Data Engineer c2c

Duration: Long Term contract

Location: San Francisco, CA

Required Skills

  • Python and PySpark. Kafka and Kafka streams. MySQL and MySQL Heat. Azure Delta Lake. ETL processes. Kafka integrations using Spring Boot Java. Data streaming with Spark.

Additional Skills

  • Experience with Delta Lake on Azure Tableau or similar data visualization tools Unity Catalog in Databricks Airflow for pipeline management

Responsibilities

Development Tasks:

  • Collect metrics based on user interactions.
  • Visualize data for business teams.
  • Develop and redesign data pipelines using Kafka streams.
  • Implement solutions using Spring Boot Java and Databricks Spark streaming.

Leadership Duties:

  • Lead the measurement processes from requirements gathering to production delivery.
  • Collaborate with other team leads, business partners, and product managers.
  • Balance between hands-on engineering (50%) and team leadership (50%).

Collaboration Structure:

  • Onsite: Lead role (this resource)

o Nearshore: Senior developer.

o Offshore: Data engineer role.

Lead Data Engineer - Job Description

Required Skills & Experience:

  • Hands-on code mindset with deep understanding in technologies / skillset and an ability to understand larger picture.
  • Sound knowledge to understand Architectural Patterns, best practices and Non-Functional Requirements
  • Overall, 8-10 years of experience in heavy volume data processing, data platform, data lake, big data, data warehouse, or equivalent.
  • 5 years of experience with strong proficiency in Python and Spark (must-have).
  • 3 years of hands-on experience in ETL workflows using Spark and Python.
  • 4 years of experience with large-scale data loads, feature extraction, and data processing pipelines in different modes near real time, batch, realtime.
  • Solid understanding of data quality, data accuracy concepts and practices.
  • 3 years of solid experience in building and deploying ML models in a production setup. Ability to quickly adapt and take care of data preprocessing, feature engineering, model engineering as needed.
  • Preferred: Experience working with Python deep learning libraries like any or more than one of these - PyTorch, Tensorflow, Keras or equivalent.
  • Preferred: Prior experience working with LLMs, transformers. Must be able to work through all phases of the model development as needed.
  • Experience integrating with various data stores, including:

o SQL/NoSQL databases

o In-memory stores like Redis

o Data lakes (e.g., Delta Lake)

  • Experience with Kafka streams, producers & consumers.
  • Required: Experience with Databricks or similar data lake / data platform.
  • Required: Java and Spring Boot experience with respect to data processing - near real time, batch based.
  • Familiarity with notebook-based environments such as Jupyter Notebook.
  • Adaptability: Must be open to learning new technologies and approaches.
  • Initiative: Ability to take ownership of tasks, learn independently, and innovate.
  • With technology landscape changing rapidly, ability and willingness to learn new technologies as needed and produce results on job.

Preferred Skills:

  • Ability to pivot from conventional approaches and develop creative solutions.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
Night Audit-Hotel Fusion-San Francisco, CA
Hotel Fusion, San Francisco, CA -
San Francisco, CA
Front Desk Clerk - Hotel Fusion, San Francisco, CA
Hotel Fusion, San Francisco, CA -
San Francisco, CA
University of San Francisco Sport Management MS in Sport Management Program (San Francisco · CA · Orange County · CA)
University of San Francisco -
San Francisco, CA

If your compensation planning software is too rigid to deploy winning incentive strategies, it’s time to find an adaptable solution. Compensation Planning
Enhance your organization's compensation strategy with salary data sets that HR and team managers can use to pay your staff right. Surveys & Data Sets

What is the career path for a Lead Data engineer - c2c - candidate must be local to San Francisco, CA (posted by Sid)?

Sign up to receive alerts about other jobs on the Lead Data engineer - c2c - candidate must be local to San Francisco, CA (posted by Sid) career path by checking the boxes next to the positions that interest you.
Income Estimation: 
$92,929 - $122,443
Income Estimation: 
$122,257 - $154,284
Income Estimation: 
$92,929 - $122,443
Income Estimation: 
$122,257 - $154,284
Income Estimation: 
$122,257 - $154,284
Income Estimation: 
$143,391 - $179,890
Income Estimation: 
$168,522 - $211,152
Income Estimation: 
$189,259 - $248,928
Income Estimation: 
$71,122 - $96,652
Income Estimation: 
$92,929 - $122,443
This job has expired.
View Core, Job Family, and Industry Job Skills and Competency Data for more than 15,000 Job Titles Skills Library

Job openings at Global Force USA

Global Force USA
Hired Organization Address Plano, TX Full Time
Job Details Required Qualifications: 5 years of software engineering experience creating and executing automated tests 2...
Global Force USA
Hired Organization Address Houston, TX Full Time
Job Details REQUIRED SKILLS: must have 5 years of Juniper experience Network routing, switching, security, IPAM, DNS Adv...
Global Force USA
Hired Organization Address Newtown, PA Full Time
Name: Clinical Appeals Reviewer Type: Staffing Status: Pending Client: AmeriHealth Caritas Services Start: 01/20/2025 En...
Global Force USA
Hired Organization Address Orlando, FL Contractor
Job Title: Specialty Pharmacy Technician This is an onsite position. Duties The Pharmacy Technician supports the pharmac...

Not the job you're looking for? Here are some other Lead Data engineer - c2c - candidate must be local to San Francisco, CA (posted by Sid) jobs in the San Francisco, CA area that may be a better fit.

Lead Data Engineer_San Francisco, CA_Only on W2_No C2C/1099

Chelsoft Solutions Co., San Francisco, CA

AI Assistant is available now!

Feel free to start your new journey!