What are the responsibilities and job description for the Senior Data Engineer position at Harnham?
Senior Data Engineer Technology United States (Remote) $150,000 - $180,000 equity
The Company Our client is revolutionizing operational safety in critical industries. They specialize in leveraging computer vision to provide real-time insights for the Construction industry. Their mission is to eliminate failures due to inadequate visual inspection, ensuring safety, reliability, and compliance.
Role Description As a Senior Data Engineer, you will play a pivotal role in designing and optimizing scalable data pipelines that power machine learning and analytics applications. You will be responsible for architecting a robust data infrastructure, collaborating with ML engineers, and ensuring efficient storage and processing of large-scale industrial and vision data. This is an opportunity to make a significant impact on a rapidly growing team that values technical excellence and innovation.
Key Responsibilities
- Architect and build scalable data pipelines using Dagster to transform and move data for analytics and ML.
- Design and optimize storage solutions for large-scale industrial and vision data, ensuring efficient retrieval and processing.
- Develop data ingestion frameworks for real-time and batch processing of images, video, and metadata.
- Work with ML engineers to structure data effectively for model training and experimentation.
- Deploy and manage data pipelines in Kubernetes-based environments.
- Automate CI / CD workflows for data infrastructure using GitLab CI / CD.
- Manage AWS-based data infrastructure, leveraging Terraform for Infrastructure as Code.
- Implement data governance best practices, ensuring data quality, lineage tracking, and metadata management.
- Optimize batch and real-time processing frameworks for scalability and performance.
- Provide technical leadership, establishing best practices and guiding future data scaling strategies.
Requirements
The Benefits As a Senior Data Engineer, you will receive a base salary of $150,000 to $180,000 plus benefits like equity, health, dental and vision.
How to Apply To apply, please submit your resume via the link on this page.
Key Words Python, Dagster, Kubernetes, Terraform, AWS, Data Pipelines, CI / CD, Streaming Data, Apache Ray, Kafka, Computer Vision, Machine Learning, SQL, Big Data, Data Engineering
Salary : $150,000 - $180,000
