What are the responsibilities and job description for the Engineer - Data III position at LanceSoft, Inc.?
Pay Rate: $70- $78 hourly on W2
Data Engineer III
Onsite: 3 Days a Week
We are currently seeking a Senior Data Engineer to join our team and play a pivotal role in leveraging the Palantir Foundry platform to enhance our data capabilities.
As a Senior Data Engineer, you will be instrumental in driving the practice of federating data ingestion, curation, and hydrating the ontology within our Palantir Foundry platform. Your expertise will guide the definition of common standards for data pipelines, ensuring high data quality, robust Role-Based Access Control (RBAC), and precise data classification. You will be at the helm of creating patterns and standards for Change Data Capture (CDC), incremental loads, and streaming data within the platform. Additionally, you will lead the design and implementation of DevOps practices, data platform monitoring, and observability to ensure operational excellence and high availability of data services. Need to solution & implement complex data pipelines orchestration for a knowledge graph build.
Responsibilities:
Federate Data Ingestion and Curation: Lead the efforts to streamline data ingestion and curation processes, ensuring efficient and scalable data handling within the Palantir Foundry platform.
Hydrate Ontology: Spearhead the development and maintenance of the ontology, facilitating a structured and intuitive understanding of the data landscape.
Define Data Pipeline Standards: Establish common standards for data pipelines, focusing on data quality, RBAC, and data classification to maintain integrity and security.
Develop Data Patterns and Standards: Create and implement patterns and standards for CDC, incremental loads, and real-time data streaming, enhancing the platform's responsiveness and flexibility.
Lead Ontology Build: Guide the construction of the ontology, ensuring it meets the complex needs of aerospace manufacturing data.
Mentor Data Engineers: Provide leadership and mentorship to other data engineers, promoting the adoption of standards, patterns, and DataOps practices for federating data pipeline work across the enterprise.
Accelerate Delivery and Self-Service: Drive the acceleration of data pipeline delivery and promote data self-service consumption, enabling faster decision-making and innovation.
DevOps Practices: Design and implement DevOps practices for data operations, ensuring continuous integration, continuous delivery, and automated testing are at the core of the data engineering workflow.
Monitoring and Observability: Establish comprehensive monitoring and observability frameworks for the data platform, enabling proactive issue detection and resolution to maintain high service availability.
Technology Stack Experience:
Programming Languages - Python, PySpark, Java, SQL
Databases - Postgres, Oracle
Cloud Services - AWS, Azure, GCP (storage, compute, databases, catalogs, streaming, replication, Queueing & Notification, Logging & Monitoring service) - any one of the cloud platforms is good but AWS is preferred.
Big Data Frameworks
Metadata Catalogs - DataHub, Informatica, Collibra (experience in any one is good)
Data Quality Platforms - Anomolo, Informatica, BigEye (experience in any one is good)
Event Platforms - Kafka, MSK
Data Platforms - Palantir Foundry, Databricks, Snowflake (experience in any one is good but Foundry experience will be a bonus)
Qualifications:
Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
5-8 years required in the data engineering & data management with a strong focus on data platforms like Databricks, Snowflake or Palantir Foundry
Extensive knowledge of data ingestion, ETL processes, and data modeling.
Strong problem-solving skills and the ability to work with complex data integration challenges.
Experience with data reconciliation, data pipeline & data monitoring, and performance optimization.
Excellent communication and teamwork skills
Leadership experience and the ability to mentor junior engineers.
Prior experience in aerospace manufacturing or a related industry is a plus
Ability to earn trust, maintain positive and professional relationships, and contribute to a culture of inclusion
Qualifications:
Locations:
Huntsville, AL
Cape Canaveral, FL
Renton, Washington
Data Engineer III
Onsite: 3 Days a Week
We are currently seeking a Senior Data Engineer to join our team and play a pivotal role in leveraging the Palantir Foundry platform to enhance our data capabilities.
As a Senior Data Engineer, you will be instrumental in driving the practice of federating data ingestion, curation, and hydrating the ontology within our Palantir Foundry platform. Your expertise will guide the definition of common standards for data pipelines, ensuring high data quality, robust Role-Based Access Control (RBAC), and precise data classification. You will be at the helm of creating patterns and standards for Change Data Capture (CDC), incremental loads, and streaming data within the platform. Additionally, you will lead the design and implementation of DevOps practices, data platform monitoring, and observability to ensure operational excellence and high availability of data services. Need to solution & implement complex data pipelines orchestration for a knowledge graph build.
Responsibilities:
Federate Data Ingestion and Curation: Lead the efforts to streamline data ingestion and curation processes, ensuring efficient and scalable data handling within the Palantir Foundry platform.
Hydrate Ontology: Spearhead the development and maintenance of the ontology, facilitating a structured and intuitive understanding of the data landscape.
Define Data Pipeline Standards: Establish common standards for data pipelines, focusing on data quality, RBAC, and data classification to maintain integrity and security.
Develop Data Patterns and Standards: Create and implement patterns and standards for CDC, incremental loads, and real-time data streaming, enhancing the platform's responsiveness and flexibility.
Lead Ontology Build: Guide the construction of the ontology, ensuring it meets the complex needs of aerospace manufacturing data.
Mentor Data Engineers: Provide leadership and mentorship to other data engineers, promoting the adoption of standards, patterns, and DataOps practices for federating data pipeline work across the enterprise.
Accelerate Delivery and Self-Service: Drive the acceleration of data pipeline delivery and promote data self-service consumption, enabling faster decision-making and innovation.
DevOps Practices: Design and implement DevOps practices for data operations, ensuring continuous integration, continuous delivery, and automated testing are at the core of the data engineering workflow.
Monitoring and Observability: Establish comprehensive monitoring and observability frameworks for the data platform, enabling proactive issue detection and resolution to maintain high service availability.
Technology Stack Experience:
Programming Languages - Python, PySpark, Java, SQL
Databases - Postgres, Oracle
Cloud Services - AWS, Azure, GCP (storage, compute, databases, catalogs, streaming, replication, Queueing & Notification, Logging & Monitoring service) - any one of the cloud platforms is good but AWS is preferred.
Big Data Frameworks
Metadata Catalogs - DataHub, Informatica, Collibra (experience in any one is good)
Data Quality Platforms - Anomolo, Informatica, BigEye (experience in any one is good)
Event Platforms - Kafka, MSK
Data Platforms - Palantir Foundry, Databricks, Snowflake (experience in any one is good but Foundry experience will be a bonus)
Qualifications:
Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
5-8 years required in the data engineering & data management with a strong focus on data platforms like Databricks, Snowflake or Palantir Foundry
Extensive knowledge of data ingestion, ETL processes, and data modeling.
Strong problem-solving skills and the ability to work with complex data integration challenges.
Experience with data reconciliation, data pipeline & data monitoring, and performance optimization.
Excellent communication and teamwork skills
Leadership experience and the ability to mentor junior engineers.
Prior experience in aerospace manufacturing or a related industry is a plus
Ability to earn trust, maintain positive and professional relationships, and contribute to a culture of inclusion
Qualifications:
- Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
- 5-8 years required in the data engineering & data management with a strong focus on data platforms like Databricks, Snowflake or Palantir Foundry
- Extensive knowledge of data ingestion, ETL processes, and data modeling.
- Strong problem-solving skills and the ability to work with complex data integration challenges.
- Experience with data reconciliation, data pipeline & data monitoring, and performance optimization.
- Excellent communication and teamwork skills
- Leadership experience and the ability to mentor junior engineers.
- Prior experience in aerospace manufacturing or a related industry is a plus
- Ability to earn trust, maintain positive and professional relationships, and contribute to a culture of inclusion
Salary : $70 - $78
