What are the responsibilities and job description for the Data Pipeline Operations Engineer position at SixMap, Inc.?
We are seeking a detail-oriented and technically skilled Data Pipeline Operations Engineer to manage and execute our weekly scanning process. This critical role ensures the timely flow of customer data through our research, scanning, and UI ingest pipeline. The ideal candidate has a mix of programming, database, and Linux system administration skills to handle the various steps in the scanning workflow.
SixMap protects some of the world’s largest and most complex government and corporate enterprises with a continuous threat exposure management (CTEM) platform. The SixMap platform is powered by an advanced computational mapping engine that quickly discovers and continuously monitors the unique contours of the extended enterprise’s Internet-facing assets across IPv4 and IPv6. Upon providing cyber defenders this comprehensive enterprise visibility, the platform complements this awareness with contextual threat intelligence and a suite of remediation measures. The SixMap team brings deep intelligence community expertise and best practices to the defense of both U.S. Federal agencies and Fortune 500 corporations. For more information, please visit : www.sixmap.io .
Responsibilities
- Manage the weekly scanning process, ensuring customer data progresses through research, scanning, and UI ingest phases according to defined SLAs
- Prepare input files and kick off processes on the scanning cluster via Airflow
- Monitor and troubleshoot jobs, adjusting parameters like rate files as needed to optimize runtimes
- Perform data ingest into production databases using SQL and Python
- Clear data artifacts and caches in between ingest cycles
- Execute post-ingest data refresh routines
- Perform quality checks on ingested data to validate contractual obligations are met
- Identify process bottlenecks and suggest or implement improvements to the automated tooling to increase speed and reliability
Requirements
SixMap is an Equal Opportunity Employer and considers applicants for employment without regard to race, color, religion, sex, orientation, national origin, age, disability, genetics, or any other basis forbidden under federal, state, or local laws. All new hires must pass a background check as a condition of employment.
Benefits
Salary : $165,000 - $175,000
