What are the responsibilities and job description for the Spark Developer / Engineer position at MBS Pro Staffing?
Job Description
Job Description
MBS is hiring a Spark Developer / Engineer. This position is a remote full-time, temp to hire opportunity.
PRINCIPAL DUTIES & RESPONSIBILITIES
- Understanding the Existing Scalding Codebase
- Analyze the current Scalding-based data pipelines.
- Document existing business logic and transformations.
- Migrating the Logic to Spark
- Convert existing Scalding jobs into Spark (PySpark / Scala) while ensuring optimized performance.
- Refactor data transformations and aggregations in Spark.
- Optimize Spark jobs for efficiency and scalability.
- Ensuring Data Parity & Validation
- Develop data parity tests to compare outputs between Scalding and Spark implementations.
- Identify and resolve any discrepancies between the two versions.
- Work with stakeholders to validate correctness.
- Writing Unit Tests & Improving Code Quality
- Implement robust unit and integration tests for Spark jobs.
- Ensure code meets engineering best practices (modular, reusable, and well-documented).
REQUIREMENTS
WHY MBS ?
ABOUT THE KABLE GROUP
For over 50 years, The Kable Group has been committed to connecting candidates looking for reliable work opportunities with companies in need of workforce solutions. We leverage our expertise in core industries to ensure that the needs of our clients and our candidates alike are satisfied. Whether it’s a professional position, skilled labor need or an education opportunity, The Kable Group’s suite of solutions caters to finding the right fit, every time.
At the Kable Group, we do not just accept diversity — we celebrate it, we support it, and we thrive on it. Read our full diversity statement here.
This job description is not intended to be all inclusive and the employee will also perform other reasonably related business duties as assigned by the immediate supervisor and other management as required.
