What are the responsibilities and job description for the Lead Data Engineer PySpark & Cloudera CDP Specialist : Chicago, IL (Onsite) position at New York Technology Partners?
Job Details
Position Details
- Title: Team Member PYSPARK Tech Lead
- Location: Chicago, IL (Onsite)
- Duration: Contract
- Experience Required: 7 11 Years
- Client: CTS/Abbvie
- Pay Rate: $75/hr C2C or $60/hr W2
Must-Have Skills
- Cloudera HUE
- Cloudera Data Platform
- PySpark
- Spark
- Apache Iceberg
Job Summary/Responsibilities
- Design and implement data solutions using Cloudera HUE and CDP
- Integrate Apache Iceberg and Spark into data pipelines
- Guide and mentor junior developers in PySpark best practices
- Collaborate with teams to gather data requirements and build scalable workflows
- Maintain documentation and ensure data quality through testing
- Optimize pipelines for performance and efficiency
- Conduct code reviews and resolve complex data issues
- Stay current on emerging data technologies and incorporate innovations
- Drive research and development initiatives
- Communicate effectively with stakeholders and contribute to project planning
- Demonstrate ownership and ensure timely delivery of high-quality solutions
Qualifications
- Proficient in PySpark and Spark-based data engineering
- Hands-on experience with Cloudera Data Platform and Apache Iceberg
- Strong understanding of data architecture principles
- Excellent problem-solving, communication, and collaboration skills
- Background in R&D is a plus
- Strong mentoring abilities and a continuous learning mindset
Aditya Jain Direct: EXT: 482 | |
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
Salary : $60 - $75
