What are the responsibilities and job description for the Python + PySpark - Junior Developer position at Compunnel?
Job Summary
The Python PySpark Junior Developer will be responsible for developing scalable data solutions using PySpark and Python. The role requires hands-on experience in data engineering, SQL, and cloud platforms such as AWS or Azure. The candidate should have strong problem-solving skills and experience in building efficient ETL pipelines for large-scale data processing.
Key Responsibilities
• Develop and maintain data processing pipelines using Python and PySpark
• Design and implement ETL workflows for structured and unstructured data
• Optimize data storage, retrieval, and transformation processes
• Work with cloud platforms (AWS or Azure) to deploy and manage data solutions
• Write complex SQL queries for data transformation and analysis
• Collaborate with data engineers, analysts, and stakeholders to ensure data integrity and reliability
• Troubleshoot and resolve performance issues in data pipelines
Required Qualifications
• 5 years of experience in data engineering
• Strong proficiency in Python and object-oriented programming
• Hands-on experience with PySpark for large-scale data processing
• Proficiency in SQL for data manipulation and query optimization
• Experience with cloud platforms (AWS or Azure)
• Knowledge of ETL processes and data warehousing concepts
• Experience with Hadoop is acceptable
Preferred Qualifications
• Experience in optimizing Spark jobs for performance and efficiency
• Familiarity with DevOps practices for data engineering
• Understanding of data governance and security best practices
Location : New York, NY
Education : Bachelors Degree
