As a Spark Developer, you will be responsible for designing, developing, and maintaining scalable data processing pipelines using Apache Spark. You will work closely with cross-functional teams to understand requirements, design solutions, and implement robust Spark applications to meet business needs. Your role will involve optimizing Spark performance, troubleshooting issues, and ensuring the reliability and scalability of Spark clusters.

*Candidates with recent financial domain experience are highly preferred*

Responsibilities:

· Design, develop, and maintain Spark applications for data processing and analytics.

· Collaborate with data engineers, data scientists, and business stakeholders to understand requirements and translate them into technical solutions.

· Optimize Spark jobs for performance and efficiency, considering factors such as data partitioning, caching, and resource management.

· Troubleshoot and debug issues in Spark applications, ensuring high availability and reliability of Spark clusters.

· Implement best practices for Spark development, including code quality standards, testing methodologies, and version control.

· Monitor Spark clusters and performance metrics, proactively identifying and addressing bottlenecks and scalability issues.

· Stay up-to-date with the latest trends and advancements in Apache Spark and big data technologies, continuously enhancing your skills and knowledge.

Qualifications:

· Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

· Proven experience developing Spark applications using Scala, Java, or Python.

· Experience with cloud platforms such as AWS, Azure, or Google Cloud is a plus.

· Strong understanding of distributed computing principles and Spark architecture.

· Proficiency in SQL and experience working with relational and non-relational databases.

· Familiarity with big data technologies such as Hadoop, Kafka, and HDFS.

· Excellent problem-solving skills and attention to detail.

· Effective communication and collaboration skills, with the ability to work in a cross-functional team environment

· Relevant certifications in Spark or big data technologies are desirable.