What are the responsibilities and job description for the Big Data Developer position at Apexsync Technologies Inc?
Job Title: Senior Big Data Engineer
Location: Phoenix, AZ (Locals Only)
Job Type: 12 Months W2
Onsite Interview
Experience Level: 10 years
Must Skills: "Java/Python, Spark/PySpark, SQL/SQL Query, Shell/Unix Scripting, Hive"
Position Overview:
As a Senior Big Data Engineer, you will play a crucial role in designing, developing, and optimizing our big data systems. You will leverage your extensive experience in Java, Python, and big data technologies to build scalable solutions and ensure the efficient processing and analysis of large datasets. Your expertise will help drive key business decisions and strategies.
Key Responsibilities:
- Architecture & Design: Lead the design and implementation of scalable big data architectures using Spark/PySpark, ensuring high performance and reliability.
- Data Processing: Develop and maintain complex data pipelines and workflows, utilizing SQL/SQL Query, Hive, and other relevant technologies to process and analyze large volumes of data.
- Programming: Write efficient and maintainable code in Java and Python for data processing, transformation, and integration tasks.
- Scripting: Create and optimize Shell/Unix scripts to automate data processing tasks and streamline workflows.
- Performance Optimization: Monitor and optimize the performance of big data systems, troubleshoot issues, and implement improvements to enhance processing speed and accuracy.
- Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver actionable insights.
- Documentation: Maintain comprehensive documentation of data processes, workflows, and system configurations.
- Leadership: Mentor and guide junior engineers, sharing best practices and providing technical support.
Qualifications:
- Education: Bachelor’s degree in Computer Science, Engineering, or a related field. Advanced degree preferred.
- Experience: 10 years of hands-on experience in big data technologies and frameworks.
- Technical Skills:
- Java: Expertise in developing and maintaining applications using Java.
- Python: Proficiency in Python for data manipulation and scripting.
- Spark/PySpark: In-depth experience with Apache Spark and PySpark for big data processing.
- SQL/SQL Query: Advanced skills in writing complex SQL queries for data extraction and manipulation.
- Shell/Unix Scripting: Strong experience with Shell and Unix scripting for automation tasks.
- Hive: Proficiency in Hive for querying and managing large datasets.