What are the responsibilities and job description for the Senior Hadoop Developer position at TekJobs?
Required Skills: Hadoop, Hive, HBase, Sqoop, PySpark, Python, SQL
Job Description
Responsibilities:
• Develop and maintain Hadoop/PySpark applications: Write efficient and scalable code for data ingestion, processing, and analysis using Hadoop ecosystem tools (HDFS, Hive, HBase) and PySpark.
• Data pipeline development: Design and implement end-to-end data pipelines for batch and real-time processing.
• Data transformation: Utilize PySpark to transform and aggregate data from various sources.
• Performance optimization: Continuously monitor and optimize Hadoop/PySpark jobs to ensure efficient resource utilization and timely processing.
• Collaboration: Work closely with business analysts and Data analysts to translate business requirements into technical solutions.
• Testing and deployment: Provide testing support during SIT/UAT phase and assist in deploying solutions to production environments.
Required Skills and Experience:
• Hadoop ecosystem: Proficiency in Hadoop core components and related tools (Hive, HBase, Sqoop).
• PySpark expertise: Strong PySpark skills with experience in developing data processing pipelines.
• Python programming: Excellent Python programming skills with a focus on data manipulation and analysis libraries (Pandas, NumPy).
• SQL proficiency: Ability to write efficient SQL queries for data extraction and analysis within Hive or other SQL-like interfaces.
• Problem-solving: Strong analytical and problem-solving skills to tackle data-related challenges.
• Communication: Effective communication skills to collaborate with diverse stakeholders.
• Other: Retail Banking domain experience and Data Stage knowledge will be good to have.
