What are the responsibilities and job description for the Python Developer with AWS and Glue position at N Consulting Ltd?
Job Title: Python Developer with AWS Glue Experience
Location: Northampton(Hybrid)
Employment Type: Contract
About the Role:
We are seeking a skilled Python Developer with a strong background in AWS services and experience in building and managing Glue jobs. The ideal candidate will design and implement scalable data pipelines and transformation solutions using AWS Glue and other AWS services, contributing to the success of data-driven projects.
Key Responsibilities:
- Develop, deploy, and maintain data pipelines using AWS Glue, Python, and related AWS services.
- Write efficient ETL scripts for data extraction, transformation, and loading across various sources and destinations.
- Collaborate with data engineers, data scientists, and stakeholders to gather requirements and ensure optimal data workflows.
- Monitor and optimize performance for Glue jobs, ensuring cost-effectiveness and scalability.
- Implement and enforce best practices for data governance, security, and compliance using AWS services.
- Utilize other AWS services like S3, Lambda, Athena, CloudWatch, IAM, DynamoDB, and Redshift for data integration and processing tasks.
- Troubleshoot and resolve issues in existing data pipelines and workflows.
- Participate in Agile ceremonies such as sprint planning, daily standups, and retrospectives.
Key Requirements:
- 6 years of experience in Python programming with a focus on data processing.
- Proficient in AWS Glue, including Glue ETL scripts and Glue Studio.
- Solid experience with other AWS services like S3, Lambda, CloudWatch, Athena, and Redshift.
- Strong understanding of data pipeline design and ETL processes.
- Hands-on experience with SQL for querying and transforming data.
- Familiarity with Spark or distributed data processing frameworks.
- Knowledge of data formats like JSON, Parquet, and Avro.
- Experience with version control tools like Git and CI/CD pipelines.
- Strong problem-solving skills and ability to troubleshoot complex data workflows.
- Familiarity with Agile methodologies and tools like Jira.
Preferred Qualifications:
- AWS certification (e.g., AWS Certified Developer, AWS Certified Solutions Architect).
- Experience with DevOps practices and Infrastructure as Code (IaC) using tools like CloudFormation or Terraform.
- Knowledge of Data Lakes, Data Warehousing, and Big Data technologies.
- Familiarity with scheduling tools like Apache Airflow or AWS Step Functions.
- Experience working with serverless architectures and microservices.
Salary : $45 - $50
