What are the responsibilities and job description for the Python Developer position at ATC?
Python developer
Location: Jersey City, NJ
Visa: GC or USC
Hybrid role
Duration : 6 months with extension
Responsibilities:
- Design and implement a scalable data warehouse on Azure Databricks using data and dimensional modeling techniques to support analytical and reporting requirements.
- Develop and optimize ETL/ELT pipelines using Python, Azure Databricks and PySpark for large-scale data processing, ensuring data quality, consistency, and integrity.
- Establish and implement best practices for data ingestion, transformation, and storage using the medallion architecture (Bronze, Silver, Gold).
- Architect and develop highly scalable data applications using Azure Databricks and distributed computing.
- Optimize Databricks clusters and ETL/ELT workflows for performance and scalability.
- Manage data storage solutions using Azure Data Lake Storage (ADLS) and Delta Lake while leveraging Unity Catalog for data governance, security, and access control.
- Develop and schedule Databricks notebooks and jobs for automated daily execution, implementing monitoring, alerting, and automated recovery processes for job failures.
- Identify and resolve bottlenecks in existing code and follow best coding practices to improve performance and maintainability.
- Use GitHub as version control tool to manage code and collaborate effectively with other developers; build and maintain CI/CD pipelines for deployment and testing using Azure DevOps and GitHub.
- Create comprehensive documentation for data architecture, ETL processes, and business logic.
- Work closely with business stakeholders to understand project goals and architect scalable and efficient solutions.
- Knowledge of user authentication on Unity Catalog and authorization between multiple systems, servers and environments.
- Ensure that programs are written to the highest standards (e.g., Unit Tests) and technical specifications.
- Ability to collaborate on projects and work independently when required.
Qualifications:
- 5 years of prior experience as a developer in the required technologies (Azure Databricks, Python, PySpark, Datawarehouse Designing)
- Solid organizational skills, ability to multi-task across different projects
- Experience with Agile methodologies
- Skilled at independently researching topics using all means available to discover relevant information.
- Ability to work in a team environment.
- Excellent verbal and written communication skills
- Self-starter with ability to multi-task and to maintain momentum.
