What are the responsibilities and job description for the Lead Data Engineer with PySpark position at Infinity Outsourcing?
About the job Lead Data Engineer with PySpark
Requisition Title : Lead Data Engineer with PySpark ( 1 Day Onsite)
Locations Mt Laurel, NJ & Charlotte, NC
Visa Status : GC / USC / H4 EAD
Client : TD Bank
Duration : Full Time / C2H
Salary pay rate to candidates : $49-$56 / hr W2
Experience : 12 Years
Primary Tech skills = Pyspark, Databricks (Hands-on experience mandatory)
Secondary Tech skills = Azure, SAS, Synapse, Databricks
MUST HAVE SKILLS :
- Techno-Managerial covers the entire life cycle of data warehouse and data cloud environment with experience in building data platforms development.
- Technical co-ordination with Product Group Technical Lead, Architecture team, Data engineering team, BSA, and QE Team for the assigned projects.
- Ability to drive more than one project in parallel.
- Experience in building end to end architecture for Data Lakes, Data Warehouses and Data Marts
- Experience of DWH, Data Integration, Cloud, Architecture, Design, Data Modelling
- Hands-on experience in Pyspark is a must.
- Experience working with structured and unstructured data.
- Extensive knowledge of ETL and Data Warehousing concepts, strategies, methodologies
- Experience in data ingestion, preparation, integration, and operationalization techniques in optimally addressing the data requirements.
- Experience in relational data processing technology like MS SQL, Delta Lake, Spark SQL, SQL Server
- Experience with Orchestration tools and GitHub
- Experience to own end-to-end development, including coding, testing, debugging and deployment.
- Must be team oriented with strong communication, collaboration, prioritization, and adaptability skill
- Need to be an individual performer as well as a Lead.
Salary : $49 - $56
