8 Lead Data Engineer-AWS, Python Jobs in Scottsdale, AZ
SORT BY
D
P
Z
N
E
E
N
E
Lead Data Engineer-AWS, Python
$134k-161k (estimate)
Full Time
3 Days Ago
Sorry! This job is no longer available. Please explore similar jobs listed on the left.
Zortech Solutions is Hiring a Lead Data Engineer-AWS, Python Near Scottsdale, AZ
Role: Lead Data Engineer-AWS, PythonLocation: Scottsdale AZ (100% onsite)Duration:: FTE and CTHJob DescriptionMust have skill set: Spark, S3, Glue, AWS Redshift , python and stream set exp6-8 years of IT experience focusing on enterprise data architecture and management.
- Experience in Conceptual/Logical/Physical Data Modelling & expertise in Relational and Dimensional Data Modelling
- Experience with Databricks & on Prem , Structured Streaming, Delta Lake concepts, and Delta Live Tables required
- Experience with Spark scala
- Data Lake concepts such as time travel and schema evolution and optimization
- Structured Streaming and Delta Live Tables with Databricks a bonus
- Experience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation / support
- Advanced level understanding of streaming data pipelines and how they differ from batch systems
- Formalize concepts of how to handle late data, defining windows, and data freshness
- Advanced understanding of ETL and ELT and ETL/ELT tools such as Data Migration Service etc
- Understanding of concepts and implementation strategies for different incremental data loads such as tumbling window, sliding window, high watermark, etc.
- Familiarity and/or expertise with Great Expectations or other data quality/data validation frameworks a bonus
- Familiarity with concepts such as late data, defining windows, and how window definitions impact data freshness
- Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design performance optimization)
- Indexing and partitioning strategy experience
- Debug, troubleshoot, design and implement solutions to complex technical issues
- Experience with large-scale, high-performance enterprise big data application deployment and solution
- Architecture experience in AWS environment a bonus
- Familiarity working with Lambda specifically with how to push and pull data, how to use AWS tools to view data for processing massive data at scale a bonus
- Experience with Gitlabs and CloudWatch and ability to write and maintain gitlabs for supporting CI/CD pipelines
- Experience working with AWS Lambdas for configuration and optimization and experience with S3
- Familiarity with Schema Registry, message formats such as Avro, ORC, etc.
- Ability to thrive in a team-based environment
- Experience briefing the benefits and constraints of technology solutions to technology partners, stakeholders, team members, and senior level of management
Job Summary
JOB TYPE
Full Time
SALARY
$134k-161k (estimate)
POST DATE
07/04/2024
EXPIRATION DATE
07/04/2024
WEBSITE
zortechsolutions.com
SIZE
<25
Show more
Similar Jobs
Popular Articles
Top 10 Things to Avoid Putting on a Job Application
Read Article
How to Describe Your Education in a Resume
Read Article
How to Write a Memorable Cover Letter in 5 Ways
Read Article
4 Personal Finance Tips to Make During Job Transition
Read Article
4 Best Camera Setups for Interview
Read Article
Keeping a Job Search Journal: 9 Types of Journaling You Can Do
Read Article
