What are the responsibilities and job description for the Data Engineer position at United IT Solutions?
Must have : Skillset : Java, Scala, S3, Glue, aws , Redshift
6-8 years of IT experience focusing on enterprise data architecture and management.
Experience in Conceptual / Logical / Physical Data Modeling & expertise in Relational and Dimensional Data Modeling
Experience with Databricks & on Prem, Structured Streaming, Delta Lake concepts, and Delta Live Tables required
Experience with Spark Scala and Java programming
Data Lake concepts such as time travel and schema evolution and optimization
Structured Streaming and Delta Live Tables with Databricks as a bonus
Experience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data
mart build, and data lakes implementation / support
Advanced level understanding of streaming data pipelines and how they differ from batch systems
Formalize concepts of how to handle late data, defining windows, and data freshness
Advanced understanding of ETL and ELT and ETL / ELT tools such as Data Migration Service etc
Understanding of concepts and implementation strategies for different incremental data loads such as tumbling window, sliding window, high watermarks, etc.
Familiarity and / or expertise with Great Expectations or other data quality / data validation frameworks a bonus
Familiarity with concepts such as late data, defining windows, and how window definitions impact data freshness
Debug, troubleshoot, design, and implement solutions to complex technical issues
Experience with large-scale, high-performance enterprise big data application deployment and solution
Architecture experience in AWS environment a bonus
Familiarity working with Lambda specifically with how to push and pull data, how to use AWS tools to view data for processing massive data at scale a bonus
Experience with Gitlabs and CloudWatch and ability to write and maintain Gitlabs for supporting CI / CD pipelines
Experience working with AWS Lambdas for configuration and optimization and experience with S3
Familiarity with Schema Registry, and message formats such as Avro, ORC, etc.
Ability to thrive in a team-based environment
Experience briefing the benefits and constraints of technology solutions to technology partners, stakeholders, team members, and senior-level of management
Keep a pulse on the job market with advanced job matching technology.
If your compensation planning software is too rigid to deploy winning incentive strategies, it’s time to find an adaptable solution.
Compensation Planning
Enhance your organization's compensation strategy with salary data sets that HR and team managers can use to pay your staff right.
Surveys & Data Sets
What is the career path for a Data Engineer?
Sign up to receive alerts about other jobs on the Data Engineer career path by checking the boxes next to the positions that interest you.
