What are the responsibilities and job description for the Sr. Data Engineer position at Wavicle Data Solutions?
About the Role
Is your CV ready If so, and you are confident this is the role for you, make sure to apply asap.
We are looking for a Senior Data Engineer with strong real-life experience in python development, including pySpark in an AWS Cloud environment. The ideal engineer will have experience with AWS Cloud on data integration with Apache Spark, EMR, Glue, Kafka, Kinesis and Lambda in S3, Redshift, RDS, and MongoDB / DynamoDB ecosystems.
Responsibilities :
Build the infrastructure required for optimal extraction, transformation and loading of data from a wide variety of sources Hadoop, Spark, AWS Lambda, etc.
Design, develop, test, deploy, maintain and improve data integration pipeline.
Develop pipeline objects using Apache Spark / Pyspark / Python or Scala.
Design and develop data pipeline architectures using Hadoop, Spark and related AWS services.
Load and performance test data pipelines built using the above-mentioned technologies.
Required Knowledge And Level Of Experience
5 years of professional work experience building data pipelines, preferably with a strategic / management consulting organization.
5 years of professional work experience with AWS and Python programming, experience with Python frameworks (e.g. Django, Flask, Bottle) is required.
Expert-level knowledge with SQL to write complex, highly-optimized queries across large volumes of data.
2 years of professional hands-on experience implementing ETL pipelines using AWS services such as Glue, Lambda, EMR, Athena, S3, SNS, Kinesis, Data-Pipelines, Pyspark is required.
2 years of hands-on professional work experience developing ETL pipelines using Scala, Python, R, or Java is required.
SQL expert, and ability to write complex, highly-optimized queries across large volumes of data.
Hands-on experience in building logical and physical data models using tools such as ErWin, Enterprise Architect or Visio is required.
Knowledge or experience in architectural best practices in building data lakes.
Strong problem solving and troubleshooting skills with the ability to exercise mature judgment.
Strong written and verbal communication skills.
Experience building strong relationships with client leaders and participating in pre-sales proposal activities is highly desired.
Bachelor or Master's degree in Computer Science, Engineering or equivalent degree is required.
Must be open to up to travel to the client location, as required by the client engagement.
Keep a pulse on the job market with advanced job matching technology.
If your compensation planning software is too rigid to deploy winning incentive strategies, it’s time to find an adaptable solution.
Compensation Planning
Enhance your organization's compensation strategy with salary data sets that HR and team managers can use to pay your staff right.
Surveys & Data Sets
What is the career path for a Sr. Data Engineer?
Sign up to receive alerts about other jobs on the Sr. Data Engineer career path by checking the boxes next to the positions that interest you.
