What are the responsibilities and job description for the Big Data Engineer position at Tek Target USA?
Job Details
Title:Big Data Engineer
Type:Contract 18 months
Location:Rockville, MD (Hybrid 2 days a week) Experience required :: 8
Job Requirements:
Intake Notes:
Java, Scala, and Python - very different languages, where do we hone in or are they language
Big data processing vs streaming uses:
Anything outside of AWS emr that they need: Hadoop
Team: last year, ETL project moved from to Market Regulation
Main technologies:
Scala, Python, SQL, AWS
PySpark is OK
Big Data
Hadoop, Hive, Spark, AWS EMR and Airflow.
Main responsibility is to understand existing process, learn the new design and make the code changes to implement that. Not a major rewrite but modernizing a part
Hadoop has own, Cloudera - and Hadoop distribution
Data bricks - same as Hadoop, it's fine
Must Have Skills/ Requirements:
This is how they communicate with users, so if they are strong in SQL, Excel can copy paste
Python - OK if they don't have
Nice to Have Skills/ Requirements:
Market knowledge is a plus to help with learning curve
Know the behaviors of the market Project:
Type:Contract 18 months
Location:Rockville, MD (Hybrid 2 days a week) Experience required :: 8
Job Requirements:
- Extensive Experience with cloud based Big Data technologies like Hadoop, Hive, Spark, AWS EMR and Airflow.
- Experience in one or more programming languages like Java, Scala, and Python.
- Experience in solving complex problems using SQL.
- Experience with developing enterprise grade solutions in an iterative or Agile environment.
- Clear, effective communication with strong interpersonal skills.
- Ability to push the frontier of technology and independently pursue better alternatives.
- Ability to maintain focus and develop proficiency in new skills rapidly.
- Ability to utilize problem solving skills in a fast-paced environment
- Experience in AWS Cloud is required.
Intake Notes:
Java, Scala, and Python - very different languages, where do we hone in or are they language
- PySpark (much easier, more bang for your buck) vs SparkScala
Big data processing vs streaming uses:
- Mainly for batch processes
- Lambdas that support the processes, but the modernization is on batch
Anything outside of AWS emr that they need: Hadoop
Team: last year, ETL project moved from to Market Regulation
- Need this person to be strong because they are operating in tight constraint on time and resources so this needs to be a producer and can't be affecting the productivity of the team
- 10-member team so there is support but once trained, expectation is they ramp up and can work indecently
Main technologies:
Scala, Python, SQL, AWS
PySpark is OK
Big Data
Hadoop, Hive, Spark, AWS EMR and Airflow.
Main responsibility is to understand existing process, learn the new design and make the code changes to implement that. Not a major rewrite but modernizing a part
- Have clear guidelines and process defined to approach the modernization
Hadoop has own, Cloudera - and Hadoop distribution
Data bricks - same as Hadoop, it's fine
Must Have Skills/ Requirements:
- SQL
- Prep data
This is how they communicate with users, so if they are strong in SQL, Excel can copy paste
Python - OK if they don't have
Nice to Have Skills/ Requirements:
Market knowledge is a plus to help with learning curve
Know the behaviors of the market Project:
- ML algorithms
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
