Bigdata engineer - Rockville, MD - 2days a week onsite - w2 role - locals preferred

• Extensive Experience with cloud based Big Data technologies like? Hadoop, Hive, Spark, AWS EMR and Airflow.
• Experience in one or more programming languages like Java, Scala, and Python.
• Experience in solving complex problems using SQL.
• Experience with developing enterprise grade solutions in an iterative or Agile environment.
• Clear, effective communication with strong interpersonal skills.
• Ability to push the frontier of technology and independently pursue better alternatives.
• Ability to maintain focus and develop proficiency in new skills rapidly.
• Ability to utilize problem solving skills in a fast-paced environment
• Experience in AWS Cloud is required.

Intake Notes:
Java, Scala, and Python - very different languages, where do we hone in or are they language

• PySpark (much easier, more bang for your buck) vs SparkScala

Big data processing vs streaming uses?

• Mainly for batch processes
• Lambdas that support the processes but the modernization is on batch

Anything outside of AWS emr that they need?
Hadoop

Team: last year, ETL project moved from to Market Regulation

• Business case for optimizing this that got approved so adding resources so they can make progress
• Scheduled for close to 5 quarters, so 1.5 years
• Lots of jobs and process that are being modernized

Need this person to be strong because they are operating in tight constraint on time and resources so this needs to be a producer and can't be affecting the productivity of the team

• 10-member team so there is support but once trained, expectation is they ramp up and can work indecently

Main technologies:
Scala, Python, SQL, AWS
PySpark is OK

Big Data
Hadoop, Hive, Spark, AWS EMR and Airflow.

Main responsibility is to understand existing process, learn the new design and make the code changes to implement that. Not a major rewrite but modernizing a part

• Have clear guidelines and process defined to approach the modernization

Hadoop has own, Cloudera - and Hadoop distribution
Data bricks - same as Hadoop, it's fine

Must Have Skills/ Requirements:

1. SQL
2. Prep data

Excel

This is how they communicate with users, so if they are strong in SQL, Excel can copy paste

Python - OK if they don't have
Nice to Have Skills/ Requirements:

Market knowledge is a plus to help with learning curve

Know the behaviors of the market Project:

• ML algorithms

Team:
Marketing technology

• Talk with internal clients and business partners
• Tech piece is for proto typing data analysis
• Business will have vague idea of what to do but they need to manipulate production data
• Can't ask engineers to do this because requirements aren't set so BAs

If business user wants X, and to do that need to go through ABC but that's not logical for the data so might need to

• Have to talk in data
• Identify the logic