Big data/Hadoop Engineer

Job Title: Big data/Hadoop Engineer

Location: Plano, TX

Required

• 1-2 years, creating, maintaining and managing Hadoop clusters
• 3-5 years of experience with development to centered around big data applications, and adhoc transformation of unstructured raw data
• 1-2 years Relational DBA experience, preferably with experience in SQLServer and/or MySQL.
• Design, build, and maintain Big Data workflows/pipelines to process continuous stream of data with experience in end-to-end design and build process of Near-Real-Time and Batch Data Pipelines.
• Demonstrated work experience in the following with Big Data and distributed programming models and technologies
• Knowledge of database structures, theories, principles and practices (both SQL and NoSQL).
• Active development of ETL processes using Spark or other highly parallel technologies, and implementing ETL/data pipelines
• Experience with Data technologies and Big Data tools, like Spark, Kafka, Hive
• Understanding of Map Reduce and other Data Query and Processing and aggregation models
• Understanding of challenges of transforming data across distributed clustered environment
• Experience with techniques for consuming, holding and aging out continuous data streams
• Ability to provide quick ingestion tools and corresponding access API’s for continuously changing data schema, working closely with Data Engineers around specific transformation and access needs
  
  

Preferred

• Experience as Database administrator (DBA) will be responsible for keeping critical tools database up and running
• Building and managing high availability environments for databases and HDFS systems
• Familiarity with transaction recovery techniques and DB Backup
  
  

Hybrid Hadoop Engineer and Hadoop Infrastructure Administrator to build and maintain a scalable and resilient Big Data framework to support Data Scientists. As an administrator, responsibility will be to deploy and maintain Hadoop clusters, adding and removing nodes using cluster management and monitoring tools like Cloudera Manager, support performance and scalability requirements, in support of our Data scientists needs. Some Relational Database administrator experience will also be desirable to support general administration of Relational Databases.

Skills And Attributes

• Ability to have effective working relationships with all functional units of the organization
• Excellent written, verbal and presentation skills
• Excellent interpersonal skills
• Ability to work as part of a cross-cultural team
• Self-starter and Self-motivated
• Ability to work without lots of supervision
• Works under pressure and is able to manage competing priorities.
  
  

Technical Qualifications And Experience Level

• At least 5 years of combined proven working experience as a Spark/Big Data developer, DBA and Hadoop Admin
• 5-10 years in development using Java, Python, Scala, and object-oriented approaches in designing, coding, testing, and debugging programs
• Ability to create simple scripts and tools.
• Development of cloud based, distributed applications
• Understanding of clustering and cloud orchestration tools
• Working knowledge of database standards and end user applications
• Working knowledge of data backup, recovery, security, integrity and SQL
• Familiarity with database design, documentation and coding
• Previous experience with DBA case tools (frontend/backend) and third party tools
• Understanding of distributed file systems, and their optimal use in the commercial cloud (HDFS, S3, Google File System, Datastax Delta lake)
• Familiarity with programming languages API
• Problem solving skills and ability to think algorithmically
• Working Knowledge on RDBMS/ORDBMS like MariaDb, Oracle and PostgreSQL
• Working knowledge on Hadoop administration.
• Knowledge of SDLC (Waterfall, Agile and Scrum)
• BS degree in a computer discipline is MUST