What are the responsibilities and job description for the Big Data PySpark Tech Lead position at Pro IT Inc?
Job Description
Skill : Big Data (PySpark) Tech Lead
10 Years Overall Experience in Data Management Data Lake and Data Warehouse.
6 Years Hadoop Hive Sqoop SQL Teradata.
6 Years PySpark(Python and Spark) Unix.
Good to have Industry leading ETL experience.
Banking Domain experience.
Key Responsibilities :
Ability to design build and unit test applications on Spark framework on Python.
Build PySpark based applications for both batch and streaming requirements which will require indepth knowledge on majority of Hadoop and NoSQL databases as well.
Develop and execute data pipeline testing processes and validate business rules and policies.
Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context SparkSQL Data Frame and Pair RDDs.
Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro Parquet ORC etc) and compression codec respectively.
Ability to design & build realtime applications using Apache Kafka & Spark Streaming.
Keep a pulse on the job market with advanced job matching technology.
If your compensation planning software is too rigid to deploy winning incentive strategies, it’s time to find an adaptable solution.
Compensation Planning
Enhance your organization's compensation strategy with salary data sets that HR and team managers can use to pay your staff right.
Surveys & Data Sets
What is the career path for a Big Data PySpark Tech Lead?
Sign up to receive alerts about other jobs on the Big Data PySpark Tech Lead career path by checking the boxes next to the positions that interest you.
