What are the responsibilities and job description for the Sr. Pyspark Data Engineer position at Sarian Solutions?
Job Details
Role: Sr. PySpark Data Engineer Fulltime
Location: Irving, TX
Job Description:
We are seeking a skilled PySpark Data Engineer to join our team. The ideal candidate will have expertise in big data processing, ETL pipeline development, and cloud-based data engineering solutions. You will work closely with data analysts, data scientists, and software engineers to design, develop, and optimize scalable data processing solutions using PySpark, Apache Spark, and other big data technologies.
Key Responsibilities:
- Develop, optimize, and maintain ETL pipelines using PySpark and Apache Spark.
- Design and implement big data solutions for processing large datasets efficiently.
- Work with structured and unstructured data sources to transform, clean, and enrich data.
- Optimize Spark jobs for performance tuning and cost efficiency.
- Collaborate with data scientists and analysts to ensure smooth data integration and availability.
- Implement data quality checks, validation, and governance best practices.
- Deploy and manage data workflows in cloud platforms (AWS, Azure, Google Cloud Platform).
- Utilize SQL and NoSQL databases for data storage and retrieval.
- Automate data ingestion and transformation processes.
- Work in an Agile environment, actively participating in sprint planning and reviews.
- Troubleshoot and resolve performance and scalability issues.
Required Skills & Qualifications:
- Extensive years of experience in Big Data Engineering with a focus on PySpark and Apache Spark.
- Strong knowledge of Python and distributed computing frameworks.
- Experience working with Hadoop, Hive, HDFS, Kafka, and other big data technologies.
- Proficiency in SQL, NoSQL, and relational databases (PostgreSQL, MySQL, Cassandra, MongoDB, etc.).
- Experience with cloud platforms (AWS, Azure, or Google Cloud Platform) and services like S3, EMR, Databricks, Glue, or Big Query.
- Knowledge of data warehousing concepts and ETL development.
- Hands-on experience with CI/CD pipelines, Docker, and Kubernetes is a plus.
- Strong problem-solving and analytical skills with attention to detail.
- Excellent communication and teamwork skills.
Muralikrishna
Sarian Solutions, Inc.
Ph: 1 X 201
|
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
