What are the responsibilities and job description for the Data Engineer position at Global Applications Solution?
Job Title : Data Engineer (CDC, Apache Spark, ETL, AWS)
Client : KFroce
Location : Plano, Texas And Reston, VA (Candidates must be local to Texas or Virginia)
Type : Onsite / Hybrid Opportunity
Job Summary :
We are seeking a highly skilled Data Engineer with expertise in Change Data Capture (CDC) and data pipeline development to join our team at KFroce. The ideal candidate will have experience setting up and managing CDC for multiple types of databases to hydrate a data lake, along with proficiency in building ETL transformations using Apache Spark. This role requires a solid understanding of both batch and streaming data pipelines, as well as hands-on experience with data processing, optimization, and performance tuning in a Big Data environment. Familiarity with AWS services and cloud-based data architectures is essential. This is an onsite and hybrid opportunity, and candidates must be local to Texas.
Key Responsibilities :
- Design and implement Change Data Capture (CDC) solutions using Debezium or other CDC tools for various databases.
- Build and maintain data pipelines for streaming and batch processing with Apache Spark using DataFrames, Spark SQL, and Spark Streaming.
- Perform data transformations and develop ETL jobs to ensure efficient data movement and integration into a data lake.
- Collaborate with data teams to design scalable, optimized solutions for large-scale data processing.
- Work with Apache Airflow to orchestrate data pipelines and automate workflows.
- Utilize AWS cloud services to build robust and scalable data pipelines.
- Work with AWS services like S3, EMR, Glue Data Catalog, Step Functions, Lambda, MWAA, and AWS Batch to optimize data workflows.
- Troubleshoot performance issues and optimize the processing of large datasets to ensure high-performance ETL workflows.
- Keep up to date with emerging technologies in Big Data and cloud services.
Skills & Qualifications :
Technical Skills :
AWS Services :
Desired Experience :
Location :
Plano, Texas (Candidates must be local to Texas; onsite / hybrid opportunity available)
