ETL Developer

Manager Notes: "The stack is very ETL heavy, maintaining orchestrator's jobs, and supported Redshift data warehouse infrastructure” Technologies: Talend, Terraform, Python, Shell Scripting, Docker, AWS: EC2, S3 Job Details: 1. ETL tools and technologies used in this project? - Involves working with but not limited to: Talend (Talend Studio, Talend Remote Engine, Talend Management Console), Terraform, Python, Shell scripting, Docker, AWS services like EC2, S3. 2. What tools are used for orchestrating the ETL processes? - Custom built solutions involving Talend. 3. Redshift Spectrum or focusing on traditional Redshift clusters? - Mainly traditional Redshift clusters. 4. What other AWS services or tools are integrated into the project? - The infrastructure consists of multiple aspects of Talend (Talend Remote Engine, Talend Managament Console, Talend Studio), EC2 instances used to run Talend, Redshift and various repositories (Python and Terraform) for adding new sources and to manage and support the infrastructure. 5. Size of data that we are working with? - 8400 GB 6. Are integrating data from multiple sources into the Redshift warehouse, or building a data pipeline for real-time analytics? - Majority of it is building the Redshift data warehouse. 7. What does the day to day look like? - Building and maintaining ETL pipelines. Also monitoring daily jobs and address the failures if any. Also work on optimizing the performance of Redshift infrastructure using Terraform. 8. Does the job include designing and building the ETL pipeline vs. maintaining existing ones? - Both 9. I generally support the Data Analytics' team's. What does “supporting Redshift infrastructure” mean in this context? - It includes managing the security aspects and fulfilling requests that come in from Cloud Engineering team. - Maintaining Talend infrastructure. - Also includes working with Terraform to maintain all the aspects of the Redshift cluster infrastructure using terraform templates