Sr Data Engineer

Job Role Sr Data Engineer

Location St Louis MO(remote)

Job Details :

The Data Engineer will work iteratively on the cloud platform to design develop and implement scalable high performance solutions that offer measurable business value to customers.

Data Solutions : Design and implementation of data pipelines using GCP services

Manage Deployments with Helm and Kubernetes : Oversee containerized deployments of data applications on Kubernetes Engine using Helm for package management ensuring efficient orchestration of services.

Automation with Jenkins : Implement CI / CD pipelines using Jenkins to automate data pipeline deployment testing and integration with other services ensuring quick iterations and deployments.

Develop and Optimize Pipelines : Write efficient Python and SQL scripts to build data pipelines and ETL / ELT processes. Continuously monitor and optimize data workflows for performance and costeffectiveness.

Data Integration and Orchestration : Design workflows to integrate data from various sources using GCP services and orchestrate complex tasks with Cloud Composer (Apache Airflow).

Security and Compliance : Ensure data security and compliance by implementing IAM policies encryption and other security measures in GCP adhering to best practices for handling sensitive data.

Collaborate Across Teams : Work closely with application developers data architects and business stakeholders to define and deliver robust datadriven solutions. Provide technical leadership and ensure alignment between business and technical objectives.

Monitoring and Logging : Set up monitoring logging and alerting using Cloud Monitoring (formerly Stackdriver) Datadog or other tools to ensure visibility into pipeline performance and quickly identify and resolve issues.

Skills

Google Cloud Platform (GCP) Tools : (Preferred) BigQuery Cloud Storage Dataflow Cloud Functions Pub / Sub Cloud Run Cloud Composer (Airflow) Cloud Spanner Bigtable.

Container Orchestration Kubernetes (preferred on GKE) and Helm for managing and deploying containerized applications.

CI / CD and Automation Jenkins for building CI / CD pipelines to automate deployment and testing of data pipelines.

Programming Languages Proficient in Python for data processing and automation SQL for querying and data manipulation. Experience with Java is a plus.

DevOps Tools Familiarity with Terraform or Deployment Manager for Infrastructure as Code (IaC) to manage GCP resources.

Monitoring and Logging Experience with Cloud Monitoring Datadog or other monitoring solutions to track pipeline performance and ensure operational efficiency.

Data Engineering Skills Expertise in ETL / ELT pipelines data modeling and data integration across large datasets.

Strong understanding of data warehousing and realtime data processing workflows.

Strong communication skills to work effectively with crossfunctional teams and mentor junior developers. Proven ability to lead in an Agile environment.

3 years of experience as a data engineer with handson experience in Kubernetes Helm Python and Jenkins.

Strong experience building and optimizing data pipelines and services in any cloud platform.

Proficiency in Python and SQL. Familiarity with Java and Docker is a plus.

Key Skills

Apache Hive,S3,Hadoop,Redshift,Spark,AWS,Apache Pig,NoSQL,Big Data,Data Warehouse,Kafka,Scala

Employment Type : Full Time

Vacancy : 1