Data Consultant

Job Summary :

The Data Engineer will be responsible for designing, developing, and maintaining scalable ETL (Extract, Transform, Load) pipelines using Vertex AI pipelines for data integration, transformation, and dataflow jobs. This role involves optimizing data models, databases, and data warehouses, ensuring efficient data storage, retrieval, and processing. The Data Engineer will collaborate with cross-functional teams to meet data needs while ensuring data quality, security, and performance.

Key Responsibilities :

Design and Develop ETL Pipelines :

Build and maintain scalable ETL pipelines using Vertex AI pipelines to integrate and transform data efficiently.

Optimize Data Models and Infrastructure :

Build, optimize, and manage data models, databases, and data warehouses to ensure efficient storage and retrieval of large datasets.

Collaboration with Stakeholders :

Work closely with data analysts, data scientists, and other stakeholders to understand data needs and deliver solutions that meet business requirements.

Ensure Data Quality and Governance :

Implement validation, governance, and security processes to ensure high-quality and reliable data.

Cloud Platform Management :

Utilize Google Cloud Platform (GCP) to manage and deploy data infrastructure, ensuring scalability and reliability of systems.

Monitor and Troubleshoot Performance :

Monitor data pipelines and systems for performance issues and troubleshoot as needed to maintain smooth operation.

Documentation Maintenance :

Develop and maintain clear documentation for data systems, processes, and workflows to ensure transparency and knowledge sharing.

Required Qualifications :

• Proven experience in building and maintaining ETL pipelines and data integration processes.
• Strong proficiency with cloud platforms, particularly Google Cloud Platform (GCP).
• Experience with dataflow jobs and pipeline orchestration tools such as Vertex AI.
• Strong understanding of database technologies and data warehousing concepts.
• Solid knowledge of data modeling, data transformation, and optimization techniques.
• Experience in ensuring data quality, reliability, and security in a cloud-based environment.
• Ability to troubleshoot and optimize data pipelines for performance and efficiency.
• Excellent communication skills and ability to collaborate with cross-functional teams.
• Strong problem-solving skills and attention to detail.
• Familiarity with tools such as BigQuery, Dataflow, and other GCP services.

Preferred Qualifications :

Certifications (if any) :

Google Cloud Certified - Professional Data Engineer or equivalent certifications are a plus.

Any relevant certifications in data engineering, cloud platforms, or ETL tools are a plus.

Education : Bachelors Degree