What are the responsibilities and job description for the GCP Data Engineer position at ASB Resources?
- Design, code, test, debug, and document technology solutions for complex projects and programs.
- Review and analyze large-scale technology solutions to align with tactical and strategic business objectives, as well as enterprise technological environments.
- Evaluate and develop companywide best practices for engineering and technology solutions, ensuring alignment with industry standards and new technologies.
- Collaborate with key technical experts and senior technology teams to resolve complex technical challenges and meet project goals.
- Contribute to the development of Data Mesh capabilities to improve data accessibility across disparate platforms such as Hadoop, Teradata, and SQL Servers.
- Support Data Science functions by preparing pre-work required for data exploration and analysis.
- Support System Testing, User Acceptance Testing (UAT), and end-to-end testing activities; develop test automation for batch processes and APIs within a CI/CD environment.
- Work with Agile methodologies, writing user stories, and participating in Agile ceremonies.
Required Qualifications:
- Extensive experience designing and delivering data solutions in Google Cloud Platform (GCP), leveraging services such as DataProc, DataFlow, BigQuery, and other GCP tools.
- Expertise in Big Data platforms (e.g., MapR, Cloudera) using technologies like HDFS, Spark, PySpark, and Hive.
- Proficiency in event-driven solutions using Java, REST APIs, Apache Flink, and Kafka technologies.
- Experience with relational and NoSQL databases, including MongoDB, SQL Server, Atlas, and Azure SQL.
- Strong familiarity with wire formats such as XML, JSON, Avro, and CSV, along with serialization/deserialization techniques.
- Knowledge of security protocols (e.g., Kerberos, JWT) and their integration into data platforms and solutions.
- Solid understanding of Agile methodologies, including writing user stories and participating in ceremonies such as sprint planning and retrospectives.
