What are the responsibilities and job description for the Lead, Data Engineering position at Gore Mutual Insurance?
Job Details
Job Description
We're now at the boldest phase of our Next Horizon journey
At Gore Mutual, we've completely transformed our business in under three years. By investing in top talent and leading technology, we've redefined what it means to be a modern mutual that does good.
Our path forward brings a sharper focus on our business' performance that's powered by innovation and an agile, high-performing culture we're built for success.
We're well on our way to becoming a purpose-driven, digitally led national insurer. Come join us.
As a Lead Data Engineer at Gore Mutual Insurance, you have a strong technical background in software engineering / computer science, you will play a pivotal role in designing, building, and maintaining our data platform. Your work will facilitate data accessibility, accuracy, and accountability, enabling data-driven decision-making across our organization. You will collaborate closely with cross-functional teams to ensure our data processes are robust and scalable.
What will you be doing in this role?
Designing Systems for Collecting and Storing Data
- Design and implement the data infrastructure and tooling that encompasses the data platform. This includes selecting the appropriate hardware and software components, configuring storage resources, and creating security policies.
Automating Data Management Processes and Handling Data Security
- Automate data management processes and handle data security.
- Ensure that security protocols and best practices are in place to protect against potential security threats.
Integrating Data Platforms with Necessary Tools
- Ensure that the necessary tools are integrated with the data platform.
Testing and Optimizing Data Platform
- Optimize data pipelines to ensure efficient data flow.
- Ensure that the data extracted from sources is accurate, complete, and usable. This might involve checking for missing values, inconsistent formats, or anomalies that could indicate errors.
- Test the efficiency and speed of data pipelines and databases. This can help identify bottlenecks and optimize performance.
- Verify that different components of the data infrastructure work together as expected. This includes checking that data flows correctly from sources to databases, and from databases to applications.
Maintenance of Data Platform
- Regularly check the health and performance of the data platform. This can involve tracking metrics like query times, error rates, and resource usage.
- Identify and resolve issues that arise in the data platform. This can involve debugging code, optimizing queries, or adjusting configurations.
- Keep the data platform up-to-date with the latest technologies and security patches. This can involve updating database software, upgrading hardware, or migrating data to new systems.
- Implement strategies to protect data from loss or corruption. This can involve regular backups, redundancy measures, and disaster recovery plan
What will you need to succeed in this role?
- Bachelor's or Master's degree in Computer Science, Data Engineering, Software Engineering or a related field.
- A minimum of 5-6 years relevant experience as a data engineer is required. This includes experience in data engineering, data system development, or related roles.
- Strong understanding of data structures, data modeling, and software architecture.
- Deep knowledge of Microsoft Azure Services (DevOps, Databricks, SQL Server, Event Hub, Web Apps, Data Factory, Azure Storage, Keyvault, etc.)
- Experience with software design patterns and test-driven development (TDD)
- Proficiency in Python, including a strong grasp of Object Oriented and Functional programming paradigms.
- A solid understanding of Spark concepts and distributed systems, including data transformations, RDDs, DataFrames, and Spark SQL.
- Excellent SQL skills and expertise in database management and tuning.
- Ability to devise and implement master data management processes.
- Experience with data lakehouse and medallion architecture using Delta Lake within Azure Databrick
- Proficiency in developing RESTful APIs
- Strong problem-solving and critical-thinking abilities.
- Strong communication and collaboration skills.
- Continuous learning and staying updated with the latest advancements in Azure Databricks and data engineering are essential for success in this role.
Nice to Have
- A graduate degree in a technical field with specialization in Analytics, Data Science, or a related subject.
- Azure certifications (Microsoft Certified: Azure Data Engineer Associate).
- Databricks certifications (Databricks Certified /Data Engineer Professional certification).
#LI-Hybrid
#IndHP
Gore Mutual Insurance is committed to providing accommodations for people with disabilities during all phases of the recruiting process, including the application process. If you require accommodation because of a disability, we will work with you to meet your needs. If you are selected for an interview and require accommodation, please advise the HR representative who will consult with you to determine an appropriate accommodation.