Data Engineer

Title: Data Engineer with AWS, Python and AWS GLUE

Fulltime

REMOTE

NO THIRD PARTY VENDORS

We are seeking an experienced Data Engineer with expertise in an AWS cloud native environment to design, develop, and maintain our data infrastructure and systems necessary for efficient data processing, storage, and analysis. This role will work closely with our product management and data analytics teams to create and manage effective data ETL processes on our application and analytics platforms.

Culture is King

The skills and experience you bring to this role include:

• Bachelor’s degree in Computer Science, Informatics, Information Systems, or another quantitative field.
• Strong Collaboration and Culture is very important.
• 5 years of experience in data engineering or a similar role with demonstrated technical leadership experience.
• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL), writing views, stored procedures and triggers in an AWS Cloud Environment with RedShift, MySQL, PostgreSQL or similar RDBMS. They do a lot of work in S3 (S3 is a cloud storage service that allows users to store and retrieve data. It’s used for a variety of applications, including backup, disaster recovery and content distribution. It stores large volumes of raw data)
• Strong database administration skills in AWS cloud databases, RedShift, RDBMS, PostgreSQL, or MySQL.
• Software development skills and prior experience using Python, SQL, and Stored Procedures.
• Experience with healthcare data (HL7, Medical Claims, Rx Claims) or similar healthcare data is preferred.
• Excellent communication and organizational skills. MUST have English as their first language

Key Responsibilities:

Data Architecture and Design:

• Design and implement scalable data architectures using AWS cloud services, including data lakes, data warehouses, bulk data ingestion, transaction processing and streaming solutions.
• Collaborate with stakeholders and other cross-functional teams, to comprehend data requirements and align data models with business needs for optimal data utilization.
• Assemble large, complex data sets that meet functional business requirements.

ETL and Data Processing: This is all done in AWS Glue. They are not using Informatica. Most of the data is structured.

• Lead the development and implementation of ETL processes to ingest, clean, and transform data from various sources into the data platform.
• Identify, design, and implement internal process improvements to automate manual processes and optimize data delivery.

AWS Cloud Services and Database Administration:

• Leverage AWS services, such as AWS Glue, AWS Data Pipeline, AWS Lambda, AWS Step Functions, Amazon Redshift, Amazon S3 and Amazon Kinesis, to build and optimize data workflows and processing pipelines.
• Administer AWS databases effectively, within Amazon RDS, RedShift, PostgreSQL or MySQL.
• Stay current with AWS services and recommend suitable tools for specific data engineering tasks.

Performance Optimization:

• Monitor and optimize data pipelines and data storage for performance, cost, and reliability.
• Implement caching mechanisms and data partitioning strategies to enhance query efficiency and reduce data processing times.

Software Development:

• Develop and support the development of internal applications software using Python, SQL, and Stored Procedures.

Technical Leadership and Collaboration:

• Technical lead for Data Engineering team, providing technical guidance, mentoring, and fostering a collaborative and innovative environment.
• Assist in establishing and managing the data security process.
• Experience in supporting and working with cross-functional teams in a dynamic environment.

Data Security and Compliance:

• Implement and enforce data security measures, ensuring compliance with relevant data regulations and industry best practices.
• Maintain data governance standards and access controls to protect sensitive data.

Continuous Improvement:

• Drive the adoption of best practices, automation, and modern data engineering techniques to improve the efficiency and reliability of data processes.
• Identify opportunities for process optimization and drive initiatives to enhance the data engineering ecosystem.
• Conduct root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.