Data Engineer

Data Engineer

Job Summary:

The Data Engineer will be responsible for designing, building, and maintaining efficient and reliable data pipelines to support data processing, analytics, and reporting. The ideal candidate will work closely with data analysts and other key cross-functional stakeholders to ensure data accuracy, security, and availability while adhering to healthcare compliance standards.

Key Responsibilities:

• Data Pipeline Development: Design, develop, and maintain scalable data architecture, infrastructure, and pipelines to collect, process, and store healthcare data from multiple sources.
• Data Integration: Integrate data from various internal and external healthcare systems, ensuring data consistency and quality.

• Data Warehousing: Build and maintain data warehouses and data lakes to support advanced analytics and reporting.  Create and maintain data models to ensure high performance, flexibility, and accuracy.
• ETL Processes: Develop ETL (Extract, Transform, Load) processes to efficiently move and transform data between systems.  Implement data integration systems (ETL processes) for batch and near-real-time data replication, extraction, transformation, and loading.
• Database Management: Administer and optimize SQL and NoSQL databases for performance and scalability.  Collaborate with business analysts to define source-to-target mappings for data warehousing.
• Data Quality and Integrity: Implement data validation and cleansing processes to maintain high data quality.  Optimize data pipelines and workflows for efficiency and reliability.
• Automation and Monitoring: Automate workflows and monitor data pipeline performance, troubleshooting issues as they arise.  Monitor and troubleshoot data systems to ensure smooth operation and minimal downtime.

• Collaboration: Work closely with data analysts, data scientists, and business stakeholders to understand data needs and deliver solutions.

• Compliance and Security: Ensure data processing and storage comply with healthcare regulations, including HIPAA and data privacy standards.  Promote best practices in data management and governance, including security compliance, etc.

• Documentation: Maintain comprehensive documentation of data models, pipelines, and system architecture.

Qualifications:

• Education: Bachelor’s or Master’s degree in Computer Science, Information Technology, Data Engineering, or related field.

Experience:

• Minimum of 3-5 years of experience as a Data Engineer or in a similar role.
• Experience in healthcare, health insurance or TPA environments is a plus.

Technical Skills:

• Proficiency in SQL and databases (e.g., MYSql, MSSql, Azure Sql).
• Experience with data pipeline and workflow management tools (e.g., Boomi, Azure Date Factory, Informatica).
• Knowledge of data warehousing solutions (e.g., Azure Datalake, Snowflake, etc.).
• Proficiency in programming languages like Python, or Java.
• Familiarity with cloud platforms (e.g., AWS, Azure, Google Cloud) and their data services.
• Expertise in ETL frameworks and data integration tools.
• Experience with data modeling, data architecture, and schema design.
• Familiarity with big data technologies (e.g., Hadoop, Spark) is a plus.
• Familiarity with BI layer tools such as PowerBI, Looker, etc.

Soft Skills:

• Strong analytical and problem-solving abilities.
• Excellent communication and teamwork skills.
• Detail-oriented with a commitment to data accuracy and quality.
• Ability to work independently and manage multiple tasks.