Data Engineer - Blue Bell, PA

We are seeking a Data Engineer to join our IT department. This role will focus on leveraging Databricks, Azure Data Factory and related tool sets to enhance our systems integration capabilities within the commercial insurance domain. The ideal candidate will have strong technical problem-solving skills, a deep understanding of Databricks, and a strong understanding of data.

Data Pipeline Development:

• Design, develop, and maintain scalable data pipelines using Databricks.
• Ensure data quality, integrity, and consistency across all data pipelines.

Data Integration:

• Integrate data from multiple sources, including Azure Data Lake, SQL databases, and any APIs.
• Leveraging Databricks to integrate with different Large and Small Language Models.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.

ETL Processes:

• Develop and maintain ETL (Extract, Transform, Load) processes using Azure Databricks.
• Implement Data Quality and Audit, Balance and Control to the data workflows.
• Maintain and optimize data workflow performance.

Collaboration and Communication:

• Work closely with cross-functional teams to understand business requirements and translate them into technical solutions.
• Provide technical support and guidance to team members and stakeholders.

Performance Optimization:

• Optimize data processing performance on Azure Databricks.
• Implement best practices for data storage, processing, and retrieval.

• Security and Compliance:
• Ensure data security and compliance with industry standards and regulations.
• Implement data governance policies and procedures.

• Documentation and Reporting:
• Document data engineering processes, workflows, and best practices.
• Generate reports and dashboards to provide insights into data pipeline performance and data quality.

• Continuous Improvement:
• Stay updated with the latest industry trends and technologies in data engineering and Azure Databricks.
• Identify opportunities for process improvements and implement best practices
• Demonstrate commitment to Company's Code of Business Conduct and Ethics, and apply knowledge of compliance policies and procedures, standards and laws applicable to job responsibilities in the performance of work.

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Proven experience as a Data Engineer with a focus on Azure Databricks.
• Strong knowledge of Azure Databricks and its ecosystem.
• Experience with data pipeline and workflow management tools.
• Proficiency in programming languages such as PySpark, Python, Scala, or SQL.
• Familiarity with big data technologies (e.g., Spark).
• Experience with Azure services such as Azure Data Lake, Azure SQL Database, and analytics capabilities available in Azure.
• Knowledge of data warehousing concepts and best practices.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills.
• Ability to work in a fast-paced, dynamic environment.