Senior Data Engineer

Job Description

Job Description

Position Overview :

The Senior Data Engineer will design, develop, and optimize our data warehouse and integration solutions. This role is pivotal to the success of our centralized data initiatives, ensuring seamless integration of partner data, building robust data pipelines, and supporting critical software applications. The ideal candidate will have a strong background in data modeling, normalization, and Python scripting, coupled with expertise in developing complex data warehouse solutions. Additionally, the role requires expertise in designing and maintaining efficient ETL processes, ensuring data is securely stored, properly normalized, and maintained to meet organizational and compliance standards.

Key Responsibilities :

Data Integration and Development :

Design and implement data warehouse solutions to integrate data from multiple systems (MS Azure, Salesforce, external partner systems, etc.).

Develop ETL / ELT processes to extract, transform, and load data into the centralized data warehouse.

Build data pipelines to facilitate real-time or scheduled data synchronization across systems using cloud-native services such as Azure Data Factory

Data Modeling and Normalization :

Create and maintain complex data models, ensuring data is normalized and structured for scalability, performance, and accuracy.

Collaborate with the Business Intelligence team to ensure data structures align with integration, application, and reporting needs, including matching and exception handling logic.

Existing Solution Support :

Enhance and optimize Python scripts for data ingestion, matching, reconciliation, and reporting within our existing solution.

Address challenges related to partner data aggregation and provide solutions for improved data accuracy and completeness.

Collaboration and Innovation :

Work closely with the Business Intelligence Manager, current developers, and other stakeholders to understand business needs and translate them into technical solutions.

Mentor current developers, including knowledge sharing with existing team members.

Quality and Performance :

Implement best practices in acquisition and storage of data, data security, governance, and compliance.

Continuously monitor, optimize, and improve system performance and data integrity.

Qualifications : Education :

Bachelor's degree in Computer Science, Data Engineering, or a related field (or equivalent experience).

Technical Skills :

Proven experience in designing and developing complex data warehouses (preferably on Microsoft Azure or a similar cloud platform).

Strong expertise in data modeling, normalization, and relational database systems (e.g., SQL Server, PostgreSQL).

Verifiable proficiency in Python, especially for scripting and data processing tasks.

Strong experience with ETL / ELT tools and frameworks (e.g., Azure Data Factory, Apache Airflow).

Familiarity with Salesforce and integrating data from CRM systems is a plus.

Experience :

More than 5 years of experience in data integration, development, or engineering roles, building solutions around data warehouses and data transporting platforms.

Experience working with relational databases such as PostrgeSQL, MySQL, SQL Server and NoSQL databases like MognoDB, etc.

Hands on experience with Data Warehousing Solutions like Snowflake or Redshift

Knowledge of big data frameworks like Hadoop, Apache Spark, etc.

Strong experience with containerization and orchestration with tools like Kubernetes.

Demonstrated ability to work with large datasets and complex business logic.

Strong problem-solving skills, with the ability to debug and optimize data processes.

Ability to work quickly and dynamically, able to adapt to change

Strong experience with Agile Development Framework

Understanding of data governance frameworks, like NIST CSF 2.0

CI / CD pipeline proficiency using GitHub or similar.

Soft Skills :

Excellent communication and collaboration skills to work effectively across teams.

Proactive, detail-oriented, and results-driven mindset.

Job Posted by ApplicantPro