Integration Engineer

Major Responsibilities:

• Build and optimize data pipelines for efficient data ingestion, transformation and loading from various sources while ensuring data quality and integrity.
• Design, develop, and deploy Spark program in data bricks environment to process and analyse large volumes of data.
• Experience of Delta Lake, DWH, Data Integration, Cloud, Design and Data Modelling.
• Proficient in developing programs in Python and SQL
• Experience with Data warehouse Dimensional data modelling.
• Working with event based/streaming technologies to ingest and process data.
• Working with structured, semi structured, and unstructured data.
• Optimize Databricks jobs for performance and scalability to handle big data workloads.
• Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks.
• Implement best practices for data management, security, and governance within the Databricks environment.

Experience designing and developing Enterprise Data Warehouse solutions.

• Proficient writing SQL queries and programming including stored procedures and reverse engineering existing process.
• Perform code reviews to ensure fit to requirements, optimal execution patterns and adherence to established standards.

Skills:

• 3 years Astronomer DAG development
• 5 years Python coding experience.
• 5 years - SQL Server based development of large datasets
• 5 years with Experience with developing and deploying ETL pipelines using Databricks Pyspark.
• Experience in any cloud data warehouse like Synapse, Big Query, Redshift, Snowflake.
• Experience in Data warehousing - OLTP, Dimensions, Facts, and Data modelling.
• Previous experience leading an enterprise-wide Cloud Data Platform migration with strong architectural and design skills.
• Experience with Cloud based data architectures, messaging, and analytics.
• Cloud certification(s).