Lead Data Engineer (Snowflake, Python and Java)

• Strong background in data engineering, platform architecture, sound technical knowledge on full stack technologies
• Requires strong Snowflake skills with background of Python and Java

Key Responsibilities

• Lead the development, design, development and deployment of data pipelines, storage solutions and analytic tools
• Ensure scalability, reliability and security of the platform to meet current and future demands.
• Oversee the implementation of data lakes, warehouses and ELT/ETL pipelines on azure and snowflake
• Develop and Review code and ensure adherence to development best practices and standards
• Foster a culture of collaboration, innovation and continous improvement within the team.
• Drive integration of real-time and batch data processing capabilities.
• Ensure compliance with data governance policies, security and regulatory requirements.
• Implement robust data quality, lineage and cataloging processes
• Build and integrate APIs and backend systems using frameworks like Django, Flask or FastAPI
• Develop scripts and tools for data analysis, transformation and automation
• Write unit tests, integration tests and conduct debugging to ensure software quality
• Work closely with cross-functional teams including developers, product managers, business analysts and other stakeholders to align the data platform with organizational goals
• Translate business requirements into scalable technical solutions

Qualifications

• Proven experience in leading the design and development of large scale data platform
• Proven experience as a full stack developer or similar role
• Experise in data platform technologies - snowflake, Azure SQL Server
• Proficiency in data engineering tools and frameworks - Apache Kafka, Spark,
• Strong understanding of cloud platforms - Azure and containerization (Docker, Kubernetes)
• Experience with both batch and real-time data processing
• Deep knowledge of SQL and programming languages - JAVA/Python
• Strong Problem-solving communication and leadership skills
• Experience of code deployments using CI/CD Pipelines
• Proficiency with code repos GitHub
• Knowledge and experience developing restful APIs
• Certifications in Snowflake, Azure
• Experience in Alternatives (Private equity, Private Capital, Co-investments, debt, venture capital, hedge and real assets) domain,
• working with private market industry solutions, Investran, Alterdomus, iCapital, Subscribe, backstop, chronograph
• Strong understanding of private market operations, investment lifecyle and structures and fund life cycle.
• Technology certifications on Azure, snowflake. Domain certifications on CFA