Senior / Lead Data Engineer (Python, Kafka, Iceberg, ClickHouse)

JOB SUMMARY

We are looking for a talented and experienced Senior/Lead Data Engineer to join our innovative team. The Senior/Lead Data Engineer will play a critical role in leading the design, development, and maintenance of our data infrastructure, pipelines, and analytical tools. The ideal candidate will have a strong technical background in data engineering, expertise in analytical tools, and proven leadership skills to mentor and guide a team of data engineers.

JOB DUTIES

• Design and implement scalable batch and streaming data pipelines using tools such as Flink, Spark, Debezium, and Seatunnel (an open-source data integration tool).
• Ingest and process high-volume data from APIs, operational databases, and semi-structured formats (e.g., JSON, CSV, XML, logs) to support diverse analytical use cases.
• Build reusable transformation pipelines to consolidate cross-domain data (e.g., user behavior, transactions) into analytics-ready marts.
• Architect and optimize data storage and modeling layers using MinIO/S3, Iceberg, ClickHouse, and other OLAP or object storage platforms to improve query performance and data reliability.
• Maintain multi-layered data warehouse architecture (staging, core, mart) aligned with business needs.
• Ensure robust CI/CD, lineage, observability, and compliance through tools like OpenMetadata and DataHub.
• Mentor junior engineers, conduct rigorous code reviews, and promote engineering best practices across the data team.
• Work cross-functionally with product managers, analysts, business stakeholders to translate data needs into scalable pipelines and business insights.
Stay current with data engineering trends and technologies, and continuously drive platform improvements.

JOB REQUIREMENTS

• Bachelor’s degree in Computer Science, Information Systems, or equivalent qualification.
• 6 years of experience in data engineering , including 2 years in a technical lead or senior IC capacity.
• Proficient in Python , Java, and SQL, with strong expertise in schema design, performance tuning , and warehouse modeling.
• Hands-on experience with lakehouse architectures (e.g., Iceberg , Delta Lake), data warehouses (e.g., Hive, ClickHouse ), and object storage (e.g., AWS S3, MinIO).
• Strong knowledge of orchestration tools (Airflow, DolphinScheduler), ETL/ELT design, and streaming frameworks ( Kafka, Flink, Spark ).
• Proven experience independently setting up and managing end-to-end data architecture in on-premise environments.
• Demonstrated success mentoring engineers in high-performance, cross-functional teams.
• Familiarity with Git-based workflows, CI/CD pipelines, and observability tools for production-grade data systems.
• Self-motivated with a strong ownership mindset, adaptability, and willingness to travel when needed.
• Fluent in English and Mandarin (written/spoken), with experience leading technical discussions in multilingual and cross-regional settings.