Job Description : Kafka Architect (Platform & Developer Focus)
Location : REMOTE
Position Overview : We are seeking a highly experienced Kafka Architect with 8-10 years of expertise in Kafka administration. This is a strategic role, not for a developer or admin, but for someone with deep architectural knowledge of Kafka and a strong ability to communicate complex concepts effectively. The ideal candidate will have extensive experience in designing, documenting, and optimizing both platform and developer elements of Kafka, with a focus on messaging and streaming capabilities. The candidate should be well-versed in implementing Kafka in diverse environments, with the ability to generalize concepts across different Kafka implementations (including Confluent and others).
Key Responsibilities :
Platform Focus : Infrastructure, Deployment, Operations, and Security
- Cluster Sizing & Capacity Planning :
Define brokers per cluster, partition count, and replication factor based on data volume, retention policies, and throughput needs.
Design scaling configurations (horizontal and vertical) for Kafka clusters.Establish optimal network configurations to ensure low-latency performance.Define Zookeeper configurations (if not using KRaft mode) or alternative metadata management.Storage Considerations :Provide recommendations for optimal disk configurations, with a focus on NVMe SSDs.
High Availability & Fault Tolerance :Define optimal replication factor (RF) strategies for both production and non-production environments.
Design broker failover strategies and leader election mechanisms.Develop multi-region and multi-AZ deployment strategies for Kafka.Security, Audit & Compliance :Implement and recommend Kafka authentication strategies (SASL, Kerberos, OAuth, TLS).
Design authorization mechanisms (ACLs, RBAC) for Kafka.Advise on key management strategies, key rotation, and secure storage.Define auditing best practices for tracking access and changes to Kafka resources.Ensure encryption for both data in-transit (TLS) and at-rest (disk encryption).Advise on compliance frameworks (e.g., SOX, CCPA) and ensure Kafka adheres to necessary standards.Monitoring & Observability :Advise on metrics collection using tools like Prometheus, Grafana, and Confluent Control Center.
Implement security monitoring tools to detect and respond to real-time threats.Provide recommendations for monitoring disk usage and log aggregation (e.g., Elasticsearch, Kibana, Splunk).Implement lag monitoring strategies using tools like Burrow or Kafka UI.Developer Focus :
Data Retention & Cleanup :Define log segment configurations and cleanup policies (delete vs. compact).
Provide recommendations for Kafka compaction processes and scheduling.Advise on time-based vs. size-based retention policies to optimize resource usage.Disaster Recovery & Backup :Define strategies for cross-cluster replication and cluster linking.
Set recovery point objectives (RPO) and recovery time objectives (RTO).Define automated backup verification and recovery procedures.Develop Kafka backup strategies, including configuration and topic-level backups.Cost Optimization :Recommend strategies for optimizing consumer group performance.
Define dynamic partition rebalancing and scaling strategies.Recommend optimal data retention policies and efficient data compression formats.Qualifications :
8-10 years of experience in Kafka architecture and administration (platform-focused).Strong knowledge of Kafka internals, cluster design, security, and monitoring tools.Proven ability to document and communicate complex Kafka platform architectures.Deep understanding of Kafkas role in both messaging and streaming use cases.Experience with different Kafka implementations (e.g., Confluent, Apache Kafka, others).Strong knowledge of distributed systems, scaling, and capacity planning.Familiarity with disaster recovery strategies and backup procedures.Experience with security best practices, including authentication, encryption, and access control.Understanding of compliance regulations and how to implement them in Kafka.Familiarity with metrics collection, monitoring, and observability tools (e.g., Prometheus, Grafana, Burrow).Excellent communication skills, both written and verbal, with the ability to engage with multiple stakeholders.Preferred Skills :
Expertise in cloud-native Kafka implementations and multi-cloud architectures.Experience with KRaft mode or alternative metadata management strategies.Familiarity with automation and orchestration tools in Kafka environments.
