Kafka Architect

Job Description: Kafka Architect (Platform & Developer Focus)

Location: REMOTE

Position Overview: We are seeking a highly experienced Kafka Architect with 8-10 years of expertise in Kafka administration. This is a strategic role, not for a developer or admin, but for someone with deep architectural knowledge of Kafka and a strong ability to communicate complex concepts effectively. The ideal candidate will have extensive experience in designing, documenting, and optimizing both platform and developer elements of Kafka, with a focus on messaging and streaming capabilities. The candidate should be well-versed in implementing Kafka in diverse environments, with the ability to generalize concepts across different Kafka implementations (including Confluent and others).

Key Responsibilities:

Platform Focus: Infrastructure, Deployment, Operations, and Security

• Cluster Sizing & Capacity Planning:
• Define brokers per cluster, partition count, and replication factor based on data volume, retention policies, and throughput needs.
• Design scaling configurations (horizontal and vertical) for Kafka clusters.
• Establish optimal network configurations to ensure low-latency performance.
• Define Zookeeper configurations (if not using KRaft mode) or alternative metadata management.
  
• Storage Considerations:
• Provide recommendations for optimal disk configurations, with a focus on NVMe SSDs.
  
• High Availability & Fault Tolerance:
• Define optimal replication factor (RF) strategies for both production and non-production environments.
• Design broker failover strategies and leader election mechanisms.
• Develop multi-region and multi-AZ deployment strategies for Kafka.
  
• Security, Audit & Compliance:
• Implement and recommend Kafka authentication strategies (SASL, Kerberos, OAuth, TLS).
• Design authorization mechanisms (ACLs, RBAC) for Kafka.
• Advise on key management strategies, key rotation, and secure storage.
• Define auditing best practices for tracking access and changes to Kafka resources.
• Ensure encryption for both data in-transit (TLS) and at-rest (disk encryption).
• Advise on compliance frameworks (e.g., SOX, CCPA) and ensure Kafka adheres to necessary standards.
  
• Monitoring & Observability:
• Advise on metrics collection using tools like Prometheus, Grafana, and Confluent Control Center.
• Implement security monitoring tools to detect and respond to real-time threats.
• Provide recommendations for monitoring disk usage and log aggregation (e.g., Elasticsearch, Kibana, Splunk).
• Implement lag monitoring strategies using tools like Burrow or Kafka UI.
  
  
  

Developer Focus:

• Data Retention & Cleanup:
• Define log segment configurations and cleanup policies (delete vs. compact).
• Provide recommendations for Kafka compaction processes and scheduling.
• Advise on time-based vs. size-based retention policies to optimize resource usage.
  
• Disaster Recovery & Backup:
• Define strategies for cross-cluster replication and cluster linking.
• Set recovery point objectives (RPO) and recovery time objectives (RTO).
• Define automated backup verification and recovery procedures.
• Develop Kafka backup strategies, including configuration and topic-level backups.
  
• Cost Optimization:
• Recommend strategies for optimizing consumer group performance.
• Define dynamic partition rebalancing and scaling strategies.
• Recommend optimal data retention policies and efficient data compression formats.
  
  
  

Qualifications:

• 8-10 years of experience in Kafka architecture and administration (platform-focused).
• Strong knowledge of Kafka internals, cluster design, security, and monitoring tools.
• Proven ability to document and communicate complex Kafka platform architectures.
• Deep understanding of Kafkas role in both messaging and streaming use cases.
• Experience with different Kafka implementations (e.g., Confluent, Apache Kafka, others).
• Strong knowledge of distributed systems, scaling, and capacity planning.
• Familiarity with disaster recovery strategies and backup procedures.
• Experience with security best practices, including authentication, encryption, and access control.
• Understanding of compliance regulations and how to implement them in Kafka.
• Familiarity with metrics collection, monitoring, and observability tools (e.g., Prometheus, Grafana, Burrow).
• Excellent communication skills, both written and verbal, with the ability to engage with multiple stakeholders.
  
  
  

Preferred Skills:

• Expertise in cloud-native Kafka implementations and multi-cloud architectures.
• Experience with KRaft mode or alternative metadata management strategies.
• Familiarity with automation and orchestration tools in Kafka environments.