Kafka Administrator

Job Overview

We are looking for a seasoned Kafka Administrator to oversee the operation and upkeep of enterprise-grade Apache Kafka clusters. The role involves installing, configuring, monitoring, optimizing performance, and ensuring the continuous availability of Kafka systems deployed in distributed environments.

Primary Responsibilities

• Administer Kafka clusters including installing, configuring, and maintenance tasks.
• Manage Kafka components such as brokers, topics, partitions, and replication processes.
• Conduct cluster upgrades, plan capacity, and scale infrastructure as necessary.
• Monitor cluster health utilizing tools like Prometheus, Grafana, and Confluent Control Center.
• Optimize the throughput, latency, and overall performance of Kafka brokers.
• Tune the configurations for Kafka producers and consumers for efficient operation.
• Handle management of real-time data pipelines and integration with frameworks including Spark, Hadoop, and Flink.
• Support streaming use cases and event-driven system architecture.
• Implement security features such as authentication and authorization via SSL, SASL, and Kerberos protocols.
• Configure Access Control Lists (ACLs) to enforce data governance and secure transmission.
• Ensure fault tolerance through replication and setup multi-cluster architectures using MirrorMaker or Cluster Linking.
• Develop and implement disaster recovery strategies.
• Troubleshoot Kafka issues including broker failures, consumer lag, and performance degradation.
• Analyze system logs to perform root cause analysis for operational problems.
• Automate Kafka deployments using tools like Ansible, Terraform, or custom scripts.
• Manage CI/CD pipelines tailored for Kafka environments and maintain version control with Git.
• Oversee Zookeeper cluster management or Kafka KRaft mode configuration if applicable.

Additional Information

• Experience with Confluent Platform is highly valued.
• Familiarity with deploying Kafka on Kubernetes (Kafka on K8s) is advantageous.
• Knowledge of DevOps tools and continuous integration/continuous deployment pipelines is expected.
• Relevant Kafka or Confluent certifications are considered beneficial but not mandatory.