How to Automate Kafka Topic Creation and Deletion, Using GitHub Actions and JavaScript

Automating Kafka topic creation and deletion is crucial for developers using event-driven architectures. Manual management becomes impractical as your Kafka deployment scales, whether you're dealing with a growing system or implementing infrastructure as code. This guide demonstrates automating this process using JavaScript and GitHub Actions, integrating it seamlessly into your DevOps workflow.

The Challenges of Manual Kafka Topic Management

Kafka topics are fundamental to event organization. As your architecture expands, you'll need diverse topics for:

• Categorizing different event types
• Storing filtered event subsets
• Maintaining transformed event versions

Manual topic management becomes complex and prone to errors with system growth. Each new service may require new topics, and each environment (development, staging, production) needs its own consistently configured set. Without automation, this manual overhead slows development and increases configuration errors.

Benefits of Automating Topic Management

Automating Kafka topic creation and deletion offers significant advantages:

1. Consistent topic configuration across all environments.
2. Reduced human error in topic management.
3. Version control for topic configurations.
4. Streamlined DevOps workflows.
5. Integration of topic changes into your CI/CD pipeline.

Choosing the Right Tools

This tutorial utilizes JavaScript for its automation solution. While other languages are possible, JavaScript offers benefits:

• Simpler scripting than Bash.
• Leverages existing team JavaScript expertise.
• Rich package ecosystem via npm.
• Clean async/await syntax for Kafka operations.

This solution is implemented as a Node.js application running as a Kubernetes Job, ideal for teams using VPC-accessible Kafka clusters.

Setting Up the Application

You'll need a Node.js project. Create one using npm init -y. If Node.js and npm aren't installed, download them from https://www.php.cn/link/0d78f6439e652fdbf801d103430d2e12.

In your project directory, run npm install kafkajs to install the Kafka JavaScript client.

Implementing the Solution

Our automation comprises three core components:

1. A JSON configuration file specifying topics to create or delete.
2. A Node.js application interacting with Kafka to perform these operations.
3. A GitHub Actions workflow triggering automation upon JSON configuration file changes.

Let's build each component.

Configuration File Setup

The application reads a JSON file listing topics to create or delete. This allows for version-controlled changes via GitHub pull requests. After merging a PR, the code reads the updated file and creates/deletes topics accordingly.

Create topics.json:

{
  "create": [],
  "delete": []
}

This structure provides a version-controlled record of topic management actions. The create array lists topics for creation, while delete lists topics for removal.

Building the Topic Management API

Create api.js:

async function createTopics(topics, kafkaAdmin) {
  if (topics.length > 0) {
    await kafkaAdmin.createTopics({
      topics: topics.map((topic) => ({
        topic,
        numPartitions: 1,        // Adjust as needed
        replicationFactor: 3,    // Adjust as needed
        configEntries: [
          { name: "min.insync.replicas", value: "2" }  // Adjust as needed
        ],
      })),
    });
  }
}

async function deleteTopics(topics, kafkaAdmin) {
  if (topics.length > 0) {
    await kafkaAdmin.deleteTopics({ topics: topics });
  }
}

module.exports = { createTopics, deleteTopics };

This module exports functions for creating and deleting Kafka topics. Adjust configuration values (partitions, replication factor, min.insync.replicas) based on your needs.

Creating the Main Application

Create index.js:

const { Kafka } = require("kafkajs");
const { createTopics, deleteTopics } = require("./api");
const topics = require("../topics.json");

// ... (Kafka connection details using environment variables) ...

const kafka = new Kafka({
  // ... (Kafka connection configuration) ...
});

const admin = kafka.admin();

admin.connect().then(async () => {
  const existingTopics = await admin.listTopics();

  // Create only non-existent topics
  const newTopics = topics.create.filter((x) => !existingTopics.includes(x));
  await createTopics(newTopics, admin);

  // Delete only existing topics
  const deletionTopics = topics.delete.filter((x) => existingTopics.includes(x));
  await deleteTopics(deletionTopics, admin);

  await admin.disconnect();
});

This application handles Kafka connection, reads topic configuration, and orchestrates topic creation/deletion. It includes safeguards against duplicate operations and ensures clean connection handling.

Automating Deployment with GitHub Actions

Integrate the topic management application into a CI/CD pipeline using GitHub Actions. This automates execution whenever the topic configuration changes.

Add .github/workflows/kafka.yml:

# ... (GitHub Actions workflow YAML configuration -  This section needs significant adaptation for a real-world scenario and would require setting up AWS credentials, ECR, and Kubernetes cluster details. The provided example is incomplete and illustrative only.) ...

This workflow automates the build, containerization (using Docker), pushing to a container registry (like Amazon ECR), and deployment to Kubernetes as a Job.

Conclusion

This guide demonstrates automating Kafka topic creation and deletion using JavaScript and GitHub Actions. This approach, using a JSON configuration file and GitHub Actions, provides a scalable solution for managing Kafka topics as your system grows. Integrating topic management into your CI/CD pipeline ensures consistent and reliable topic configurations across your Kafka deployment. This transforms manual topic management into a streamlined, automated process aligned with modern DevOps practices. The provided code serves as a foundation for customization and adaptation to your specific requirements.

The above is the detailed content of How to Automate Kafka Topic Creation and Deletion, Using GitHub Actions and JavaScript. For more information, please follow other related articles on the PHP Chinese website!