How to delete Topic from Kafka : Topic marked for deletion issue

Mostly while working with Kafka topics we create, drop and recreate them but sometimes Kafka topics behaves in an intended way. For example after executing the drop command when we get the same “Gold Standard Message” that Topic is marked for deletion but when you check the topic is still present. This situation ideally doesn’t come when you have successfully deleted the topics in past but due to certain issues this problem can still pop-up. The list below would give you right sequence of steps to follow while deleting topic from Kafka (0.10.0). How to delete topic from Kafka:

  1. Ensure that you have the topic deletion privileges enabled by setting:

    This Enables topic deletion using admin tools. If you don’t have this set to true, then deleting topics through the admin tools has no effect at all.

  2. Disable auto topic creation by setting

    This setting enables auto creation of topics. If this is set to true, then any attempt to produce, consume, or fetch metadata for a non-existent topic would automatically create the topic with the default replication factor and number of partitions which is what we don’t want. Because if this is true and you try to get configuration detail of any specific topic or some other job (assuming being in the development environment) consuming the messages from or publishing the messages to the same topic then it would be recreated even if it was deleted earlier.

  3. Finally execute the command to delete the topic as:

    You should now be able to see the message stating that topic is marked for deletion. If you have followed above steps sequentially then the topic should have been deleted.

  4. But sometimes due to network issues (or metafile corruption, metafile lock) the Kafka topic’s metadata is not deleted from zookeeper servers. The only option left here is to hard delete the leftover metadata information which can be easily done using following commands:

    Diagram below depicts zookeeper znodes used/maintained by kafka and you can easily identify which znodes hierarchy we want to remove to permanently hard delete a kafka topic:

    Kafka znode hierarchy on Zookeeper
    Kafka znode hierarchy on Zookeeper

 

 

For more detailed information on Kafka znodes and their purpose watch this space for my upcoming blog.

 

Hopefully you now know how to delete topic from kafka better and your issues with Kafka topic deletion should be resolved with this last step (if it was still a concern after normal deletion). 

Do let me in case you still face any issue.

 

Leave a Reply

Your email address will not be published.