Intermittently consuming kafka messages in a springboot application

Issue

I have the following use-case:

Messages which my application could not process, possibly due to some intermittent issues related to network, etc, are being written to a kafka topic.

I want to periodically read these messages using a cron/scheduled task, and then process them.

So far I've used the KafkaListener in my springboot for my kafka-related needs, and has worked well.

However, I could not find a way to read messages intermittently using KafkaListener.

Is there even a way to do this in spring? Would be glad if someone could link to documents or tutorials or describe a way to achieve what I want to do.

Solution

You need to manage consuming process manually. It is possible if you disable autostartup feature of KafkaListener.

There are two discussions about this. See below:

Is there any example of Spring Schedule that reads Kafka topic?

Spring Boot Job scheduler with Kafka consumer

Answered By - Alper Derya