This screencast shows an example program reading a large directory and publishing the offsets to a Kafka instance. It displays it consuming batches of 50 files, then stopping and then resuming from where it stopped.
Note: the code is deliberately slowed down for a better screencast.