Kafka Topic Consumer

Overview

The Kafka Topic Consumer function is a source stream which reads records off of a Kafka topic and turns each into a valid tuple object. Every call to read() will return a single tuple.

Expression

The kafkaConsume function allows you to provide a topic name, a client id, and a connection string to the Kafka broker from which records should be read. You can also provide any arbitrary parameters which will be passed, as is, to the Kafka consumer client.

kafkaConsume(
  topic = topicName,
  groupId = clientGroupId,
  bootstrapServers = kafkaBootstrapServers
)

kafkaConsume(
  topic = topicName,
  groupId = clientGroupId,
  bootstrapServers = kafkaBootstrapServers,
  enable.auto.commit = true
)

Parameters

• topic: (required) This is the name of the Kafka topic
• groupId: (required) Any valid consumer group id
• bootstrapServers: (required) Connection string to the Kafka brokers

You can also provide any other name/value pair which will be passed directly to the Kafka Consumer client object. A list of available conection config parameters can be found here.

Registering with Solr

As with any Solr Streaming function, it can be registered by adding the following line to the standard solrconfig.xml file.

<expressible name="kafkaConsumer" 
             class="com.dennisgove.streaming.expressions.kafka.KafkaTopicConsumerStream"/>