Enhancement to the IBM MQ Source Connector to expose a two configurable values to adjust the blocking of the JMS receive methods when retrieving messages from MQ.

Opening this idea to expose two configurable parameters to IBM MQ Source Connector. We will like to have an configurable parameter to adjust the blocking wait time on the JMS receive() methods when the Connector task is getting messages from MQ.

Currently the connector will do an initial receive with a wait of 3 seconds for a message. If a message is returned back from MQ, any proceeding messages will be received using the receiveNoWait() method until no messages are returned. If a message is not returned, it will send a partial batch of messages to Kafka.  

We would like an enhancement that will allow us to configure both the initial and proceeding blocking.

1. Initial receive  – allows to adjust the receive timeout for x (value time in ms) amount of time.
2. Proceeding receive – allows to adjust the receive timeout for x (value time in ms) amount of time.
   1. If value is set to 0, do receiveNoWait() (default value of what it does today)
   2. If integer value greater than 0, do receive(<value time in ms>)

Reasons for this enhancement:

·         We found that during a batch of workload that is putting a message at a rate of 1 to 5 ms between puts was consistently sending partial batches of 1 to Kafka. This resulted in an increase in CPU usage because the connector was sending a single batch of messages to Kafka.

·         Increasing the timeout of the proceeding receive will allow to get a higher chance of filling up the batch of messages. Having a bigger batch size before sending to Kafka will increase performance, throughput and efficiency of the Connector.

·         Allowing adjustment to both the initial and proceeding receive will help reduce the number of “failed” GETs from MQ. Lowering the CPU usage on both MQ and the Connector.  

·         If it is an application that doesn’t have tight SLA, we would want to get as many full batches. Any business-critical application that requires no delay, we would configure to do a receive with no wait.