Hi,
I'd like to submit a feature request on behalf of our webMethods API Gateway customers who have encountered the following operational issue in production environments.
Background
In webMethods API Gateway, the internal Elasticsearch client uses Apache HttpAsyncClient to communicate with the Elasticsearch cluster for analytics data persistence. When the I/O Reactor enters a STOPPED state, the Elasticsearch client becomes permanently unavailable until the entire API Gateway is manually restarted.
This issue was confirmed in a production environment running API Gateway 10.5 (httpasyncclient-4.1.4), where 3 out of 4 cluster nodes were simultaneously affected. IBM Support confirmed through case investigation that this behavior is consistent with a known defect in httpasyncclient-4.1.4, specifically HTTPASYNC-155:
"Runtime exception in AbstractClientExchangeHandler can cause the I/O reactor to shut down."
This defect is documented in the official Apache HttpAsyncClient release notes:
https://downloads.apache.org/httpcomponents/httpasyncclient/RELEASE_NOTES-4.1.x.txt
HTTPASYNC-155 was fixed in httpasyncclient-4.1.5, which ships with API Gateway 11.1. However, IBM Support confirmed that even with 4.1.5, the risk of the I/O Reactor entering the STOPPED state is reduced but not eliminated.
Problem
Once the I/O Reactor enters the STOPPED state, the following operational impact occurs:
Analytics data (transactionalEvents) cannot be saved to Elasticsearch and is permanently lost
API Gateway Administrator UI becomes inaccessible ("Data store is down")
The only recovery option is a full manual restart of API Gateway, which is disruptive in production environments
There is no way to recover without human intervention, regardless of API Gateway version
Additionally, in httpasyncclient-4.1.4 (used in API Gateway 10.5 and 10.15), the exception that causes the I/O Reactor to shut down is not always written to the log, making root cause identification extremely difficult.
Why This Matters Even in API Gateway 11.1
API Gateway 11.1 ships with httpasyncclient-4.1.5, in which HTTPASYNC-155 is fixed and exception logging behavior is improved. This means the STOPPED state can now be properly detected through the logged exception. The technical foundation for implementing an automatic recovery mechanism is therefore already in place in API Gateway 11.1. However, no such mechanism has been implemented.
Since the risk is not fully eliminated even in 11.1, a transient network disruption or runtime exception could still cause the I/O Reactor to enter the STOPPED state, leaving manual restart as the only option.
Proposed Enhancement
We request that API Gateway implement an automatic recovery mechanism that detects when the Elasticsearch client's I/O Reactor enters the STOPPED state and automatically recreates the Elasticsearch client instance without requiring a full API Gateway restart.
The recovery mechanism could work as follows:
Monitor the Elasticsearch client's I/O Reactor state
Upon detecting the STOPPED state, log an appropriate error message
Automatically recreate the HttpAsyncClient instance and re-establish the Elasticsearch connection
Resume normal analytics data persistence without requiring a restart
Impact
Without this enhancement, even after upgrading to API Gateway 11.1, a rare but possible condition can still cause permanent loss of Elasticsearch connectivity that requires manual intervention. This is a fundamental operational stability concern for production environments running API Gateway.
This issue affects API Gateway 10.5, 10.15, and potentially 11.1.
Thanks and regards,
Sunjin Yim
