ACE Embedded Global Cache: Graceful Peer Failure Detection and Planned Maintenance Notification for Replication Peers

During planned guest reboots, ACE processes are stopped cleanly (by disabling them in DataPower) before the OS reboot is initiated. However, when the HiperSocket interface goes offline, peer replication connections do not receive a TCP FIN or RST. As a result, ACE replication threads remain blocked waiting on existing SSL/TCP connections until the underlying TCP stack detects that the connection has failed.

This results in a measurable TPS (Transactions per second) degradation across the remaining ACE servers during planned maintenance activities.

IBM Support has confirmed that: 

ACE Embedded Global Cache does not implement application-level heartbeat or peer-failure detection. 

Dead-peer detection is entirely dependent on the underlying TCP layer. 

Replication threads remain blocked until a TCP socket timeout or connection failure is detected. 

TCP keepalive tuning is the recommended mitigation today. While TCP keepalive tuning reduces the recovery window, it does not eliminate the degradation period during planned maintenance events.

We were running ACE v12 WXS (WebSphere eXtreme Scale) prior to this and had two isolated global cache guests acting as Primary and secondary. When Primary goes down, the secondary switch to primary and we never saw any downtime during these planned reboots but that architecture was deprecated and we had to switch to Embedded cache.

Requested Enhancement:

Implement a more resilient peer-failure detection mechanism within ACE Embedded Global Cache so that planned or unplanned peer outages can be detected independently of TCP socket timeout behavior. 

Potential approaches include:

Provide heartbeat exchanges between cache peers.

Peer Membership Monitoring : Introduce cluster membership monitoring similar to other distributed cache products.

Graceful Maintenance Notification: Provide an explicit maintenance mode or shutdown notification mechanism.

Configurable Cache-Level Failure Detection: Provide ACE cache-specific settings such as peerFailureDetectionInterval/peerHeartbeatInterval/peerFailureThreshold allowing us to tune recovery behavior without relying solely on OS-level TCP parameters.

The ability to proactively notify peers would reduce throughput degradation during: 

OS patching, infrastructure maintenance, guest reboots and planned failovers