ILMT Playbooks with big inventory

In our organisation over 10,000 servers are reporting to ILMT, it does consists of different OS, Regions etc. But the inventory that we use to collect results for ILMT consists over 4000 servers. The problem (although technically it is not a problem as the playbook is working in almost most of the cases and the right data is being exported to the lmt_server) is that the playbook runs for very long(approx 9 or 10 hrs) on the inventory. 

We have often noticed a time out issue on the job as well when it runs over 24hrs, specially for windows regional servers. We have even raised few IBM tickets as well in the past, when manually ran, issue resolves.

 As we use Ansible Automation Platform (AAP) and it have some mechanics build in which can mitigate these problems. Ideally, you want to implement a method whereby you can split your inventory in (equal) pieces. One way to achieve this is to implement serials in the header of your play / playbook. See the documentation regarding serials: Controlling playbook execution: strategies and more — Ansible Community Documentation.

The Automation Platform also has a mechanic built in with a similar feature namely job slicing.When setting this number, multiple jobs are being started in AAP (the amount is based on the value of job slicing if enabled). The inventory will be equally split as well.

Unfortunately, both options do not work with the lmt_collect_results playbook. The playbook consists of several play whereby the last play has the “lmt_server” hard-coded in the header of the playbook. This means, the lmt_server must always be present in the inventory. If you split the inventory, the lmt_server will also be part of the hosts which are being split. For instance: if you configure job slicing to be four, three out of four jobs will fail because the lmt_server is not available hence the data cannot be exported. 

 Hence to summarize our issue:

• The lmt_collect_result does not support job slicing / has no serial build in.
• Therefore, we cannot split out inventory.
• With large inventories (our inventory is over 3k in size) it is not best practice to perform on playbook once in terms of efficiency, resource utilization and playbook execution.
• We would like to see the lmt_collect_result playbook to be able to be compatible with job slicing if possible.

It would be nice if IBM ILMt playbooks can incorporate these adjustments to be more efficient on larger inventories for organisation like us.

Thanks in advance.