Improve DataPower JSV Compiler for Optimizing Schema Compilation

Description: I would like to request an enhancement for IBM DataPower's JSON Schema Validation (JSV) compiler to optimize the handling of large schemas with repetitive substructures. This request is based on performance challenges encountered during the compilation of extensive JSON schemas in a DataPower environment with limited memory resources.

Background: During the development process, we have experienced significant memory consumption and extended compilation times when using the JSV compiler. Specifically, compiling a schema with 6716 objects, which contains repetitive substructures, resulted in high resource consumption and long compilation times (55 minutes and 28.6 GB of RAM on a 64 GB DataPower instance). For environments with smaller memory capacities, such as our 16 GB DataPower instance, this led to throttle reload issues.

We were advised that no APAR will be created for this, but we should raise an RFE to improve the JSV compiler's efficiency.

Requested Enhancement: We propose the following improvements to the DataPower JSV compiler:

Optimization of Identical Subparts: The JSV compiler should identify identical subparts within the schema and automatically move these repetitive subparts to /definitions, utilizing $ref references to reduce redundant compilation efforts.

In our case, out of the 6716 objects, there are only 60 unique patterns. With optimization, only these 60 unique subparts should be compiled, rather than compiling the same structure 6716 times.

Reduced Compilation Time and Memory Usage: With the suggested improvement, we observed that a rewritten JSV schema (using a custom GatewayScript) reduced the initial compilation time to 16 minutes, and memory usage dropped to 25.5 GB, which would work within the limits of a 32 GB DataPower appliance. We request that similar optimization be incorporated directly into the DataPower JSV compiler to improve the experience for all users, especially those working with large schemas.

Business Impact: This enhancement would significantly reduce the time and memory required to compile large JSON schemas. It would improve performance for systems with limited memory resources and enhance the overall reliability and efficiency of the DataPower JSV compiler.