Support Schema registry to integrate with Kafka

Our specific use case is related to facilitate interaction with serialised Avro on Confluent, but any kind of improvement in that area would be welcome. At the moment almost everything we do to requires custom logic in java compute nodes, turning ACE into a dumb wrapper for java code, which is less than desirable.

Support for Avro schema in a similar vein to what is available for XSD. Having a direct link to the schema registry would be nice, but honestly, not as important. Based on those, conversion from/to either a new Avro parser or just the regular Json one. And then being able to link a topic to the current schema, so it knows how to serialise/deserialise the message tree. Some support for magic bytes might be required here, but seeing as different vendors have different solutions for them, I would settle for being able to just provide a setting to denote how much has to be trimmed from the front.
More fine grained control on the commit when acting as a consumer would be nice, as the node currently commits on propagation and you have to manually build in transactionality through other means.
On the schema lifecycle, I don't expect a fully managed solution around this. Honestly, it can already be worked around with a byte stream reader and interpreting the magic bytes and routing, but maybe a slightly more OOTB way of working might be useful. For example, if the consumer node can propagate some info on the magic bytes in the local environment or message properties, that would make life easier.
With XSD, generation of java classes with Jaxb is provided out of the box and the runtime comes with Jackson, so something similar for Avro would be greatly appreciated.

I assume most of these will already be on their radar, but regardless, these are some of the features we could really use.

Are we considering this feature to make the Kafka Producer a schema-aware client?

This benefits so many ACE users and adds value to ACE ecosystem.

RFE Review. Thank you for taking the time to submit this idea for enhancement which we agree would benefit the product. In the past we have extended the Kafka message flow nodes to assist with use cases involving schema registries - the provision of options to set and retrieve Kafka custom header properties helps flow developers deal with the association between a Kafka message and the schema which describes its format:

https://www.ibm.com/docs/en/app-connect/12.0?topic=enterprise-setting-retrieving-kafka-custom-header-properties

One popular serialization/deserialization format is Avro, and being able to interrogate Kafka custom header properties in a JavaCompute node can help a user write code to select a relevant schema when interpreting the content of a Kafka message.

Regarding the specifics of the idea, we like the concept of tying together the process of reading/writing the message with the process of injecting/retrieving the schema from the registry, but also note that in many circumstances flow developers might want a built-in function for doing these tasks but not necessarily want to execute the injection/retrieval of the schema into the registry at the same time (potentially this could be controlled by separate flow logic, or only invoked for certain messages depending on their characteristics, or perhaps a set of commonly used schemas cached in memory). Any thoughts on these aspects would be appreciated as it would help us devise the best externals for this feature (eg separate node for the purpose of injection/retrieval, or extra properties on the existing KafkaConsumer / KafkaRead / KafkaProducer?)

Status of the Idea is updated to Future Consideration.