Integration


This is an IBM Automation portal for Integration products. To view all of your ideas submitted to IBM, create and manage groups of Ideas, or create an idea explicitly set to be either visible by all (public) or visible only to you and IBM (private), use the IBM Unified Ideas Portal (https://ideas.ibm.com).


Shape the future of IBM!

We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:

Search existing ideas

Start by searching and reviewing ideas and requests to enhance a product or service. Take a look at ideas others have posted, and add a comment, vote, or subscribe to updates on them if they matter to you. If you can't find what you are looking for,

Post your ideas

  1. Post an idea.

  2. Get feedback from the IBM team and other customers to refine your idea.

  3. Follow the idea through the IBM Ideas process.


Specific links you will want to bookmark for future use

Welcome to the IBM Ideas Portal (https://www.ibm.com/ideas) - Use this site to find out additional information and details about the IBM Ideas process and statuses.

IBM Unified Ideas Portal (https://ideas.ibm.com) - Use this site to view all of your ideas, create new ideas for any IBM product, or search for ideas across all of IBM.

ideasibm@us.ibm.com - Use this email to suggest enhancements to the Ideas process or request help from IBM for submitting your Ideas.


Add a new idea Subscribe
Status Submitted
Workspace App Connect
Created by Guest
Created on Apr 11, 2024

RELATED IDEAS

Provide the ability to enable the automatic, continuous loading of data streams into Snowflake as the data becomes available.

Many of our event messages varying between 2k and 4k in size (with an average closer to 2k) streamed today are based a pub/sub model that focuses on a variety of transactional data related to intraday customer balances, positions/holdings, as well as trade confirmations (maximum of about 15 topics per client).  This information is sourced from an on prem MQ manager with network connectivity to the queues hosted by the customer's on prem MQ infrastructure.  In order to support the data modernization requirements, the expectation is that we can leverage the IIB/ACE platform to route the messages from MQ through an adapter or connector to Snowflake.  Requirements include: 1) support for keypair auth as alternative to basic auth, 2) use of offset tokens, 3) message format consisting of XML and JSON, 4) with the ability to propagate the streaming payload into a table with the variant data type for semi-structured data, and 5) support target use case to stream data from IBM MQ series to Snowflake with the peak velocity of 300k messages per minute.  

Please refer to the following links:
Snowpipe Streaming | Snowflake Documentation
The SDK includes classes for streaming: Overview (Snowflake Ingest SDK 2.1.0-unshaded API) (javadoc.io)

Idea priority High
  • Admin
    Ben Thompson
    Reply
    |     Apr 15, 2024

    Idea review. Thank you very much for taking the time to raise this enhancement and for directly explaining to us in detail your usecases in the webex sessions held last week. As discussed on those calls, it was very helpful to provide us some rough throughput requirements for the interaction with Snowflake, which I'm adding here for future reference. We spoke about there being a inbound data from multiple MQ clients for sourcing messages destined for the Snowflake platform. For one particular inbound MQ queue or topic the ballpark data rates were about 10MB per second average throughput and possibly up to approximately 20MB per second for peak throughput. In total the solution would need to be able to cope with about 14 million messages per day. As discussed on the last call, the next step will be for Varun Jain to consider this requirement. Once that happens, the status of the idea will be updated (from "Submitted" status, most likely either to "Future Consideration" or "Not under consideration").

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.