Integration

Shape the future of IBM!

We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:

Post your ideas

Start by posting ideas and requests to enhance a product or service. Take a look at ideas others have posted and upvote them if they matter to you,

  1. Post an idea

  2. Upvote ideas that matter most to you

  3. Get feedback from the IBM team to refine your idea

Help IBM prioritize your ideas and requests

The IBM team may need your help to refine the ideas so they may ask for more information or feedback. The offering manager team will then decide if they can begin working on your idea. If they can start during the next development cycle, they will put the idea on the priority list. Each team at IBM works on a different schedule, where some ideas can be implemented right away, others may be placed on a different schedule.

Receive a notification on the decision

Some ideas can be implemented at IBM, while others may not fit within the development plans for the product. In either case, the team will let you know as soon as possible. In some cases, we may be able to find alternatives for ideas which cannot be implemented in a reasonable time.

If you encounter any issues accessing the Ideas portal, please send email describing the issue to ideasibm@us.ibm.com for resolution. For more information about IBM's Ideas program visit ibm.com/ideas.

Status Future consideration
Workspace DataPower Gateway
Created by Guest
Created on Sep 13, 2021

Datapower syslog - GROK filter for Logstash

Datapower has delimeters in syslog messages, but when certain data is missing from the syslog, delimiters are also missing from the message so it is hard to figure out, what part is missing.


Example:

1. missing: DomainSub, Direction, ClientIP, GlobalTID

- Syslog:

<15>2021-08-20T13:56:31.831414+02:00 \\DP-TES\Domain1 [0x80e003ca][network][debug] xmlmgr(default): trans(4806034): Attempting TCP connect to ClientIP.


- Pattern:

%{TIMESTAMP_ISO8601:Time} \\\\%{GREEDYDATA:Device}\\%{GREEDYDATA:Domain} \[%{GREEDYDATA:MessageCode}\]\[%{GREEDYDATA:Category}\]\[%{GREEDYDATA:Severity}\] %{GREEDYDATA:ObjectType}\(%{GREEDYDATA:ObjectName}\): trans\(%{INT:TransactionID}\): %{GREEDYDATA:Message}

2. missing: Direction

- Syslog:

<14>2021-08-25T15:30:04.617934+02:00 \\\DP-TES\Domain1 [Domain1][0x80e0015b][network][info] mpgw(MPGW-Test): trans(16644771)[112.21.12.122] gtid(8b94ee1a612645dc00fdfaa3): HTTP response code 200 for 'https://CalloutDestination:915/Service1'


- Pattern:

%{TIMESTAMP_ISO8601:Time} \\\\%{GREEDYDATA:Device}\\%{GREEDYDATA:Domain} \[%{GREEDYDATA:DomainSub}\]\[%{GREEDYDATA:MessageCode}\]\[%{GREEDYDATA:Category}\]\[%{GREEDYDATA:Severity}\] %{GREEDYDATA:ObjectType}\(%{GREEDYDATA:ObjectName}\): trans\(%{INT:TransactionID}\)\[%{GREEDYDATA:ClientIP}\] gtid\(%{GREEDYDATA:GlobalTID}\): %{GREEDYDATA:Message}


Idea Priority Low
Why is it useful?

Currently it is not possible to parse Datapower syslogs in ELK with predefined GROK patterns.

Who would benefit from this IDEA? Everyone trying to parse Datapower syslogs
  • Guest
    Nov 29, 2021

    Hello,

    my final filters (I couldn't join these two, so i left them seperate):


    firstFilter ^(<%{INT:PRIORITY}>)?%{TIMESTAMP_ISO8601:Time} \\\\%{GREEDYDATA:Device}\\%{GREEDYDATA:Domain} (\[%{USERNAME:DomainSub}\])?\[(%{BASE16NUM:MessageCode})?\]\[%{USERNAME:Category}\]\[%{LOGLEVEL:Severity}\]( %{DATA:ObjectType}\(%{DATA:ObjectName}\):)? (trans\(%{INT:TransactionID}\))?(\[%{WORD:Direction}\])?(\[%{IP:ClientIP}\])?( gtid\(%{BASE16NUM:GlobalTID}\))?: %{GREEDYDATA:Message}

    secondFilter ^(<%{INT:PRIORITY}>)?%{TIMESTAMP_ISO8601:Time} \\\\%{GREEDYDATA:Device}\\%{GREEDYDATA:Domain} (\[%{USERNAME:DomainSub}\])?\[(%{BASE16NUM:MessageCode})?\]\[%{USERNAME:Category}\]\[%{LOGLEVEL:Severity}\]( %{DATA:ObjectType}\(%{DATA:ObjectName}\):)? (trans\(%{INT:TransactionID}\))?(\[%{WORD:Direction}\])?(\[%{IP:ClientIP}\])?( gtid\(%{BASE16NUM:GlobalTID}\))?%{GREEDYDATA:Message}


    That is all for now, I will focus on other tasks.

  • Guest
    Nov 2, 2021

    Hi,


    It seems this task is doable. I tried merging all the filters into one, so far I have come the pattern below - it still needs a few corrections, but it mostly works fine. "\\Device\Domain" format is needed as Local Identifier in DP Log target for this to work):


    %{TIMESTAMP_ISO8601:Time} \\\\%{GREEDYDATA:Device}\\%{GREEDYDATA:Domain} (\[%{GREEDYDATA:DomainSub}\])?\[%{GREEDYDATA:MessageCode}\]\[%{GREEDYDATA:Category}\]\[%{GREEDYDATA:Severity}\]( %{GREEDYDATA:ObjectType}\(%{GREEDYDATA:ObjectName}\):)? trans\(%{INT:TransactionID}\)(\[%{WORD:Direction}\])?(\[%{GREEDYDATA:ClientIP}\])?( gtid\(%{GREEDYDATA:GlobalTID}\))?: %{GREEDYDATA:Message}


    To be continued...