Ingesting LOGs questions

[alex-replicant]

Hi All,

Running the latest version of heplify and homer-app (Docker Hub images )

./homer-app --version
VERSION: 1.4.53

./heplify-server --version
Could not find toml config file, use flag defaults.
VERSION: heplify-server 1.59.5

We are looking to capture Freeswitch ESL (sipcapture/hepipe.js) events in Homer7
Got the basics working, heppipe gets JSON events and sends them to Heplify, Heplify stores received JSON payload to 'raw' field (var_char)

Have several questions about LOG events:

1)Default Flow view titles
HEP-LOG event type seem to display the first line of the raw field in DB, however does not parse JSON
/

LOG event type -
parses JSON properly from raw, however does not display any 'header' ie event-name etc...

Is it possible to modify this behavior?
Either add some kind of title/label to LOG entries in default flow view or make HEP-LOG entries parse the JSON properly..

2. JSON payload processing
   The raw field in the database is a var_char type, however payload comes in as JSON.

Is there any documentation that I can refer to how to store this JSON in data_header field?

I tried loading the LUA module; however it seem to crash the Heplify server completely.
Are there any alternatives?

3. It appears that Flow logic sorts events order based on 'create_date' value in hep_proto-XXX tables.
   However there is a possibility of events arriving to heplify out-of-order since they are coming from different nodes.

Is it possible to extract the actual event timestamp from JSON and override local create_date value?

Thank you in advance.

[lmangani]

1)Default Flow view titles
Please attach screenshots, etc to demonstrate your report

[alex-replicant]

Thank you for prompt reply. I just updated the original comment.

[lmangani]

Could you clarify the motivation for this request in more detail? I cannot understand the reasoning other than the cosmetic parts.

HEP logs are nice because they're simple and almost any HEP enabled system can generate them easily. The content of HEP type 100 is always a char string but the API can detect and parse the content as JSON when required and only on-demand. And as you know, the only association is the correlation_id header which allows Logs to be optionally related to a session/method/message.

The JSON content of HEP 100 is not meant to be parsed and is not designed to feed the data_header which is controlled by the agent.

I doubt you can achieve your goals with HEP logs. On the other hand, the upcoming version of HOMER is entirely based on our stack qryn, which has dedicated and sophisticated support for logs, metrics and traces alongside the packets.

[alex-replicant]

Realistically, they are cosmetic changes which will allow easy view of the order of things on a specific call (SIP packets+ESL events).
The challenge that it hard to identify which ESL event the user is looking at unless it's clicked on..

as a simple hack, would it be maybe possible to increase character limit displayed on the line for HEP-LOG events?
ie: currently, it cuts off

Would you be able to suggest where we can change this limit in our deployment?

Thank you.

[lmangani]

This should be very easy to do and definitely possible. Please open a feature request with this specific item and we'll check it out.

[lmangani]

@alex-replicant

parses JSON properly from raw, however does not display any 'header' ie event-name etc...

When JSON objects don't have a schema, data cannot be promoted to display fields since it might not exist.

[alex-replicant]

also, just one more question about timestamps:
It appears that Flow logic sorts events order based on 'create_date' value in hep_proto-XXX tables.
However there is a possibility of events arriving to heplify out-of-order since they are coming from different nodes.
Is it possible to extract the actual event timestamp from JSON and override local create_date value?

[lmangani]

@alex-replicant Homer knows how to sort and yes, you could hack the mapping to pick a different time header possibly , but ultimately the correct timestamp should be set by the agent. There's no other way so please do this in the extraction of heppipe.js or if you prefer using pastash and its transformers to customize the hep packets.

[alex-replicant]

Makes sense.. Thank you very much