Memory stream , Memory sink , memory lookup doubts.

[Ivelraj1997]

Please describe your problem in detail, if necessary, you can upload the log file through the attachment:

Hi ,

Memory Stream, is it possible to Join with another stream or might need memory look up table in this case ?

memory lookup table, is real table and that to store the data in memory ? or just virtual table ?

also data template is it only work for rest sink ? how to use datatemplate for redis sink?

[ngjaying]

Memory streams are the same as other streams, they can join within a window.
Memory lookup table is like a real table in memory.
Data template works for most sinks. Currently for redis sink, data template only takes effect when using string value type. If it doesn't meet you requirement, feel free to open a feature request with detail explanation of your requirement.

[Ivelraj1997]

Thanks for the answer , so the memory lookup table is real one then is it persist data in disk or will save data in memory only? or kind of pub sub concept ,is works?

https://ekuiper.org/docs/en/latest/concepts/sources/table.html#lookup-table

this above link definition say lookup table , it kind of interface and not persistence the data and help to query data from sources like redis , sql , memory sources

https://ekuiper.org/docs/en/latest/guide/sources/builtin/memory.html#create-a-lookup-table-source

this above link seems like , memory lookup table is persist in memory the data across stream processing .

this above two context of document seems like more different and confusing.

Redis sink with data template , is it only works for key type multiple and doesn't for key type single? if key type single to support data template then how to define key in data template?

Which latest version currently stable?

[ngjaying]

Memory lookup table is in memory only, a snapshot of history data. If "persist" means the data are kept after restart, then it is not persisted. Memory stream is in memory pub/sub.

Redis sink with data template , is it only works for key type multiple and doesn't for key type single? if key type single to support data template then how to define key in data template?

I was wrong, dataTemplate supports all key types. It will run prior to other transformations. So if you receive {"key":"value"} and apply a dataTemplate to convert it to {"key1":"value1"}, it will work as you receive the transformed json.

Current stable version is 1.11.x

[Ivelraj1997]

Thank you for the updates.

What is the different between schema less and schema type stream ?

If stream is defined schema types , when publish data to MQTT with wrong data type . currently checking type and throwing error and this is happen only in json_path function otherwise don't getting any problem . so I need to know both difference

Ex :

I have some property type defined as string and this is optional field , If the property value is nil then it is breaking

[ngjaying]

Schemaless will not check the schema at all. If schema is defined

• SQL parser will validate SQL based on Schema. For example, SELECT a FROM stream; In schema stream, if a is not defined, there will be a SQL validation error.
• When using JSON format, schema stream will be optimized to only parse the defined fields which will be faster
• If strictValidation=true is set in the stream, a preprocess operator will run to validate the data type of the data against the schema. There are performance overhead for the validation so it is turned off by default.

[Ivelraj1997]

When rule creation in old kuiper version the property triggered automatically add in rule section but the latest kuiper version that keyword no more and it doesn't add when rule create.

is it not required ?

[ngjaying]

If triggered is not set, it is default to true.