limit_execution and run_every functionality

[CaGix22]

Hi! We are currently working in a flatline rule and we saw that according to what Jertel mentioned here: 1094 if we use run_every on a single rule it overrides the global param. The problem is that if the rule has a limit_execution configured the run_every parameter takes the global configuration ignoring the run_every IN the rule. Can these params be used together in a rule?

Also, we cannot understand the proper use of limit_execution. Even though Jertel explained that the parameter must be used as a range, we keep receiving alerts outside that range.

For example:
My logs are at: 5:10

Parameters in rule:

limit_execution: "30 5-6 * * 1-5"
run_every: 1 hour (in the global config)
use_count_query: true (so buffer_time is ignored)
timeframe: 2 hours

Elastalert not only checks between 5:30 AND 6:30 but also checks from 6:30 Day "A" to 5:30 Day "B" and throws an alert that between 7:30 and 9:30 on Day "B" there weren't any logs (which is true but ends up being a false alert because I just want to check the logs from the 5:10) We thought that limit_execution only processed the data at 5:30 (two hours backwards due to timeframe to find the 5:10 logs) and then again at 6:30 (two hours backwards also bc of timeframe). Also, the schedulet has set 1-5, but on Mondays I always receive an alert saying that on Saturday there weren't logs, again it is true, but I just want to be alerted if there are no logs from Mon to Fri at 5:10.

I hope you can understand my questions, thank you very much!

[jertel]

In general ElastAlert 2 is designed to not miss any gaps in time. So I think this behavior is intentional, to always pick up where it left off on the prior run.

That said, there is a hidden option called limit_execution_coverage which might be what you need. I haven't personally used it, but the code appears to prevent searching any further back in time than the timeframe.

limit_execution_coverage: true

[CaGix22]

Thanks Jertel for the answer! I'll take a look to the code and give a try to the param you mentioned.

[CaGix22]

As for my other question about overriding run_every and having limit_execution in the same rule, do you have any idea why run_every is being ignored?

logs are at: 2:00UTC

run_every: 12 hours
timeframe: 15 hours
limit_execution: 0 3/12 * * 1-5
use_count_query: true

The idea with this config was to try to "trick" EA and not to use limit_execution as a one time scheduler, so we decided to put 12 hours range. So, everytime that limit_execution processed the rule, run_every runs the rule and checks 15 hours back (timeframe) so the run will always find the logs at 2:00.

BUT we received an alert that between 3:00 and 18:00 there weren't logs, then run_every: 12hs is not being taken, because if it took the timeframe between those hours, then the rule ran at 18:00 and not at 15:00 as expected according to our cron. We think that then it MUST be taking run_every: 1 hour (from the global config) instead of overriding it.

Note: It was expected that run_every executed the rule at 15:00 as we observed that when a rule is deployed it executed the rule the same moment as the cron in limit_execution. We also took into account what we read here in discussions that the range of limit_execution must be equal or higher than run_every.

Has anyone had a similar issue with using both params together in the same rule?? Thanks!

[CaGix22]

run_every wasn't being ignored. We detected that every time that run_every is present in a rule, if it is larger than the run_every in the global config, it splits by the parameter in the config.

Example:
rule:
run_every: 12 hours
global config:
run_every: 1 hour

The queried rule splits by 1 hour, for a total run of 12 hours

INFO:elastalert:Queried rule TestRule from 2024-05-07 13:00 UTC to 2024-05-07 14:00 UTC: 0 hits
INFO:elastalert:Queried rule TestRule from 2024-05-07 14:00 UTC to 2024-05-07 15:00 UTC: 0 hits
.... (and so on)
INFO:elastalert:Queried rule TestRule from 2024-05-08 00:00 UTC to 2024-05-08 01:00 UTC: 0 hits
INFO:elastalert:Ran TestRule from 2024-05-07 13:00 UTC to 2024-05-08 01:00 UTC: 0 query hits (0 already seen), 2 matches, 1 alerts sent

Maybe this info would be useful for some EA users.