Elastalert is using up to much CPU and memory

[fast-turtle-9]

I am using an ubuntu VM, and running elastalert on it, there is nothing else being ran on the Ubuntu except fo relastalert. I had configure the machine to use 8 GB and 2 cpus and the machien started crashing alot, increased it to 16 GB and 4 cpus and it is working better. But the memory being used is quite a lot, is it supposed to be as such? I know elastalert keeps an instance per rule in memory to store the data, if so is there a way to flush that data periodically through any rule category?

[nsano-rururu]

Please note that if you do not regularly close the indexes accumulated in Elasticsearch, the Java heap memory will be exhausted and Elasticsearch will become unresponsive, even if there is enough disk space.

[fast-turtle-9]

So the elasticsearch is running on a separate machine and the index is being flushed regularly. The machine that has elastalert running is the issue, it is taking up a lot of ram, the process itself shows it taking over majority of the ram

[jertel]

If ElastAlert 2 is consuming excessive memory then the most likely culprit is your rules are attempting to collect excessive data from Elasticsearch, and storing it in memory in order to compare subsequent rule runs with the previously collected data. If you are using very large sliding windows, or if your queries are returning massive numbers of records that would do it.

[fast-turtle-9]

So you were right, it was the large sliding windows but changing the window to a shorter time only delayed it, apparently elastalert is storing everything in memory and so the memory keeps rising day by day util it maxes out and crashes the machine. Am I missing a config option to release memory after certain time period? Thank you
@jertel

[jertel]

The memory usage depends on the rule configurations and the amount of data each rule returns on every query. I've not personally run into a scenario where ElastAlert 2 runs out of memory, and I've had it running for many months at a time. I'm not saying there's no memory leak, but just that I've not observed it and so it will be difficult to find it without seeing it firsthand.

If you can rewrite your rules to avoid querying large result sets, such as switching to use_count_query to only return counts instead of the actual documents, that would likely eliminate the problem. Or if you can continue to isolate the problem then perhaps we have a better shot of finding out specifically what's going on. But that will require you to continue adjusting rule and config settings, watching logs, etc.