Deleted Rule without disabling it

[CaGix22]

Hello! I recently encountered an issue where I deleted a rule without disabling it first.... I read later the documentation and also this post. So, I am in this scenario where I deleted it and the alerts keep arriving.

How can I restore this rule or ensure it is fully deleted from ElastAlert? If I copy the rule again with the same name will I be able to disable it? Or is restarting ElastAlert the only solution?

I have other scenarios where I deleted rules without disabling and did not receive any more alerts or notices in the ElastAlert indexes. I thought that maybe this new issue with the alerts might be because the rule has limit_execution and since the cron didn't end will I notice it on Friday? ( This is the cron: "25 2-22/4 * * 0-5" yes, I know the s wrongly configured but is the one that was in the rule at the time I deleted it).

I am quite lost here... has anybody encountered with something similar? Thank you for your assistance!

Update: Problem Timeline:
rule name: My Flatline Alert (everything working fine)

Made an update and changed it to: MyFlatlineAlert

Kept receiving alerts with the name "My Flatline Alert". Also, noticed on index, that "another rule" called: MyFlatlineAlert was constinously Pausing. (it was like two instances of the same rule with different names)

Deleted the rule without disabling. MyFlatlineAlert disappeared, but alerts with the name "My Flatline Alert" continue to arrive.
And saw in the console: INFO:elastalert:Rule file my_flatline_alert not found, stopping rule execution

But I keep receiving alerts with "My Flatline Alert" hahaha. I understand that the alert with the name "My Flatline Alert" stayed in memory, but i want it to make it dissapear 😂 pls help me

[jertel]

I just deleted a rule with a limit_execution and it successfully removed the scheduled job and the rule from memory.

2024-06-04 09:17:45,784     INFO apscheduler.executors.default Job "Rule: flatline11 (trigger: interval[0:00:10], next run at: 2024-06-04 09:18:00 EDT)" executed successfully
2024-06-04 09:17:47,408    DEBUG apscheduler.scheduler Looking for jobs to run
2024-06-04 09:17:47,409    DEBUG apscheduler.scheduler Next wakeup is due at 2024-06-04 09:17:57.408415-04:00 (in 9.999236 seconds)
2024-06-04 09:17:47,409     INFO apscheduler.executors.default Running job "Internal: Handle Pending Alerts (trigger: interval[0:00:10], next run at: 2024-06-04 09:17:57 EDT)" (scheduled at 2024-06-04 09:17:47.408415-04:00)
2024-06-04 09:17:47,410     INFO apscheduler.executors.default Running job "Internal: Handle Config Change (trigger: interval[0:00:10], next run at: 2024-06-04 09:17:57 EDT)" (scheduled at 2024-06-04 09:17:47.408542-04:00)
2024-06-04 09:17:47,411     INFO           elastalert Rule file r2/f.yaml not found, stopping rule execution
2024-06-04 09:17:47,411     INFO apscheduler.scheduler Removed job flatline11
2024-06-04 09:17:47,411     INFO           elastalert Background configuration change check run at 2024-06-04 09:17 EDT
2024-06-04 09:17:47,411     INFO apscheduler.executors.default Job "Internal: Handle Config Change (trigger: interval[0:00:10], next run at: 2024-06-04 09:17:57 EDT)" executed successfully
2024-06-04 09:17:47,412     INFO           elastalert Disabled rules are: []
2024-06-04 09:17:47,412     INFO           elastalert Sleeping for 9.999853 seconds

Notice the log lines that states the scheduler removed the job. The rule did not fire after that.

Restarting ElastAlert 2 to clear out the rule is the way forward. I can't recommend an alternative approach with any confidence, since I don't know how your rule got into this state. E.g., I can't seem to reproduce it.

[CaGix22]

Thanks Jertel for taking the time to try to reproduce it!

The chaos began when I updated only the name parameter in the rule (same .yaml file). Then, I deleted it thinking that would solve my problem.

I've added the rule again with the same old name "My Flatline Alert" that sends the email alerts and also the same file_name.yaml name, but this time with is_enabled: false . I'm still waiting for the Background check interval to check if it worked or not. Fingers crossed! I'll be back with news.

[CaGix22]

Hi again! So I come with bad news (for me haha)

I reproduced locally the scenario I described, long story short for whoever reads this: Do not change ever the name of the rule, if you want to do it, create it with a different name both the file and the name parameter value.

my rule config:

name: My Flatline Rule
  type: flatline
  run_every: 
    minutes: 5

Started with my rule file my_flatline_rule.yaml with the rule name: My Flatline Rule
INFO:elastalert:Queried rule My Flatline Rule from 2024-06-05 19:04 UTC to 2024-06-05 19:09 UTC: 0 hits

On the same rule file, updated the name to name: MyFlatlineRule And two instances of the same rule (with the two names) started running:

INFO:elastalert:Reloading configuration for rule rules/my_flatline_rule.yaml
INFO:elastalert:Queried rule My Flatline Rule from 2024-06-05 19:15 UTC to 2024-06-05 19:20 UTC: 0 hits
INFO:elastalert:Queried rule My Flatline Rule from 2024-06-05 19:20 UTC to 2024-06-05 19:20 UTC: 0 hits
INFO:elastalert:Sent email to ['dummy@maill']
INFO:elastalert:Ignoring match for silenced rule My Flatline Rule.all
INFO:elastalert:Ran My Flatline Rule from 2024-06-05 19:15 UTC to 2024-06-05 19:20 UTC: 0 query hits (0 already seen), 4 matches, 1 alerts sent
INFO:elastalert:My Flatline Rule range 301
INFO:elastalert:Queried rule MyFlatlineRule from 2024-06-05 19:15 UTC to 2024-06-05 19:20 UTC: 0 hits
INFO:elastalert:Ran MyFlatlineRule from 2024-06-05 19:15 UTC to 2024-06-05 19:20 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:MyFlatlineRule range 300

I deleted the file rule my_flatline_rule.yaml (with the last updated name:MyFlatlineRule but the rule with "My Flatline Rule" kept ongoing

INFO:elastalert:Rule file rules/my_flatline_rule.yaml not found, stopping rule execution
INFO:elastalert:Background configuration change check run at 2024-06-05 19:39 UTC
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-06-05 19:39 UTC
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 299.999801 seconds
INFO:elastalert:Queried rule My Flatline Rule from 2024-06-05 19:35 UTC to 2024-06-05 19:40 UTC: 0 hits
INFO:elastalert:Queried rule My Flatline Rule from 2024-06-05 19:40 UTC to 2024-06-05 19:40 UTC: 0 hits
INFO:elastalert:Sent email to ['dummy@maill']

Added again the file my_flatline_rule.yaml with the rule name: My Flatline Rule hoping that maybe ElastAlert will take both names again. The rule is not recognized, or yes but there is a conflict ID:

ERROR:apscheduler.executors.default:Job "Internal: Handle Config Change (trigger: interval[0:05:00], next run at: 2024-06-05 20:09:57 UTC)" raised an exception
Traceback (most recent call last):
...
apscheduler.jobstores.base.ConflictingIdError: 'Job identifier (My Flatline Rule) conflicts with an existing job'

Not even that stopped the rule from querying, I still receive the alerts... I see the "existing job", is the one I can't disable! hahaha So nothing else to do here but Restarting ElastAlert 2 to clear out the rule is the way forward. as Jertel said.

Today I learned something new about ElastAlert haha, thanks for your time Jertel!