Flatline alerts sending alerts even though logs are there

[eeH9ahso]

Hi there. This could be just me not understanding how flatline alerts are being triggered. I have a rule that looks for Suricata logs and what I'm wanting to do is send a Slack notification if there's 0 alerts in 5 minutes. Here's my rule:

name: Suricata log ingestion flatline
type: flatline
index: .ds-logs-suricata.eve-default-*
timestamp_field: timestamp
use_count_query: true
is_enabled: true

threshold: 1
timeframe:
  minutes: 5
filter:
- query:
    query_string:
      query: "*"

alert:
  - "slack"

alert_subject: "0 logs found for log source: Suricata"
alert_text: "No Suricata logs found within the defined threshold period."
alert_text_type: alert_text_only

slack_webhook_url: "https://hooks.slack.com/..."

Here's the elastalert output:

INFO:elastalert:Loaded new rule rules/flatline_suricata.yaml
INFO:elastalert:Background configuration change check run at 2024-04-22 20:59 UTC
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-04-22 20:59 UTC
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999421 seconds
INFO:elastalert:Queried rule Suricata log ingestion flatline from 2024-04-22 20:58 UTC to 2024-04-22 20:59 UTC: 0 hits
INFO:elastalert:Ran Suricata log ingestion flatline from 2024-04-22 20:58 UTC to 2024-04-22 20:59 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Suricata log ingestion flatline range 60
INFO:elastalert:Background configuration change check run at 2024-04-22 21:00 UTC
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-04-22 21:00 UTC
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999865 seconds
INFO:elastalert:Queried rule Suricata log ingestion flatline from 2024-04-22 20:59 UTC to 2024-04-22 21:00 UTC: 0 hits
INFO:elastalert:Queried rule Suricata log ingestion flatline from 2024-04-22 21:00 UTC to 2024-04-22 21:00 UTC: 0 hits
INFO:elastalert:Ran Suricata log ingestion flatline from 2024-04-22 20:59 UTC to 2024-04-22 21:00 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Suricata log ingestion flatline range 61
INFO:elastalert:Background configuration change check run at 2024-04-22 21:01 UTC
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-04-22 21:01 UTC
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999864 seconds
INFO:elastalert:Queried rule Suricata log ingestion flatline from 2024-04-22 21:00 UTC to 2024-04-22 21:01 UTC: 0 hits
INFO:elastalert:Queried rule Suricata log ingestion flatline from 2024-04-22 21:01 UTC to 2024-04-22 21:01 UTC: 0 hits
INFO:elastalert:Ran Suricata log ingestion flatline from 2024-04-22 21:00 UTC to 2024-04-22 21:01 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Suricata log ingestion flatline range 61
INFO:elastalert:Background configuration change check run at 2024-04-22 21:02 UTC
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-04-22 21:02 UTC
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999874 seconds
INFO:elastalert:Queried rule Suricata log ingestion flatline from 2024-04-22 21:01 UTC to 2024-04-22 21:02 UTC: 0 hits
INFO:elastalert:Queried rule Suricata log ingestion flatline from 2024-04-22 21:02 UTC to 2024-04-22 21:02 UTC: 0 hits
INFO:elastalert:Ran Suricata log ingestion flatline from 2024-04-22 21:01 UTC to 2024-04-22 21:02 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Suricata log ingestion flatline range 62
INFO:elastalert:Background configuration change check run at 2024-04-22 21:03 UTC
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999854 seconds
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-04-22 21:03 UTC
INFO:elastalert:Queried rule Suricata log ingestion flatline from 2024-04-22 21:02 UTC to 2024-04-22 21:03 UTC: 0 hits
INFO:elastalert:Queried rule Suricata log ingestion flatline from 2024-04-22 21:03 UTC to 2024-04-22 21:03 UTC: 0 hits
INFO:elastalert:Ran Suricata log ingestion flatline from 2024-04-22 21:02 UTC to 2024-04-22 21:03 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Suricata log ingestion flatline range 61
INFO:elastalert:Background configuration change check run at 2024-04-22 21:04 UTC
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-04-22 21:04 UTC
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.99986 seconds
INFO:elastalert:Queried rule Suricata log ingestion flatline from 2024-04-22 21:03 UTC to 2024-04-22 21:04 UTC: 0 hits
INFO:elastalert:Queried rule Suricata log ingestion flatline from 2024-04-22 21:04 UTC to 2024-04-22 21:04 UTC: 0 hits
INFO:elastalert:Alert 'Suricata log ingestion flatline' sent to Slack
INFO:elastalert:Ignoring match for silenced rule Suricata log ingestion flatline.all
INFO:elastalert:Ignoring match for silenced rule Suricata log ingestion flatline.all
INFO:elastalert:Ignoring match for silenced rule Suricata log ingestion flatline.all
INFO:elastalert:Ran Suricata log ingestion flatline from 2024-04-22 21:03 UTC to 2024-04-22 21:04 UTC: 0 query hits (0 already seen), 4 matches, 1 alerts sent
INFO:elastalert:Suricata log ingestion flatline range 62

But I get a Slack notification every minute:

What am I doing wrong here?

Thanks!

[jertel]

Is the concern that you are receiving Slack alerts every 1 minute instead of every 5 minutes?

[eeH9ahso]

Sorry, I didn't include more information. I was just wondering why it's sending alerts when there are results returned.

Maybe 5 minute timeframe is too small of a window? Although I have a CrowdStrike flatline rule similar to the one above (but pointing to another index obviously) where the timeframe is set to 60 minutes and it's generating alerts:

Even though you can see there are definitely logs. This is the CrowdStrike index over the last 24 hours:

Here are the logs:

INFO:elastalert:Queried rule CrowdStrike log ingestion flatline from 2024-04-23 14:46 UTC to 2024-04-23 14:47 UTC: 0 hits
INFO:elastalert:Queried rule CrowdStrike log ingestion flatline from 2024-04-23 14:47 UTC to 2024-04-23 14:47 UTC: 0 hits
INFO:elastalert:Alert 'CrowdStrike log ingestion flatline' sent to Slack
INFO:elastalert:Ignoring match for silenced rule CrowdStrike log ingestion flatline.all
INFO:elastalert:Ignoring match for silenced rule CrowdStrike log ingestion flatline.all
INFO:elastalert:Ignoring match for silenced rule CrowdStrike log ingestion flatline.all
INFO:elastalert:Ran CrowdStrike log ingestion flatline from 2024-04-23 14:46 UTC to 2024-04-23 14:47 UTC: 0 query hits (0 already seen), 4 matches, 1 alerts sent
INFO:elastalert:CrowdStrike log ingestion flatline range 64

Here's the rule:

name: CrowdStrike log ingestion flatline
type: flatline
index: .ds-logs-crowdstrike.falcon-default-*
timestamp_field: timestamp
use_count_query: true
is_enabled: true

threshold: 1
timeframe:
  minutes: 60
filter:
- query:
    query_string:
      query: "*"

alert:
  - "slack"
alert_subject: "0 logs found for log source: CrowdStrike"

slack_webhook_url: "https://hooks.slack.com..."

I'm sure it's something I'm doing wrong, I'm just not sure what.

Thanks again

[jertel]

Try changing your timestamp_field to @timestamp instead of timestamp.

[eeH9ahso]

Hey there and thanks again for the reply. No love when changing out timestamp to @timestamp. It throws errors when trying to load the rule:

# docker run --name elastalert -v /opt/elastalert2/config.yaml:/opt/elastalert/config.yaml -v /opt/elastalert2/rules:/opt/elastalert/rules jertel/elastalert2 --verbose                                                                                                                    [24/1943]
Reading Elastic 8 index mappings: 
Reading index mapping 'es_mappings/8/silence.json'     
Reading index mapping 'es_mappings/8/elastalert_status.json'
Reading index mapping 'es_mappings/8/elastalert.json'
Reading index mapping 'es_mappings/8/past_elastalert.json'                                                                                        
Reading index mapping 'es_mappings/8/elastalert_error.json'
Index elastalert_status already exists. Skipping index creation.
2024-04-25 14:48:20,605    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_USE_SSL' casted as 'None'/'None' with default 'None'
2024-04-25 14:48:20,605    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_BEARER' casted as 'None'/'None' with default 'None'
2024-04-25 14:48:20,605    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_PASSWORD' casted as 'None'/'None' with default 'None' 
2024-04-25 14:48:20,605    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_USERNAME' casted as 'None'/'None' with default 'None' 
2024-04-25 14:48:20,605    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_API_KEY' casted as 'None'/'None' with default 'None'
2024-04-25 14:48:20,605    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_HOST' casted as 'None'/'None' with default 'None'                                                                                                                                                      
2024-04-25 14:48:20,605    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_HOSTS' casted as 'None'/'None' with default 'None'
2024-04-25 14:48:20,605    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_PORT' casted as 'None'/'None' with default 'None'
2024-04-25 14:48:20,606    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_URL_PREFIX' casted as 'None'/'None' with default 'None'
2024-04-25 14:48:20,606    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'STATSD_INSTANCE_TAG' casted as 'None'/'None' with default 'None'
2024-04-25 14:48:20,606    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'STATSD_HOST' casted as 'None'/'None' with default 'None' 
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/elastalert/loaders.py", line 597, in get_yaml
    return read_yaml(filename)                                                                                                                                                                                                                                                                      
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/yaml.py", line 8, in read_yaml
    return yaml.load(yamlContent, Loader=yaml.FullLoader)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/composer.py", line 64, in compose_node
    if self.check_event(AliasEvent):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
                         ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/parser.py", line 449, in parse_block_mapping_value
    if not self.check_token(KeyToken, ValueToken, BlockEndToken):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/scanner.py", line 116, in check_token
    self.fetch_more_tokens()
  File "/usr/local/lib/python3.12/site-packages/yaml/scanner.py", line 258, in fetch_more_tokens
    raise ScannerError("while scanning for the next token", None,
yaml.scanner.ScannerError: while scanning for the next token
found character '@' that cannot start any token
  in "<unicode string>", line 4, column 18:
    timestamp_field: @timestamp
                     ^

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/elastalert/loaders.py", line 172, in load
    rule = self.load_configuration(rule_file, conf, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/loaders.py", line 237, in load_configuration
    rule = self.load_yaml(filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/loaders.py", line 261, in load_yaml
    loaded = self.get_yaml(current_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/loaders.py", line 599, in get_yaml
    raise EAException('Could not parse file %s: %s' % (filename, e))
elastalert.util.EAException: Could not parse file rules/flatline_crowdstrike.yaml: while scanning for the next token
found character '@' that cannot start any token
  in "<unicode string>", line 4, column 18:
    timestamp_field: @timestamp
                     ^

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/elastalert", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/elastalert.py", line 1916, in main
    client = ElastAlerter(args)
             ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/elastalert.py", line 128, in __init__
    self.rules = self.rules_loader.load(self.conf, self.args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/loaders.py", line 184, in load
    raise EAException('Error loading file %s: %s' % (rule_file, e))
elastalert.util.EAException: Error loading file rules/flatline_crowdstrike.yaml: Could not parse file rules/flatline_crowdstrike.yaml: while scanning for the next token
found character '@' that cannot start any token
  in "<unicode string>", line 4, column 18:
    timestamp_field: @timestamp
                     ^

[jertel]

Needs to be wrapped in quotes:

timestamp_field: "@timestamp"

Or just get rid of the line altogether since it's the default. I wasn't sure why you were trying to override it in the first place, since your Kibana screenshot shows that the timestamp field starts with @.