Using aggregation_key and summary_table_fields, but the generated alerts are not displaying the table content

[aisuhua]

Why the alerts are not displaying the summary table content and are instead sending each alert separately?

Using Docker image jertel/elastalert2:2.19.0.

ElasticSearch index

POST applog/_doc
{
  "appname": "cms",
  "level": "error",
  "message": "i am cms 1",
  "extra": "abc",
  "@timestamp": "2024-09-07T12:37:12.000+08:00"
}

POST applog/_doc
{
  "appname": "cms",
  "level": "error",
  "message": "i am cms 3",
  "extra": "abc",
  "@timestamp": "2024-09-07T12:37:14.000+08:00"
}

Rules configuration

name: alert by level field 
type: blacklist
index: applog

compare_key: level
blacklist:
  - error
  - err
  - fatal 

aggregation:
  minutes: 1

aggregation_key:
  - appname.keyword

summary_table_fields:
  - appname.keyword
  - count
  - message

alert:
  - debug

Output debug logs

Key log fragments

...
elastalert-elastalert2-1  | INFO:elastalert:New aggregation for alert by level field, aggregation_key: cms. next alert at 2024-09-07 04:39:04.332983+00:00.
elastalert-elastalert2-1  | INFO:elastalert:Adding alert for alert by level field to aggregation(id: 7GfGypEB5rLNh48YYVpW, aggregation_key: cms), next alert at 2024-09-07 04:39:04.332983+00:00
...
elastalert-elastalert2-1  | INFO:elastalert:alert by level field
elastalert-elastalert2-1  | 
elastalert-elastalert2-1  | @timestamp: 2024-09-07T12:37:12+08:00
elastalert-elastalert2-1  | _id: 6mfGypEB5rLNh48YUlre
elastalert-elastalert2-1  | _index: applog
elastalert-elastalert2-1  | appname: cms
elastalert-elastalert2-1  | appname.keyword: cms
elastalert-elastalert2-1  | extra: abc
elastalert-elastalert2-1  | level: error
elastalert-elastalert2-1  | message: i am cms 1
elastalert-elastalert2-1  | num_hits: 2
elastalert-elastalert2-1  | num_matches: 2
elastalert-elastalert2-1  | 
elastalert-elastalert2-1  | INFO:elastalert:Alert for alert by level field at 2024-09-07T12:37:14+08:00:
elastalert-elastalert2-1  | INFO:elastalert:alert by level field
elastalert-elastalert2-1  | 
elastalert-elastalert2-1  | @timestamp: 2024-09-07T12:37:14+08:00
elastalert-elastalert2-1  | _id: 62fGypEB5rLNh48YUlr-
elastalert-elastalert2-1  | _index: applog
elastalert-elastalert2-1  | appname: cms
elastalert-elastalert2-1  | appname.keyword: cms
elastalert-elastalert2-1  | extra: abc
elastalert-elastalert2-1  | level: error
elastalert-elastalert2-1  | message: i am cms 3
elastalert-elastalert2-1  | num_hits: 2
elastalert-elastalert2-1  | num_matches: 2
elastalert-elastalert2-1  | 

The full log

elastalert-elastalert2-1  | INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-09-07 04:37 UTC
elastalert-elastalert2-1  | INFO:elastalert:Background configuration change check run at 2024-09-07 04:38 UTC
elastalert-elastalert2-1  | INFO:elastalert:Disabled rules are: []
elastalert-elastalert2-1  | INFO:elastalert:Sleeping for 9.999761 seconds
elastalert-elastalert2-1  | INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-09-07 04:38 UTC
elastalert-elastalert2-1  | INFO:elastalert:Queried rule alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:38 UTC: 2 / 2 hits
elastalert-elastalert2-1  | INFO:elastalert:New aggregation for alert by level field, aggregation_key: cms. next alert at 2024-09-07 04:39:04.332983+00:00.
elastalert-elastalert2-1  | INFO:elastalert:Adding alert for alert by level field to aggregation(id: 7GfGypEB5rLNh48YYVpW, aggregation_key: cms), next alert at 2024-09-07 04:39:04.332983+00:00
elastalert-elastalert2-1  | INFO:elastalert:Ran alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:38 UTC: 2 query hits (0 already seen), 2 matches, 0 alerts sent
elastalert-elastalert2-1  | INFO:elastalert:alert by level field range 84
elastalert-elastalert2-1  | INFO:elastalert:Background configuration change check run at 2024-09-07 04:38 UTC
elastalert-elastalert2-1  | INFO:elastalert:Disabled rules are: []
elastalert-elastalert2-1  | INFO:elastalert:Sleeping for 9.999764 seconds
elastalert-elastalert2-1  | INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-09-07 04:38 UTC
elastalert-elastalert2-1  | INFO:elastalert:Queried rule alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:38 UTC: 2 / 2 hits
elastalert-elastalert2-1  | INFO:elastalert:Ran alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:38 UTC: 2 query hits (2 already seen), 0 matches, 0 alerts sent
elastalert-elastalert2-1  | INFO:elastalert:alert by level field range 98
elastalert-elastalert2-1  | INFO:elastalert:Background configuration change check run at 2024-09-07 04:38 UTC
elastalert-elastalert2-1  | INFO:elastalert:Disabled rules are: []
elastalert-elastalert2-1  | INFO:elastalert:Sleeping for 9.999756 seconds
elastalert-elastalert2-1  | INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-09-07 04:38 UTC
elastalert-elastalert2-1  | INFO:elastalert:Queried rule alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:38 UTC: 2 / 2 hits
elastalert-elastalert2-1  | INFO:elastalert:Ran alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:38 UTC: 2 query hits (2 already seen), 0 matches, 0 alerts sent
elastalert-elastalert2-1  | INFO:elastalert:alert by level field range 110
elastalert-elastalert2-1  | INFO:elastalert:Background configuration change check run at 2024-09-07 04:38 UTC
elastalert-elastalert2-1  | INFO:elastalert:Disabled rules are: []
elastalert-elastalert2-1  | INFO:elastalert:Sleeping for 9.999747 seconds
elastalert-elastalert2-1  | INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-09-07 04:38 UTC
elastalert-elastalert2-1  | INFO:elastalert:Queried rule alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:38 UTC: 2 / 2 hits
elastalert-elastalert2-1  | INFO:elastalert:Ran alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:38 UTC: 2 query hits (2 already seen), 0 matches, 0 alerts sent
elastalert-elastalert2-1  | INFO:elastalert:alert by level field range 122
elastalert-elastalert2-1  | INFO:elastalert:Background configuration change check run at 2024-09-07 04:38 UTC
elastalert-elastalert2-1  | INFO:elastalert:Disabled rules are: []
elastalert-elastalert2-1  | INFO:elastalert:Sleeping for 9.999706 seconds
elastalert-elastalert2-1  | INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-09-07 04:38 UTC
elastalert-elastalert2-1  | INFO:elastalert:Background configuration change check run at 2024-09-07 04:38 UTC
elastalert-elastalert2-1  | INFO:elastalert:Disabled rules are: []
elastalert-elastalert2-1  | INFO:elastalert:Sleeping for 9.999658 seconds
elastalert-elastalert2-1  | INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-09-07 04:38 UTC
elastalert-elastalert2-1  | INFO:elastalert:Queried rule alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:38 UTC: 2 / 2 hits
elastalert-elastalert2-1  | INFO:elastalert:Ran alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:38 UTC: 2 query hits (2 already seen), 0 matches, 0 alerts sent
elastalert-elastalert2-1  | INFO:elastalert:alert by level field range 134
elastalert-elastalert2-1  | INFO:elastalert:Background configuration change check run at 2024-09-07 04:39 UTC
elastalert-elastalert2-1  | INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-09-07 04:39 UTC
elastalert-elastalert2-1  | INFO:elastalert:Disabled rules are: []
elastalert-elastalert2-1  | INFO:elastalert:Sleeping for 9.99982 seconds
elastalert-elastalert2-1  | INFO:elastalert:Queried rule alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:39 UTC: 2 / 2 hits
elastalert-elastalert2-1  | INFO:elastalert:Ran alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:39 UTC: 2 query hits (2 already seen), 0 matches, 0 alerts sent
elastalert-elastalert2-1  | INFO:elastalert:alert by level field range 146
elastalert-elastalert2-1  | INFO:elastalert:Background configuration change check run at 2024-09-07 04:39 UTC
elastalert-elastalert2-1  | INFO:elastalert:Disabled rules are: []
elastalert-elastalert2-1  | INFO:elastalert:Sleeping for 9.999734 seconds
elastalert-elastalert2-1  | INFO:elastalert:Alert for alert by level field at 2024-09-07T12:37:12+08:00:
elastalert-elastalert2-1  | INFO:elastalert:alert by level field
elastalert-elastalert2-1  | 
elastalert-elastalert2-1  | @timestamp: 2024-09-07T12:37:12+08:00
elastalert-elastalert2-1  | _id: 6mfGypEB5rLNh48YUlre
elastalert-elastalert2-1  | _index: applog
elastalert-elastalert2-1  | appname: cms
elastalert-elastalert2-1  | appname.keyword: cms
elastalert-elastalert2-1  | extra: abc
elastalert-elastalert2-1  | level: error
elastalert-elastalert2-1  | message: i am cms 1
elastalert-elastalert2-1  | num_hits: 2
elastalert-elastalert2-1  | num_matches: 2
elastalert-elastalert2-1  | 
elastalert-elastalert2-1  | INFO:elastalert:Alert for alert by level field at 2024-09-07T12:37:14+08:00:
elastalert-elastalert2-1  | INFO:elastalert:alert by level field
elastalert-elastalert2-1  | 
elastalert-elastalert2-1  | @timestamp: 2024-09-07T12:37:14+08:00
elastalert-elastalert2-1  | _id: 62fGypEB5rLNh48YUlr-
elastalert-elastalert2-1  | _index: applog
elastalert-elastalert2-1  | appname: cms
elastalert-elastalert2-1  | appname.keyword: cms
elastalert-elastalert2-1  | extra: abc
elastalert-elastalert2-1  | level: error
elastalert-elastalert2-1  | message: i am cms 3
elastalert-elastalert2-1  | num_hits: 2
elastalert-elastalert2-1  | num_matches: 2
elastalert-elastalert2-1  | 
elastalert-elastalert2-1  | INFO:elastalert:Background alerts thread 1 pending alerts sent at 2024-09-07 04:39 UTC
elastalert-elastalert2-1  | INFO:elastalert:Queried rule alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:39 UTC: 2 / 2 hits
elastalert-elastalert2-1  | INFO:elastalert:Ran alert by level field from 2024-09-07 04:36 UTC to 2024-09-07 04:39 UTC: 2 query hits (2 already seen), 0 matches, 0 alerts sent
elastalert-elastalert2-1  | INFO:elastalert:alert by level field range 160
elastalert-elastalert2-1  | INFO:elastalert:Background configuration change check run at 2024-09-07 04:39 UTC
elastalert-elastalert2-1  | INFO:elastalert:Disabled rules are: []
elastalert-elastalert2-1  | INFO:elastalert:Sleeping for 9.999739 seconds
elastalert-elastalert2-1  | INFO:elastalert:Background alerts thread 0 pending alerts sent at 2024-09-07 04:39 UTC

[jertel]

Per the docs, aggregation_key isn't supported by the Blacklist or Whitelist rule types:

[aisuhua]

I try to use the any rule type，but the debug output is the same as above.
Something I am missing?
Thanks.

[jertel]

I recommend enabling debug logging to watch the queries and the results coming back to ElastAlert 2.

Also, there are two info (non-debug) log lines that mention when an event is aggregated:

New aggregation for ...
Adding alert for XYZ to aggregation...

Assuming there are no errors, if you aren't seeing those log lines then it means the rule does not appear to have aggregation setup correctly. That would happen if the wrong rule file is being loaded or the rule file failed to properly load (there would be errors at startup).