Opensearch set to 7.10.2 and not working with Elastalert2

[rajk1000]

Hi there, We are trying to use elastalert2 (docker version) with Opensearch (where opensearch has its backward compatibility mode set to TRUE). This means that the cluster version details returned from Opensearch are elastic search 7.10.2.

However, when elastalert container runs for the first time, it tries to create the index and returns this error. it does indeed create the index, BUT this 400 error does not look good. (last line of the code/logs block below. This is an excerpt from the logs).

elasticsearch:PUT [https://opensearch.sandbox:443/elastalert_status/_doc/_mapping?include_type_name=true](https://opensearch.sandbox/elastalert_status/_doc/_mapping?include_type_name=true) [status:400 request:0.007s]
Reading Elastic 7 index mappings:
Reading index mapping 'es_mappings/7/silence.json'
Reading index mapping 'es_mappings/7/elastalert_status.json'
Reading index mapping 'es_mappings/7/elastalert.json'
Reading index mapping 'es_mappings/7/past_elastalert.json'
Reading index mapping 'es_mappings/7/elastalert_error.json'
Traceback (most recent call last):
  File "/usr/local/bin/elastalert-create-index", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/create_index.py", line 242, in main
    create_index_mappings(es_client=es, ea_index=index, recreate=args.recreate, old_ea_index=old_index)
  File "/usr/local/lib/python3.12/site-packages/elastalert/create_index.py", line 78, in create_index_mappings
    es_client.indices.put_mapping(index=ea_index, doc_type='_doc',
  File "/usr/local/lib/python3.12/site-packages/elasticsearch/client/utils.py", line 152, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elasticsearch/client/indices.py", line 408, in put_mapping
    return self.transport.perform_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elasticsearch/transport.py", line 392, in perform_request
    raise e
  File "/usr/local/lib/python3.12/site-packages/elasticsearch/transport.py", line 358, in perform_request
    status, headers_response, data = connection.perform_request(
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elasticsearch/connection/http_requests.py", line 199, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "/usr/local/lib/python3.12/site-packages/elasticsearch/connection/base.py", line 315, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
**elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'request [/elastalert_status/_doc/_mapping] contains unrecognized parameter: [include_type_name]')**

SO, looking at the code in https://github.com/jertel/elastalert2/blob/master/elastalert/create_index.py#L77, it does seem that, if elastalert2 detects the ES cluster is 7.x, (which in our case is 7.10.2), it will try to add the ?include_type_name=true, BUT the Opensearch cluster returns a 400, as it clearly does not like that query format.

(1) Is there a way around this please? If , for example, we set an es_version value to 8.2, would it get around this? But, then would it create further problems, because actually our Opensearch backward compatible mode cluster is at 7.10.2.
(2) We could of course change our Opensearch cluster to be NOT backward compatible, in which case it may successfully create this index without error. However, other apps/components we use demand that our Opensearch cluster must be set to backward compaitible, in other words we need it to be set to 7.10.2
(3) As the index is actually created, do we need to worry about this 400 error? Is it serious or is it innocuous , and can we continue?
(a) By the way, if we do ignore it and continue, then we get another error:
Something like : elasticsearch.exceptions.RequestError: RequestError(400, ‘search_phase_execution_exception’, ‘No mapping found for [alert_time] in order to sort on’)

Can you please put a fix in your code to deal with Openseach in backward compatible mode, OR suggest a solution if possible.

Many Thanks

[jertel]

You do seem to be in a bit of a pickle. It sounds like you're running OpenSearch 2, which is equivalent to Elasticsearch 8, yet you're currently faking the OpenSearch version to be an older version 7. ElastAlert 2 is creating its indexes correctly, based on the detected version.

1. I don't know the answer to this. I don't have the capacity to test ElastAlert 2 in all various forms of version compatibility between old versions of Elasticsearch and OpenSearch. You would need to do this testing yourself.

2. You could try changing the cluster to not use backward compatability just for the index creation, and the set it back to behave like 7, but again you would need to do the testing.

3. Yes, it appears the index is not correctly being created, and is causing the mapping issue.

Unfortunately, I do not have capacity to try and make ElastAlert 2 run in all different version modes against a large number of different remote cluster vendors and versions. You would need to do this testing on your own time.