ingest pipeline

[Alisher-Nabiev]

Hi, I am having trouble using an ingest pipeline. I can manually run the pipeline when reindexing the index, but I can't automatically ingest my index during creation.

this is my conf:
output {

elasticsearch {
host => 0.0.0.0
port => 9200
bulk_limit => 1000
bulk_timeout => 100
index_prefix => netflow
pipeline => imei-pipeline
data_type => netflow

testing my pipeline :
POST /_ingest/pipeline/imei-pipeline/_simulate
{
"docs": [
{
"_source": {
"ipv4_src_addr": "10.1.1.1"
}
}
]
}

res:
{
"docs" : [
{
"doc" : {
"_index" : "_index",
"_type" : "_doc",
"_id" : "_id",
"_source" : {
"ipv4_src_addr" : "10.1.1.1",
"customer" : {
"ipv4_src_addr" : "10.1.1.1",
"IMEI" : "89898989898989",
"CustomerName" : "some_brend",
"ThingId" : "ThingId_ICCID_89898989898989"
}
},
"_ingest" : {
"timestamp" : "2024-08-05T12:10:28.478468965Z"
}
}
}
]
}

when using reindex command:
POST /_reindex?wait_for_completion=false
{
"source": {
"index": "netflow-2024.08.02"
},
"dest": {
"index": "enriched_netflow-2024.08.02",
"pipeline": "imei-pipeline"
}
}

---

so my main goal is to ingest the netflow index to be created from the beginning with the data from the pipeline (IMEI, customer,CustomerNam, ThingId) and avoid reindexing.

[lmangani]

I'm afraid we can't help with any such aspect but If you have a proposed solution, we might be able to implement it.

[Alisher-Nabiev]

this is the same as in logstash :
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-pipeline

[lmangani]

I don't know what you mean or require here. I guess this is not about pastash but rather about Elasticsearch since we don't do anything about the request body and/or its fields.