[TI_MISP] Transform logs-ti_misp.latest_ioc can enter on a FAILED state because of mapping conflicts.

[leandrojmp]

Hello,

The MISP integration has a transform of the type latest associated to it, but it looks like that the transform can enter a failed state if it hits some mapping conflict on the destination index.

What happens is that in some rare cases some fields in the source index can have invalid values, for example a network on a IP field, the document is indexed on the source index because the template for the misp datastream has index.mapping.ignore_malformed set to true, but in the destination index of the transform index.mapping.ignore_malformed is set to false, so a document with a mapping conflict that ignored the field on the source index will be rejected on the destination index, and this creates a permanent error on the transform.

For example, if you have the field threat.indicator.ip with the value 178.21.14.0/23 in the source index, when the transform hits this document it will fail and stop working because this will be rejected in the destination index.

You will then have a message like the following one in the transforms page.

Failed to index documents into destination index due to permanent error: [org.elasticsearch.xpack.transform.transforms.BulkIndexingException: Bulk index experienced [1] failures and at least 1 irrecoverable [org.elasticsearch.xpack.transform.transforms.TransformException: Destination index mappings are incompatible with the transform configuration.; org.elasticsearch.index.mapper.DocumentParsingException: [1:1396] failed to parse field [threat.indicator.ip] of type [ip] in document with id 'dF8zCer1ngjD1STWly66UgJMNQAAAAAA'. Preview of field's value: '178.21.14.0/23'; java.lang.IllegalArgumentException: '178.21.14.0/23' is not an IP string literal.].; org.elasticsearch.xpack.transform.transforms.TransformException: Destination index mappings are incompatible with the transform configuration.; org.elasticsearch.index.mapper.DocumentParsingException: [1:1396] failed to parse field [threat.indicator.ip] of type [ip] in document with id 'dF8zCer1ngjD1STWly66UgJMNQAAAAAA'. Preview of field's value: '178.21.14.0/23'; java.lang.IllegalArgumentException: '178.21.14.0/23' is not an IP string literal.]

To make the transform working again you need to fix the source index by deleting the conflicting document or directly updating it on the backing indice of the data stream.

The following query will list all documents that have ignored fields:

GET logs-ti_misp.threat_attributes-*/_search
{
    "query": {
        "exists": {
            "field": "_ignored"
        }
    }
}

I believe that the fix is to change index.mapping.ignore_malformed to also be true in the destination index, but I'm not sure from where the mappings for this index are coming from.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[jamiehynds]

Thanks for reporting @leandrojmp. Mind taking a look at this @kcreddy?

[kcreddy]

The package-spec only allows few fields to be set under index.mapping: https://github.com/elastic/package-spec/blob/main/spec/integration/data_stream/manifest.spec.yml#L211-L221 . Hence index.mapping.ignore_malformed cannot be changed.
The dynamic_templates configuration however allows setting this ignore_malformed: https://github.com/elastic/package-spec/blob/main/spec/integration/data_stream/manifest.spec.yml#L347-L348

But adding MISP destination index dynamic templates configuration, with threat.indicator.ip having ignore_malformed: true like this is still causing the transform to fail with same error:

      - _embedded_ecs-threat_indicator_ip:
          mapping:
            ignore_malformed: true
            type: ip
          path_match: threat.indicator.ip

I also tried adding the threat.indicator.ip field manually into fields.yml with ignore_malformed: true, but that didn't work either and still leading to transform failing.

---

To make the transform working again you need to fix the source index by deleting the conflicting document or directly updating it on the backing indice of the data stream.

This is definitely one of the workaround. The other workaround I can think of is defining mappings under source datastream for the field threat.indicator.ip with ignore_malformed: false to not ingest documents containing malformed IPs.
Something like this: https://github.com/mrodm/package-spec/blob/main/test/packages/good_input/manifest.yml#L94-L100

I also created elastic/package-spec#730 for adding property index.mapping.ignore_malformed for package definition so that it can be modified.

[simonbjorzen-ts]

Also seeing this, any progress?

I tried the workaround with setting ignore_malformed: false for the threat.indicator.ip mapping in the component template and it appears to be working.

[kcreddy]

In the issue elastic/package-spec#730, we concluded that instead of changing/setting up the property index.mapping.ignore_malformed at integration level, it should be changed inside the Fleet for all transform's destination indices and thus tracking it here: elastic/kibana#179445.