[Filebeat] Load JSON template 405 Method Not Allowed

[rschirin]

Hey there,
I am trying to import a custom template using a json file from Filebeat.
Below you can find the section related to this operation and also the faced error.

Elasticsearch: 8.2.2
Filebeat: 8.2.2
OS: Container

Step to reproduce:
filebeat.yml configuration file

# # #======TEMPLATE
setup.template.enabled: true
setup.template.name: "logtmp"
setup.template.pattern: "logtmp*"
setup.template.json.enabled: true
setup.template.json.path: "/usr/share/filebeat/template.json"

error message:

{"log.level":"error","@timestamp":"2022-06-10T10:21:18.519Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://es-3:9200)): Connection marked as failed because the onConnect callback failed: error loading template: failure while checking if template exists: 405 Method Not Allowed: ","service.name":"filebeat","ecs.version":"1.6.0"}

If I try just with setup.template.enabled:true , the standard Filebeat template will be loaded properly. If I try only with setup.template.json.enabled:true nothing will happen.
Taking a look to ES Forum, I found also this discussion without any reply

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

Hey @rschirin, you're missing the setup.template.json.name setting. By default it's an empty string, this makes the request to be routed to the wrong place, hence the method not allowed error.

Try like this:

setup.template.name: "filebeat-test"
setup.template.pattern: "filebeat-test*"
setup.template.json.enabled: true
setup.template.json.path: "set-foo.json"
setup.template.json.name: "foo-filebeat"

And for set-foo.json this file:

{
  "index_patterns": ["foo-filebeat*"],
  "template": {
    "settings": {
      "number_of_shards": 1
    },
    "mappings": {
      "_source": {
        "enabled": true
      },
      "properties": {
        "host_name": {
          "type": "keyword"
        },
        "created_at": {
          "type": "date",
          "format": "EEE MMM dd HH:mm:ss Z yyyy"
        }
      }
    },
    "aliases": {
      "mydata": { }
    }
  },
  "priority": 500,
  "version": 3,
  "_meta": {
    "description": "my custom"
  }
}

Then run Filebeat with debug logs:

./filebeat setup --index-management -v -e -d "*" 

I'll close the issue for now. If it does not work for you, please post here the debug logs and re-open the issue.

[casesolved-co-uk]

@belimawr I don't suppose you could advise on this same problem for me?

Version 7.17.23

I just have nginx, system and mysql modules enabled plus a custom json log

I believe auto ILM sorts out the index template?

filebeat.config:
  inputs:
    enabled: true
    path: ${path.config}/inputs.d/*.yml
    reload:
      enabled: true
      period: 10s
  modules:
    enabled: true
    path: ${path.config}/modules.d/*.yml
    reload:
      enabled: true
      period: 10s

# ================================== General ===================================

name: "hostname-blah.co.uk"

# ================================== Outputs ===================================

output.elasticsearch:
  enabled: true
  hosts: ["https://hostname-blah.co.uk:443"]

  compression_level: 0
  protocol: "https"

  username: "namehere"
  password: "pwhere"

  path: "/elasticsearch"

  ssl.enabled: true
  ssl.verification_mode: strict
  ssl.supported_protocols: [TLSv1.3]

# ================================== Logging ===================================

logging.to_files: true
logging.files:
  path: /var/log/filebeat

root@ip-10-1-2-101:/etc/filebeat# filebeat setup --index-management -v -e "*"
2024-11-17T14:13:31.654Z	INFO	instance/beat.go:698	Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2024-11-17T14:13:31.654Z	INFO	instance/beat.go:706	Beat ID: e4a9c6b5-fa42-4367-97d6-
2024-11-17T14:13:31.655Z	INFO	[beat]	instance/beat.go:1052	Beat info	{"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "e4a9c6b5-fa42-4367-97d6-"}}}
2024-11-17T14:13:31.655Z	INFO	[beat]	instance/beat.go:1061	Build info	{"system_info": {"build": {"commit": "bd0ffc1af3bb6c6b0b61e4324f8d9e905b60b45a", "libbeat": "7.17.23", "time": "2024-07-25T13:43:03.000Z", "version": "7.17.23"}}}
2024-11-17T14:13:31.655Z	INFO	[beat]	instance/beat.go:1064	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.21.12"}}}
2024-11-17T14:13:31.656Z	INFO	[beat]	instance/beat.go:1070	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2024-11-16T22:46:48Z","containerized":false,"name":"ip-10-1-2-101","ip":["127.0.0.1","::1","10.1.2.101","fe80::93:17ff:fecf:8903","172.17.0.1","fe80::42:e9ff:fe34:4c96","fe80::d447:10ff:fe79:f83b","fe80::147f:91ff:fee2:bf96","fe80::648a:3eff:fe04:5abd","fe80::30f2:10ff:fea5:9c66"],"kernel_version":"5.15.0-1072-aws","mac":["02:93:17:cf:89:03","02:42:e9:34:4c:96","d6:47:10:79:f8:3b","16:7f:91:e2:bf:96","66:8a:3e:04:5a:bd","32:f2:10:a5:9c:66"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.6 LTS (Focal Fossa)","major":20,"minor":4,"patch":6,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0,"id":"ec239e9adac8694fcbf1432"}}}
2024-11-17T14:13:31.656Z	INFO	[beat]	instance/beat.go:1099	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 178406, "ppid": 176470, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2024-11-17T14:13:30.720Z"}}}
2024-11-17T14:13:31.657Z	INFO	instance/beat.go:292	Setup Beat: filebeat; Version: 7.17.23
2024-11-17T14:13:31.657Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'filebeat-7.17.23' as ILM is enabled.
2024-11-17T14:13:31.657Z	WARN	[cfgwarn]	tlscommon/config.go:100	DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2024-11-17T14:13:31.657Z	INFO	[esclientleg]	eslegclient/connection.go:105	elasticsearch url: https://hostname-blah.co.uk:443/elasticsearch
2024-11-17T14:13:31.658Z	INFO	[publisher]	pipeline/module.go:113	Beat name: hostname-blah.co.uk
2024-11-17T14:13:31.660Z	INFO	[esclientleg]	eslegclient/connection.go:105	elasticsearch url: https://hostname-blah.co.uk:443/elasticsearch
2024-11-17T14:13:31.692Z	INFO	[esclientleg]	eslegclient/connection.go:285	Attempting to connect to Elasticsearch version 7.17.23
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

2024-11-17T14:13:31.731Z	INFO	[index-management]	idxmgmt/std.go:260	Auto ILM enable success.
2024-11-17T14:13:31.749Z	INFO	[index-management.ilm]	ilm/std.go:170	ILM policy filebeat exists already.
2024-11-17T14:13:31.749Z	INFO	[index-management]	idxmgmt/std.go:396	Set setup.template.name to '{filebeat-7.17.23 {now/d}-000001}' as ILM is enabled.
2024-11-17T14:13:31.749Z	INFO	[index-management]	idxmgmt/std.go:401	Set setup.template.pattern to 'filebeat-7.17.23-*' as ILM is enabled.
2024-11-17T14:13:31.749Z	INFO	[index-management]	idxmgmt/std.go:435	Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.17.23 {now/d}-000001} as ILM is enabled.
2024-11-17T14:13:31.749Z	INFO	[index-management]	idxmgmt/std.go:439	Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2024-11-17T14:13:31.783Z	INFO	template/load.go:197	Existing template will be overwritten, as overwrite is enabled.
2024-11-17T14:13:34.481Z	INFO	template/load.go:131	Try loading template filebeat-7.17.23 to Elasticsearch
2024-11-17T14:13:34.576Z	INFO	template/load.go:123	Template with name "filebeat-7.17.23" loaded.
2024-11-17T14:13:34.576Z	INFO	[index-management]	idxmgmt/std.go:296	Loaded index template.
2024-11-17T14:13:34.636Z	ERROR	[index-management.ilm]	ilm/std.go:133	Index Alias filebeat-7.17.23 setup failed: failed to create alias: {"error":"Incorrect HTTP method for uri [/<filebeat-7.17.23-{now/d}-000001>] and method [PUT], allowed: [POST]","status":405}: 405 Method Not Allowed: {"error":"Incorrect HTTP method for uri [/<filebeat-7.17.23-{now/d}-000001>] and method [PUT], allowed: [POST]","status":405}.
2024-11-17T14:13:34.636Z	ERROR	instance/beat.go:1027	Exiting: failed to create alias: {"error":"Incorrect HTTP method for uri [/<filebeat-7.17.23-{now/d}-000001>] and method [PUT], allowed: [POST]","status":405}: 405 Method Not Allowed: {"error":"Incorrect HTTP method for uri [/<filebeat-7.17.23-{now/d}-000001>] and method [PUT], allowed: [POST]","status":405}
Exiting: failed to create alias: {"error":"Incorrect HTTP method for uri [/<filebeat-7.17.23-{now/d}-000001>] and method [PUT], allowed: [POST]","status":405}: 405 Method Not Allowed: {"error":"Incorrect HTTP method for uri [/<filebeat-7.17.23-{now/d}-000001>] and method [PUT], allowed: [POST]","status":405}

EDIT:
It looks like the uri used is invalid:

  329 35.604452380    127.0.0.1 → 127.0.0.1    HTTP 764 PUT /<filebeat-7.17.23-{now/d}-000001> HTTP/1.0  (application/json)
  331 35.606396402    127.0.0.1 → 127.0.0.1    HTTP 341 HTTP/1.0 405 Method Not Allowed  (application/json)
  340 35.611672411    127.0.0.1 → 127.0.0.1    HTTP 630 GET /_alias/filebeat-7.17.23 HTTP/1.0 
  342 35.613715974    127.0.0.1 → 127.0.0.1    HTTP 568 HTTP/1.0 404 Not Found  (application/json)

[belimawr]

Hi @casesolved-co-uk,

I've tried to reproduce it with Elasticsearch 8.16.0 and 7.17.23 and in both cases I got a success, I copy+pasted your configuration, the only thing I adjusted was the credentials and SSL settings to match my deployment.

Which version of ES are you using?
Have you edited the modules files in any way?

I noticed that you have output.elasticsearch.path: "/elasticsearch" set, is Filebeat going through a proxy to reach Elasticsearch? Is there any change the proxy is blocking the requests?

Where did you get the request logs you posted at the end from?

I also noticed a typo in your CLI regarding enabling debug logs, you need -d "*", but you forgot the -d, effectively having log level info.

Could you run it again with the following CLI?

filebeat setup --index-management -v -e -d "*"

This will enable debug logs and log the full URL that Filebeat is calling, on my tests I got:

2024-11-20T17:33:23.298-0500    DEBUG   [esclientleg]   eslegclient/connection.go:340   HEAD https://es.elastic-cloud.com:443/_index_template/filebeat-7.17.23  <nil>

2024-11-20T17:33:23.410-0500    DEBUG   [esclientleg]   eslegclient/connection.go:340   PUT https://es.elastic-cloud.com:443/_index_template/filebeat-7.17.23 ...

2024-11-20T17:33:25.002-0500    DEBUG   [esclientleg]   eslegclient/connection.go:340   GET https://es.elastic-cloud.com:443/_alias/filebeat-7.17.23  <nil>

[casesolved-co-uk]

@belimawr
Thanks for looking into it
I logged a bug here: #41686
This is a build issue affecting the official 7.x debian builds (at least)