Skip to content

Latest commit

 

History

History
324 lines (257 loc) · 20.8 KB

CHANGELOG.next.asciidoc

File metadata and controls

324 lines (257 loc) · 20.8 KB

Beats version HEAD

Breaking changes

Affecting all Beats

  • Remove the non-ECS agent.hostname field. Use the agent.name or agent.id fields for an identifier. 16377 18328

  • Remove the deprecated xpack.monitoring. settings. Going forward only monitoring. settings may be used. 9424 18608

  • Remove deprecated/undocumented IncludeCreatorMetadata setting from kubernetes metadata config options 28006

  • Remove deprecated fields from kubernetes module 28046

  • Remove deprecated config option aws_partition. 28120

  • Improve stats API 27963

  • Libbeat: logp package forces ECS compliant logs. Logs are JSON formatted. Options to enable ECS/JSON have been removed. 15544 28573

  • Update docker client. 28716

  • Remove auto from the available options of setup.ilm.enabled and set the default value to true. 28671

  • add_process_metadata processor: Replace usage of deprecated process.ppid field with process.parent.pid. 28620

  • add_docker_metadata processor: Replace usage of deprecated process.ppid field with process.parent.pid. 28620

  • Use data streams instead of indices for storing events from Beats. 28450

  • Remove option setup.template.type and always load composable template with data streams. 28450

  • Remove several ILM options (rollover_alias and pattern) as data streams does not require index aliases. 28450

  • Index template’s default_fields setting is only populated with ECS fields. 28596 28215

  • Remove deprecated --template and --ilm-policy flags. Use --index-management instead. 28870

  • Remove options logging.files.suffix and default to datetime endings. 28927

Auditbeat

  • File integrity dataset (macOS): Replace unnecessary file.origin.raw (type keyword) with file.origin.text (type text). 12423 15630

  • Change event.kind=error to event.kind=event to comply with ECS. 18870 20685

Filebeat

  • Fix parsing of Elasticsearch node name by elasticsearch/slowlog fileset. 14547

  • With the default configuration the following modules will no longer send the host field that contains information about the host on which Filebeat is running. You can revert this change by configuring tags for the module and omitting forwarded from the list. 13920

  • With the default configuration the cloud modules (aws, azure, googlecloud, o365, okta)

  • With the default configuration the cef and panw modules will no longer send the host

  • Add while_pattern type to multiline reader. 19662

Heartbeat

Metricbeat

  • Remove deprecated fields in Docker module. 11835 27933

  • Remove deprecated fields in Kafka module. 27938

  • Remove deprecated config option default_region from aws module. 28120

  • Remove network and diskio metrics from ec2 metricset. 28316

  • Rename read/write_io.ops_per_sec to read/write.iops in rds metricset. 28350

  • Remove linux-only metrics from diskio, memory 28292

  • Remove deprecated config option perfmon.counters from windows/perfmon metricset. 28282

  • Remove deprecated fields in Redis module. 11835 28246

  • system/process metricset: Replace usage of deprecated process.ppid field with process.parent.pid. 28620

Packetbeat

  • Redis: fix incorrectly handle with two-words redis command. 14872 14873

  • event.category no longer contains the value network_traffic because this is not a valid ECS event category value. 20556

  • Remove deprecated TLS fields in favor of tls.server.x509 and tls.client.x509 ECS fields. 28487

  • HTTP: The field http.request.method will maintain its original case. 28620

  • Unify gopacket dependencies. 29167

Winlogbeat

  • Add support to Sysmon file delete events (event ID 23). 18094

  • Improve ECS field mappings in Sysmon module. related.hash, related.ip, and related.user are now populated. 18364

  • Improve ECS field mappings in Sysmon module. Hashes are now also populated to the corresponding process.hash, process.pe.imphash, file.hash, or file.pe.imphash. 18364

  • Improve ECS field mappings in Sysmon module. file.name, file.directory, and file.extension are now populated. 18364

  • Improve ECS field mappings in Sysmon module. rule.name is populated for all events when present. 18364

  • Fix unprefixed fields in fields.yml for Powershell module 18984

  • Remove top level hash property from sysmon events 20653

Functionbeat

Bugfixes

Affecting all Beats

  • Fix a race condition with the Kafka pipeline client, it is possible that Close() get called before Connect() . 11945

  • Allow users to configure only cluster_uuid setting under monitoring namespace. 14338

  • Update replicaset group to apps/v1 15802

  • Fix missing output in dockerlogbeat 15719

  • Fix issue where TLS settings would be ignored when a forward proxy was in use. 15516

  • Update replicaset group to apps/v1 15802

  • Add ssl.ca_sha256 option to the supported TLS option, this allow to check that a specific certificate is used as part of the verified chain. 15717

  • Improve some logging messages for add_kubernetes_metadata processor https://github.com/elastic/beats/pull/16866{16866}

  • Do not rotate log files on startup when interval is configured and rotateonstartup is disabled. 17613

  • Fix setup.dashboards.index setting not working. 17749

  • Fix Elasticsearch license endpoint URL referenced in error message. 17880 18030

  • Change decode_json_fields processor, to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

  • Gives monitoring reporter hosts, if configured, total precedence over corresponding output hosts. 17937 17991

  • Change decode_json_fields processor, to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

  • [Autodiscover] Check if runner is already running before starting again. 18564

  • Fix an issue where error messages are not accurate in mapstriface. 18662 18663

  • Fix regression in add_kubernetes_metadata, so configured indexers and matchers are used if defaults are not disabled. 18481 18818

  • Fix the translate_sid processor’s handling of unconfigured target fields. 18990 18991

  • Fixed a service restart failure under Windows. 18914 18916

  • Fix terminating pod autodiscover issue. 20084

  • Fix seccomp policy for calls to chmod and chown. 20054

  • Output errors when Kibana index pattern setup fails. 20121

  • Fix issue in autodiscover that kept inputs stopped after config updates. 20305

  • Add service resource in k8s cluster role. 20546

  • Allows disable pod events enrichment with deployment name 28521

  • Fix fingerprint processor to give it access to the @timestamp field. 28683

  • Fix the wrong beat name on monitoring and state endpoint 27755

Auditbeat

  • system/package: Fix parsing of Installed-Size field of DEB packages. 16661 17188

  • system module: Fix panic during initialisation when /proc/stat can’t be read. 17569

  • system/package: Fix an error that can occur while trying to persist package metadata. 18536 18887

  • system/socket: Fix bugs leading to wrong process being attributed to flows. 29166 17165

Filebeat

  • cisco/asa fileset: Fix parsing of 302021 message code. 14519

  • Fix filebeat azure dashboards, event category should be Alert. 14668

  • Fix s3 input with cloudtrail fileset reading json file. 16374 16441

  • Add queue_url definition in manifest file for aws module. 16640

  • Add queue_url definition in manifest file for aws module. https://github.com/elastic/beats/pull/16640{16640}

  • Fix elasticsearch.gc fileset to not collect all logs when Elasticsearch is running in Docker. 13164 16583 17164

  • Fixed a mapping exception when ingesting CEF logs that used the spriv or dpriv extensions. 17216 17220

  • Remove migrationVersion map 7.7.0 reference from Kibana dashboard file to fix backward compatibility issues. 17425

  • Fix issue 17734 to retry on rate-limit error in the Filebeat httpjson input. 17734 17735

  • Fixed cloudfoundry.access to have the correct cloudfoundry.app.id contents. 17847

  • Fixing ingress_controller. fields to be of type keyword instead of text. 17834

  • Fixed typo in log message. 17897

  • Fix o365 module ignoring var.api settings. 18948

  • Fix netflow module to support 7 bytepad for IPFIX template. 18098

  • Update container name for the azure filesets. 19899

  • Fix o365 module ignoring var.api settings. 18948

  • Fix S3 input to trim delimiter /n from each log line. 19972

  • Fix s3 input parsing json file without expand_event_list_from_field. 19902 19962 20370

  • Fix millisecond timestamp normalization issues in CrowdStrike module 20035, 20138

  • Fix support for message code 106100 in Cisco ASA and FTD. 19350 20245

  • Fix fortinet setting event.timezone to the system one when no tz field present 20273

  • Fix okta geoip lookup in pipeline for destination.ip 20454

  • Fix mapping exception in the googlecloud/audit dataset pipeline. 18465 20465

  • Fix cisco asa and ftd parsing of messages 106102 and 106103. 20469

  • Resolve issue with @timestamp for defender_atp. 28272

  • Fix threatintel.misp filters configuration. 27970

  • Fix handling of escaped newlines in the decode_cef processor. 16995 29268

  • Fix panw module ingest errors for GLOBALPROTECT logs 29154

  • aws-s3: Stop trying to increase SQS message visibility after ReceiptHandleIsInvalid errors. 29480

  • Fix handling of IPv6 addresses in netflow flow events. 19210 29383

Heartbeat

  • Remove accidentally included cups library in docker images. pull

  • Fix broken monitors with newer versions of image relying on dup3. pull

Metricbeat

  • Fix checking tagsFilter using length in cloudwatch metricset. 14525

  • Log bulk failures from bulk API requests to monitoring cluster. 14303 14356

  • Fix skipping protocol scheme by light modules. pull

  • Revert changes in docker module: add size flag to docker.container. 16600

  • Fix detection and logging of some error cases with light modules. 14706

  • Fix imports after PR was merged before rebase. 16756

  • Reduce memory usage in elasticsearch/index metricset. 16503 16538

  • Fix issue in Jolokia module when mbean contains multiple quoted properties. 17375 17374

  • Fix issue in Jolokia module when mbean contains multiple quoted properties. 17375 17374

  • Fix azure storage dashboards. 17590

  • Metricbeat no longer needs to be started strictly after Logstash for logstash-xpack module to report correct data. 17261 17497

  • Fix pubsub metricset to collect all GA stage metrics from gcp stackdriver. 17154 17600

  • Add privileged option so as mb to access data dir in Openshift. 17606

  • Fix "ID" event generator of Google Cloud module 17160 17608

  • Add privileged option for Auditbeat in Openshift 17637

  • Fix storage metricset to allow config without region/zone. 17623 17624

  • Fix overflow on Prometheus rates when new buckets are added on the go. 17753

  • Remove specific win32 api errors from events in perfmon. 18292 18361

  • Fix application_pool metricset after pdh changes. 18477

  • Fix panic on metricbeat test modules when modules are configured in metricbeat.modules. 18789 18797

  • Fix getting gcp compute instance metadata with partial zone/region in config. 18757

  • Add missing network.sent_packets_count metric into compute metricset in googlecloud module. 18802

  • Fix compute and pubsub dashboard for googlecloud module. 18962 18980

  • Fix crash on vsphere module when Host information is not available. 18996 19078

  • Modify doc for app_insights metricset to contain example of config. 20185

  • Add required option for metrics in app_insights. 20406

  • Groups same timestamp metric values to one event in the app_insights metricset. 20403

  • Use xpack.enabled on SM modules to write into .monitoring indices when using Metricbeat standalone 28365

  • Fix in rename processor to ingest metrics for write.iops to proper field instead of write_iops in rds metricset. 28960

  • Enhance filter check in kubernetes event metricset. 29470

Packetbeat

  • Prevent incorrect use of AMQP protocol parsing from causing silent failure. 29017

  • Fix error handling in MongoDB protocol parsing. 29017

Winlogbeat

  • Add source.ip validation for event ID 4778 in the Security module. 19627

Functionbeat

Elastic Logging Plugin

Added

Affecting all Beats

  • Decouple Debug logging from fail_on_error logic for rename, copy, truncate processors 12451

  • Fingerprint processor adds a new xxhash hashing algorithm 15418

  • Update RPM packages contained in Beat Docker images. 17035

  • Update documentation for system.process.memory fields to include clarification on Windows os’s. 17268

  • When using the decode_json_fields processor, decoded fields are now deep-merged into existing event. 17958

  • Update documentation for system.process.memory fields to include clarification on Windows os’s. 17268

  • Add keystore support for autodiscover static configurations. {pull]16306[16306]

  • When using the decode_json_fields processor, decoded fields are now deep-merged into existing event. 17958

  • Add keystore support for autodiscover static configurations. {pull]16306[16306]

  • Add TLS support to Kerberos authentication in Elasticsearch. 18607

  • Add config option rotate_on_startup to file output 19150 19347

  • Set index.max_docvalue_fields_search in index template to increase value to 200 fields. 20215

  • Upgrade prometheus library. 28716

  • Name all k8s workqueue. 28085

  • Add options to configure k8s client qps/burst. 28151

  • Update to ECS 8.0 fields. 28620

  • Add http.pprof.enabled option to libbeat to allow http/pprof endpoints on the socket that libbeat creates for metrics. 21965

  • Support custom analyzers in fields.yml. 28540 28926

  • SASL/SCRAM in the Kafka output is no longer beta. 29126

  • Discover changes in Kubernetes nodes metadata as soon as they happen. 23139

  • Support self signed certificates on outputs 29229

  • Update k8s library 29394

  • Add FIPS configuration option for all AWS API calls. 28899

  • Add default_region config to AWS common module. 29415

Auditbeat

  • Reference kubernetes manifests include configuration for auditd and enrichment with kubernetes metadata. 17431

Filebeat

  • container and docker inputs now support reading of labels and env vars written by docker JSON file logging driver. 8358

  • Add index option to all inputs to directly set a per-input index value. 14010

  • move create-[module,fileset,fields] to mage and enable in x-pack/filebeat 15836

  • Work on e2e ACK’s for the azure-eventhub input 15671 16215

  • Add a TLS test and more debug output to httpjson input 16315

  • Add an SSL config example in config.yml for filebeat MISP module. 16320

  • Update filebeat httpjson input to support pagination via Header and Okta module. 16354

  • Add a TLS test and more debug output to httpjson input 16315

  • Add an SSL config example in config.yml for filebeat MISP module. 16320

  • Added documentation for running Filebeat in Cloud Foundry. 17275

  • Release Google Cloud module as GA. 17511

  • Improve ECS categorization field mappings for nats module. 16173 17550

  • Enhance elasticsearch/slowlog fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17729

  • Added documentation for running Filebeat in Cloud Foundry. 17275

  • Release Google Cloud module as GA. 17511

  • Update filebeat httpjson input to support pagination via Header and Okta module. 16354

  • Change the json.* input settings implementation to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

  • Add support for array parsing in azure-eventhub input. 18585

  • Add support for array parsing in azure-eventhub input. 18585

  • Improved performance of PANW sample dashboards. 19031 19032

  • Add event.ingested for CrowdStrike module 20138

  • Add support for additional fields and FirewallMatchEvent type events in CrowdStrike module 20138

  • Azure signinlogs - Add support for ManagedIdentitySignInLogs, NonInteractiveUserSignInLogs, and ServicePrincipalSignInLogs. 23653

  • Add text/csv decoder to httpjson input 28564

  • Update aws-s3 input to connect to non AWS S3 buckets 28222 28234

  • Add support for '/var/log/pods/' path for add_kubernetes_metadata processor with resource_type: pod. 28868

  • Add documentation for add_kubernetes_metadata processors log_path matcher. 28868

  • Add support for parsers on journald input 29070

  • Add support in httpjson input for oAuth2ProviderDefault of password grant_type. 29087

  • Update Cisco module to enable TCP input. 26118 28821 26159

Heartbeat

Metricbeat

  • Move the windows pdh implementation from perfmon to a shared location in order for future modules/metricsets to make use of. 15503

  • Add database_account azure metricset. 15758

  • Add database_account azure metricset. 15758

  • Release Zookeeper/connection module as GA. 14281 17043

  • Add dashboard for pubsub metricset in googlecloud module. 17161

  • Added documentation for running Metricbeat in Cloud Foundry. 17275

  • Added documentation for running Metricbeat in Cloud Foundry. 17275

  • Remove required for region/zone and make stackdriver a metricset in googlecloud. 16785 18398

  • Add memory metrics into compute googlecloud. 18802

  • Preliminary AIX support 27954

  • Add option to skip older k8s events 29396

  • Add add_resource_metadata configuration to Kubernetes module. 29133

Packetbeat

Functionbeat

Winlogbeat

  • Add more DNS error codes to the Sysmon module. 15685

  • Add configuration option for registry file flush timeout 29001 29053

  • Add support for custom XML queries 1054 29330

Elastic Log Driver

  • Fixed docs for hosts 23644

Deprecated

Affecting all Beats

Filebeat

Heartbeat

Metricbeat

Packetbeat

Winlogbeat

Functionbeat

Known Issue

Journalbeat