[Filestream] migrate state from previous ID to current #42624

belimawr · 2025-02-07T00:02:02Z

Proposed commit message

This commit enables Filestream inputs to migrate file states from previous IDs to its current ID, this is done by adding a previous_ids entry to the input configuration. We look in the store for all states that match an active file with one of the previous IDs and migrate this state to the new ID. The migrated old states are marked for removal from the store. States are only migrated if they have the same file identity.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

Test changing file identity and ID at the same time.
Investigate any possible side effects of migrating ID + file identity at the same time.
Ensure the integration tests passes on Linux, Mac and Windows)

How to test this PR locally

Run the tests (on all platforms: Linux, Mac and Windows)

cd filebeat
mage buildSystemTestBinary
go test -tags=integration -run=TestFilestreamIDMigration -v -count=1 ./tests/integration

Manual test

Create a file with more than 1kb in size: docker run -it --rm mingrammer/flog -f rfc5424 -n 15 > /tmp/flog.log

Run Filebeat with the following configuration:

filebeat.yml

filebeat.inputs:
  - type: filestream
    id: "first-id"
    paths:
      - /tmp/flog.log

queue.mem:
  flush.timeout: 0s

output.file:
  path: ${path.home}
  filename: "output-file"
  rotate_on_startup: false

filebeat.registry:
  cleanup_interval: 5s
  flush: 1s

logging:
  level: debug
  selectors:
    - input
    - input.filestream
    - input.filestream.prospector
  metrics:
    enabled: false

Wait until the file is fully ingested (15 lines in the output file)
Stop Filebeat
Optionally, remove the logs, this will make the next steps easier. rm -rf logs

Run Filebeat with the following configuration (note the change in id and the new previous_ids):

filebeat.yml

filebeat.inputs:
  - type: filestream
    id: "second"
    paths:
      - /tmp/flog.log
    previous_ids:
      - "first-id"

queue.mem:
  flush.timeout: 0s

output.file:
  path: ${path.home}
  filename: "output-file"
  rotate_on_startup: false

filebeat.registry:
  cleanup_interval: 5s
  flush: 1s

logging:
  level: debug
  selectors:
    - input
    - input.filestream
    - input.filestream.prospector
  metrics:
    enabled: false

Wait until Filebeat "read" the files to the end. Look for End of file reached: /tmp/flog.log; Backoff now. in the second execution logs.
Ensure no new data was added to the output file

You can also look for log entries like:

Migrating input ID: 'filestream::first-id::fingerprint::6fb3cb6c565bdba1354f64a42dd47ef937964019400dd571f25c2cd13a9fb5be' -> 'filestream::second-id::fingerprint::6fb3cb6c565bdba1354f64a42dd47ef937964019400dd571f25c2cd13a9fb5be'
migrated entry in registry from 'filestream::first-id::fingerprint::6fb3cb6c565bdba1354f64a42dd47ef937964019400dd571f25c2cd13a9fb5be' to 'filestream::second-id::fingerprint::6fb3cb6c565bdba1354f64a42dd47ef937964019400dd571f25c2cd13a9fb5be'. Cursor: map[offset:309]

Related issues

Closes Allow changing filestream IDs without duplication by providing the previous ID values #42472

~~## Use cases~~
~~## Screenshots~~

Logs

{
  "log.level": "info",
  "@timestamp": "2025-02-07T15:12:34.224-0500",
  "log.logger": "input.filestream.prospector",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileProspector).Init.func4",
    "file.name": "filestream/prospector.go",
    "file.line": 214
  },
  "message": "Migrating input ID: 'filestream::first-id::fingerprint::6fb3cb6c565bdba1354f64a42dd47ef937964019400dd571f25c2cd13a9fb5be' -> 'filestream::second-id::fingerprint::6fb3cb6c565bdba1354f64a42dd47ef937964019400dd571f25c2cd13a9fb5be'",
  "service.name": "filebeat",
  "filestream_id": "second-id",
  "ecs.version": "1.6.0"
}
{
  "log.level": "info",
  "@timestamp": "2025-02-07T15:12:34.224-0500",
  "log.logger": "input",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*sourceStore).updateIdentifiers",
    "file.name": "input-logfile/store.go",
    "file.line": 273
  },
  "message": "migrated entry in registry from 'filestream::first-id::fingerprint::6fb3cb6c565bdba1354f64a42dd47ef937964019400dd571f25c2cd13a9fb5be' to 'filestream::second-id::fingerprint::6fb3cb6c565bdba1354f64a42dd47ef937964019400dd571f25c2cd13a9fb5be'. Cursor: map[offset:309]",
  "service.name": "filebeat",
  "input_type": "filestream",
  "ecs.version": "1.6.0"
}
{
  "log.level": "info",
  "@timestamp": "2025-02-07T15:12:34.224-0500",
  "log.logger": "input.filestream.prospector",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileProspector).Init.func4",
    "file.name": "filestream/prospector.go",
    "file.line": 214
  },
  "message": "Migrating input ID: 'filestream::first-id::fingerprint::db8399294e69089070405b13d4f057672f3852fa8e0f56ce4b6c92398aef1b6a' -> 'filestream::second-id::fingerprint::db8399294e69089070405b13d4f057672f3852fa8e0f56ce4b6c92398aef1b6a'",
  "service.name": "filebeat",
  "filestream_id": "second-id",
  "ecs.version": "1.6.0"
}
{
  "log.level": "info",
  "@timestamp": "2025-02-07T15:12:34.224-0500",
  "log.logger": "input",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*sourceStore).updateIdentifiers",
    "file.name": "input-logfile/store.go",
    "file.line": 273
  },
  "message": "migrated entry in registry from 'filestream::first-id::fingerprint::db8399294e69089070405b13d4f057672f3852fa8e0f56ce4b6c92398aef1b6a' to 'filestream::second-id::fingerprint::db8399294e69089070405b13d4f057672f3852fa8e0f56ce4b6c92398aef1b6a'. Cursor: map[offset:296]",
  "service.name": "filebeat",
  "input_type": "filestream",
  "ecs.version": "1.6.0"
}

This commit enables Filestream inputs to migrate file states from previous IDs to its current ID, this is done by adding a `previous_ids` entry to the input configuration. We look in the store for all states that match an active file with one of the previous IDs and migrate this state to the new ID. The migrated old states are marked for removal from the store.

elasticmachine · 2025-02-10T20:18:11Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

filebeat/tests/integration/filestream_test.go

…-from-other-filestream-id

belimawr · 2025-02-20T22:15:30Z

I've handled the accidental file identity migration on cc980a2. Now if the file identity is different, then no entry will be migrated from the registry.

pierrehilbert · 2025-02-21T07:35:17Z

@belimawr your new test is failing:

=== FAIL: filebeat/tests/integration TestFilestreamIDMigrationDoesNotMigrateFileIdentity (2.74s)
--
  | filestream_test.go:656:

belimawr · 2025-02-21T16:47:06Z

@belimawr your new test is failing:

=== FAIL: filebeat/tests/integration TestFilestreamIDMigrationDoesNotMigrateFileIdentity (2.74s)
--
  | filestream_test.go:656:

Thanks!

I accidentally made the test rely on a hardcoded inode 🤦‍♂️

I'm gonna put it back to draft until I sort out all the tests. I also have some other things failing on Windows 😞

…-from-other-filestream-id

AndersonQ · 2025-02-25T13:02:31Z

CHANGELOG.next.asciidoc

@@ -424,6 +424,7 @@ otherwise no tag is added. {issue}42208[42208] {pull}42403[42403]
 - The journald input is now generally available. {pull}42107[42107]
 - Add metrics for number of events and pages published by HTTPJSON input. {issue}42340[42340] {pull}42442[42442]
 - Filestram input now supports migrating state when changing its ID, for that set `previous_ids`. {issue}42472[42472] {pull}42624[42624]
+- Add `etw` input fallback to attach an already existing session. {pull}42847[42847]


Is it supposed to be here? It seems unrelated to this PR

That came when I merged main onto my branch, I'll fix it. Thanks for spotting that!

Looking at the whole PR diff, the only new entry on the changelog is the one related to this PR. Maybe you reviewed a subset of changes since the last review?

AndersonQ · 2025-02-25T13:11:51Z

libbeat/tests/integration/framework.go

+				if runtime.GOOS == "windows" {
+					t.Logf("[WARN] Could not remove temporatry directory '%s': %s", tempDir, err)
+				} else {
+					t.Errorf("could not remove temp dir '%s': %s", tempDir, err)
+				}


Is it happening that often? I'm wondering if it'd be worth retrying within a given timeout and/or perhaps getting a callback that should try to kill the beat once again. For any test that does not want/need/can try to stop the beat again, the callback can be a noop. That way this function does not depend on the BeatProc but still can attempt to stop it and retry the clean up.
It could receive the BeatProc if it wouldn't compromise other tests

Is it happening that often?

Often enough. While testing this PR on Windows, every time I added an runtime.GOOS == "windows" to fix a failure, another would pop up. They all together make the test fail all the time.

I'm a bit confused on what would be the function of the callback you're suggesting. One of the ideas of this integration test framework is to remove as much work from the engineer writing the test as possible, the clean up part being one of the key automation. So it feels that adding a callback for the clean up would be adding burden on the engineer writing tests.

belimawr · 2025-02-26T17:34:16Z

Moving it back to draft while we discuss: #36777

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 7, 2025

belimawr self-assigned this Feb 7, 2025

belimawr added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team backport-skip Skip notification from the automated backport with mergify and removed needs_team Indicates that the issue/PR needs a Team:* label labels Feb 7, 2025

belimawr added 7 commits February 7, 2025 10:00

Update mocks/tests

959a260

Update interface

ae14864

Add integration test

0a662f4

Add changelog and docs

d6b42db

update docs

21c3b5e

Add docs and non-functional changes

032b9c3

Update generated files

da7921d

belimawr marked this pull request as ready for review February 10, 2025 20:18

belimawr requested a review from a team as a code owner February 10, 2025 20:18

belimawr requested review from AndersonQ and rdner February 10, 2025 20:18

AndersonQ approved these changes Feb 11, 2025

View reviewed changes

filebeat/tests/integration/filestream_test.go Show resolved Hide resolved

filebeat/tests/integration/filestream_test.go Outdated Show resolved Hide resolved

belimawr added 4 commits February 19, 2025 18:09

Merge branch 'main' of github.com:elastic/beats into 42472-copy-state…

7f5dd77

…-from-other-filestream-id

Add missing 'waitForEOF'

e6f7d8c

Do not migrate registry if identity has changed

cc980a2

Merge branch 'main' of github.com:elastic/beats into 42472-copy-state…

4f491a7

…-from-other-filestream-id

belimawr requested a review from AndersonQ February 20, 2025 22:15

Update file identity to path to increase test compatibility

5158721

belimawr marked this pull request as draft February 21, 2025 16:48

run integration tests on Windows

9ec9a07

belimawr added 4 commits February 21, 2025 18:30

Update framework and tests to work on Windows

db45b4c

fix buildkite pipeline

965ce0b

Update test name and remove Windows integration test

f746740

Merge branch 'main' of github.com:elastic/beats into 42472-copy-state…

4a20bbf

…-from-other-filestream-id

belimawr marked this pull request as ready for review February 24, 2025 17:21

rdner mentioned this pull request Feb 25, 2025

Extend "take over" mode in filestream to cover the changing ID use-case #42884

Open

AndersonQ reviewed Feb 25, 2025

View reviewed changes

belimawr requested a review from AndersonQ February 25, 2025 16:25

belimawr added the windows-11 Enable builds in the CI for windows-10 label Feb 26, 2025

belimawr marked this pull request as draft February 26, 2025 17:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filestream] migrate state from previous ID to current #42624

[Filestream] migrate state from previous ID to current #42624

belimawr commented Feb 7, 2025 •

edited

Loading

elasticmachine commented Feb 10, 2025

belimawr commented Feb 20, 2025

pierrehilbert commented Feb 21, 2025

belimawr commented Feb 21, 2025

AndersonQ Feb 25, 2025

belimawr Feb 25, 2025

belimawr Feb 25, 2025

AndersonQ Feb 25, 2025

belimawr Feb 25, 2025

belimawr commented Feb 26, 2025

[Filestream] migrate state from previous ID to current #42624

Are you sure you want to change the base?

[Filestream] migrate state from previous ID to current #42624

Conversation

belimawr commented Feb 7, 2025 • edited Loading

Proposed commit message

Checklist

Disruptive User Impact

Author's Checklist

How to test this PR locally

Run the tests (on all platforms: Linux, Mac and Windows)

Manual test

Related issues

Logs

elasticmachine commented Feb 10, 2025

belimawr commented Feb 20, 2025

pierrehilbert commented Feb 21, 2025

belimawr commented Feb 21, 2025

AndersonQ Feb 25, 2025

Choose a reason for hiding this comment

belimawr Feb 25, 2025

Choose a reason for hiding this comment

belimawr Feb 25, 2025

Choose a reason for hiding this comment

AndersonQ Feb 25, 2025

Choose a reason for hiding this comment

belimawr Feb 25, 2025

Choose a reason for hiding this comment

belimawr commented Feb 26, 2025

belimawr commented Feb 7, 2025 •

edited

Loading