[Flaky Test]: TestFleetAirGappedUpgradeUnprivileged – Agent status is not online #6968
Assignees
Labels
Comments
pkoutsovasilis
added
flaky-test
|
Pinging @elastic/elastic-agent (Team:Elastic-Agent) |
|
another occurrence of it here |
|
I am running this using the integration testing suite from my machine with:
I will let it run until I can reproduce a failure. This will leave the cloud running and the Elastic Agent running so I can debug better in what state Fleet is in, as the Elastic Agent is clearly not online or even shown in the list. |
|
Ran for a total of 78 times. Didn't fail during any of those tests. To me this implies the issue is not with this test specifically when running in isolation. It is more of an issue when the whole test suite is running. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Failing test case
TestFleetAirGappedUpgradeUnprivileged
Error message
Agent status is not online
Build
https://buildkite.com/elastic/elastic-agent-extended-testing-bk/builds/2018#01952793-9424-4cda-90dc-1097d18dd49b/336-1012
OS
Linux
Stacktrace and notes
=== Failed === FAIL: testing/integration TestFleetAirGappedUpgradeUnprivileged (795.22s) fetcher_artifact.go:213: Downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz fetcher_artifact.go:292: Downloading artifact progress 47.21% fetcher_artifact.go:292: Downloading artifact progress 94.46% fetcher_artifact.go:292: Downloading artifact progress 100.00% fetcher_artifact.go:259: Completed downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz fetcher_artifact.go:213: Downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512 fetcher_artifact.go:259: Completed downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512 fetcher_artifact.go:213: Downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc fetcher_artifact.go:259: Completed downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc fixture.go:290: Extracting artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestFleetAirGappedUpgradeUnprivileged4025483309 fixture.go:308: Completed extraction of artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestFleetAirGappedUpgradeUnprivileged4025483309 fixture.go:908: Components were not modified from the fetched artifact fetcher_artifact.go:213: Downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz fetcher_artifact.go:292: Downloading artifact progress 94.90% fetcher_artifact.go:292: Downloading artifact progress 100.00% fetcher_artifact.go:259: Completed downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz fetcher_artifact.go:213: Downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512 fetcher_artifact.go:259: Completed downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512 fetcher_artifact.go:213: Downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc fetcher_artifact.go:259: Completed downloading artifact from https://snapshots.elastic.co/9.0.0-3e08d0a8/downloads/beats/elastic-agent/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc ArtifactsServer root dir "/tmp/TestFleetAirGappedUpgradeUnprivileged2312044965/001", served files ["elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz" "elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc" "elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz.sha512"] upgrade_fleet_test.go:495: found [34.120.127.130 2600:1901:0:1d7::] IPs for "artifacts.elastic.co", blocking them... upgrade_fleet_test.go:514: added iptables rule [OUTPUT -j DROP -d 34.120.127.130] upgrade_fleet_test.go:518: use "iptables -D OUTPUT -j DROP -d 34.120.127.130" to remove it upgrade_fleet_test.go:514: added iptables rule [OUTPUT -j DROP -d 2600:1901:0:1d7::] upgrade_fleet_test.go:518: use "ip6tables -D OUTPUT -j DROP -d 2600:1901:0:1d7::" to remove it fixture.go:290: Extracting artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestFleetAirGappedUpgradeUnprivileged680566699 fixture.go:308: Completed extraction of artifact elastic-agent-9.0.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestFleetAirGappedUpgradeUnprivileged680566699 fixture.go:908: Components were not modified from the fetched artifact upgrade_fleet_test.go:325: Testing Elastic Agent upgrade from 9.0.0-SNAPSHOT to 9.0.0-SNAPSHOT with Fleet... upgrade_fleet_test.go:333: creating download source "local-air-gaped-7d77d981-2ce0-441a-9498-82d25360c82d", using "http://127.0.0.1:39873/downloads/beats/elastic-agent/". fixture.go:665: >> running binary with: [/tmp/TestFleetAirGappedUpgradeUnprivileged680566699/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent version --binary-only --yaml] fixture.go:665: >> running binary with: [/tmp/TestFleetAirGappedUpgradeUnprivileged4025483309/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent version --binary-only --yaml] upgrade_fleet_test.go:375: Creating Agent policy... upgrade_fleet_test.go:380: Creating Agent enrollment API key... upgrade_fleet_test.go:387: Getting default Fleet Server URL... upgrade_fleet_test.go:391: Installing Elastic Agent (unprivileged: true)... fixture_install.go:199: [test TestFleetAirGappedUpgradeUnprivileged] Inside fixture install function fixture_install.go:227: [test TestFleetAirGappedUpgradeUnprivileged] Inside fixture installNoPkgManager function fixture.go:665: >> running binary with: [/tmp/TestFleetAirGappedUpgradeUnprivileged680566699/elastic-agent-9.0.0-SNAPSHOT-linux-x86_64/elastic-agent install --force --non-interactive --unprivileged --url https://e3716922f8594cb543422b381ab510f1.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token OE5XZko1VUJuczJTOHFGbW4zTjY6QUZMeDlxeGpIdDZiemFLeWZWUWUwdw==] upgrade_fleet_test.go:408: Waiting for Agent to be correct version and healthy... fixture.go:665: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json] upgrade_fleet_test.go:412: Waiting for enrolled Agent status to be online... upgrade_fleet_test.go:420: Upgrading from version "9.0.0-SNAPSHOT-02ee98b3525f22a84aaa7911c5374b77e9f6362f" to version "9.0.0-SNAPSHOT-8200a86e0d3c7431b020aefbfe02d7f673254c19"... upgrade_fleet_test.go:426: Waiting from upgrade details to show up in Fleet upgrade_fleet_test.go:439: Waiting for upgrade watcher to start... upgrade_fleet_test.go:442: Upgrade watcher started fixture.go:665: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json] upgrade_fleet_test.go:448: Waiting for enrolled Agent status to be online... check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] check.go:57: unable to determine agent status: unable to find agent with hostname [bk-agent-prod-gcp-1740125969302575864.c.elastic-ci-prod.internal] for policy [096ac28f-cda0-4d38-bbfc-4adafb9ff90f]. Found: [] upgrade_fleet_test.go:449: Error Trace: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1740125969302575864/elastic/elastic-agent-extended-testing-bk/testing/integration/upgrade_fleet_test.go:449 /opt/buildkite-agent/builds/bk-agent-prod-gcp-1740125969302575864/elastic/elastic-agent-extended-testing-bk/testing/integration/upgrade_fleet_test.go:341 /opt/buildkite-agent/builds/bk-agent-prod-gcp-1740125969302575864/elastic/elastic-agent-extended-testing-bk/testing/integration/upgrade_fleet_test.go:120 Error: Condition never satisfied Test: TestFleetAirGappedUpgradeUnprivileged Messages: Agent status is not onlineThe text was updated successfully, but these errors were encountered: