[v1.33.1] Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device 'emulator-5554' not found/device offline #1525

DavidREntwistle · 2023-10-13T08:46:03Z

Describe the bug
Whilst running our android test folder containing 70~ flows taking around 40mins on CI, the below Exception in thread "pool-4-thread-1" occurs. This then causes an execution timeout, stopping the build on agent.

  Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device 'emulator-5554' not found
    at dadb.adbserver.AdbServer.send$dadb(AdbServer.kt:99)
    at dadb.adbserver.AdbServerDadb.open(AdbServer.kt:137)
    at dadb.forwarding.TcpForwarder.handleForwarding$lambda-1(TcpForwarder.kt:64)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)

To Reproduce

Running the emulator headless on CI
Run a test folder containing a large number of flows
The Flow fails around the 40 min mark because of the Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device 'emulator-5554' not found

Expected behaviour
The adb server to continue running after lengthy execution times.

Environment information:

Maestro version [v1.33.1]
Platform: [Android 13]
Framework: [Native Android]
Emulator - headless
Host [AWS linux-x86_64]

The text was updated successfully, but these errors were encountered:

DavidREntwistle · 2023-11-17T11:27:01Z

Update:

The error now appears to display as:

[Passed] test_flow29 (14s)
[Failed] test_flow30 (1m 29s) (Android driver unreachable)
[Failed] test_flow31 (41s) (Android driver unreachable)
Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device offline
   at dadb.adbserver.AdbServer.send$dadb(AdbServer.kt:99)
   at dadb.adbserver.AdbServerDadb.open(AdbServer.kt:137)
   at dadb.forwarding.TcpForwarder.handleForwarding$lambda-1(TcpForwarder.kt:64)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   at java.base/java.lang.Thread.run(Thread.java:833)
[Failed] test_flow32 (1m 20s) (Android driver unreachable)
[Failed] test_flow33 (1m 20s) (Android driver unreachable)

It appears that now in v1.34.0 after this issue occurs, the flows continue to run with the "Android driver unreachable" message as the reason (this I feel is better).

I can't think of any reason why the device would go offline mid-execution.

Possibly the related to the issue what I've raised here #1570

sanyar123 · 2024-01-04T11:58:25Z

Hi @DavidREntwistle, I am also facing the same issue. Did you find any workaround for this issue?

DavidREntwistle · 2024-01-04T12:08:55Z

Hi @DavidREntwistle, I am also facing the same issue. Did you find any workaround for this issue?

@sanyar123 Sadly not yet.

I did add some logging and handling at the start of the flows to check if an ANR/Crash occurred in my test app on the previous flow. But this doesn't seem to be the cause of the issue.

DavidREntwistle · 2024-01-12T13:22:16Z

I have been able to simulate this issue consistently, here are some steps to reproduce:

Run a flows folder containing any valid flow (for example purposes).
Allow the 1st flow to pass (for example purposes).
Run adb reconnect device whilst the second flow is running.
Observe the below failure in terminal:

Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device 'emulator-5554' not found
        at dadb.adbserver.AdbServer.send$dadb(AdbServer.kt:99)                                
        at dadb.adbserver.AdbServerDadb.open(AdbServer.kt:137)                                
        at dadb.forwarding.TcpForwarder.handleForwarding$lambda-1(TcpForwarder.kt:64)         
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

Observe the 2nd flow fail after 2 minutes with Android driver unreachable.
Run adb reconnect device after the second flow fails with Android driver unreachable.
Observe then the 3rd & 4th flow each fail after 2 minutes with Android driver unreachable.
Observe then the 5th flow run and pass with no error.

See the error in the mastro test log:

[ERROR] maestro.cli.runner.TestRunner - Failed to run flow
io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
	at maestro_android.MaestroDriverGrpc$MaestroDriverBlockingStub.deviceInfo(MaestroDriverGrpc.java:634)
	at maestro.drivers.AndroidDriver$deviceInfo$1.invoke(AndroidDriver.kt:170)
	at maestro.drivers.AndroidDriver$deviceInfo$1.invoke(AndroidDriver.kt:169)
	at maestro.drivers.AndroidDriver.runDeviceCall(AndroidDriver.kt:989)
	at maestro.drivers.AndroidDriver.deviceInfo(AndroidDriver.kt:169)
	at maestro.ViewHierarchy$Companion.from-8JJjmZI(ViewHierarchy.kt:28)
	at maestro.Maestro.viewHierarchy-c1iYVAs(Maestro.kt:416)
	at maestro.Maestro.viewHierarchy-c1iYVAs$default(Maestro.kt:415)
	at maestro.Maestro$findElementWithTimeout$element$1.invoke(Maestro.kt:445)
	at maestro.Maestro$findElementWithTimeout$element$1.invoke(Maestro.kt:444)
	at maestro.utils.MaestroTimer.withTimeout(MaestroTimer.kt:16)
	at maestro.Maestro.findElementWithTimeout-AE3g2Tc(Maestro.kt:444)
	at maestro.Maestro.findElementWithTimeout-AE3g2Tc$default(Maestro.kt:438)
	at maestro.orchestra.Orchestra.findElement(Orchestra.kt:863)
	at maestro.orchestra.Orchestra.access$findElement(Orchestra.kt:44)
	at maestro.orchestra.Orchestra$evaluateCondition$3$result$1.invoke(Orchestra.kt:521)
	at maestro.orchestra.Orchestra$evaluateCondition$3$result$1.invoke(Orchestra.kt:519)
	at maestro.utils.MaestroTimer.withTimeout(MaestroTimer.kt:16)
	at maestro.orchestra.Orchestra.evaluateCondition(Orchestra.kt:519)
	at maestro.orchestra.Orchestra.evaluateCondition$default(Orchestra.kt:496)
	at maestro.orchestra.Orchestra.runFlowCommand(Orchestra.kt:489)
	at maestro.orchestra.Orchestra.executeCommand(Orchestra.kt:268)
	at maestro.orchestra.Orchestra.executeCommands(Orchestra.kt:201)
	at maestro.orchestra.Orchestra.runFlow(Orchestra.kt:103)
	at maestro.orchestra.Orchestra.runFlow$default(Orchestra.kt:75)
	at maestro.cli.runner.MaestroCommandRunner.runCommands(MaestroCommandRunner.kt:185)
	at maestro.cli.runner.TestRunner$runSingle$result$1.invoke(TestRunner.kt:54)
	at maestro.cli.runner.TestRunner$runSingle$result$1.invoke(TestRunner.kt:51)
	at maestro.cli.runner.TestRunner.runCatching(TestRunner.kt:145)
	at maestro.cli.runner.TestRunner.runSingle(TestRunner.kt:51)
	at maestro.cli.command.TestCommand$call$1.invoke(TestCommand.kt:177)
	at maestro.cli.command.TestCommand$call$1.invoke(TestCommand.kt:136)
	at maestro.cli.session.MaestroSessionManager.newSession(MaestroSessionManager.kt:101)
	at maestro.cli.session.MaestroSessionManager.newSession$default(MaestroSessionManager.kt:58)
	at maestro.cli.command.TestCommand.call(TestCommand.kt:136)
	at maestro.cli.command.TestCommand.call(TestCommand.kt:46)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
	at picocli.CommandLine.access$1200(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)
	at maestro.cli.DisableAnsiMixin$Companion.executionStrategy(DisableAnsiMixin.kt:22)
	at picocli.CommandLine.execute(CommandLine.java:2058)
	at maestro.cli.AppKt.main(App.kt:117)

DavidREntwistle · 2024-01-12T14:32:06Z

@sanyar123 are you able to reproduce this issue with these steps above?

sanyar123 · 2024-01-12T18:29:12Z

@DavidREntwistle I haven't replicated the same steps but I am getting the same error that you have shared

simon-gilmurray · 2024-01-16T11:41:16Z

@axelniklasson Also hitting this issue on v1.35.0

Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device offline
	at dadb.adbserver.AdbServer.send$dadb(AdbServer.kt:99)
	at dadb.adbserver.AdbServerDadb.open(AdbServer.kt:137)
	at dadb.forwarding.TcpForwarder.handleForwarding$lambda-1(TcpForwarder.kt:64)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

yuseiatlas · 2024-01-19T10:55:20Z

A temporary fix that worked for me on #1573 (comment). TL;DR: I set the emulator API level to 32.

DavidREntwistle · 2024-01-29T09:35:20Z

I found the Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device offline was being caused by our CI build timing out because of the length of execution. But I also think emulator crashes could still be a factor.

DavidREntwistle · 2024-07-15T12:22:29Z

As of 15th July 2024, this remains an issue.

DavidREntwistle · 2024-07-15T15:48:49Z

I believe this PR could resolve this issue.

KeunwooPark · 2024-09-22T13:46:37Z

Still seeing this error with Android API 31 on 22nd Sep 2024

gy6543721 · 2024-11-14T08:50:49Z

Issue still exists in 2024/11
#1733 (comment)

DavidREntwistle · 2025-01-21T15:53:18Z

Issue still exists 21st Jan 2025

woelmer · 2025-01-29T18:21:23Z

Has anyone found a viable workaround other than retrying the tests 10 times?

DavidREntwistle · 2025-01-30T12:39:26Z

I've looked into this a lot more, so in advance for the long comment...

What have I tried so far:
Changing the Android System Image API version had varying results, not consistent with one API version causing more of an issue than the next. I feel it's not a specific system image issue.
Increasing the emulator's memory seemed to help, but maybe this was a bit of a red herring and did not actually help too much.

What more have I found out:

This seems to be a long-standing Android ADB server issue - e.g. this from 14 years ago...
This doesn't seem to be a specific app issue.

Steps to reproduce manually:

Kick off the Maestro tests and then run adb detach (ignoring the adb: failed to detach: transport type doesn't support detach message), followed by adb devices displaying the emulator-5554 offline.
This then triggers this same Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device offline error we experience on CI raised in this issue, with the following test marked as (Android driver unreachable).
Run adb kill-server && adb start-server turns the offline device "* daemon not running; starting now at tcp:5037" back online (as expected).
Note: Maestro does seem to initially force close the app/interact again with the re-connected device (Unable to launch app com.yourAppName.android.test: am force-stop com.yourAppName.android.test), closing the app, returning to the Android home screen.
After the above error, each flow fails with (Unable to launch app com.yourAppName.android.test: Android driver unreachable) taking ~6 minutes to fail.

What do I think?
The Maestro connection to the previous server doesn't reconnect to the new one. @felipevolpone Maybe Maestro could try regaining the connection again? Or end throw the exception and stop running all the flows. My preference would be for Maestro to try restarting the adb server and regain connection, continuing the Maestro test.

What I'm trying next:

I'm experimenting with different Android SDK platform tool versions, not a surprise, that different versions have varying differences with the Android system images.
Additionally, I'm going to try including the Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device offline as a build exception in our TeamCity script, failing the build and automatically triggering a new build.

DavidREntwistle added the bug Something isn't working label Oct 13, 2023

This was referenced Oct 13, 2023

[v1.33.0] Exception in thread pool-4-thread-1 with Github actions #1495

Closed

[Maestro version, e.g. v1.30.4] - Exception in thread while running ui test #1304

Open

DavidREntwistle mentioned this issue Nov 6, 2023

[v1.34.0] Android driver unreachable whilst running flow #1570

Open

DavidREntwistle mentioned this issue Nov 17, 2023

[BUG] - Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device offline #1289

Closed

DavidREntwistle mentioned this issue Jan 12, 2024

[v1.34.0][BUG] - Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device offline #1573

Open

DavidREntwistle mentioned this issue Jan 16, 2024

When one of the devices is offline, Exception has been throwed mobile-dev-inc/dadb#62

Open

This was referenced Jan 22, 2024

[v1.35.0] "io.grpc.StatusRuntimeException: UNKNOWN" Failed to take screenshot on android flow failure #1647

Open

Android Driver - io.grpc.StatusRuntimeException: UNAVAILABLE: io exception #998

Open

DavidREntwistle mentioned this issue Jan 30, 2025

Exception in thread "pool-4-thread-1, v1.31.0 #1412

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v1.33.1] Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device 'emulator-5554' not found/device offline #1525

[v1.33.1] Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device 'emulator-5554' not found/device offline #1525

DavidREntwistle commented Oct 13, 2023 •

edited

Loading

DavidREntwistle commented Nov 17, 2023

sanyar123 commented Jan 4, 2024

DavidREntwistle commented Jan 4, 2024 •

edited

Loading

DavidREntwistle commented Jan 12, 2024 •

edited

Loading

DavidREntwistle commented Jan 12, 2024

sanyar123 commented Jan 12, 2024

simon-gilmurray commented Jan 16, 2024 •

edited

Loading

yuseiatlas commented Jan 19, 2024

DavidREntwistle commented Jan 29, 2024

DavidREntwistle commented Jul 15, 2024

DavidREntwistle commented Jul 15, 2024

KeunwooPark commented Sep 22, 2024 •

edited

Loading

gy6543721 commented Nov 14, 2024 •

edited

Loading

DavidREntwistle commented Jan 21, 2025

woelmer commented Jan 29, 2025

DavidREntwistle commented Jan 30, 2025

[v1.33.1] Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device 'emulator-5554' not found/device offline #1525

[v1.33.1] Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device 'emulator-5554' not found/device offline #1525

Comments

DavidREntwistle commented Oct 13, 2023 • edited Loading

DavidREntwistle commented Nov 17, 2023

sanyar123 commented Jan 4, 2024

DavidREntwistle commented Jan 4, 2024 • edited Loading

DavidREntwistle commented Jan 12, 2024 • edited Loading

DavidREntwistle commented Jan 12, 2024

sanyar123 commented Jan 12, 2024

simon-gilmurray commented Jan 16, 2024 • edited Loading

yuseiatlas commented Jan 19, 2024

DavidREntwistle commented Jan 29, 2024

DavidREntwistle commented Jul 15, 2024

DavidREntwistle commented Jul 15, 2024

KeunwooPark commented Sep 22, 2024 • edited Loading

gy6543721 commented Nov 14, 2024 • edited Loading

DavidREntwistle commented Jan 21, 2025

woelmer commented Jan 29, 2025

DavidREntwistle commented Jan 30, 2025

DavidREntwistle commented Oct 13, 2023 •

edited

Loading

DavidREntwistle commented Jan 4, 2024 •

edited

Loading

DavidREntwistle commented Jan 12, 2024 •

edited

Loading

simon-gilmurray commented Jan 16, 2024 •

edited

Loading

KeunwooPark commented Sep 22, 2024 •

edited

Loading

gy6543721 commented Nov 14, 2024 •

edited

Loading