Amulet deployment hitting timeout when model is deployed

[javacruft]

We've hit this a few times in OpenStack CI; specifically Amulet raises:

DEBUG:runner:Timeout occurred, printing juju status...environment: osci-sv16

The model is deployed and all units are in the started agent state:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline/openstack/charm-percona-cluster/344162/4/2016-07-21_08-46-40/test_charm_amulet_smoke_1730/juju-stat-yaml-collect.txt

so I'm not quite sure why this happens; full test run details here:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline/openstack/charm-percona-cluster/344162/4/2016-07-21_08-46-40/index.html

[javacruft]

I guess it might be todo with the default wait timeout of 300 seconds; but I can see test runs where the deployer phase takes longer than that still:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline/openstack/charm-percona-cluster/344162/3/2016-07-21_08-45-20/consoleText.test_charm_amulet_smoke_1729.txt

[ryan-beisner]

@javacruft Linking here to track/inspect another possibly impactful timeout location, which isn't configurable or announced:

http://bazaar.launchpad.net/~charm-helpers/charm-helpers/devel/view/head:/charmhelpers/contrib/amulet/deployment.py#L79

Whether that is the cause of this issue or not, it seems like we should listen for and use env var AMULET_SETUP_TIMEOUT if it's set.

[ryan-beisner]

Re: AMULET_SETUP_TIMEOUT
https://github.com/juju/amulet/blob/master/amulet/deployer.py#L642