HalfAtATime deployment with an ELB/ASG

[mattejames]

We currently use these scripts during deployment to deregister our instances from an ASG during deployment. However, we've discovered what appears to be a race condition when using HalfAtATime deployment in CodeDeploy and using more than 2 EC2 instances in the ASG.

For example, if you have 4 EC2 instances behind an ASG and run a HalfAtATime deployment on them from CodeDeploy, it will run the deployment scripts against 2 of the 4 at the same time. From there, one of them will instantly fail citing:

[stderr]An error occurred (ValidationError) when calling the EnterStandby operation: AutoScalingGroup [DeploymentGroup] has min-size=3, max-size=70, and desired-size=3. To place into standby 1 instances, please update the AutoScalingGroup sizes appropriately.

I believe this is because both EC2 instances are running autoscaling_enter_standby() at the same time. So they both scale the ASG min and desired down to 3 at the same time and then attempt to move into standby, however only one gets in. When the other makes the attempt, it fails because by that time there are 3 in the ASG and you can't take another one away.

Is there any way to adjust the script to account for this issue? Without that, HalfAtATime deployment is pretty useless given that you're limited to just 2 instances in the ASG at a time.

[michaelwittig]

I just researched this issue and found the following statement in the README

WARNING: If you are using this functionality you should only use CodeDeployDefault.OneAtATime deployment configuration to ensure a serial execution of the scripts. Concurrent runs are not supported.

[mattejames]

@michaelwittig Ha! OK, that explains it. Go figure. Thanks for finding that!

I would love to see an update to the code there, but obviously this probably isn't the highest item on the AWS team's list. :)