Ops Manager Security Group for Private blocks access to AWS

[voor]

https://github.com/pivotal-cf/terraforming-aws/blob/master/modules/ops_manager/security_group.tf#L29

When the ops manager is deployed privately, this egress rule makes it so that the Ops Manager cannot reach AWS itself, causing all of the calls to fail.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[voor]

This is resolved by https://github.com/pivotal-cf/terraforming-aws/pull/86 unless you want to figure out how to only allow traffic to AWS endpoints.

[voor]

An alternative that was proposed was to use an internal Interface VPC Endpoint to access the EC2 API:
https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html

[voor]

https://www.terraform.io/docs/providers/aws/r/vpc_endpoint.html

[voor]

As another potential alternative you can get all of the public AWS IP addresses with the following:

curl -sSL https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '[.prefixes[] | select(.service=="AMAZON").ip_prefix] - [.prefixes[] | select(.service=="EC2").ip_prefix] | .[]'

Source: https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html

[voor]

Had some time today so took a first crack at creating the VPCE using terraform. This works perfectly, and the routes for elasticloadbalancing.${region}.amazonaws.com and ec2.${region}.amazonaws.com properly resolve to internal IP Addresses. Except the second you hit them, they just redirect you to aws.amazon.com which is still resolving as a public IP Address inaccessible to Ops Manager. 👎

voor@f1623cb

[voor]

There are some new options that might help this work now:

https://bosh.io/jobs/aws_cpi?source=github.com/cloudfoundry/bosh-aws-cpi-release&version=73#p%3daws.region
https://bosh.io/jobs/aws_cpi?source=github.com/cloudfoundry/bosh-aws-cpi-release&version=73#p%3daws.elb_endpoint
https://bosh.io/jobs/aws_cpi?source=github.com/cloudfoundry/bosh-aws-cpi-release&version=73#p%3daws.ec2_endpoint

If I get another opportunity, I'll take a crack at it.

[voor]

We encountered this issue elsewhere, and the solution was to turn off all the verifiers:

curl -k https://$om_ip/api/v0/staged/director/verifiers/install_time -H "Authorization: Bearer $token" | jq .

echo

verifiers=(
    IaasConfigurationVerifier
    AvailabilityZonesVerifier
    NetworksExistenceVerifier
    NetworksPingableVerifier
    PrivilegeVerifier
    NtpVerifier
    MysqlDatabaseVerifier
    AllocatedIpAddressVerifier
    BlobstoreVerifier
)

for v in ${verifiers[*]}; do
    curl -k https://$om_ip/api/v0/staged/director/verifiers/install_time/$v \
         -X PUT \
         -H "Authorization: Bearer $token" \
         -H "Content-Type: application/json" \
         -d '{"enabled":false}'
done

Might be worth checking out again, also related to https://www.pivotaltracker.com/n/projects/803319/stories/162745406