aws-sdk-core > 3.78.0 slows down the fetching of credentials

[grdw]

Issue description

Ever since moving from aws-sdk-core version 3.78.0 to 3.79.0 the way credentials are fetched have changed from:

curl -H "User-Agent aws-sdk-ruby3/3.78.0" "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
curl -H "User-Agent aws-sdk-ruby3/3.78.0" "http://169.254.169.254/latest/meta-data/iam/security-credentials/<response-from-first-get>"

to:

curl -v -H "User-Agent aws-sdk-ruby3/3.79.0" -H "x-aws-ec2-metadata-token-ttl-seconds 21600" -X PUT http://169.254.169.254/latest/api/token

I guess that has it's reasons, but this PUT request returns a 400 at my end, which in turn causes it to retry 5 times (?). After that unsuccessful chain of events it falls back to the old way of fetching credentials. This is of course painfully slow. Is there some special magic required to not make it return a 400?

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-core 3.79.0 and up

Version of Ruby, OS environment

Ruby 2.6.0
Kubernetes/Docker

Code snippets / steps to reproduce

See curl examples

[mullermp]

duplicate of #2174

[mullermp]

@grdw How are you invoking that curl request? Is that directly from the SDK? If so that's broken behavior. For the header, you're missing a : between the key and value. On an EC2 host, the following command works:

curl -v -H "User-Agent aws-sdk-ruby3/3.79.0" -H "x-aws-ec2-metadata-token-ttl-seconds:21600" -X PUT http://169.254.169.254/latest/api/token

Regarding the issue, could you try increasing the hop limit? It fails on 1 hop by design, but if you're running kubernetes, the container will need to forward your request, adding an additional hop. See the SDK GO issue as well as the other Ruby SDK issue (#2174).

aws ec2 modify-instance-metadata-options --instance-id <instance ID> --http-put-response-hop-limit 3 --http-endpoint enabled --region <region>

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

[mullermp]

Hop limits should be increased for EKS/ECS users automatically, but that appears not to be the case. I'm working with other SDK language teams and forwarding that feedback to relevant EC2/ECS teams to see what the appropriate fix should be.

[btalbot]

I'd like to point out again that the hop limit is just one of several anti-proxy protections added to the PUT token request. Tuning the number of TCP hops is meant to accommodate layer-3 proxies (like NAT) but does nothing to enable support for HTTP (layer 7) proxies.

In our case, we're running Kubernetes clusters on EC2 (which we have had deployed long before AWS offered any k8s support at all so please don't say we should have just used EKS in 2016 instead). For such clusters the jtblin/kube2iam#241 proxy is a very popular way to manage roles apps may use. However, the kube2iam runs as an HTTP proxy which adds the X-Forwarded-For header. These are rejected if the IMDSv2 PUT token call is made and currently any use of a current AWS SDK prevents the app from working (without long delays). The only work-around is to fall back to older SDK versions.

[grdw]

curl -v -H "User-Agent aws-sdk-ruby3/3.79.0" -H "x-aws-ec2-metadata-token-ttl-seconds:21600" -X PUT http://169.254.169.254/latest/api/token

@mullermp if I make the change to the curl-request as suggested, by adding a : I see that the repsonse goes from a 400 Bad request to a 403 Forbidden.

[mullermp]

@grdw Are you running that from within the container or on the ec2 host? What type of EC2 host is it?