InstanceProfileCredentialsProvider is slow to get credentials since 1.11.678

[joelittlejohn]

For apps that have any kind of indirect network configuration (e.g. docker, k8s, ECS, etc), the InstanceProfileCredentialsProvider now hangs for 10 seconds before providing credentials. This has been reported in other SDKs:

aws/aws-sdk-ruby#2177
aws/aws-sdk-go#2972

The problem appears to be that the SDK now uses PUT /latest/api/token to create credentials, but this PUT API is subject to a hop limit so the request does not succeed. Instead it hangs waiting for a response. After 10 seconds a timeout exception is thrown and the SDK falls back to the usual GET /latest/meta-data/iam/security-credentials/... which succeeds.

This can be fixed like:

aws ec2 modify-instance-metadata-options --instance-id <instance ID> --http-put-response-hop-limit 3 --http-endpoint enabled

however it seems like a major issue that all instances will now need to be modified in this way.

[KaibaLopez]

Hi @joelittlejohn, thanks for pointing this out, we'll take a look into it too.

[chrisshafer]

We have encountered this issue as well.

[dagnir]

The timeouts for IMDS have been lowered so this should be fixed as of 1.11.700.

[joelittlejohn]

@dagnir That actually seems like it will make this bug harder to see, even though requests are still being delayed. People won't even know they need to change the instance metadata options. There should also be a warning logged when this happens.

I raised this because changing the instance metadata options seems like a bad requirement for the SDK to have. This was not required before 1.11.678.