March 19, 2019
We're going to dig in to why AWS has taken a strange stance on the status code
returned for a file that doesn't exist in an S3 bucket. You may have observed
that if you have permissions to read from a bucket, but not to list the
contents of the bucket, then you will get
403 Forbidden instead of
404 Not Found.
If you want to get the proper
404 status for AWS S3, you need to give the user
or role the
It's starts with security.
Enumeration attacks are where the attacker is able to make the target list the resources available. A resource name itself might contain sensitive information. Or the existence of a certain file can indicate what software is running on a server that would provide the attacker with a new attack vector to try.
To prevent enumeration attacks, we don't want the user to know whether the resource doesn't exist, or the resource exists and they do not have permission to view it. The solution is to return the same status code for both cases to prevent leaking information.
If we look at the 403 status code spec we see:
"If the server does not wish to make [why the request has not been fulfilled] available to the client, the status code 404 (Not Found) can be used instead."
And in the 404 status code spec there is nothing suggesting that 403 should be returned instead.
My guess would be that AWS has some legacy reason for always returning
instead of always returning
404 and is choosing to keep backwards
compatibility over spec compliance.
The result of the AWS S3 behaviour is that properly configured S3 buckets can prevent enumeration attacks, which is good.
For your own APIs you should not replicate the AWS S3 behaviour. Instead, always
404 Not Found.
If you want to see more things from Graham McGregor in the future, you should follow him on twitter!