This is a timeless subject and one I'm pretty passionate about. There are many reasons you might want to offload your file storage from the local file system and onto Amazon S3.
If you've never performance tested a website before stop what you are doing, if you have, do the same. I'm pretty sure you're doing it wrong and wasting everyone's time.
To performance test a Drupal website correctly you need a number of things:
There are a couple of facts you need to understand.