we've 3 webservers (apache + PHP) on three different machines, speaking to some database machine. Our application enables customers to upload their very own images. The submitted images are saved and planned to some local drive around the machines. Allow me to first admit that people never designed our application to become distributed. In tha above scenario, what we should see would be that the images are submitted and saved on a single server and therefore are unavailable towards the others.

What's the current condition from the art for upload content - could it be something similar to S3 or conveying personal files system over NFS (security issues ?), or perhaps is it something similar to a picture server which enables you to achieve that ?

Take a look at a couple of of numerous solutions:

  1. Storing these questions Database
  2. Separate server, with Varnish/Squid in-front. (in-memory cache) you should use rsync as well as other tool to help keep this current.
  3. A CDN (although, i'd mix it with #2)

Combine!

This might be overkill for you personally.. but this is the way Facebook get it done: http://www.facebook.com/note.php?note_id=76191543919

We simply map shared drives on our SAN. CDN's will also be a possible option. You simply need shared disk space somewhere. Whether that's within the 'cloud', or on the SAN, or perhaps a simple shared network drive somewhere.. it does not matter.