I'm writing a python service (pyamf) by which a person can access images. All images are saved on the central server. The python services is going to be running on satellite machines that have network use of server. The service should work the following:

  1. check in your area to ascertain if the file is available, if that's the case, utilize it.
  2. check in your area to ascertain if file is presently being moved from server ( file.part is available and dimensions are altering ). If that's the case, watch for download to complete, then use file.
  3. if file doesn't exist and file isn't being downloaded, download the file via urlretrieve.

The issue is with Apache's multiple threads. Threads are reaching the file presence check simultaneously and for that reason all of them think the file must be downloaded. Obviously, this isn't good.

What's the proper way to handle this race condition?


I am speculating its whether threaded or perhaps a forked apache, however the effect will be the same being that they are being able to access an online resource.

This issue may also be known as the "dog pile" problem and it is among the issues addressed through the Beaker caching library (http://beaker.groovie.org)And supplies a system bywhich you may create a callable that "produces" a brand new cached value, within this situation a URL akin to some image that's fetched, if your value does not already exist. Securing can be used so that concurrent threads or processes wait for a single process chosen because the "creator" to complete what its doing. Beaker uses lockfiles if set up on the unix-like multi-process oriented system or mutexes if on the home windows system.

I am the initial author of Beaker's guts together with Ben Bangert who packed up for usage with Pylons.