I have lately began experimentation with using Python for web design. To date I have had some success using Apache with mod_wsgi and also the Django web framework for Python 2.7. However have encounter some difficulties with getting processes constantly running, upgrading information and the like.

I've written a script I call "daemonManager.py" that may stop and start any individual python update loops (Must I give them a call Daemons?). It will that by forking, then loading the module for that specific functions it will run and beginning an infinite loop. It saves a PID file in /var/run to keep an eye on the procedure. To date so great. The issues I have experienced are:

  • Occasionally among the processes will just quit. I check ps each morning and the operation is just gone. No errors were drenched (I am while using logging module), and I am covering every exception I'm able to think about and logging them. Also I do not think these giving up processes has anything related to my code, because my processes run different code and exit at pretty similar times. I possibly could be wrong obviously. Could it be normal for Python ways to just die after they have run for several daysOrdays? How must i tackle this issue? Must I write another daemon that periodically inspections when the other daemons continue to be running? Let's say that daemon stops? I am baffled regarding how to handle this.

  • How do i programmatically determine if a procedure continues to be running or otherwise? I am saving the PID files in /var/run and checking when the PID file can there be to find out set up process is running. But when the procedure just dies of unpredicted causes, the PID file will stay. I therefore need to remove these files whenever a process crashes (a few occasions each week), which kind of defeats the reason. I suppose I possibly could see if a procedure is running in the PID within the file, but let's say another process has began and was designated the PID from the dead process? My daemon would believe that the operation is running fine even when it's lengthy dead. Again I am baffled just how to approach this.

Any helpful answer regarding how to best run infinite Python processes, hopefully also losing some light around the above problems, I'll accept

I am using Apache 2.2.14 with an Ubuntu machine.
My Python version is 2.7.2

I'll open by proclaiming that this really is one method to run a lengthy running process (LRP) -- not p facto by any stretch.

In my opinion, the perfect product originates from focusing around the specific problem you are coping with, while assigning supporting tech with other libraries. Within this situation, I am mentioning to the action of backgrounding processes (the skill of the double fork), monitoring, and log redirection.

My personal favorite option would be http://supervisord.org/

Utilizing a system like supervisord, you essentially write a regular python script that works an activity while stuck within an "infinite" loop.


import sys
import time

def main_loop():
    while 1:
        # do your stuff...

if __name__ == '__main__':
    except KeyboardInterrupt:
        print >> sys.stderr, '\nExiting by user request.\n'

Writing your script by doing this causes it to be easy and easy to develop and debug (it is simple to start/stop it inside a terminal, watching the log output as occasions unfold). When the time comes to throw into production, you just define a supervisor config that calls your script (here's the full example for determining a "program", a lot of that is optional: http://supervisord.org/configuration.html#program-x-section-example).

Supervisor has a lot of configuration options and so i will not enumerate them, however i will state that it particularly solves the issues you describe:

  • Backgrounding/Daemonizing
  • PID monitoring (could be set up to restart a procedure should it terminate suddenly)
  • Log normally inside your script (stream handler if using logging module instead of printing) but let supervisor redirect to some apply for you.