My server was doing all right up to yesterday. It had been running Redmine (a ruby development monitoring application), also it was the most joyful little server until my "friend" imported a sql table that my little guy could not take. Regrettably after an hour or so of looking to get the lil guy to reply, we needed to energy cycle him.

Now after restart, we obtain a 503 error when attempting to go to the domain attached to Redmine. It's connected to some mongrel daemon, and that we use Apache Proxy to direct all connections towards the port Redmine is running on.

Using Lynx around the server (http://localhost:8000) you can observe the happy ruby application working fine. But this bit isn't employed in my apache conf:

<VirtualHost *:80>
    ServerName sub.example.com
    ProxyPass / http://localhost:8000
    ProxyPassReverse / http://localhost:8000
    ProxyPreserveHost on
    LogLevel debug
</VirtualHost>

Here's the mistake log output for Apache:


[debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //localhost:8000

[debug] proxy_util.c(1335): [client 216.27.137.51] proxy: http: found worker http://localhost:8000 for http://localhost:8000/

[debug] mod_proxy.c(756): Running plan http handler (attempt )

[debug] mod_proxy_http.c(1687): proxy: HTTP: serving URL http://localhost:8000/

[debug] proxy_util.c(1755): proxy: HTTP: has acquired connection for (localhost)

[debug] proxy_util.c(1815): proxy: hooking up http://localhost:8000/ to localhost:8000

[debug] proxy_util.c(1908): proxy: connected / to localhost:8000

[debug] proxy_util.c(2002): proxy: HTTP: fam 2 socket produced for connecting to localhost

[error] (13)Permission refused: proxy: HTTP: make an effort to connect with 127...1:8000 (localhost) unsuccessful

[error] ap_proxy_connect_after sales crippling worker for (localhost)

[debug] proxy_util.c(1773): proxy: HTTP: has launched connection for (localhost)

To be able to get this work, I desired to show of SELinux, after which add trailing slashes towards the "localhost:8000" lines.

Here's the code to show off SELinux:

echo 0 >/selinux/enforce

Are you certain they are restarting within the correct order? I have had strange issues where Apache begins, then Mongrel begins and although Mongrel is running, Apache still throws the proxy error.

I have solved this previously with assorted incantations and restarts of Apache and finally the gods are pleased. It appears that sometimes the Mongrel processes don't correctly shut lower so you've to by hand kill them. Here is a link with a few [possible] help.

I wound up adding a "kill" choice to my /etc/init.d/ mongrel script since it happened a lot. It stop Mongrel, wiped out all Mongrel periods, began Mongrel and restarted Apache.

<snip>
    kill)
      echo "Stopping, killing, starting, and restarting Apache..."
      mongrel_cluster_ctl stop -c $CONF_DIR --clean
      killall -u mongrel
      mongrel_cluster_ctl start -c $CONF_DIR --clean
      /etc/init.d/httpd restart
      RETVAL=$?
  ;;
</snip>

Most likely not an excellent solution however the evil disappeared.

Try running monit to watch your mongrels behind Apache, which way it may restart mongrels for you personally when they die or get too hungry for memory. If unconditionally Apache still will get confused you might have to beautifully restart apache also it should resolve itself, however for 99% of cases getting monit keep an eye on your mongrels should avoid this happening again. Another choice is consider Phusion Passenger.

Run following command

# /usr/sbin/setsebool httpd_can_network_connect 1

OR

# /usr/sbin/setsebool httpd_can_network_connect true

and then restart httpd

# service httpd restart