I've got a highly trafficked application on a single debian machine and apache has began acting strange.

Each time I start apache, a lot of apache processes are created, the application does not load whatsoever, and incredibly rapidly the entire machine freezes and should be powercycled to reboot.

Here's what I recieve to find the best soon after beginning apache:


top -   20:14:44    up         1:16,      2 customers,    load average: .48, .10, .03

Tasks:  330 total,  5 running, 325 sleeping,    stopped,    zombie

Cpu(s): 12.%us,    21.4%sy,   .%ni,        65.7%id,   .2%wa,  .1%hi,  .7%si,  .%st

Mem:    8179920k    total,     404984k used,  7774936k free,    60716k buffers

Swap:   2097136k    total,     0k used,       2097136k free,    43424k cached

10251 www-data  15     467m 8100 4016 S    6  .1   :00.04 apache2

10262 www-data  15     467m 8092 4012 S    6  .1   :00.05 apache2

10360 www-data  15     468m 8296 4016 S    6  .1   :00.05 apache2

10428 www-data  15     468m 8272 3992 S    6  .1   :00.05 apache2

10241 www-data  15     467m 8256 4012 S    4  .1   :00.03 apache2

10259 www-data  15     467m 8092 4012 S    4  .1   :00.04 apache2

10274 www-data  15     467m 8056 4012 S    4  .1   :00.03 apache2

10291 www-data  15     468m 8292 4012 S    4  .1   :00.03 apache2

10293 www-data  15     468m 8292 4012 S    4  .1   :00.03 apache2

10308 www-data  15     468m 8296 4016 S    4  .1   :00.02 apache2

10317 www-data  15     468m 8292 4012 S    4  .1   :00.02 apache2

10320 www-data  15     468m 8292 4012 S    4  .1   :00.04 apache2

10325 www-data  15     468m 8292 4012 S    4  .1   :00.04 apache2

And so on.. with increased apache2 processes.

Under one minute later, you can observe below the load went from .48 to two.17. If I don't stop apache at this time, the burden keeps rising on the couple of minutes or less before the machine dies.


top -    20:15:34 up 1:17,       2 customers,  load average: 2.17, .62, .21

Tasks:   1850 total,  5 running, 1845 sleeping,    stopped,    zombie

Cpu(s):  .3%us,      2.1%sy,    .%ni, 96.4%id,  .%wa,  .1%hi,  1.%si,  .%st

Mem:     8179920k     total,     1938524k used,  6241396k free,    60860k buffers

Swap:    2097136k     total,     0k used,  2097136k free,    44196k cached

There exists a firewall where we whitelist the addresses we all know are permitted hitting our website.

Any ideas by what the issue may be are extremely welcome.

Thanks!

Perhaps you have transformed your configuration file lately? If so, I trust you retain that old version for diffing?

Otherwise, look for the "StartServers", "MaxSpareServers" and "MinSpareServers" directives. Generally you need to leave these at defaults, but it is possible that they are deliberately set high (bad idea) or accidentally set this way because of a poor config edit.

If the does not help, you're ready to look outdoors Apache, for many process that's opening connections quickly (might be that you have a testing process that's run amok).

Initial step may be the access log. Next step would be to run netstat, to determine in which the connections may be originating from. And when it's running on a single system, you are able to try looking in /proc/*/fd to obtain the two finishes from the connection.