Monday, July 09, 2007

Apache MaxClients reached, all connections in CLOSE_WAIT

Our Extreme Lab web server (Solaris running Apache 1.3) has recently developed a problem. Twice yesterday it got into a state where it had hit its MaxClients limit (of 128, apparently) and then was unable to service any further requests. Running netstat -f inet showed the all existing connections were in the CLOSE_WAIT stage. I can't tell at this stage if there is some denial of service attack going on or just a problem with the server preventing it from finally closing these connections.

Update: Googling around, and with some help from our local Unix guru, Rob Henderson, I found out that if you have connections stuck in the CLOSE_WAIT stage, this usually indicates that the server side is having trouble closing the connection. Rob says that usually when he sees this problem it is because of an NFS server being down. You get requests for something on that NFS mount, and the process hangs there for a long time. With that NFS server remounted, things are much better. Whereas before I was seeing steadily increasing numbers of CLOSE_WAIT connections, I now see none.

1 comment:

brunis said...

Hey, i'm having that exact problem! But i don't have any NFS running, just apache and mysql.