I'm wondering how do i problem a HTTP Status code, which will cause spiders to test later for your content, because the system presently is running on high load.

Does a 304 status with no content display resolves this problem?

You need to problem a 503 (Service Not available) using the Retry-Later option set. This appears to become a precise solution.

See: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html and http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

A 304 HTTP Status code means Not Modified.

In case your content is not modified because the before the crawler found its page (that's presuming the crawler send a HTTP header to point if this came), I guess it's OK to transmit a 304 HTTP status -- actually, it might be the best factor to complete.

However, you should most likely not send a Not Modified status code if :

  • You can't determine once the crawler came during the last time -- and, so, you can't pick which version of the content continues to be last indexed
  • Your articles continues to be modified

Rather, in case your webserver is overloaded, the best HTTP status code could be 503 Service Unavailable (quoting) :

The server is presently not available (since it is overloaded or lower for maintenance).
Generally, this can be a temporary condition.

Well, should you answer having a 304 for an unconditional request, a customer will probably just sort the way to go as "generic error" (like 500). Why not send the status code meant for this type of situation, 503 Service Unavailable?.