I'm going through server load issues on the public facing site. After analysis it seems the 'Google Web Preview' bot is consuming the great majority from the server assets between your hrs of 16:00-23:00 GMT. This really is now affecting the website ease of access for my customers throughout peak time.
I'll be re-architecting the website and searching at beefing in the hardware, but meanwhile I wish to have the ability to throttle demands from certain search bots.
The best choice presently seems to lie within the Apache Retry-After Header which may be used to defer demands with a other (off-peak) time. I wish to have the ability to apply this on the daily moving basis, using the directive to trap all demands from all of these bots between your hrs of 15:00-23:00 and ask for that matching user-agents retry after 23:00. What's the easiest way of approaching this or perhaps is there another cleaner solution?