I am pretty a new comer to caching methods and implementations. I am focusing on a project that'll be database intensive, but additionally have information being up-to-date and transformed very regularly.

I have found enough info to understand generally how you can develop the caching function, but what I am unsure about may be the general strategy.

Basically cache all query results and group them by logical things will be able to obvious on triggers which make sense, I'll most likely have hundreds of 1000's (a minimum of) small files during my cache. Wouldn't it be preferable to cache only large query results?

I understand that this can be a somewhat hardware specific question, usually at what amount of files does caching become somewhat pointless? Meaning, if you are loading in the file system wonderful these small files, does use of them eventually be slow enough which you may too have simply not cached the data to begin with?

Thanks all, I am thinking about any opinions you are offering

EDIT: In line with the reactions in regards to this being absolutely application specific, allow me to pose the question by doing this that ought to be universal:

Presuming which i come with an application that is dependent on a single table with 1,000,000 products inside it...

Will it be faster to perform a query to retrieve among individuals products from the database, in order to retrieve among individuals products from my cache directory with 1,000,000 files, each that contains the particulars of 1 of individuals products?

EDIT: Apparently 100,000 wasn't enough to obtain a valid answer, let us allow it to be 1,000,000. Anybody want to choose 1,000,000,000? Because I'm able to get it done...

Use MySQL's built-in query cache rather than attempting to keeping it yourself. It'll instantly obvious cached queries to tables when they're written to. Plus, it really works in memory so it ought to be extremely powerful...

Also, don't just cache queries. Attempt to cache entire segments from the application at different procedures in the rendering cycle. To help you let MySQL cache the queries, then you definitely cache every individual view (made), every individual block, and every page. Then, you are able to choose if you should pull from cache based on the request.

For instance, a non-drenched-in user could get the entire page from cache. But a drenched-in user might not have the ability to (because of username, etc). So for him, you might have the ability to render 1/2 your sights around the page from cache (given that they don't rely on the consumer object). You'll still get the advantage of caching, but it will be tiered based on need.

If you are really expecting lots of traffic, it's certainly worth considering Memcached. Let MySQL store your queries for you personally, after which store all user-land cache products in memcache...

Edit: To reply to your edit:

Filesystems may become slow if your single directory develops large. As lengthy as you are "namespacing" by directory (so each directory has only a little part of cache files), you ought to be fine from that perspective. For the precise threshold, it will rely on your hardware and filesystem a lot more than other things. I understand EXT3 will get quite slow if you will find a lot of files in one directory (I've sites with literally 100s of 1000's of files, also it can require half another to merely stat() among the files, not to mention inflict type of directory listing)...

But understand that should you add another server, you are likely to either have duplication of cache (which isn't a positive thing), or will have to rewrite your whole cache layer. It is possible to reason not to choose Memcached immediately?

EDit 2: To reply to your latest edit:

Will still be too difficult to call. I've a credit card applicatoin which has a database with around 1.5 billion rows (growing around 500k daily). We do not use any caching onto it whatsoever because we do not have concurrency issues. And even when we did, we'd be best tossing more MySQL servers in internet marketing instead of adding caching since any kind of cache might have this type of low hit rate it would not be well worth the development time for you to add it.

And for this reason I'm so adamant about not caching for speed. There'll always be an item that's not in cache. If you hit a webpage and among individuals objects, still it must be fast. Usually of thumb, I attempt to cache anything that'll be utilized again within the next couple of minutes (I have a time for you to live around a few minutes in production on other programs anyway). Therefore if products aren't getting greater than a couple of hits for the reason that time period, or even the hit minute rates are really low (under 90%), I do not bother caching that item....