I'm appearing this searching for practical advice regarding how to design something.
Sites like amazon . com.com and the planet pandora have and keep huge data sets to operate their core business. For instance, amazon . com (and each other major e-commerce site) has countless items available, images of individuals items, prices, specifications, etc. etc. etc.
Disregarding the information arriving from third party retailers and also the user produced content everything "stuff" needed to originate from somewhere and it is maintained by someone. It is also incredibly detailed and accurate. How? How can they are doing it? Can there be just an military of information-entry clerks or they have devised systems to handle hard work?
My opportunity is within an identical situation. We conserve a huge (10-of-countless records) catalog of automotive parts and also the cars they can fit. We have been in internet marketing for some time now and also have develop numerous programs and procedures to help keep our catalog growing and accurate however, it appears prefer to grow the catalog to x products we have to grow they to y.
I have to figure some methods to improve the efficiency from the data team and hopefully I'm able to gain knowledge from the work of others. Any suggestions are appreciated, more though could be links to content I possibly could take the serious time reading through.
Use site visitors.
Even when you've one individual per item, you will see wrong records, and clients will think it is. So, allow them to mark products as "inappropiate" making a short comment. Bear in mind, they are not the employees, don't request them as well much see Facebook's "like" button, it's not hard to use, as well as little energy in the user. Good performance/cost. If there will be a mandatory area in Facebook, which asks "so why do you want it?", nobody should use that function.
Site visitors likewise helps you implicite way: they visit item pages, and employ search function (I am talking about both internal internet search engine and exterior ones, like Google). You will gain information from visitors' activity, say, setup an order of the very visited products, then you definitely should concentrate more human forces on top of their email list, and fewer for that "lengthy tail".
As this is much more about controlling theyOrsignalOrinformation instead of implementation and also, since you pointed out Amazon . com I think you will find this helpful: http://highscalability.com/amazon . com-architecture.
Particularly, follow the link to Werner Vogels interview.
Construct it right to begin with. Make sure that you use every integrity checking method obtainable in the database you are using, as appropriate as to the you are storing. Better that the upload fail than bad data get quietly introduced.
Then, evaluate which you are likely to do when it comes to your personal integrity checking. DB integrity inspections make the perfect start, but rarely are all that's necessary. Which will also pressure you to definitely think, right from the start, about which kind of data you are dealing with, how you have to store it, and just how to identify and flag or reject bad or questionable data.
I can not let you know the quantity of discomfort I have seen from attempting to rework (or simply daily use) old systems filled with garbage data. Doing the work right and testing it completely in advance may appear just like a discomfort, also it can be, however the reward is getting something that typically hums along and requires virtually no intervention.
For a hyperlink, if there's anybody who's needed to consider and design for scalability, it's Google. You will probably find this instructive, it's good quality items to bear in mind: http://highscalability.com/google-architecture