I've a credit card applicatoin that's comprised of the next:

A central database that contains 100k+ records Numerous "client" databases each that contains around 10-20k records

The customer databases contain particulars of contacts that every possess a unique ID (contactID).
The central database consists of a few of these contacts recognized through the same contactID.

Overnight we have to iterate with the client databases querying the central database for updates to every contact then take it in to the client database.
The central database is held by a 3rd party therefore we cannot change anything.
The organization holding the central database do over web services iterating through each contact.

My concerns are this is very slow to complete over web services given the amount of records.

Presently I'm considering producing personal files on each client which consists of a listing of all of the contacts for your client. This could then be delivered to the central database. The central database would then process this file and send back another file that contains all updates.

How does one create this therefore it runs as quick as you possibly can?

I wouldn't undergo each record within the scenario you describe. Web services are fine however, you could as quickly utilize them for bulk updates.

From the top my mind, something similar to this could work just a little better:

  1. Obtain a 'diff' that contains all the new changes within the master database. This may be done prior to the synchronisation begins and just needs to be achieved once daily.
  2. Send a complete listing of contacts in the client towards the server. With a listing that large it doesn't really matter how it's send but when you do not like Cleaning soap web services consider JSON. I'd opt for web-services for convenience. Again, their email list of contacts might be prepared ahead of time.
  3. Create a listing of records which have to be update by evaluating the 'diff'with their email list in the client.
  4. Get all of the up-to-date particulars all at once (if you are using MS SQL you could do this by using XML, other SQL services offer different pathways)
  5. Send the particulars towards the client

Exactly the same mechanism might be employed for upgrading one record only, you need to simply use different list (that contains 1 ID) within the p.2

As you are relatively restricted through the 3rd party, you might like to do among the following:

  1. Possess the 3rd party produce a web service which returns a listing of id's that have particulars transformed within the last day.

  2. After this you iterate through this reduced list utilizing a web service (which hopefully is a lot more compact then your entire list), increase your contacts as appropriate.

or

  1. Possess the 3rd party produce a web service which consists of an xml document that contains a dump from the contacts particulars within their system that have transformed inside a certain period. You would process this document, upgrading your particulars.

Opt for bulk transfers whenever we can.

Opening a TCP/IP connection is an extremely slow process. Moving 20K contact ID's (even when the ID is big) in one WS request is faster than 20K web service demands.

You won't want to wait to allow them to reply. You've 2 kinds of potential workflows.

  1. Busy Waiting. You signal a load. You poll every couple of minutes before the batch is performed. This really is rather icky, but totally one-on the sides. You need to do everything, they simply respond with "dirty yet" or "done". Then you definitely request your batch results.

  2. Notification. You signal a load including some address at which you'll be informed. It may be their email, or it may be a URL (Ip, port and path) where you would like notification.

    • Should you want to use email (or similar) notification after that you can perform a proper WS request to retrieve the batch.

    • Should you want to use WS notification, you've got a small web service that either will get an easy notification, and does a WS request to obtain the result, or they give the entire result.

  3. If you wish to be rude, open hundreds of threads making hundreds of concurrent demands. This can take less passed time, and can swamp their server.

You are able to request for any file (XML) that contains all of the ID particulars in the central DB and you will have it converted directly into some object mappings. Then getting object list at the finish you are able to compare increase accordingly.