I'm using Delphi 2009. I've got a quite simple data structure, with 2 fields:

  1. A string that's the important thing area I have to retrieve by and it is usually 4 to fifteen figures long.
  2. A string that's the information area that might be any size, from 1 character as much as say 10,000 figures.

The problem is the fact that I might have into the millions of those records, so that they may total at least as much than 10 GB in dimensions. Clearly, I am searching for an on-disk solution instead of an in-memory solution.

My program must at random retrieve this info, in line with the key area. That's that part that should be made as efficient as you possibly can.

Must I make use of a database for this type of simple structure, and when so, which database might be best additional and become easiest to implement?

Alternatively, it is possible to simple on-disk data structure not needing a complete-blown database that will work equally well?

Well, all I desired was the main one response to kick me back to reality. I had been searching for something simpler than a simple database. However when no-duh answer is by using a database, i quickly realize I have already clarified this with my very own response to another question: Best database for small applications and tools.

My answer was DISQLite3 for the reasons I specified there. And that is what I'll most likely with for my implementation.

A couple of more good solutions with a few options. That's great. I'll have the ability to consider using a couple of different techniques to determine the things that work best.

More contemplation, and that i have needed to alter the recognized response to the GpStructuredStorage solution.

During my situation, millions of records adding up to several Gb will put stress on the database structure. Particularly, the B* tree that's accustomed to keep index in many databases is fast, and can decelerate for many procedures for example reindexing millions of values.

The only real factor you will find faster than B* to have an index is really a hash table. Which is exactly what is provided in gabr's recommended accessory for the GpStructuredStorage solution. I believe the correct answer is elegant the way in which he segmented the hash value to provide a 4 level directory structure.

Exactly why I can turn to a hash option would be which i just have random access through the secrets. I don't need to sort through the secrets. If sorting was needed, then your gains from the speed from the hash table could be lost and also the database system will be a no-brain champion.

After I get lower to applying this, I ought to perform a comparison of the technique versus a database. Maybe I'll match up against both Firebird and SQLite which may both be worthy competitors.

Yet another followup:

I simply discovered Synopse Big Table by A. Bouchez which is made for speed and meets the specs of my question almost precisely. I'm going to be giving it a go out first after i do my implementation inside a couple of several weeks and can report on their behavior here with my results.

In excess of 10GB in data, a database is what you'll need. It'll handle indexing for quickly finding data (your random retrieval), the functionality for adding, modifying, and removing data, and also the actual storage, in addition to a lot more should you so choose.

You will find a large number of posts here associated with which databases are for sale to use within Delphi, including built-inches and FOS ones like Firebird.