Do you know the quickest search technologies aside from Relational DB Searches ?

I've got a assortment of Text Files from varied sources (Banks/Ledgers/Stock Marketplaces). Each line during these Text Files is really a Record. Each line can further be parsed into some DB Posts(Stock Title/Date of purchase/Owner/...). There is no need that every line has all of the fields. If some text isn't mathcing the fields, I store it because it is inside a seperate column(columnA). When the parser does not find any column whatsoever I simply keep Line in columnA.

I than perform a DB search according to user Inputs. I've found the search becomes slow following a couple of million records. I wish to move to a different storage like File System as well as other Indexed storage.

What is your opinion will be the quickest storage technology with this type of requirement ?

I'm presently using SQLite DB. Since I Have am going to rewrite the whole code, I'll be more than pleased if every other storage technology can provide me faster results.

  1. Apache Lucene is really a high-performance, full-featured text internet search engine library written entirely in Java. It's a technology appropriate for pretty much any application that needs full-text search, especially mix-platform.
  2. Lucene.Net is really a source code, class-per-class, API-per-API and algorithmatic port from the Java Lucene internet search engine towards the C# and .Internet platform utilizing Microsoft .Internet Framework.

Relational databases aren't efficient at text searching. Possibly you could attempt some thing suitable for the job like solr. Nevertheless, it appears like you are putting lots of data in SQLite (it may be a little "lite" for the purpose). Should you switched to something similar to MySQL or Postgres you can find the rate you'll need without a rewrite.