I have got a comparatively small (<100K) statistical CSV dataset that I wish to process and graph with a few numpy and pylab utilities, also it happened in my experience that there are most likely an easy method of processing the information than absurd custom if-steps for siphoning the relevent experimental situations and evaluations.

If the data were inside a DB as opposed to a CSV this would not be an issue, but tossing together a 'real' db instance with regard to this appears to become overkill. It is possible to pythonic means to fix what I am searching for?

TLDR Wish to query CSV files just like a DB / move CSV's right into a small-db.

Not understanding any sort of particulars (whatsoever) of the situation, I'll expect that you will find eventually among the following steps like a dominant one for the situation:

  • Just make use of the built-in Python sqlite3.
    • However, when the relational model isn't a necessity then pytables might be what you want on.
      • Possibly still, structured arrays can offer the required functionality.
        • But plenty could be still accomplished with only proper use of plain logic functions.
          • In the end, become familiar to reside together with your ridiculous custom if-ladders.

Clearly, the steps drew above will posses its specific benefits and drawbacks, with respect to the actual situation. Thus a very careful mixture of them may eventually yield to best 'overall' result.

I remember when i began to create a library of utilities known as wavemol. One subpackage I developed was wavemol.fileaccess, which consists of a CSV parsing class, which enables to gain access to the file inside a more practical way. Check here the techniques provided.

check here the source

you may want to install wavemol.core first. I'm not positively developing this code any longer, but when you have an interest which stuff does the secret for you personally, I might find a while to refocus onto it a little and restore it on the right track (obviously assistance is welcome, although not necessary to really make it just a little better). I kind of lost interest in it because I transformed job and that i did not need these items any longer.