I'm storing a table using python, and that i need persistance.

Basically I'm storing the table like a dictionary string to amounts. And also the whole is saved with shelve


I personally use writeback to true when i found the machine are usually unstable basically don't.

So following the computations the machine must close the database, and store it back. The database (the table) is all about 540MB, which is taking age range. Time skyrocketed following the table increased to around 500MB. However I require a larger table. Actually I want a couple of them.

I'm most likely while using wrong type of persistance. Any suggestions?

For storing a sizable dictionary of string : number key-value pairs, I'd advise a JSON-native storage solution for example MongoDB. It features a wonderful API for Python, Pymongo. MongoDB is lightweight and extremely fast, and json objects will natively be dictionaries in Python. What this means is which you can use your string key because the object ID, permitting for compressed storage and quick research.

To illustrate how easy the code could be, begin to see the following:

d = {'string1' : 1, 'string2' : 2, 'string3' : 3}
from pymongo import Connection
conn = Connection()
db = conn['example-database']
collection = db['example-collection']
for string, num in d.items():
    collection.save({'_id' : string, 'value' : num})
# testing
newD = {}
for obj in collection.find():
    newD[obj['_id']] = obj['value']
print newD
# output is: {u'string2': 2, u'string3': 3, u'string1': 1}

You'd have to convert back from unicode, that is trivial.

Perhaps you have attempted pickle/cPickle or json?

Just how much bigger? Do you know the access designs? What types of computation must you do onto it?

Bear in mind that you will possess some performance limits if you cannot keep your table in memory regardless of how you're doing so.

You might want to take a look at likely to SQLAlchemy, or directly using something similar to bsddb, but each of individuals will sacrifice simplicity of code. However, with SQL you might have the ability to offload a few of the try to the database layer with respect to the workload.

Look for BerkeleyDB it's free, fast and also have many features. And... it have python bindings.