I'm writing an application to perform a file conversion and a part of that's changing old account amounts with a brand new account amounts.
At this time I've got a CSV file mapping that old and new account amounts with around 30K records. I just read this in and store it as being dict so when writing the brand new file grab the brand new account in the dict by key.
My real question is what's the easiest method to do that when the CSV file increases to 100K+ records?
Will it be more effective to transform the account mappings from the CSV to some sqlite database instead of storing them like a dict in memory?
As lengthy as everyone will easily fit in memory, a dict would be the most effective solution. It is also a great deal simpler to code. 100k records should not be a problem on the modern computer.
You're right that switching for an SQLite database is a great choice when the amount of records will get large.