Sunday, 12 June 2011

A Python module providing thread safe SQLite backed persistent storage

In a previous article I wrote about some research I did on Python modules that could provide me with a persistent storage solution for dictionaries. I didn't quite find what I needed especially as none of the modules provided a thread safe solution. In the end I decided to write my own.

Thread safe, SQLite backed persistent storage

The module I wrote, persistentdict, is is freely available on my website along with some notes and examples. It also has a fairly extensive test suite. It provides a single class PersistentDict that behaves almost exactly like a native Python dict but stores its keys and values in a SQLite table instead of keeping it in memory.

5 comments:

  1. Thanks for your work, I've been look for something like this for a long time, and just FYR there's another approach for persistent data is ATpy but it come with an numpy based low level data structure but not pure python, and it is "table" based.

    Have you ever considing to extend your "hash" based module support a table\field based structure, I means, to make it sort of persistent "table" with autoincr ID, with field name ... by pure python dict?

    ReplyDelete
  2. @anonymous
    thanks for your suggestions. I'll have a look.

    You may want to check my new article. It shows that unfortunately in Python the combination multithreading/SQLite back-end has some additional problems.

    ReplyDelete
  3. The Python standard library has the shelve module. It implements persistent dictionaries based on the pickle protocol.

    ReplyDelete
  4. There's a module called y_serial designed for any arbitrary Python object -- not just dict.

    "y_serial = serialization + persistance. In a few lines of code, compress and annotate Python objects into SQLite; then later retrieve them chronologically by keywords without any SQL. Highly useful NoSQL “standard” module for a database to store schema-less data."

    See http://yserial.sourceforge.net/ for further details.

    ReplyDelete
  5. @y_Developers: certainly a interesting and useful module. Why not port it to 3.x?

    ReplyDelete