I wanted to take a moment and introduce my new open source project named Speedy.
I had a need for a embedded key value repository that performed well for both writes and reads. I tried using other already existing key value database but I could not get any to work well with both read and writes or others would actually corrupt when we prototyped it. It was a very frustrating week due to the fact that I wanted to use an existing solution rather than providing my own.
However it turns out I went on vacation at the perfect time. During my vacation I decided this would be a great time to spend some time working on a solution for this need. I put in a few hours and got a very small and simple solution to the problem.
Here is a quick example on how to start a repository and add three items.
var repository = new Repository("C:\\Speedy", "Items");
Here are some numbers from a few quick benchmarks.
Let's create a repository with 100000 items @ 100 at a time.<br /> Done: 00:00:23.2290542
Let's create a repository with 100000 items @ 1000 at a time.<br /> Done: 00:00:04.2213932
Let's create a repository with 100000 items @ 2500 at a time.<br /> Done: 00:00:01.7740462
Let's create a repository with 100000 items @ 10000 at a time.<br /> Done: 00:00:00.5225688
Let's create a repository with 100000 items @ 50000 at a time.<br /> Done: 00:00:00.1917974
You'll need to balance how often to save you repository based on how much memory you want to use. More items written before saving will help speed but will require more memory. If you have low memory requirements then save often but it'll take longer to write the full repository.
Reads using 100 random keys in a repository of 100,000 items.
Let's read randomly into the repository @ 1 at a time.<br /> Total: 00:00:02.9124129
Let's read randomly into the repository using all keys.<br /> Total: 00:00:00.0293204
Now I'm sure there are much better ways to do what I'm doing but I really have enjoyed working on this simple solution. I'll continue to work on it as a fun little side project and continue to learn more about how to efficiently store data in a single file.