riak_mongo - making Riak act like MongoDB

Being in SF for the Erlang Factory this year, I've been listening to Andy Gross speaking about riak_core abstraction and how different sorts of distributed applications can be built upon it. Even during his talk I asked him if, having such a flexible architecture, Basho has ever considered to implement a MongoDB clone on top of Riak. They didn't, so I've started hacking on riak_mongo in the same night in the conference hotel. Very early in the beginning Kresten thrown himself on the project and gave it a crucial push.

The code is on GitHub.

So, what's riak_mongo? The idea is simple:

• create a layer on top of Riak which would speak Mongo's TCP based protocol (Wire)
• translate commands and queries into Riak's equivalents and run them, if necessary or applicable, against the K/V store
• try to store objects in the K/V store the way that both, Mongo and Riak can benefit from

Existing Mongo clients that speak the Wire protocol, even including the Mongo shell, will connect and speak to a Riak node without even knowing it's Riak behind the protocol.

Why implement something like that? There is a bunch of possible reasons, driven by possible users' needs:

• MongoDB users can seamlessly switch to a Riak backend. They get a Dynamo style distributed store with different storage strategies, being very reliable and almost constantly predictably scaling
• MongoDB users who need to migrate to Riak have a migration help. There would be no need to modify the existing client code to be able to start
• MongoDB users can start experimenting with reliability and eventual consistency and C/A knobs still having P as a given, and thus learn about the Dynamo approach before actually migrating to Riak
• One side effect for Riak users is to comfortably make ad-hoc queries in the Riak store
• ... you might find further reasons to use it ...

How does this technically work? Well, you can find all the details in the code. For some basic overview:

• riak_mongo speaks Mongo Wire Protocol. That's the one Mongo drivers speak with the process they connect to
• Wire protocol carries BSON, the binary JSON if you like. Internally, riak_mongo decodes this to Erlang terms to better deal with
• Since there is no equivalent for database in the K/V store, Mongo's "db.collection" pair is used as bucket name for objects in the K/V store
• what actually gets stored in the K/V store, is also BSON, even encoded is "application/bson"
• querying is done using pipelined mepred in the K/V store. The query coming in gets encoded first for easier processing in Erlang
• cursors are implemented through separate Erlang processes keeping the query results for their lifetime
• riak_mongo is an OTP application which should be loaded into the same nodes you run Riak on
• riak_mongo accesses the K/V store through the local low level Erlang API, so there is no waste for translation or explicit networking at this point
• also "application/json" encoded objects can be queried from the K/V store through the same path

What can also be done later: a MongoDB K/V backend. This would also have some interesting aspects:

• People who need to migrate to Riak could access their existing data in the MongoDB without explicit data migration
• It can even be used to migrate data, for example when only one node in the group would be backed by MongoDB. During the background data replication, data can be seamlessly and asynchronously migrated to "real" Riak nodes. Auto-sharding might be a challenge though
• One can think of combining the protocol adapter with the MongoDB backend. This way around, the user can use Riak's distribution functionality only, leaving the client and the backend as is. Existing auto-sharding stores might be a challenge here as well
• One can play with C/A knobs having existing data
• ... whatever you like ...

The project is still work in progress and doesn't introduce all possible features yet. Right now it's possible to connect to the store through a MongoDB client (we tested it with the mongo shell) and fire some basic commands and queries, like insert, update or findOne. And of course, optimizations still have to be done. We will also provide  a test suite on different levels. Now, it's a good basis for research and it also might be an interesting option for users to migrate step by step.

Any constructive feedback except ranting/bashing is welcome. We also appreciate any hands-on help.

If you want to contact developers:

Pavlo Baron (pb at pbit dot org)
Kresten Krab Thorup (krab at trifork dot com)