Friday, May 15, 2009

Why I don't use CouchDB

CouchDB is lots of fun. It's really easy to install on a mac using the CouchDBX package. It comes with a nice web UI so you can play around with it straight away. It leverages REST and JSON to provide a simple API that you can use from virtually any language. It has a great transactional model which lets you have full ACID semantics in a very lightweight way. So why don't I use it? Well, several reasons. I'll try to skip the standard flaming I've heard on the 'tubes before. Here goes…

Views

The concept of CouchDB's views is actually very elegant. It's a purely functional map of the documents in the database. This means you can process the data any way you like using javascript, but CouchDB can make assumptions about data freshness, and safely cache and index the results to provide efficient queries and updates (at least in theory ;-).

Unfortunately you can only create a view from the original data, there is no way to create views whose input is other views. This means that you cannot do anything really interesting with values from multiple documents. You can aggregate data from several documents using the reduce functionality into buckets, but you can't process that data further.

This means that you have to live with the same limitations of SQL queries (the fact that they are non recursive, so they can't express transitive relationships), but you don't get the freedom to write queries ad hoc and have them execute efficiently (ad hoc views are supported, but there are no general purposes indexes).

The reduce functionality alleviates this somewhat, but personally I feel this is a bit of a kludge (reduce is really just a special case of map: map takes data and outputs data to buckets using a key. Reduce is a map where the data is the resulting buckets from the previous pass and the output).

The overview implies that CouchDB contains a port of Google's MapReduce framework, but the "real" MapReduce is much more flexible than CouchDB's implementation.

Replication

The replication subsystem is also heavily hyped, but it's hard to find details about the way it actually works. My understanding is that each conflicting version is kept in storage, but that one of these "wins" and becomes the default version of a document. This is rationalized in the CouchDB technical overview as follows:

The CouchDB storage system treats edit conflicts as a common state, not an exceptional one

If I understand correctly since a conflict is not an error, without explicitly seeking out these conflicts your keep working with the "winner". From the user's point of view if your application is not defensive about conflicts but the user decides to deploy it with replication it could lead to apparent data loss (the data is still there, but not viewable in the application) and inconsistencies (if two different documents' "winners" have a conflicting assumption about the state of the database, without actually conflicting in the data, though if fully serializable transactions are used this might not be an issue).

In short, color me skeptical. The replication subsystem could be a useful start to building distributed apps, but there is still a lot of effort involved in doing something like that.

Out of the box replication support is useful for taking data sets home on your laptop as a developer, and being able to push changes back later. I see no compelling evidence for the claims about scalability and clustering.

To me this seems like a niche feature, not really relevant for most applications, but one in which significant effort was invested. The presence of a feature I don't quite care for doesn't really mean I shouldn't use something, but for a project which is still under heavy development this comes at the expense of more important features.

Latency

If I recall correctly CouchDB supports upwards of 2000 HTTP requests per second on commodity hardware, but this is only optimal if you have many concurrent dumb clients, wheras most web applications scale rather differently (a handful of server side workers, not thousands).

Even if you use non blocking clients the latency of creating a socket, connecting, requesting the data and waiting for it is very high. In KiokuDB's benchmarks CouchDB is the slowest backend by far, bested even by the naive plain file backend by a factor of about 2-3, and by the more standard backends (Berkeley DB, DBI) by a factor of more than 10. To me this means that when using KiokuDB with Berkeley DB backend I don't need to think twice about a request that will fetch several thousand objects, but if that request takes 5 seconds instead of half a second the app becomes unusable. Part of the joy of working with non linear schemas is that you can do more interesting things with tree and graph traversals, but performance must be acceptable. Not all requests need to fetch that many objects, but for the ones that do CouchDB is limiting.

If you have data dependencies, that is you fetch documents based on data you found in other documents this can quickly become a bottleneck. If bulk fetching and view cascades were supported a view that provides a transitive closures of all relevant data for a given document could be implemented by simply moving the graph traversal to the server side.

So even though CouchDB performs quite when measuring throughput it's quite hard to get low latency performance out of it. The simplicity gained by using HTTP and JSON is quickly overshadowed by the difficulties of using nonblocking IO in an event based or threaded client.

To be fair a large part of the problem is probably also due to AnyEvent::CouchDB's lack of support for the bulk document API's include_docs feature (is that a recent addition?). KiokuDB's object linker supports bulk fetching of entries, so this could have the potential to make performance acceptable for OLTP applications requiring slightly larger transient data sets. Update: this has since been added to AnyEvent::CouchDB. I will rerun my benchmarks and post the results in the comments tomorrow.

No authentication or authorization

Authorization support could make a big performance difference for web applications. If the mechanisms to restrict access were in place the CouchDB backend could be exposed to the browser directly, removing the server side application code as a bottleneck.

If the server side could provide the client with some trusted token allowing it to only view (and possibly edit) a restricted set of documents. There is lots of potential in the view subsystem for creating a flexible authorization framework.

This would also make CouchDB a serious platform for writing pure javascript applications, without needing a fully trusted sandbox environment. If all you need is CouchDB and static files then deploying your app would be a breeze.

LDAP authentication is on the roadmap, but authentication and authorization are really separate features, and there doesn't seem to be any work toward flexible access control yet.

Apparent lack of development focus

I guess I have no business complaining about this since I don't actually contribute code, but it seems to me like the focus of the team was to improve what already exists, instead of adding important missing features (or at least features I feel are important). This makes me pessimistic about having any of the issues I raised resolved.

When I was last on the IRC channel there were discussions of a second rewrite of the on disk BTree format. Personally I would much rather see feature completeness first. Rewriting the on disk format will probably not provide performance improvements an order of magnitude better than the current state, so I think it's more than acceptable to let those parts remain suboptimal until the API is finalized, for instance. CouchDB's performance was definitely more than acceptable when I was using pre-rewrite, so this strikes me as a lack of pragmatism and priorities, especially when the project does have an ambitious roadmap.

Alternatives

We've been using the excellent Berkeley DB as well as SQLite and unfortunately MySQL for "document oriented storage", and all of these work very well. Connectivity support is fairly ubiquitous, and unlike CouchDB the APIs are already stable and complete.

Other alternatives worth exploring include MongoDB (which unfortunately lacks transactions), key/value pair databases (lots of these lately, many of them distributed), RDF triplestores, and XML databases.

One alternative I don't really consider viable is Amazon SimpleDB. It exhibits all of the problems that CouchDB has, but also introduces complete lack of data consistency, and a much more complex API. Unless you need massive scaling with very particular data usage patterns (read: not OLTP) SimpleDB doesn't really apply.

I think the most important thing to keep in mind when pursuing schema free data storage the "you are not google" axiom of scaling. People seem to be overly concerned about scalability without first having a successful product to scale. All the above mentioned technologies will go a long way both in terms of data sizes and data access rates, and by using a navigational approach to storing your data sharding can be added very easily.

Anyway, here's hoping CouchDB eventually matures into something that really makes a difference in the way I work. At the moment once the store and retrieve abstractions are in place there's nothing compelling me to try and use it over any other product, but it does show plenty of promise.

18 comments:

knowtheory said...

I have it on good authority that modern sqls can do transitive queries, and you can do recursive queries, in manners like this: http://pastie.org/477488 (i'm not the author of the pastie, this was just shown to me the other day).

Ah in fact i've been cited these two links as a demonstration of transitive queries, and recursive queries:
http://msdn.microsoft.com/en-us/library/ms190766.aspx
http://msdn.microsoft.com/en-us/library/ms186243.aspx

Clifford Heath said...

It's simply not true that SQL can't do recursive (transitive) queries. See http://www.dba-oracle.com/t_sql_patterns_recursive.htm for example... and there are older non-standard ways to do it.

nothingmuch said...

Sorry, I guess I really was generalizing.

Unfortunately at least for the databases our clients seem stuck on even ANSI SQL recursive queries are not yet supported. Hopefully by the time I retire this problem will be solved =P

That said, IIRC the recursive selects are still not able to do transitive relations, but only recursive ones. The relationship is a traversal, and isn't parameterized on the origin of the traversal (the ancestor) but simply the direct parent. If I'm not mistaken it still can't do things things like computing context graphs or extracting the trusted closure out of a web of trust using diminishing values and trust metric, for instance.

Unknown said...

hmm, aren't you mixing apples with peaches when you compare a db server like CouchDB to file-based db engines (sqlite, berkeley db et al).

Of course file-based db engines save you from the network overhead, but they quickly come to their limits in multi-user environments.

jasonW said...

I'm curious how you use sqlite, mysql and berkleydb for 'document oriented storage'. Do you have some links you could provide? interested.

grant said...

Build some applications on the Lotus Notes platform. That'll make you see the virtue and appeal of CouchDB.

@jasonW: I've heard of using json (or like) to store a "document" in a field of a relational database.

nothingmuch said...

PicPURLPy - MySQL performs about the same as both SQLite and BerkeleyDB in those tests. Not quite as good as BDB, but actually better than SQLite.

Also, even if the comparison isn't fair, I still need to make it if I decide to forego BDB for CouchDB.

jasonw - See earlier posts on this blog. All of these are KiokuDB backends: http://www.iinteractive.com/kiokudb/

grant - erm, I think I see the virtue and appeal, I've been exploiting the obvious advantages in other products, it's just that CouchDB's more unique features (e.g. views vs. indexing, replication) are not in and of themselves reason enough to switch to it, because there are also limitations.

Unknown said...

In addition to CouchDB also worth checking out

Tokyo Cabinet
http://tokyocabinet.sourceforge.net/

MemcacheDB
http://memcachedb.org/

Damien said...

What BTree rewrite? The btree code hasn't changed substantially in nearly a year.

Unknown said...

Have you taken a look at Persevere?
http://docs.persvr.org/

It can store json much the same way as couchdb, but has support for fast ad-hoc queries, references between json objects, optional schema constraints, and both user authentication and authorization.

Since queries are so important, I'll just through out some examples of Persevere queries:

/Product/[?price<15.00 & rating>3][0:10]

To break that down: everything in '[]' are individual queries. The first query retrieves all Products with price less that 15.0 and rating > 3. The second query is a "slice" operation that returns the first ten items of the original query.

/Product/[?price<15.00][=name]

Here we get Products < $15.00 and then apply a map operation that will return the list of products names.

Persevere supports references between items, and you can query on those too:

/Product/?manufacturer=/Manufacturer/5

nothingmuch said...

That's when I was referring too. I think it was fall of last year when I was last on the channel.

nothingmuch said...

Actually yes, I've heard of Persevere (and that's why I decided to use JSPON for JSON serialization in KiokuDB) but unfortunately I haven't actually implemented a backend for it yet.

CouchDB had the distinct advantage of having about 5 client libraries on the CPAN, but for Persevere I would have to implement my own.

JanL said...

I'd just like to add that comparing an alpha project with mature solutions is somewhat skewed, but the CouchDB team could do well with documenting how far down the roadmap they are.

(I'm on that team :)

Anonymous said...

I'll also echo JanL. Please keep in mind that CouchDB is alpha, it isn't finished. What it does have is great and I've found it fits well with my current ruby/merb project. Allows me to get on with it, so to speak. Pick the correct tool for the job ;)

Anonymous said...

I think you should also try your tests on the latest couchdb build. I haven't run it yet, but my understanding is that there have been some major speed ups.

I do agree with you that views only on data can be limiting. I'd love to have 1 view that returned say the sum of all the records in a database (perhaps wordcount frequencies). Then I could have a 2nd view which could normalize specific word instances.

i.e.
All words = 152
The = 31

I could get a normalized view of "the" if the views could cascade calls.

nothingmuch said...

Hey, my comments feed seems to be lagging, I didn't see these responses till now.

So, first off: I don't expect CouchDB to be as solid projects that have been around for more than a decade (some of them longer than me =P). Don't get me wrong, I'm thoroughly impressed with CouchDB's progress so far, but a girl can dream, eh? ;-)

However, since CouchDB is already an attractive platform and pretty popular as a result of that, I thought this post is useful for two reasons:

a. It raises awareness to the issues that I care about and are preventing me from using it successfully.

b. It might be useful for people with similar data and data access pattens to my own.

And now a small update:

I've been very busy with work (over the weekend too unfortunately) but I did get a chance to switch the KiokuDB backend over to the bulk fetching api, and update for various 0.9 changes.

Unfortunately the changes in the transactional model are much more off-putting, and so far I've not found a way to get atomicity and isolation for the transactional support. Since my time is limited I don't feel that I have investigated it fully, but after I've done my research (and hopefully managed to discuss it with the people behind the design, too -- I think CouchDB can still offer both even in the new model), I will post a followup with the new results. I will also verify that the library I am leverages Keep Alive properly.

If it turns out that the transactional guarantees I need cannot be provided then I guess I'll need to give up on it, because it will have become fundamentally incompatible with the type of data I use, instead of just being temporarily incompatible with my data access patterns.

Anyway, more to come... =)

Rob Tweed said...

Also check out the M/DB family of databases at http://www.mgateway.com.

M/DB started life as an Open Source API-compatible alternative to SimpleDB. As such you can use M/DB as a local backup to SimpleDB and/or a failsafe against the proprietary nature of SimpleDB.

It's now spawned M/DB:X which is an open source, lightweight Native XML Database. Currently it's HTTP requests in, XML out, but development is now underway to add JSON as an input option and JSON as an output option, with M/DB:X mapping between JSON and XML automatically for you. In effect it will allow you to store Javascript objects in XML DOM format them, search them using XPath, and output them as either XML or JSON.

Security is currently based on the SimpleDB model, but I like the idea of providing an additional mechanism of trusted tokens to allow a defined level of access to specified documents, so that should be an up-coming feature.

Anonymous said...

THIS IS OLD AND SHOULD BE TAKEN DOWN.

On a side note checkout kanso if you need some structure to help you understand couchdb and the beauty of couchapps.