A blog about ideas relating to philoinformatics (or at least that have something to do with computer science or philosophy)
Showing posts with label datastore. Show all posts
Showing posts with label datastore. Show all posts

Wednesday, December 8, 2010

Sparql on Riak

Graph Data Stores
There are generally two different goals people have in mind when using graph data stores.
  1. Graph-walking from node to node, by following edges in the graph. People usually have social networks or The Social Web in mind.
  2. Dynamic schema or schemaless triplestores. People usually have mass fact databases (aka "knowledgebases") or The Semantic Web in mind.
But under the covers, these two concerns generally overlap as graph data stores.

Riak is a very interesting, open source, homogeneously clustered, document data store. But Riak also supports "links" between documents, which makes it a graph data store as well. These links were designed for the goal of graph-walking. An interesting feature is that this graph-walking is reduced to a series of map-reduce operations so queries are fulfilled in parallel across the cluster.

SPARQL on Riak
SPARQL is a query language that is designed for the triplestores (the 2nd goal) but I see no reason why you couldn't use it for graph data stores in general, at least in theory. So if you can reduce SPARQL queries down to a Riak queries then you automatically get your SPARQL queries reduced down to map-reduce operations. Riak even supports something similar to SPARQL property paths where you can keep intermediate results while following links, so it might not be too difficult reduce most types of SPARQL queries. One concern I have (after the main concern of whether the reduction is possible) is whether Riak can handle billions of tiny "documents", which would essentially just be URIs unless you wanted to store associated documents with each URI.

Infinitely Scalable Triplestore
One goal I would like to see achieved is an infinitely scalable triplestore. Or if "infinite" is too strong of a word, let's say a triplestore that can handle an order of magnitude more triples than the biggest triplestores out there. This SPARQL on Riak proposal might actually be able to pull this off. The query might be unbearably slow, but it should complete reliably even if it takes hours, days, or months. Creating some sort of major plug-in for Apache Hive that handles SPARQL-like queries (Hive currently supports SQL-like queries) might be the more ideal way to build an infinitely scalable distributed triplestore, but doing this would be much more difficult.

Tuesday, December 7, 2010

Subjective Consistency

I've been researching all kinds of data stores (well, actually relational, key-value, and document data stores) and I've become aware of the interesting constraint on distributed data stores known as Brewer's CAP Theorem. The idea is that you can't have Consistency, Availability, and Partition tolerance, simultaneously in any distributed data store. It looks like it's difficult to get complete consistency on a single node (see: Isolation Levels) and it's thought to be impossible to get it on a network scale (because of CAP theorem). This is where "Eventual Consistency" usually comes in, relaxing consistency for availability and partition tolerance.

Interaction-Centric Consistency
Hopefully I can frame my idea properly now that I've confused you with some terminology. My initial thought was: What kind of guarantees can a data store offer if a single user or application talks to the same node in the network? We could call this a "data session" or an "interaction". It's a kind of network transaction idea, looser than a data transaction. Anyways, I wonder if  you could guarantee a stronger level of consistency by using your distributed network in this way. There might be a way to offer an apparent or subjective temporary consistency. Ultimately, the idea is that maybe if we make use of the patterns of access we expect from our users, then we won't need strict distributed consistency in the first place for a good number of applications.