Sparql on Riak

Graph Data Stores
There are generally two different goals people have in mind when using graph data stores.

1. Graph-walking from node to node, by following edges in the graph. People usually have social networks or The Social Web in mind.
2. Dynamic schema or schemaless triplestores. People usually have mass fact databases (aka "knowledgebases") or The Semantic Web in mind.

But under the covers, these two concerns generally overlap as graph data stores.

Riak is a very interesting, open source, homogeneously clustered, document data store. But Riak also supports "links" between documents, which makes it a graph data store as well. These links were designed for the goal of graph-walking. An interesting feature is that this graph-walking is reduced to a series of map-reduce operations so queries are fulfilled in parallel across the cluster.

SPARQL on Riak
SPARQL is a query language that is designed for the triplestores (the 2nd goal) but I see no reason why you couldn't use it for graph data stores in general, at least in theory. So if you can reduce SPARQL queries down to a Riak queries then you automatically get your SPARQL queries reduced down to map-reduce operations. Riak even supports something similar to SPARQL property paths where you can keep intermediate results while following links, so it might not be too difficult reduce most types of SPARQL queries. One concern I have (after the main concern of whether the reduction is possible) is whether Riak can handle billions of tiny "documents", which would essentially just be URIs unless you wanted to store associated documents with each URI.

Infinitely Scalable Triplestore
One goal I would like to see achieved is an infinitely scalable triplestore. Or if "infinite" is too strong of a word, let's say a triplestore that can handle an order of magnitude more triples than the biggest triplestores out there. This SPARQL on Riak proposal might actually be able to pull this off. The query might be unbearably slow, but it should complete reliably even if it takes hours, days, or months. Creating some sort of major plug-in for Apache Hive that handles SPARQL-like queries (Hive currently supports SQL-like queries) might be the more ideal way to build an infinitely scalable distributed triplestore, but doing this would be much more difficult.

Subjective Consistency

I've been researching all kinds of data stores (well, actually relational, key-value, and document data stores) and I've become aware of the interesting constraint on distributed data stores known as Brewer's CAP Theorem. The idea is that you can't have Consistency, Availability, and Partition tolerance, simultaneously in any distributed data store. It looks like it's difficult to get complete consistency on a single node (see: Isolation Levels) and it's thought to be impossible to get it on a network scale (because of CAP theorem). This is where "Eventual Consistency" usually comes in, relaxing consistency for availability and partition tolerance.

Interaction-Centric Consistency
Hopefully I can frame my idea properly now that I've confused you with some terminology. My initial thought was: What kind of guarantees can a data store offer if a single user or application talks to the same node in the network? We could call this a "data session" or an "interaction". It's a kind of network transaction idea, looser than a data transaction. Anyways, I wonder if  you could guarantee a stronger level of consistency by using your distributed network in this way. There might be a way to offer an apparent or subjective temporary consistency. Ultimately, the idea is that maybe if we make use of the patterns of access we expect from our users, then we won't need strict distributed consistency in the first place for a good number of applications.

Bridging Ontologies: The Key to Scalable Ontology Design

It's been years since I've created an ontology (in the computing/informatics sense) but I'm going to give some advice on creating them anyways. When creating an ontology, it can be helpful to connect it up to other related ontologies. In fact, I think this is a requirement for building the semantic web (taking 'ontology' in a broad sense). You may want to ground your ontology (i.e. connect it to more generic or foundational ontologies; towards upper ontologies) or connect it to well known ontologies increasing the potential usefulness and adoption of your ontology. Whatever the reason, there are benefits in doing so if you want the data that your ontology schematizes to be more easily and automatically reusable. The potential downside is that you are forcing your users to endorse the ontology you're connecting to. So how exactly should one connect their ontology into the ontology ecosystem?

Most ontologies out there seem to me to be part of a stack of ontologies built by a single group of people. The ontologies tend to build directly on top of each other, meaning "lower" ontologies directly reference "upper" ontologies. Since the ontologies are developed by a single organization, it seems to make sense to directly connect to them because the organization (arguably) knows exactly what they are attempting to represent or what they mean. The fact that organizations tend to keep their ontologies rather isolated may be caused by a fear to commit to ontologies they didn't create.

The way ontologies are (or at least should be) developed allows the possibility for changes and updates. To accommodate this, one should develop ontologies with versioning. This way, someone using your ontology won't ever have it change on them and the developers can still maintain and change the ontology by introducing new versions. It's as simple as adding a version number to your ontology's url.

But this brings up the problem we face by directly referencing other ontologies. Let's imagine you have an ontology X that makes reference to another ontology Y and that ontology Y has a newer version available. You're planning on updating a term in X to make reference to essentially the same term but in the newer version of Y to keep X up to date. So you update X to the new version of Y even though it basically hasn't changed its meaning. The role an ontology fulfills is to describe a certain subject or topic and this intrinsic meaning has not changed. Yet you still need to change your ontology. Under these conditions, no matter how much concensus is formed around the accuracy of your ontology, you will never know when it is stable. In fact, this leads to a cascading of updates and changes required by upstream ontologies that reference your ontology, and so on. This is not a distributed web-scale ontology design pattern. We need a way to decouple our ontologies.

So, is there a design pattern we can use to avoid these dangers and burdens of connecting to other ontologies? Can we do better than simply identifying good stable ontologies and directly referencing only those ontologies in our own ontology? Yes!

Introducing: Content vs Bridging Ontologies

The key to scalable ontology design is what I call Bridging Ontologies. You write your intended ontology without referencing other ontologies and then create a separate ontology that is mainly made up of owl:sameAs and rdfs:subClassOf relationships between your terms and the target ontology's terms. I call these ontologies Content Ontologies and Bridging Ontologies, respectively. You only need to update your Bridging Ontology when either the source or target ontology changes. The nature of a Bridging Ontology makes it useless for anyone to reference in their own ontologies, which stops any potential cascade of changes throughout the web of ontologies. Of course users would still need to use the Bridging Ontologies and would likely need to collapse/deflate the owl:sameAs relationships into single terms for most visualizing, processing, or reasoning purposes.

I'll go out on a limb here and say that every ontology anyone creates should be isolated in this manner. The vision then becomes a web of ontologies of small Content Ontology nodes that satisfy specific "semantic roles" and then Bridging Ontology edges definied between these Content Ontologies. Since you don't need to adopt all of the Bridging Ontologies that are built for a Content Ontology, it is much easier to reach concensus on the Content Ontologies and then to pick and choose your Bridging Ontologies, choosing to commit (or not) to exactly how that content fits into the big picture. This allows for decoupled semantics rather than traditional inflexible semantics.

The Universal Model

Imagine a computer model of the universe. The past, the present, the future, every event that has or will ever occur. This is The Universal Model.

Clearly there would be some major limitations on the quality this model. You can't have a perfect model of the present stored in the present (not to mention storing the past 13.7 billion years or the next googols of years predicted by our current best cosmological theories). What I have in mind is a kind of massive service where you can ask for information about the universe and retrieve those answers if they are available. The Universal Model is an abstract model that is, at best, partly built, as needed, in order to fulfill a query about it.

Let me try to flesh the idea out a little more. Imagine any system that needs to "answer a question" of some sort. Maybe you're typing a question into a search box. Maybe you're "asking" Google Earth to display a useful image of downtown Seattle. Maybe you're checking the weather for next weekend. Now, imagine a system that can fulfill the query in terms of questions about the way the universe is, plus some processing. In other words, imagine converting or compiling down the question into explicit questions about the universe. For Google Earth this could amount to rendering the image of what would be seen from a specific point and orientation above Seattle, right now. For the weather, you need to look at the 4D volume of space above the location you're at and from Saturday morning until Sunday evening. You'd then need to process the physical description to extract the amount of cloud and rain that's present, and then convert that into a label saying "sunny" or "partly cloudy" based on the context.

Now, again, since we can't store the whole universe, we need methods to fulfill the queries about The Universal Model. This could be done in a number of ways. If only a small subsection of the model is required, which would be the case in the vast majority of queries, then you could invoke specialized services designed to answer queries of that type. Queries about the nature of the weather on the weekend could be fulfilled by invoking weather models, for example.

Consider an analogy. When one wants to automatically translate a document from one language to another, one method would be to build a language-agnostic conceptual representation of the meaning expressed by the sentence and then to express that conceptualization in the other language. This solution would be extremely powerful because it preserves all relevant information and is scalable. (For any new language, you only need to be able to map it to The Language Model and then express that, in any language you want.) The downside of this approach is that we haven't been able to achieve any kind of mapping like this, partly because we don't know exactly what The Language Model would look like and because language is very context-dependent. Instead we build lower level representations, but not as "low" as The Language Model and then take a shortcut on that level of representation.

This is what would need to be done with The Universal Model as well. We take shortcuts while "translating" our query into an answer. This is a general problem for any sort of "conversion".

(Forgive the MS Paint). The boxes are representations of queries (red) and answers (green). As you get lower in layers, the representations are more abstract, more powerful, and more fully answerable. Each arrow is a conversion. The horizontal arrows are hacky conversions that skip the lower more ideal, powerful, but unobtainable layers and conversions. For both ideal models, we get as close as we can to the ideal, so that we can use "hacks" and shortcuts and ingenious tricks to jump to the other side of the gap. For the NLP example, the top layer could start with a document in spanish and end with a document in english. The bottom layer would be The Language Model.

If, like me, you believe that all propositions are made true or false by the universe, then in the case of The Universal Model, the top layer would be any coherent question. So even our idealistic Language Model could be seen as a special case "hack", albeit at one of the lower levels of this hypothetical universal question answering device, that has The Universal Model at the bottom. The Universal Model is the most ideal model possible.


That's the idea anyways; I'm not sure it fully holds up; Take it with a grain of salt; etc; etc.
I wanted to get the idea out there rather than dwell on it too much and then end up not posting anything. Please comment and refine and clarify the idea further for me if you find it interesting!