Towards a Smooth Semantic Web

This is an almost complete draft I wrote almost 2 years ago. I thought I should publish it because it is basically complete, and because my next post will be about the new language and technologies that will make the "smooth semantic web"  a reality. And for the record, 'smooth semantic web' was a provisional name to be changed before publishing.
Here it is:

The Semantic Web is slowly building up and will eventually grow to a critical mass where it will become useful. How useful is another question. Will it revolutionize the web? Web 3.0? Maybe. Maybe not. But it will definitely have a use and an effect. The current standards such as RDF, OWL, SWRL, SPARQL, and SKOS are good, and still have a ton of potential that is growing exponentially as we speak. But these standards are not enough. OWL and SKOS can only capture so much knowledge. They can capture crystalized categorizations of well defined concepts. But much, if not the vast majority, of knowledge is not found in strict categorization and necessary relationships. So what is missing?

People have made attempts to extend OWL in a few different ways to make it cover a wider range of statable knowledge. People have looked into adding probability, non-monotonicty (time), belief. I propose a different addition which, if adopted, would add a prior layer to OWL, just as the above attempts do.

Consider a music ontology. OWL can support certain relationships such as: Rock ISA Genre, Track isPublishedOn CD, etc. But the certain pieces of knowledge cannot be represented because they haven't been crystallized to the point of being definable, especially when you consider the categorization of instances. Imagine the daunting task of deciding whether certain border line songs count as belonging to a specific genre. But consider the task (still daunting, but not as much) of deciding whether a song is "better classified as Rock than Rap" for instance. It may be difficult to say that a song is Rock, or is Rap, but it may be clear which one it is closer to. Much more can be said.

Notice that we are not talking about probabilities here. It isn't semantically correct to say that a song is "more likely" categorized as Rock than Rap. We are saying that if we were to categorize it as either Rock or Rap, it would be better categorized as Rock than Rap.

You may be wondering what the criterion for "better" is or should be. This itself should be represented within a knowledgebase.

So how would we use this knowledge? SWRL and SPARQL can only handle deductive reasoning, so they won't help. There are two options, the way I see it. Take my smooth knowledge and use some kind of classifying step to derive a regular rigid knowledgebase. The other option is to invent smooth reasoners. I think vector space representations and abduction would be important for this step.

Large and Small Controlled Vocabularies

I've been thinking about something to make using a project called Entity Describer (ED) that my co-workers have been working on. I thought, and still think, I should make a recommender because ED should contain better tag data than basically all other tag sets because the tags are taken as a controlled vocabulary from Freebase. A controlled vocabulary gives you spelling correction and possibly disambiguation, but because the tags are from Freebase, ED also has a very large user generated vocabulary, and connections between the tags (aka semantics). But I realized that the tag data generated by ED doesn't contain information about what the users like and dislike (as of July 2008), though you could guess that most users tag things they like much more often than things they dislike.

Fortunately, ED will soon allow you to choose which vocabulary you would like to use if Freebase topics aren't right for your tagging needs. So I thought I would create a few vocabularies including:

• Thumbs up and down
• Agree/Disagree (more concrete, but less usable than thumbs)
• Consistent, Valid, Sound, Straw Man, etc (for arguments, and maybe good for philosophy papers)
• Stars (0 or 1) to (4, 5, or 10) seems popular
• "Effects on mentality" like frightening, enlightening, and annoying (kind of like TED talk ratings)
  
• Emotions (similar to effects on mentality)
  or my personal favourite
• "like this", "want people to see this", "want to see more like this", "agree with this", and the inverses (a small set of reasons I've used thumbs up or thumbs down with the Firefox StumbleUpon add-on)
  

I could create these vocabularies, but why not add them to Freebase and let people use them from there? This got me thinking about something more general.

One great reason to use Freebase for tags is that there are so many. But when trying to use the tag data to make (say) a recommender, the user may not have found their ideal tags to use. So what a person actually felt about something slightly differs from what they expressed through tagging it. This doesn't happen with a sufficiently small vocabulary. You know that the user found the best description they could, but then they may not have been able to express themselves with the limited vocabulary. So, either way, big or small vocabulary, tags may not be reflecting what the user has in mind.

There are definitely ways to alleviate this problem, and the best size totally depends on exactly how and what your users are tagging. I think the above small vocabularies would all work better outside of Freebase, but tags describing the subject of a website work better within Freebase. What do you think? How significant is this problem? When is this problem most significant? Do you think a vocabulary of content types (i.e. Blog, Video, News, Article, Game, etc) is better inside or outside of Freebase?

Getting rough data into the semantic web

It's nice when regular relational databases, excel spreadsheets, and other sources that people have created can be attached to the semantic web. But there are problems with the way most data is recorded, especially for data about time. In short, the problem is that people round values. This may seem like a trivial unimportant problem, but I don't think so if you want to be able to use the rough data that makes up most of the world's data. Here are the issues:

1) Datatype granularity. Datatypes allow for a wide range of possible values, but not all values. We may need to know in some situations whether the data value written is an exact match to the value that was intended. For instance, 1/3 can't be represented fully as a double and 3pm today can't be represented as a date without dropping the 3pm part.

2) Granularity used vs datatype granularity. This is a much bigger problem than (1). People often write things to a granularity that is much less fine grained than the datatype granularity. For instance, you may be recording distance to 2 decimal places and storing it as a float. This needs to be taken into account for situations where values are compared. We don't want to say that two things have the same height just because they are almost the same height. Also, people round times to the minute, 5 minutes, 10 minutes, 15 minutes, hour, and many other ways. Are they rounding down? Are they rounding to the nearest? Some other recording method?

An ontology that has commonly used recording methods to describe the data recording process would allow more justified inferences about the data to be made. This would allow a higher level of trust (in the second sense, explained here) when using the data which is going to be of great importance as the semantic web grows.