I’ve just returned from the European Semantic Web Conference, where I gave a keynote talk on “The Semantic Web for End Users”. My talk addressed the problem that has interested me for eighteen years: making it easier for end users to manage their information. The thesis was that
- The current state of tools for end users to capture, manage, and communicate their information is terrible (last week), and
- The Semantic Web presents a key part of the answer to building better tools (also last week), but
- Not enough work is being directed toward this problem by the community (today’s post).
Since I had a lot to say (217 slides) I’ve broken the summary into three separate posts aligned with these three bullets; today’s is the last.
Our Story So Far
In my first post, I argued (using the work of Voida et al.) that our traditional application, designed around fix schemas, fail users when those user wish to use their own schemas or connect information from different schemas. In the second, I argued that the Semantic Web suggest an exciting alternative, of applications that can be used with any schema, permitting end users to adapt their schemas as they see fit. I gave four examples of such Semantic Web applications.
Papers in Our Semantic Web Conferences
Unfortunately, research on these end-user applications is almost completely absent from our Semantic Web conferences.
Preparing for this keynote, I looked over the program of ESWC 2013. Of the 36 main-track papers, the vast majority were devoted to the generic underlying technologies of the semantic web—ontology matching (9), entity linking (6), information extraction (6), rdf querying (6), inference (3), and specific rdf ontologies (3). Amidst all this infrastructure, there’s only one I’d label an end-user application: Kaljurand and Kuhn’s paper “Multilingual semantic wiki based on Attempto Controlled English and Grammatical Framework.” Indeed, only 5 of the 36 main-track papers had any screenshot of a working system at all! ESWC also offers an “in use track”, it is particularly odd that only 2 of the 6 papers in this track came with a screenshot—the other 4 were basically narratives about experts putting their data on the Semantic Web—useful, but certainly not end-user applications.
A glance back at ISWC 2012 shows a similar lack of end user applications; much the same topics are covered as in ESWC 2013.
I have a number of possible explanations for what is going on here; I suspect the truth is some mixture of them.
Less Semantic, More Web
The Semantic Web is a relatively new research community, and one would imagine that it was established to tackle a relatively new research direction. But for many of the papers at this conference, it isn’t clear that’s the case. The mainstays of the conference–knowledge representation, inference, and ontologies—have seen decades of study within the artificial intelligence community. That’s because these topics are all essential for the long-term objective of modeling cognition and developing true artificial intelligence.
This is obviously an important goal, but I question the importance of its connection to the Semantic Web. Isn’t such work on knowledge representation and reasoning still going on in the AI community? Given the fundamental nature of these problems, does the fact that we are doing our inferences over web data rather than (say) an expert system knowledge base change the problem at all? And if there is nothing specific to the Semantic-Web about this work, what is the value of partitioning it from the AI community?
The major novelty in the Semantic Web arises from its innovation over the web, rather than its innovation over semantics. The Web’s revolution was in making it easy for everyone to author, manage and share information. It wasn’t really about novel systems—all the pieces already existed. Instead, it was about a novel arrangement of those pieces that empowered end users. The introduction of structured data can drive that revolution forward, but only if we continue to think about how end users will use that technology.
Hammers and Nails
A common risk in academic research is getting too caught up in our hammers—powerful solution techniques—and losing track of the nails—the problems that need to be solved. I fear this may have happened in our community. Back in the early days, we convinced ourselves that a web of structured data would be useful. Now, we’re devoting all our energy to a hypothetical infrastructure for that web.
But we ever fully worked out just how that structured web would actually be used. Sure, if we solve AI then we can send off our autonomous agents to do all our work for us on the Semantic Web—but on the other hand, if we solve AI, those agents will be able to understand text and won’t need the Semantic Web. Anyway, that solution seems a long way off.
I think we have to do a much better job of demonstrating, to ourselves and to others, the more immediate benefits of the Semantic Web. And we can only do this by showing how the Semantic Web can solve problems that end users have right now. I don’t mean the generic utopian vision—I mean nails. We have to describe specific end-user problems and demonstrate specific Semantic Web applications that will solve those problems. If we fail to do that, if we create hammers without nails, I doubt we’ll ever build the right hammers. Someone else will solve those problems without using Semantic Web tools, and the Semantic Web will be left behind.
More of our research should start with the identification of a current end-user problem, because then we’ll know that there’s a real reason to create the solution. Each of the four example applications I described last time did so: personal information management and integration under your own schema (Haystack), making spreadsheets work better (Related Worksheets), publishing cool interactive data visualizations (Exhibit), and automatically coping with incoming information streams (Atomate).
I’ll specifically highlight Atomate, because it’s actually quite close in spirit to the agents proposed in the Semantic Web visioning paper, but with an important difference. We don’t know how to build autonomous Semantic Web agents yet. But we can lower our sights, give people a simple controlled natural language for query specification and get something that we can build and use right now. As such, we can point to it and say “look, if the Semantic Web were widespread, everyone could immediately benefit from something like this!”
And supporting my warning that the Semantic Web might be left behind as irrelevant, I’ll remind you of the If This Then That tool I described in the last post. This is a tool that is tackling the same problem, but solving it without using Semantic Web technology. This makes it inferior in ways I discussed last time. But it is clearly superior in the most important way: it’s actually out there, being used by people to solve their current problems. If we wait too long to offer something better based on the Semantic Web, people will get comfortable with the lesser solution. An O’reilly post last year speaks approvingly of “API-centric architectures” (like IFTTT) and disparagingly of the Semantic Web—if we don’t take steps to demonstrate the superiority of our approach, then theirs will win by default.
As an aside, I’ll propose a specific remediation here: next year’s Semantic Web Challenge should be to create a Semantic Web version of If This Then That. We could compare different instantiations of the conveniently names SWIFTTT on usability, power, and usefulness. IFTTT shows that if we actually created one that worked, there’d be clear demand for it.
Demos
I’ve been focusing on the papers, but I also want to talk about the demo track and two pervasive issues I saw there. The demo track had plenty of applications. But it suffered from the same problem that I wrote about in a blog post last year, which initiated my thinking that led to this keynote. Far too many of these demos called themselves Semantic Web applications because they stored their data in RDF. This makes no sense as the definition of a Semantic Web application. From the perspective of the end user, the particular storage model ought to be invisible. Any system that stores its data in RDF could store it in a SQL database instead with its user none the wiser (indeed, the user is unlikely to know what RDF and SQL are). Instead, I’ll state once more what I consider the key characteristic of Semantic Web applications:
A Semantic Web application is one whose schema is expected to change.
Few of the applications demoed at ESWC met this description. Rather, the majority seemed to have a fixed schema for their data which was hard-coded into the user interface. The fact that the underlying data was RDF might make it easy to change the schema in the data model, but there’s no hint of how that change might easily be propagated to the user interface.
Evaluation
Another serious problem with the demos was the lack of evaluation. At first glance this is understandable: evaluation is the difference between a demo and research paper. But consider this question: where are all the evaluations of last year’s demos, which should have shown up as research papers in this year’s conference? There aren’t any. Which suggests that our builders are stopping at demo and never actually evaluating what they built.
If true, this is a fundamental flaw in our community. As a scientific discipline we must have evaluation; without it we can make no progress. Can anyone other than the creator actually use the system they built? Does anyone see any benefit to doing so? Without positive answers to these questions, our demos tell us nothing. As academics, we aren’t building tools for use; we’re building tools as experiments that yield data for us to analyze. If we don’t do that, we’re wasting our time.
This problem extended to some of the in-use papers as well. I attended the in-use track and, after several of the talks, asked some variant of “how did making this a Semantic Web application make it better than a traditional application?” Some speakers didn’t really have an answer; other did but hadn’t put it in their paper. I particularly remember a paper on Hafslund Sesam, whose presenter, when prompted, went on at length about how much easier it had been to build their system using Semantic Web technologies instead of traditional databases. This was great to hear, but (1) this argument is barely made in the paper itself and (2) there is no (evaluation-based) proof—only opinion. Where is the comparable system that tried to do things the old fashioned way and was harder? How do we know that Semantic Web technologies were actually better here, as opposed to being what the developers found most familiar?
The three examples I sketched last time all included some form of evaluation. In Related Worksheets, we concentrated on usability: could end-user answer questions faster with these than with traditional spreadsheets? Similarly, with Atomate we asked whether end users could actually create rules using our controlled natural language, as well as asking whether end user perceived value in having such a tool. With Exhibit, we’ve begin (and are continuing) to ask what kind of data and visualizations end users want (and are able) to publish on the web. With Related Worksheets and Atomate we did “lab studies”—bringing users into our lab to carry out specific tasks we defined for them. These are generally great for understanding usability, pretty easy to set up, and not too time consuming. There are simple metrics like task correctness and time to completion that can be used to compare different tools. Exhibit is more of a “field study”—we’ve put the tool out in the world and gotten some adoption, and are now asking what users are doing with it. These studies tend to be messier, requiring more time and effort to deploy the tool and analyze the data, but also more realistic.
Conclusion
Winding up this long post, I hope I’ve convinced you that the Semantic Web reflects some of the key insights, on schema variability, that I think are critical to improving people’s ability to manage information. But we aren’t doing the work. We’re devoting far too much energy to studies of knowledge representation, reasoning, and information extraction that have traditionally appeared in artificial intelligence conferences, and perhaps should continue to do so. We build applications, but we call them demos and don’t evaluate them. Many of them aren’t really Semantic Web applications; they’re just traditional applications that happen to be storing their data in RDF. We’re letting a great opportunity pass us by; I hope we can think about ways to seize it.
Finally, for those who’ve made it this far, here are my slides and those of Voida et al.
