Sophox vs QLever

[westnordost]

To me, Sophox seemed to be kind of unmaintained / dead during the last years because the demo instance was kind of broken, but it just started working again some days ago, so I was wrong. Now, that there are two projects which (from my point of view) seem to do a very similar thing, I do wonder what's the difference between the two.

In particular, it would be helpful if you described how these two projects differ (in the project's readme?). I mean, not necessarily as far as currently supported features are concerned but concerning... X = different vision, different technology, different goals. If you knew that Sophox existed before, that is.

Are there any common components from which both projects could profit in case due to different X, both projects should be developed further independently? In other words, any possibility for cooperation?

[hannahbast]

The two projects have in common that they provide RDF data and a SPARQL endpoint for the OpenStreetMap data. Concerning the features and the performance they are very different. Here are the most obvious differences.

0. First note that we maintain three different projects. First, there is osm2rdf, which provides weekly updated RDF dumps of the OSM data as well as a tool to do it yourself. Second, there is QLever, a SPARQL engine which provides endpoints for many large datasets, including the OSM data. Third, there is Petrimaps, which provides the functionality to interactively show large numbers of geometric objects on a map.

1. osm2rdf provides the full geometry for every OSM object, as a WKT literal. Sophox provides only POINT information and only for objects which themselves are not composed of other objects. For example, in Sophox there is no triple that gives you the geometry of a particular country like so: https://qlever.cs.uni-freiburg.de/osm-planet/O3KNj5 .

2. osm2rdf provides predicates for ogc:sfContains and ogc:sfIntersects. For example, here is a query that gives you all streets in Hamburg: https://qlever.cs.uni-freiburg.de/osm-planet/3Rm2WX . Note that that is a very hard query. Sophox provides no such functionality.

3. QLever let's you visualize very large result sets. For example, click on the Map view button above the result for the previous query. That Map view works even for millions of complex objects. Sophox is based on Blazegraph and can display at most a few thousand points on a map before getting very laggy.

4. QLever is much much more efficient. Here is a query to obain all postboxes and their location: https://qlever.cs.uni-freiburg.de/osm-planet/cn7N1u . The query takes around 100 ms. The corresponding query on Sophox takes over 10 seconds, that is over a 100 times slower . This is due to Blazegraph being much slower than QLever.

5. QLever offers interactive autocompletion to construct your SPARQL query. Try it yourself on https://qlever.cs.uni-freiburg.de/osm-planet .

[westnordost]

That sounds very cool! So, you'd say QLever is better in every regard? Or can you think of a use case or aspect in which Sophox might be the better tool for the job? (Well, I guess I should rather ask this question the author(s) of Sophox)

A few more questions:

1. What do you mean with

Sophox provides only [...] information [...] for objects which themselves are not composed of other objects.

?

2. How come Blazegraph is so slow (for this use case) and what did you use instead to make it faster? (Is it a fixed part of QLever or is it a dependency that other projects may use as well?)

3. Can you write something about the system requirements of hosting a QLever instance? E.g. on what system are you hosting the demo instance? (That info could also go into the readme at the installation instructions)

[1ec5]

5. QLever offers interactive autocompletion to construct your SPARQL query. Try it yourself on https://qlever.cs.uni-freiburg.de/osm-planet .

Sophox’s GUI frontend also provides interactive autocompletion, but it’s less aggressive: you need to press Ctrl+Space to see suggestions. To be honest, I’ve been struggling with QLever’s autocompletion, because it often pops up when I intend to add a newline, responding to the Return key by inserting whatever was the first suggestion. Delaying the popup slightly would probably mitigate that issue. But I appreciate the thoughtful suggestions that it provides, such as example OSM elements to use as subjects or objects. That’s something Sophox doesn’t have.

[hannahbast]

@1ec5 Yes, the UI code is still a bit buggy, which is a pity and can be annoying at times. We really need to fix that because for some users this seems to overshadow how useful the autocompletion is. (Once you get used to it, it's easy to work around the bugs, but of course the bugs shouldn't be there in the first place).

I know that Blazegraph offers autocompletion by pressing Ctrl+Space, but I didn't get this to give me any useful suggestions on https://sophox.org . For example, how do I complete to osmrel:51477 by typing a prefix of "Germany"?

[1ec5]

Right, this is a very nice advantage of QLever over Sophox. Sophox can only autocomplete SPARQL syntax and predicates in the osmt:, osmdt:, and osmd: prefixes, but not the osmnode:, osmway:, or osmrel: prefix, and definitely not based on names of elements.

[hannahbast]

@1: I was wrong. It seems that every OSM object has a location via the osmm:loc predicate. But the location is always just a point, even for complex objects like the border of a country. For example, try this query on https://sophox.org : SELECT * WHERE { osmrel:51477 osmm:loc ?loc }.

@2: QLever and Blazegraph are two completely different SPARQL engines, created by completely different groups of people. Your question is like asking why is a Porsche faster than a 2CV. Well, they are both cars that get you from A to B, but two completely different cars. Also, QLever is in active development, while Blazegraph ist not anymore for many years now. There are also many other SPARQL engines, see this comparison.

@3: It is one of QLever's design principles to not require special hardware, in particular, no expensive hardware. In https://github.com/ad-freiburg/qlever-control/tree/python-qlever/Qleverfiles you find config files (called Qleverfiles) for various datasets. At the top of each Qleverfile, an estimate is given of the resources needed on a typical PC. For example, you can load the complete Wikidata (19 B triples) and run a server on a 1000 € PC in 4 - 5 hours.

[1ec5]

An OSM user pointed out that QLever currently relies on TTL files that can be a couple weeks old. By comparison, as long as it’s healthy, Sophox typically keeps up with OSM’s minutely diffs. For example, it’s only 2 minutes behind as I’m writing this comment. Do you think this is an inherent tradeoff to being able to add the (very useful) GeoSPARQL triples, or could QLever eventually get closer to querying OSM and Wikidata in real time?

[hannahbast]

@1ec5 Good point. We currently rebuild the index weekly, but could also do it daily. So far, we download the PBF of the whole data. We haven't looked at the diffs yet. Good to hear that Sophox does because that means it's possible. Our goal is certainly to be able to query both datasets in real time. Though our experience is that the majority of users does not care if the dataset is a day or even a few days old.

[1ec5]

Less frequent updates are probably fine for some use cases. Real-time updates matter more to OSM contributors for their mapping workflows. Mappers routinely use Overpass API queries to track their progress or download the (more or less) latest OSM data to use in JOSM, as an alternative to the more limited OSM API. Sophox includes OSM editing functionality, with the idea that a mapper might use it as a tasking manager, sharing a query that others can use to fix OSM elements one by one. This use case requires more current data in order to avoid conflicts. However, in practice, the editing functionality is seldom used compared to the data visualization functionality.

[westnordost]

Overpass API is often used for sparse editing. I.e. download only the objects of one kind and edit those in isolation.

This use case is however not possible in Sophox because for that to work, Sophox would need to be able to return results in OSM format.

[hannahbast]

@westnordost What would be the advantage for such expert users in using a SPARQL engine vs. using the Overpass API? One could argue that both approaches have their place. The SPARQL engine for convenient querying and visualization (for experts as well as non-experts) and the Overpass API for editing stuff.

[westnordost]

Hmm, SPARQL is faster and uses a query language that may be more familiar to many. However, the only real use case I can think of where sparql could outperform overpass would be to prepare (worldwide) mass edits. Overall, it does not seem to be a use case worth optimizing for as there is not really a niche to fill in that regard.

[1ec5]

The Overpass API is incapable of federated queries. That’s the main source of demand for SPARQL engines such as QLever in the OSM community.

Sophox’s editing functionality doesn’t resemble an Overpass workflow at all; the workflow is similar to MapRoulette, in that you go one by one through the results and make a decision about each individual element, committing that change atomically. It’s appropriate for cleanup on the level of retagging, not more substantial edits. But again, this functionality is obscure and seldom used, even back before Sophox became unusable in the last couple years.