Structured Data Exploration BenchMark (SDEBM) based on the Berlin SPARQL Benchmark (BSBM) but intended for benchmarking the user experience while exploring a structured dataset, not the performance of the query engine.
Roberto García, Rosa Gil, Juan Manuel Gimeno, Eirik Bakke, David R. Karger. (2016). Structured Data Exploration Benchmark (SDEBM). Available from: http://w3id.org/SDEBM
Please, look at CONTRIBUTING and LICENSE to know how you can contribute to the benchmark.
Task 1. Find products for a given set of features combined
Task 2. Find products for a given set of alternative features
Task 3. Retrieve basic information about a specific product for display purposes
Task 4. Find products having some specific features and not having one feature
Task 5. Find products matching two different sets of features
Task 6. Find products that are similar to a given product
Task 7. Find products having a name that contains some text
Task 8. Retrieve in-depth information about a specific product including offers and reviews
Task 9. Give me recent reviews in English for a specific product
Task 10. Get information about a reviewer
Task 11. Get offers for a given product which fulfill specific requirements
Task 12. Export the chosen offer into another information system which uses a different schema
For the moment, three Quality in Use metrics have been used, one about effectiveness and two about efficiency:
- Capacity (effectiveness): what proportion of one task is completed. 0% if not possible to complete or 100% otherwise.
- Operation Count (efficiency): how many KLM Operators are required to complete the task.
- Speed (efficiency): each KLM Operator has a corresponding average time to complete it as detailed in the table below. For a task, this metric is computed by multiplying, for each operator type, the time for each operator by the operator count. Then, summing them all together.
| KLM Operator | Time (seconds) |
|---|---|
| K: button press or keystroke, keys and not characters. | 0.2 |
| P: pointing to a target on a display with a mouse. Time differs depending on distance and size of the target, but is held constant. | 1.1 |
| H: homing the hand(s) on the keyboard or other device, this includes movement between any two devices. | 0.4 |