API listing responses don't include counts or paging information

[jackdos]

As a user of the API listing endpoints I want to have some paging information, including how many entities in total match my request, how many entities are in this response, where I am in the paging or how the next equivalent page of results is requested.

[carlwilson]

This is releated to #31, I think that some work on counting and paging would be valuable in 1.3/

[jackdos]

Current Behaviour

Our existing listing endpoints (GET /{entity-name}) already accept MySQL style "offset" and "limit" parameters. The documentation of default behaviour is not entirely clear, as they both list "0" as a default value, but what happens in the Preservica implementation (and I suspect any sane implementation) is that if they are not provided explicitly, the offset is interpreted as "0" and the limit as "unlimited".

Currently, if you wanted paging behaviour, you would specify for example: offset 0, limit 10 for your first query, then offset 10, limit 10 for your second, and so on until the query returns fewer than 10 items. At that point, you can assume you have reached the end of the list.

Our existing listing endpoints also specify a return object that is not specified in the domain model, but which consists of a single property "{entityNamePluralised}", which contains a list of entities. This object feels like a natural holder for version information as well as count/paging information.

Proposal

1. That the documentation be modified to clarify that not specifying an offset results in 0 being assumed
2. That the documentation be modified to clarify that not specifying a limit results in the assumption that all results be returned
3. That the documentation be modified to clarify that to explicitly set no limit, you can pass an integer less than zero (i.e. limit=-1)
4. That the API specification be modified to require that limit=0 returns an object with an empty list of entities (for example to perform a low bandwidth count of the number of results)
5. That the listing endpoint object be documented in a json schema
6. That the listing endpoint be extended to include API endpoint version, query playback information, paging information indicating the url and parameters for the next page of results.

Example return object:

{
    "apiVersion": "1.3",
    "query": {
        "url": "https://example.com/par/business-rules?offset=0&limit=100",
        "parameters": [
            {"file-format-name", "fmt/39,fmt/40"},...
        }
    },
    "paging" : {
        "totalResults": 150,
        "returnedResults": 100,
        "nextPage": "https://example.com/par/business-rules?offset=100&limit=100",
        "parameters": [
            {"file-format-name", "fmt/39,fmt/40"},...
        }
    },
    "businessRules": [
        {...}, ...
    ]
}

Implications

Although not a documented requirement, sensible paging requires that the list of objects be returned in a consistent, predictable order. This can be implementation specific. An "orderBy" parameter to let clients determine the order they wish to receive the results in is a separate enhancement.

The response will always include the nextPage information, it is a client implementation detail to determine when it has received the last page of results (i.e., when the returnedResults is less than the limit).

Outstanding Questions

1. Where should we put the object schema(s), they don't feel like part of the core domain data model, they are an API implementation detail.