RFC: gluon-config-api #2296

lemoer · 2021-08-12T00:51:30Z

This is my suggestion to support a REST API for the config-mode.

It basically works like this:

A single JSON object called "config" is exposed via http.
Packages can add config definitions in /lib/gluon/config-api/parts/*.lua.
- Such a file contains a json-schema for their config options,
- a method to get the config,
- and a method set the config.
The complete config definition is merged from all snippets in /lib/gluon/config-api/parts/*.lua.
The json-schema verification is based on libucl.

Here is an example usage. To get the config, use the following command:

$> curl http://localhost:6080/cgi-bin/api/v1/config -X GET
{
   "wizard": {
     "contact": "Leo",
     "location": {
       "lon": 9.7192900000000009,
       "lat": 52.386696000000001,
       "share_location": false
     }
   }
 }

To get the complete schema, use:

#> curl http://localhost:6080/cgi-bin/api/v1/config -X OPTIONS
{
   "schema": {
     "type": "object",
     "additionalProperties": false,
     "properties": {
       "wizard": {
         "properties": {
           "contact": {
             "type": "string"
           },
           "location": {
             "type": "object",
             "required": [
               "lat",
               "lon",
               "share_location"
             ],
             "properties": {
               "lon": {
                 "type": "number"
               },
               "lat": {
                 "type": "number"
               },
               "share_location": {
                 "type": "boolean"
               }
             }
           }
         },
         "additionalProperties": false,
         "type": "object"
       }
     },
     "required": [
       "wizard"
     ]
   },
   "allowed_methods": [
     "GET",
     "POST",
     "OPTIONS"
   ]
 }

Change contact from Leo to Hans:

#> curl http://localhost:6080/cgi-bin/api/v1/config -X POST --data @- <<EOI
{
   "wizard": {
     "contact": "Hans",
     "location": {
       "lon": 9.7192900000000009,
       "lat": 52.386696000000001,
       "share_location": false
     }
   }
 }
EOI
{
   "status": 200,
   "error": "Accepted"
 }

See the result:

#> curl http://localhost:6080/cgi-bin/api/v1/config -X GET
{
   "wizard": {
     "contact": "Hans",
     "location": {
       "lon": 9.7192900000000009,
       "lat": 52.386696000000001,
       "share_location": false
     }
   }
 }

Feel free to comment this approach.

(This is based on the ideas in #2238)

Fix bugs, ...

As of now, we prefer to use the existing uhttpd instance from the config-mode instead of creating our own.

lemoer · 2021-12-27T17:13:42Z

This has not yet received a lot of attention. Can I provide additional information? Is there anything missing?

neocturne · 2022-01-25T23:36:55Z

Some unsorted thoughts:

I like the JSON-based data structure
Having the ability to download configuration as a whole is probably generally a good thing for performance. While I'd love to have some GraphQL-like filtering capabilities, the increased processing time would likely more than outweigh the lower transferred data volume
I'm not a fan of JSON schemas, but I guess I can live with it if it works well for our use case.
- What's the binary size of libucl?
- How are config schema definitions combined?
- See also https://gitlab.com/neoraider/ece/-/wikis/design/schema, which was the simplified/modified JSON schema approach of my GSoC project a few years ago - in particular it was designed to allow merging of modular schemas. Use a more maintained library like libucl is definitely preferable to my old code though.
We should not abuse the OPTIONS method, there should be a separate schema URL you can GET
Is deleting values possible, or do we expect that no values are optional and must be set to "null" when they don't have a value?
Should modifying array values be possible in an atomic way, or do you need to GET + POST the whole array? Should a way to detect or prevent concurrent changes be provided?

I have not looked at the implementation yet.

lemoer · 2022-01-25T23:59:39Z

Some unstructured answers...

What's the binary size of libucl?

IIRC, libucl is ~51kb. I decided to use it for my RFC, since it worked more or less without much effort.

How are config schema definitions combined?

I used (more or less) the merging algorithm that you developed for ece.

See also https://gitlab.com/neoraider/ece/-/wikis/design/schema, which was the simplified/modified JSON schema approach of my GSoC project a few years ago ...

Yes. I am aware of that code and looked into it. As it uses blobs instead of json, I decided not to use it as of now. Also I think, some parts were even unfinished.

lemoer · 2022-01-26T00:12:05Z

Is deleting values possible, or do we expect that no values are optional and must be set to "null" when they don't have a value?
Should modifying array values be possible in an atomic way, or do you need to GET + POST the whole array?

Should a way to detect or prevent concurrent changes be provided?

In the current version of this RFC, there is no patching of the old config based on the POST request. The new config is actually directly the body of the POST request. So there is no merging of the old config with the new config.

This means that if you want to change just a single value, you have to retrieve the entire config and post it again.

I know this is unperformant and ugly (especially for larger applications), but I think it could be useful for this API:

It's easy to understand for users of the API.
The code stays simple. (The old state is irrelevant. No merge between old and new state necessary).

lemoer · 2022-01-26T00:19:58Z

Btw. an example of how a config could look like can be found here: Link

lemoer · 2022-11-21T16:17:42Z

@NeoRaider This is lying around for over a year without any feedback. Is there any progress?

AiyionPrime

takeaways

frontend and backend should be able to verify against the same schema
should still be possible to have config objects depending on installed packages (like batman-adv, or community specific packages)
have a look at gluon-web-model for current implementation of data attributes and logic
- https://github.com/freifunk-gluon/gluon/blob/master/package/gluon-web-model/files/lib/gluon/web/view/model/section.html
- js in frontend evaluates that
- types and dependencies of fields should be exposed as well
way to go:
- go step by step.
- gluon-web-model already provides information about dependencies/type/....
- expose this as API.
- try to build on this.
javascript code should reference fields from backend.
- based on the information about dependencies/type/..., the frontend code could render a field.
Most of the model definition could be reused.
- The biggest problem is, that fields are not well defined by their identifiers.
- One of the first steps would be to replace the enumeration of forms and sections.

@lemoer will also have a look at OpenAPI / swagger.

AiyionPrime · 2023-02-22T19:15:53Z

package/gluon-config-api-core/luasrc/lib/gluon/config-api/controller/schema.lua

+		end
+	end
+
+	-- TODO: implement array


AiyionPrime · 2023-02-22T19:16:21Z

package/gluon-config-api-geo-location/Makefile

@@ -0,0 +1,16 @@
+# Copyright (C) 2021 Leonardo Moerlein <me at irrelefant.net>


AiyionPrime · 2023-02-22T19:16:43Z

package/gluon-config-api-core/Makefile

@@ -0,0 +1,16 @@
+# Copyright (C) 2021 Leonardo Moerlein <me at irrelefant.net>


AiyionPrime · 2023-02-22T19:19:06Z

package/libucl/Makefile

@@ -0,0 +1,51 @@
+#
+# Copyright (C) 2021 OpenWrt.org


spdx no further license comments

AiyionPrime · 2023-02-22T19:20:42Z

package/libucl/Makefile

+
+PKG_SOURCE_PROTO:=git
+PKG_SOURCE_URL:=https://github.com/vstakhov/libucl
+PKG_SOURCE_VERSION:=e6c5d8079b95796099693b0889f07a036f78ad77


https://github.com/vstakhov/libucl/releases/tag/0.8.2

lemoer added 12 commits August 11, 2021 02:23

Initial tests

60aa731

gluon-config-api: First working version

64ed50c

Fix bugs, ...

gluon-config-api: use http method OPTIONS

7ae6a2b

gluon-config-api: "Not Implemented" is 501

c19e083

gluon-config-api: add allowed_methods to OPTIONS

493fe33

gluon-config-api: use config-mode uhttpd instance

299d6b8

As of now, we prefer to use the existing uhttpd instance from the config-mode instead of creating our own.

gluon-config-api: simplify merged_keys() -> keys()

3ded15d

gluon-config-api*: split to packages

8d5d753

gluon-config-api: cleanup dupplicate files

2e13ead

gluon-config-api-core: cleanup and simplify

b917439

gluon-config-api*: cleanup

2d2c71f

gluon-config-api-core: fixup function order

ea3cfbe

github-actions bot added the 3. topic: package Topic: Gluon Packages label Aug 12, 2021

lemoer added 2 commits August 12, 2021 03:05

gluon-config-api-geo-location: simplify set

71ee67e

gluon-config-api-core: commit all configs after set

3f739d5

lemoer mentioned this pull request Aug 12, 2021

RFC: A New Concept for Gluon Config Generation #2238

Open

AiyionPrime added the 2. status: waiting-on-review Awaiting review from the assignee but also interested parties. label Jan 8, 2023

AiyionPrime requested a review from neocturne January 8, 2023 14:28

AiyionPrime mentioned this pull request Jan 21, 2023

check existing PRs for linting issues #2765

Closed

25 tasks

AiyionPrime requested changes Feb 22, 2023

View reviewed changes

AiyionPrime added 2. status: waiting-on-author Waiting on some action from the author and removed 2. status: waiting-on-review Awaiting review from the assignee but also interested parties. labels Feb 22, 2023

maurerle mentioned this pull request Dec 30, 2023

x86: Using an x86 image, not all the interfaces are picked up & cannot be set in UI #3129

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: gluon-config-api #2296

RFC: gluon-config-api #2296

lemoer commented Aug 12, 2021

lemoer commented Dec 27, 2021

neocturne commented Jan 25, 2022

lemoer commented Jan 25, 2022

lemoer commented Jan 26, 2022

lemoer commented Jan 26, 2022

lemoer commented Nov 21, 2022

AiyionPrime left a comment •

edited by lemoer

Loading

AiyionPrime Feb 22, 2023

AiyionPrime Feb 22, 2023

AiyionPrime Feb 22, 2023

AiyionPrime Feb 22, 2023

AiyionPrime Feb 22, 2023

		@@ -0,0 +1,16 @@
		# Copyright (C) 2021 Leonardo Moerlein <me at irrelefant.net>

RFC: gluon-config-api #2296

Are you sure you want to change the base?

RFC: gluon-config-api #2296

Conversation

lemoer commented Aug 12, 2021

lemoer commented Dec 27, 2021

neocturne commented Jan 25, 2022

lemoer commented Jan 25, 2022

lemoer commented Jan 26, 2022

lemoer commented Jan 26, 2022

lemoer commented Nov 21, 2022

AiyionPrime left a comment • edited by lemoer Loading

Choose a reason for hiding this comment

AiyionPrime Feb 22, 2023

Choose a reason for hiding this comment

AiyionPrime Feb 22, 2023

Choose a reason for hiding this comment

AiyionPrime Feb 22, 2023

Choose a reason for hiding this comment

AiyionPrime Feb 22, 2023

Choose a reason for hiding this comment

AiyionPrime Feb 22, 2023

Choose a reason for hiding this comment

AiyionPrime left a comment •

edited by lemoer

Loading