feat(Core): system for extending Docq with external modules (aka plug…

…ins) (#148)

* feat(extensions): extension system v0.1 (alpha) to handle UI and data layer extension. See [docs](docs/developers-guides/extensions.md)
* docs: add docs for Extensions feature
* chore: instrument extension loading
* chore: fix run with Otel task

* feat(core): system-level settings (original system settings became org level  setting when the concept of organisations was introduced)

*fix(core): initialising `settings` table with defaults for system settings and org settings

*fix(setting page UI): if no persisted settings, stop showing options in the selectbox.

* chore(observability): add more OTel instrumentation

* feat(core): toggle free user signup UI with system feature. `FREE_USER_SIGNUP` feature toggle enables the signup UI

* build(docs): remove mkapi plugin. It breaks docs build due to a lack of support for Python 3.11. MkApi project seems abandoned

* docs: fix code formatting.
* chore: update lock file
* build: pin some dependencies. update lockfile

.gitignore

@@ -190,4 +190,7 @@ docker.env
 /docs/tests
 
 # ignore Infisical config file
-.infisical.json
+.infisical.json
+
+# ignore default docq extensions
+.docq-extensions.json

docs/developer-guide/extensions.md

@@ -0,0 +1,51 @@
+# Extensions (alpha)
+
+Note: this interface is highly unstable. Expect breaking changes.
+
+Extensions is a type of plugin system that enables you to extend Docq functionality. This enables developing custom extensions as external modules. Then add to Docq through configuration (so imported dynamically at build time or run).
+
+Extensions are Python modules that implement one of the `DocqExtension` classes:
+
+- `DocqWebUiExtension` - Web UI extensions
+- `DocqWebApiExtension` - Web API extensions (in the future as there's no web API at present)
+- `DocqDalExtension` - Database layer extensions
+
+Docq implements extensions as a event hooks system. In places that can be extended Docq fires the `callback_handler()` method on all registered extensions. The `event_name` is used identify which hook was fired. Together with `ExtensionContext` allows implementing logic.
+
+## Configure Docq Extensions
+
+Drop a `.docq-extensions.json` file into the root folder of the Docq app deployment. In the future we'll develop easier way to deploy extensions without having to redeploy the entire Docq app.
+
+## `.docq-extensions.json` Schema
+
+```json
+{
+  "unique_key": {
+    "name": "any friendly name",
+    "module_name": "<the full module path. Same as `from` in an static import",
+    "source": "<location of the source. Relative path or git url",
+    "class_name": "<name of your class that inherits from `DocqExtension`"
+  }
+}
+```
+
+Example
+
+```json
+{
+  "docq_extensions.web.layout.example": {
+    "name": "Docq example web extension",
+    "module_name": "docq_extensions.web.layout.example",
+    "source": "../docq-extensions/source/docq_extensions/web/layout/example.py",
+    "class_name": "Example"
+  },
+  "docq_extensions.web.layout.example_dal": {
+    "name": "Docq example DAL extension",
+    "module_name": "docq_extensions.docq.example_dal",
+    "source": "../docq-extensions/source/docq_extensions/docq/example_dal.py",
+    "class_name": "ExampleDal"
+  }
+}
+```
+
+

docs/developer-guide/getting-started.md

@@ -7,7 +7,6 @@ To run this project locally, you will need to install the prerequisites and foll
 This Project depends on the following projects.
 
 - Poetry
-
   ```sh
   pip install --user --upgrade poetry
   ```
@@ -20,26 +19,19 @@ This Project depends on the following projects.
 ### Installation
 
 1. Clone the repo
-
    ```sh
    git clone https://github.com/docqai/docq
    cd docq
    ```
-
 2. Install Poe the Poet and Poetry
-
    ```sh
    pip install --user --upgrade poethepoet poetry
    ```
-
 3. Install requirements for development
-
    ```sh
    poe install-dev
    ```
-
 4. Run tests
-
    ```sh
    poe test
    ```
@@ -58,20 +50,20 @@ Some useful examples of how this project can be used:
 
 ### Run tests
 
-- Unit tests
+#### Unit tests
 
 ```sh
 poe test
-``` 
+```
 
-- Integration tests
+#### Integration tests
 
- - Prepare env vars by supplying a pytest.env file required for the integration tests
+Prepare env vars by supplying a pytest.env file required for the integration tests
 
-   ```sh
-   cp misc/pytest.env.template pytest.env
-   ## Make edits on pytest.env with the correct values for the test environment then run the tests
-   ```
+```sh
+cp misc/pytest.env.template pytest.env
+## Make edits on pytest.env with the correct values for the test environment then run the tests
+```
 
 ```sh
 poe test-integration
@@ -80,23 +72,17 @@ poe test-integration
 ### Run the project
 
 - Prepare env vars by supplying a Streamlit secrets file
-
   ```sh
   cp misc/secrets.toml.template .streamlit/secrets.toml
   ## Make edits on .streamlit/secrets.toml
   ## Customise other files in .streamlit/ directory to influence Streamlit behaviour
   ```
-
 - TIP: Make sure `.streamlit` directory is in the `.gitignore` file so it isn't checked in with secrets.
-
 - Run the application and it will be available at http://localhost:8501
-
   ```sh
   poe run
   ```
-
 - To change the port number
-
   ```sh
   poe run --port PORT
   ```
@@ -105,8 +91,9 @@ poe test-integration
 
 This runs the `poe run` task above with the `opentelemetry-instrument` command which auto instruments several libraries with tracing and sends traces to the configured exporter. Honeycomb env vars see the `/misc/secrets.toml.template`
 
-- `poe run-otel`
-  
+```sh
+poe run-otel
+```
 
 ### Run doc site locally
 
@@ -122,24 +109,25 @@ This runs the `poe run` task above with the `opentelemetry-instrument` command w
 
 ### Build a docker image for tests
 
-- `poe docker-build --target test --build-tag 3.10-alpine --test true`
-- `poe docker-run --target test`
+```sh
+poe docker-build --target test --build-tag 3.10-alpine --test true
+```
+
+```sh
+poe docker-run --target test
+```
   
 ### Build a container image to run the root files only without running any test
 
 - Build image
-
   ```sh
   poe docker-build
   ```
-
 - Set env vars
-
   ```sh
   cp misc/docker.env.template docker.env
   ## Make edits on docker.env
   ```
-
 - TIP: Make sure `docker.env` is in the `.gitignore` file so it isn't checked in with secrets. It's also safer if you only have the environment variable name for secrets and set the value in your shell with `export`.
 
 ### Build a container image with cached dependencies

docs/index.md

@@ -6,7 +6,7 @@ Think of Docq as **your employees' private and secure ChatGPT, a second-brain th
 
 Docq is Open Source Software (OSS AGPLv3) that is self-hosted "on-premises" within your cloud provider account (Azure/AWS/GCP). That means all systems, including machine learning models and data, lives within your boundary of control.
 
-For the developers and other tech-savvy audience, Docq is like **WordPress for generative AI**. Out-of-the-box you get a private ChatGPT and a platform that can be customised and built upon using a plugin system and APIs (coming soon).
+For the developers and other tech-savvy audience, Docq is like **WordPress for generative AI**. Out-of-the-box you get a private ChatGPT and a platform that can be customised and built upon using a plugin system (see [extensions](./developer-guide/extensions.md)) and APIs (coming soon).
 
 ## Learn More
 

misc/docq-extensions.json.template

@@ -0,0 +1,7 @@
+{
+  "docq_extensions": {
+    "name": "friendly name",
+    "module_name": "fully qualified module name ref",
+    "source": "path to module *.py file. relative to 'docq' repo root"
+  }
+}

mkdocs.yaml

@@ -9,7 +9,7 @@ repo_url: https://github.com/docqai/docq
 
 plugins:
   - search
-  - mkapi
+  # - mkapi # doesn't work with python 3.11. project not maintained
   - awesome-pages
   - social
   - gen-files:
@@ -103,6 +103,7 @@ nav:
       - "Getting Started": developer-guide/getting-started.md
       - "Architecture": developer-guide/architecture.md
       - "Contributing": developer-guide/contributing.md
+      - "Extensions": developer-guide/extensions.md
       - "FAQ": developer-guide/faq.md
   #- ... | glob=readme.md
   #- ... | regex=scenarios/.+.md