Epic: Jan 0.6.0 that uses Cortex Platform

[Van-QA]

Motivation

Main Decisions:

• Integrate Cortex as an Experimental Backend
• Overall: Nitro -> Cortex
• Today: Jan Roadmap for v0.6, 0.6.1, -> path to Cortex
• Blog Post: share the roadmap with the community

v0.6 scope

• Cortex as additional Engine
• Existing feature improvements, bug fixes @ashley
• UI to introduce Cortex as experimental
• Roadmap for that incremental migration @van @louis

v0.6 out-of-scope

• Conversation threads in DB
• Migration from legacy to DB

Specs

• https://www.notion.so/jan-ai/Jan-0-5-2-with-Cortex-platform-extensions-5fa7b164ecd24a178ec4c2a406fccf7a

Tasklist

• feat: Remote APIs can fetch Model List #3374
• feat: Hub - Imported model will be default to Cortex engines #3375
• feat: Engine - New Cortex extensions supported with 3 new engines #3376
• feat: Thread - Support Cortex extensions and allow preserving thread/model settings to new threads #3377
• feat: Log - App.log should include and label log record per engines #3378
• feat: Local API server - Local API server start based on model engine #3379
• feat: Settings - Support new Cortex extension / 3 new engines with config #3380
• epic: V0.6.0 Migration Wizard #3325
• feat: Jan supports most llama.cpp params #3508

Discontinued

• feat: Hub - Jan should support Hugging Face Cortexso hub models with YAML #3373

Appendix

[louis-jan]

@marknguyen1302 to attach the specs here

[dan-homebrew]

@louis-jan @marknguyen1302 I'm adding the #3325 to this Tasklist

• we can discuss later if we don't need a migration wizard (if we will continue supporting legacy filesystem)

[louis-jan]

@dan-homebrew @marknguyen1302 I'm listing somepoints to consider / concerns here so we can address together

• Model prepopulation - One important point to consider when using the HF/Provider model is fetching (3373, 3374) feat: Remote APIs can fetch Model List #3374 (comment)
• Cortex process handling - For now, Cortex serves the chat function and also serves the server API. There are possible side effects such as:
  • Load model fail -> cortex process crashed -> server stopped working
  • Run Cortex on app load so the API server can always be on (on message send before)
  • Change thread settings (context length / ngl) / Stop / Delete a running model -> Unload instead of just kill as before
  • New UX on state update? E.g. Cortex stopped working -> Restart by process watchdog -> Running
• Need a better UX for running multiple models simultaneously
  • Loading so many models could break the user's machine
  • Unloading a model could affect a running server (or not, since sending a message could load the model again)
• Need a better UX for engine setup (the UI update should also work with our extension architecture but not direct users to many screens)
  • Llama.cpp is bundled by default, but not for others, also CUDA dependencies (should some how shared among engines)
• Models can be run through the API Server
  • Legacy models are routed to Nitro (even eventually route to cortex-cpp), new models are routed to Cortex. Is it a good UX to force Cortex API server to serve legacy nitro models?
• Pre-populate HF cortex models: there should be duplicated models where the 0.5.3 models list is updated. New branch / model repo to serve Jan hub only? So we can filter out which models could be pre-populated instead of everything.
• Preserve thread settings - confusing feature? since we will work on Presets soon, let's take it out?
• Sync settings (& dtos?) across projects (https://discord.com/channels/1107178041848909847/1239846009258119178/1279001259613093910)
• Cortex release process - e.g. We are updating the cortex repository's CI/CD for interim CLI development
• Cortex provides a backward-compatible FS module, so this means all requests from Jan go through the Cortex API? Back to the previous integration, there are so many edge cases to handle such as:
  • The Cortex server should be ON before showing threads/models. Data caching can improve the experiment for a better user experience, as users cannot wait for loading every time they reopen the app.
  • Cortex process status, as described above ^
  • Models import should also go thru cortex server
  • Deprecate core FS from Jan
  • Advanced Settings from Jan -> configure Cortex?
  • HTTP Proxy for model downloading?
  • HuggingFace API Token input from Jan -> Cortex
  • Logs settings from Jan > Cortex (on/off, cleaning interval..), how to log from Jan app -> cortex?
  • Tools support (Actually just PDF retrieval for now) - Depends on Jan core FS, if we don't deprecate that is fine
• IMPORTANT: Multiple instructions set support - Multiple engine binaries distribution

[marknguyen1302]

thanks @louis-jan for this. about Cortex provides a backward-compatible FS module, I think it's for the Jan API server only, so we keep the current app the same, instead of spawning a new port like the current Jan, we will use Cortex Platform API for that server. what do you think?

[marknguyen1302]

about Cortex process handling, I think when load/inference/start the Jan API server, we should check the health of the Cortex server and attempt to spawn if the server is not on.

[louis-jan]

Yes @marknguyen1302, scope down the other parts would help.
For now:

• I think just trying to sync the state between threads and the API server could be a good option when the Cortex process is terminated -> update the state from both places, and the user can either send a new message to try to load again or turn on the server.

But when we migrate full requests to Cortex endpoints, there should be something to addres.

[dan-homebrew]

@louis-jan Thanks for your points here - I'll likely need to break them up into separate discussions - it covers a lot of issues
#3365 (comment)

[0xSage]

to be rescoped into #3599

[dan-homebrew]

Discontinuing this due to Cortex Platform deprecation