Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle categorical slots in command generator - [ENG 569] #12876

Closed

Conversation

varunshankar
Copy link
Contributor

@varunshankar varunshankar commented Sep 26, 2023

Proposed changes:

  • Skips slot set command if the LLM proposes a slot value that is not part of the categorical list of values defined in the domain
  • For each collection step in a workflow, the default pattern, namely "pattern_collect_information," is executed. The initial step defined within this pattern is the ActionRunSlotRejections action. I've implemented extra validations within this action to reset any invalid categorical values set by the LLM. This approach allows for the reuse of code responsible for fetching slot values and generating bot responses. Additionally, it ensures that user-defined rejections within the flow take precedence.

Among the potential solutions considered but not yet implemented are:

  • Adding a new default action to implement the categorical slot check as the initial step in the pattern_collect_information. This method doesn't leverage most of the existing code in ActionRunSlotRejections, and its necessity for introducing a new default action is uncertain.

  • Implementing the categorical slot check before the slotset event in the slot set command. The skip information for the slot is stored in a default slot, triggering the bot utterance step in the default pattern - pattern_collect_information. This approach, however, lacks computational efficiency.

Status (please check what you already did):

  • added some tests for the functionality
  • updated the documentation
  • updated the changelog (please check changelog for instructions)
  • reformat files using black (please check Readme for instructions)

@varunshankar varunshankar requested a review from a team as a code owner September 26, 2023 21:30
@varunshankar varunshankar self-assigned this Sep 26, 2023
Copy link
Contributor

@twerkmeister twerkmeister left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a small typing thing to take care of :)

rasa/dialogue_understanding/commands/set_slot_command.py Outdated Show resolved Hide resolved
@sonarcloud
Copy link

sonarcloud bot commented Oct 17, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug C 5 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot E 3 Security Hotspots
Code Smell A 53 Code Smells

0.0% 0.0% Coverage
0.9% 0.9% Duplication

warning The version of Java (11.0.20) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

Copy link
Contributor

@twerkmeister twerkmeister left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the idea here, I think this is close to a really good way of handling this issue 😎! A few things that I'd like to see improved:

  • I think the completely right approach is to have all type checks be a default rejection
    • don't just execute this type check rejection when no others are there. It should be always there
    • apply it here for all types, like floats and bools and categorial slots, not only the categorial slots. This means moving some code from the command generator that catches these cases for floats, bools, etc. here
    • The message should be something like "Sorry, it seems 'X' is not a valid 'Type'"
      • x being the value
      • type being something like "number" if the slot is float, "option" if bool and categorial
      • just first suggestions, feel free to improve upon this yourself. But I think it would be really nice to see the value and the slot this is about
  • Maybe you can also undo the big indentation change, it makes the diff really challenging to read

@varunshankar varunshankar deleted the ENG-569-handle-categorical-slots-llm-cmd-generator branch October 27, 2023 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants