Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add another download queue #111

Open
jrdh opened this issue Jul 18, 2023 · 0 comments
Open

Add another download queue #111

jrdh opened this issue Jul 18, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@jrdh
Copy link
Member

jrdh commented Jul 18, 2023

The ckanpackager had a slow and fast queue which had jobs added based on number of records (can't remember the exact threshold but something like <10,000 records goes on the fast queue, everything else is slow).

It would be good to add something like this/add a locking mechanism to the code so that we can run multiple workers (this would be my preference tbh, it's easier to scale the number of workers up and have them all handle the same kind of jobs than create some two-tiered worker system).

The "locking" (this could probably take different forms and doesn't have to be locks) would presumably just need to be around specific parts of the request, perhaps the query hash? I don't know if RQ has any of that kind of functionality built in, if not we'll have to think about how to do it.

This needs a bit of thought/investigation and then an implementation after.

@jrdh jrdh added the enhancement New feature or request label Jul 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

No branches or pull requests

1 participant