Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve handling endpoint rate limits #2230

Open
3 tasks done
livthomas opened this issue Jan 29, 2024 · 0 comments
Open
3 tasks done

Improve handling endpoint rate limits #2230

livthomas opened this issue Jan 29, 2024 · 0 comments

Comments

@livthomas
Copy link

Prerequisites

  • Are you running the latest version(s)?
  • Have you searched the documentation for your issue?
  • Did you search for an existing issue or pull request?

Description

Having written my own indexer before, I must say that SubQuery is very aggressive with endpoints. I'm using free tiers of 6 different RPC providers which are quite generous in both monthly quota and RPS. But I still hardly see anything expect for rate limit errors in the SubQuery logs. I have also received emails from multiple providers saying that most of my requests were rate limited and I should do something about it. And I only ran this app once for a few minutes. (It is an Ethereum app that uses only event handlers.)

I would very much appreciate if SubQuery respected rate limits. I would propose a three step solution you could gradually implement, starting from the simplest one:

  1. Add an option to configure rate limit globally across endpoints.
  2. Enable users to specify rate limits per endpoint as @stwiname suggested in Improve multiple endpoints configuration #1925.
  3. Detect endpoint rate limits automatically, then respect them in all requests but test it again once in a while.

Details

These details can help to reproduce the environment the issue is occurring

Local Environment: @subql/cli/4.2.5 darwin-arm64 node-v18.18.2
Query Version: @subql/[email protected]
Indexer Version: @subql/[email protected]
Network Details:

  • Ethereum Goerli testnet

Steps to Reproduce

  1. Use multiple free-tier endpoints.
  2. Add a few event handlers.
  3. Run the indexer.

Expected behavior: There should be almost no rate limit errors in the logs. SubQuery should know about these limits and respect them.

Actual behavior: Logs are full of rate limit errors. There is barely anything else in there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant