"Failed to process query" errors

[rosnfeld]

Production repro case: "truthtelling" search seems to hang for a long time, and then brings up an error. What is going on?

[rosnfeld]

Looks like it is timing out..? The same query will often work if re-tried.

Indexes were rebuilt, and unfortunately not much has changed.

Concurrent users? Something about postgresql versions?

[rosnfeld]

Maybe the connection (rather than the query) is timing out and needs to be re-established. Once it's working, it seems to stay working. This is not something you would normally see in development unless you let it stay up for a long time.

[rosnfeld]

https://docs.djangoproject.com/en/1.6/ref/databases/ even says that the development server doesn't keep persistent connections, since a new thread handles each request.

[rosnfeld]

Unfortunately still happening.

[rosnfeld]

Given more experience with it, I don't think it's due to the database connections going stale. Current best guess is that the postgres instance is overloaded (probably shared with too many other hobby users, or a few 'hyperactive' ones) and the jQuery ajax call is timing out.

[rosnfeld]

Hopefully disabling jQuery timeout fixes this, if not I can always add debug logging to the jQuery ajax error handler.

[rosnfeld]

Sadly just saw this again.

[rosnfeld]

Now I suspect it's actually happening between the Heroku router and the web app:
https://devcenter.heroku.com/articles/request-timeout

The Heroku logs have various alerts with the text "[CRITICAL] WORKER TIMEOUT" in them.

Not exactly sure what the next course of action is.