Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read only configuration mode #215

Merged
merged 4 commits into from
Nov 6, 2023
Merged

Read only configuration mode #215

merged 4 commits into from
Nov 6, 2023

Conversation

bitner
Copy link
Collaborator

@bitner bitner commented Sep 25, 2023

Fixes #191

This adds a configuration parameter "readonly".

The "readonly" configuration mode prevents searches and context functions to store data in the _searches and _search_wheres table. Setting "readonly" prevents the lookup of searches using a search hash (as used in TiTiler-PGStac) and does not cache context lookups, so use of this with context turned on will be slow for paged results.

@bitner bitner requested a review from mmcfarland October 2, 2023 13:07
Copy link
Collaborator

@mmcfarland mmcfarland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. There are some important consequences to enabling this mode, consider adding a section to the readme/docs that explain its existence, why you might want it enabled, and what you can't do. As it relates to read replicas, the linked issue mentions that the stac-fastapi-pgstac implementation uses the read host for executing searches (which should use the write host). If that change was suitable to support read-replicas, what's the use case for a system setting that covers all connections?

I believe that titiler-pgstac just executes a search to "register" it, so that's why that function will no longer be possible. In future work, would it be possible for a specific write operation to hash and save to searches so that one could still register a search but still do a read-only search operation when consuming it?

@bitner
Copy link
Collaborator Author

bitner commented Oct 2, 2023

Looks good. There are some important consequences to enabling this mode, consider adding a section to the readme/docs that explain its existence, why you might want it enabled, and what you can't do.

As it relates to read replicas, the linked issue mentions that the stac-fastapi-pgstac implementation uses the read host for executing searches (which should use the write host). If that change was suitable to support read-replicas, what's the use case for a system setting that covers all connections?

I think that perhaps we have a bit of a misnomer that we are using in stac-fastapi-pgstac. When I think read vs write in the context of "read host" and "write host", I am thinking about the actual records themselves, so the "write host" is used with the transactions api and the "read host" is used for everything else. I've never really counted the statistics, cache, or registering of a search that get written to the database as "writes" from a stac perspective. It might make sense to make some changes in stac-fastapi-pgstac that allow for a flag to be used that indicates whether a search should go to the "read host" (which could be a read replica if the readonly setting was turned on) or to the configured "write host" or create yet a third host and make the setting names a bit more clear. This flag could then be set by titiler-pgstac only when registering a search.

I believe that titiler-pgstac just executes a search to "register" it, so that's why that function will no longer be possible. In future work, would it be possible for a specific write operation to hash and save to searches so that one could still register a search but still do a read-only search operation when consuming it?

Yes, it is only the registering of a search that would not be possible. It would be possible to direct the register only to the primary database and then all titiler-pgstac calls that use that hash to the read replica.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

STAC Server responding with SQL Create Error when using /collections/<id>/items with dedicated STAC Reader
2 participants