Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"core" filter removes QQBrowser usage #56

Open
jByrneSpringer opened this issue Dec 14, 2022 · 1 comment
Open

"core" filter removes QQBrowser usage #56

jByrneSpringer opened this issue Dec 14, 2022 · 1 comment

Comments

@jByrneSpringer
Copy link

QQBrowser is a browser used in China. An example user agent field will be "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3641.400 QQBrowser/10.4.3284.400"

Do you have a list of robots targeted by the "core" field?

@mdio
Copy link

mdio commented May 30, 2023

In addition, the smartphone Crosscall Core X4 is marked as a crawler due to this:

Mozilla/5.0 (Linux; Android 10; Core-X4 Build/QKQ1.200407.002; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/113.0.5672.76 Mobile Safari/537.36

According to https://www.robotstxt.org/db/core.html this is/was a robot developed by Minho Univeristy in Portugal in 1995.
I think it's safe to assume that it is no longer in existence, but I've messaged one of the authors to confirm.
If it's still around I think the string "core" should at least be made more specific to avoid false positives.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants