BUG: fetch_string_and_headers compat: raise in and out of pyodide

[Carreau]

Currently only the not_in_pyodide will raise on non-success, because this is the default behavior of urllib, the in_pyodide will not, so I added a raise_for_status.

It is better to raise, as otherwise the package parser will potentially get proper URL and not manage to parse it, and decide there is no wheels, while we actually just got an error (404, or maybe 500).

In addition wraps both case in a custom local HttpStatusError, so that we can actually catch these errors in the right places when we encounter them.

I think we could define HttpStatusError on the ABC and not make it an ABC itself, but it small enough that I think duplication is ok.

[Carreau]

Do I just need to throw a @run_in_pyodide here to make sure this is tested in and out of pyodide ? Or is that a bit more complicated ?

[hoodmane]

It's a bit more complicated. Honestly, the easiest thing might be just to copy paste the code. At some point we'll have to think about a better way...

[ryanking13]

Yep, testing something both in and out of Pyodide is a bit complex. Actually, I think you don't need to test out-of-pyodide case. We don't care about using micropip out of Pyodide and not_in_pyodide exists mostly for test purpose and typing.

[Carreau]

Moving to draft, I think HttpStatusError is not yet in latest released pyodide.

[Carreau]

Ok, I've added relevant fixes now that PyPI properly set CORS on 404s, and added conditional for wether we are on pyodide 0.26.x or 0.27+.

[Carreau]

SO now some of the fixture that return 500 now need to return 404... I'm not sure how to get the httpserver fixture to return 404 on wildcard url without expecting requests.

[ryanking13]

ValueError: Function test_404_raise's first argument name 'httpserver' should start with 'selenium'

@run_in_pyodide should be the inner-most decorator.

[ryanking13]

Let's release Pyodide 0.27.0a1 and use it in the micropip CI so we don't need to put complex version checks. Hood released 0.27.0a1 yesterday but the deploy was failed because of the CI timeout (pyodide/pyodide@5231ac0). Let me see if rerunning the job would work.

[Carreau]

Let's release Pyodide 0.27.0a1 and use it in the micropip CI

I was also just concern micropip might be broken with current pyodide/warehouse, but a new version of pyodide would be great :-)

[Carreau]

Ok, I've tried multiple variation of the test to @run_in_pyodide, but can't figure out how to actually have one that works.

[hoodmane]

Suggested change

	async def _inner_test_404_raise(httpserver):
	async def _inner_test_404_raise(selenium, httpserver):

[hoodmane]

Also the httpserver fixture isn't going to work with @run_in_pyodide. I'm not sure what the best way is to do this, @ryanking13 do you have any suggestions?

[ryanking13]

You can do this way (passing only string not httpserver fixture)

def test_404(httpserver):
    @run_in_pyodide
    async def _inner_test_404_raise(url):
        from ... import fetch_string_and_headers
        import pytest
        with pytest.raises(HttpStatusError):
            await fetch_string_and_headers(url, {})

    httpserver.expect_request("/404").respond_with_data(
        "Not found", status=404, content_type="text/plain"
    )
    url_404 = httpserver.url_for("/404")
    _inner_test_404_raise(url_404)

[Carreau]

Ah... let me try that, thanks for the pointers.

[hoodmane]

I released 0.27.0a2 instead.

[Carreau]

I released 0.27.0a2 instead.

BTW should CI be updated

- pyodide-version: [0.24.1, 0.25.0]
+ pyodide-version: [0.24.1, 0.25.0, 0.26.0, 0.27.0a2]

?

[Carreau]

Any idea on how to actually change the timeout to something else the 10 minutes... I've tried various options with no success.

[ryanking13]

Any idea on how to actually change the timeout to something else the 10 minutes... I've tried various options with no success.

I think this is a chrome (+selenium) issue.. for now I just pinned the version of the chrome to some older version to see if it pass. #135

[Carreau]

I think this is a chrome (+selenium) issue.. for now I just pinned the version of the chrome to some older version to see if it pass. #135

More generally when I write a test with an error the timeout is also 10 min, and I could not find how to make it timeout in less... but thanks for #135