Change exitcode-stdio-1.0 description to match current status quo

[thomasjm]

Hi, I'm hoping to solicit comments/design discussion on an idea to add a new test suite interface.

I'm the author of the Sandwich test framework, which provides a way of running and visualizing tests in a terminal UI interface. It would be nice to add first-class support for this way of working to tooling like Cabal, so you could type cabal test from your terminal and get dropped into the interface.

This is currently not possible because Cabal's existing test suite types don't allow the tests to use stdin; see the exitcode-stdio-1.0 documentation.

This PR proposes to add a new TestSuiteInterface constructor called TestSuiteInteractiveV10, which behaves just like TestSuiteExeV10 except for the restriction on stdin. I haven't filled in any other code and I'm not sure how hard the rest of the implementation would be.

Possible alternative: simply change the semantics of exitcode-stdio-1.0 to allow stdin when used in interactive settings. It might be painful to add a new test suite type, because most tests in the wild assume exitcode-stdio-1.0 (and some tools like hpack currently bake it in by default).

This PR was emboldened by @Mikolaj 's email to Haskell-Cafe :)

---

Please include the following checklist in your PR:

• Patches conform to the coding conventions.
• Any changes that could be relevant to users have been recorded in the changelog.
• The documentation has been updated, if necessary.

[Mikolaj]

Sandwich certainly looks cool and I see no reason not try supporting it better. AFAIK, the test types are an abandoned project. E.g., the detailed test type never got to the point of being usable (was it supposed, by chance, to introduce anything useful for Sandwich?).

I think you are asking the right questions. To be properly lazy, I'd rather lift the restriction on exitcode-stdio-1.0 than revisit the test types idea, if that can be done in a backward compatible manner. But I have no idea what the purpose of the restriction is. Ease of implementation? Some benefit to users? To other tools?

[thomasjm]

was it supposed, by chance, to introduce anything useful for Sandwich?

Nope! It predates Sandwich by many years.

But I have no idea what the purpose of the restriction is. Ease of implementation? Some benefit to users? To other tools?

I'm not sure, but having just delved into Stack I can report that Stack follows Cabal's lead here. I think the tricky part about allowing stdin is that it only makes sense in certain circumstances. You wouldn't want tests to try reading from stdin when running in a CI environment. It would only make sense when running tests in an "interactive" scenario like a terminal (but not when running multiple test suites in parallel/interleaved mode). There's also the question of whether your stdin is connected to a TTY or not. In short, I think the restriction probably exists in order to guarantee that test suites run to completion without trying to ask for input.

However, we've just landed a PR in Stack to relax the restriction! commercialhaskell/stack#5951 allows test access to stdin by default, and the original behavior can be brought back by passing --no-tests-allow-stdin. What do you think about a similar change for Cabal?

[ulysses4ever]

My 2¢: the feature makes sense, and I agree with Mikolaj that it's better not to multiply the number of test types, so extending exit-code-1.0 would be preferable. At the same time, it has potential of breaking someone's workflow. I am more of move-fast-break-things person, so I'd embrace this risk. Only, it'd be nice to have a flag either enable or disable the feature; probably, the former is safer, not sure if it satisfies the topic starter.

[Mikolaj]

We can't be outdone by stack! :D Let's add the flag to cabal as well. Can we play the older, more conservative brother, and have the flag in backward compatible state by default? What are the pros and cons? What are the failure modes?

E.g., when somebody runs all tests in parallel and one of them needs stdin, what error is the user going to get? And how would it be recorded in a CI log? And the other way around, if stdin is forbidden and somebody tries to run a test with Sandwitch, what message is going to appear?

[thomasjm]

Actually, after some further investigation it seems that the problem with Cabal is the opposite of the problem with Stack. cabal test already connects the terminal stdin to the test executable stdin. What it doesn't connect is the output streams! [0]. AFAICT Cabal's behavior is the following:

1. Run the test executable to completion while writing the stdout/stderr contents to a log file.
2. If it exits with code 0, indicate success and print the log file path.
3. Otherwise, indicate failure and print the stderr/stdout contents.

The actual proposal, then, would be to make cabal test behave more like stack test. In the particular case of an interactive, single test scenario, we should start the test executable with the output streams forwarded to the terminal. IMO this is better anyway, because many test suites print incremental output that can be good to look at as it proceeds. What do you all think?

---

[0]: As a result, if a test does a getLine, it blocks, and you can actually type characters in and hit enter and the test will receive them. (So Cabal appears not to be following the exitcode-stdio-1.0 contract!) If you try to use Sandwich, it starts the TUI but you don't see it because the output streams are going into the log file. But you can hit 'q' to exit it :)

[Mikolaj]

However, please consider test-show-details: direct (that's what I use in all my cabal.project.local files) and similar. How does that change the picture?

Regarding cabal not following the exitcode-stdio-1.0 contract, all the better for us, I suppose. Let's change the contract to match the status quo, if that's useful. In any case, let's get both in sync.

[fgaz]

I feel like cabal run is more suited to run interactive tests (or interactive anything). I'd personally make it so my testsuites runs noninteractively by default (so cabal test works well even in parallel) but can run interactively with a -i/--interactive flag, so that I can just cabal run my-test -- -i if needed.

Are there any disadvantages to that usage of run?

[thomasjm]

Thank you both for the suggestions, they both work! Both test-show-details: direct and cabal run allow you to start the Sandwich TUI successfully.

As you can probably tell, I'm not a heavy Cabal user. At this point there's a working solution I can put in the Sandwich docs, so I'm happy. My only remaining concern would be to make starting Sandwich as ergonomic/obvious as possible for Cabal users. I would love if --test-show-details=direct became the default. But I just saw in #7817 that this appears to be the plan anyway!

Let's change the contract to match the status quo, if that's useful.

I'd suggest maybe a text-only change?

Test interface "exitcode-stdio-1.0". The test-suite takes the form of an executable. It returns a zero exit code for success, non-zero for failure. The stdout and stderr channels may be logged. It takes no command line parameters and nothing on stdin.

could become

Test interface "exitcode-stdio-1.0". The test-suite takes the form of an executable. It returns a zero exit code for success, non-zero for failure. The stdout and stderr channels may be logged. Test tooling may pass command line arguments and/or connect the stdin channel to the test.

[Mikolaj]

I'd suggest maybe a text-only change?

Yes, please. :)

[ulysses4ever]

@thomasjm could you perhaps update the initial posting and the title of the issue to say that after discussion here you're seeking an improvement in documentation and which one, please?

[ulysses4ever]

@Mikolaj

However, please consider test-show-details: direct (that's what I use in all my cabal.project.local files)

Is this documented in the manual? I can't seem to find either test-show-details or direct there. If that's so, it looks like a pretty major omission and worth a ticket. There's Setup.hs test show-details but it doesn't list direct as a possible value.

[Mikolaj]

You are right. Definitely worth a ticket.

[thomasjm]

I've updated the PR title and added a commit for the documentation change we discussed.

[Mikolaj]

LGTM

[Mikolaj]

@thomasjm: I've given you Triage rights to the repo. Should you accept, you will be able to set the merge_me label that's going to merge this PR after 2 days of no activity. As soon as you are ready. If you see anything related that you'd like to fix or anything that would help consumers of Sandwich, don't hesitate to open tickets. :)

[thomasjm]

Will do, thanks!

[ulysses4ever]

@thomasjm we do merges via rebases, so our merge bot needs permission to update your branch, as indicated by this error message here:

The merge-queue pull request can't be update
Details: `Pull request can't be updated with latest base branch changes

Mergify needs the author permission to update the base branch of the pull request.
https://github.com/codedownio?type=source needs to authorize modification on its head branch.
err-code: F74AF`

Could you fix it? The way to switch is described here, I think.

[thomasjm]

Ah I don't have that checkbox since my fork is on an organization repo. I can re-open the PR from a personal repo if necessary. I just hit the rebase button myself so maybe that will work?

[ulysses4ever]

It may help but I doubt it, let's wait for updated mergify status. In the past people had to reopen PR in such cases.

[ulysses4ever]

Oh, but it did work! Thanks a lot for your contribution, @thomasjm!