cabal run a test-suite in project could change working directory to that of package, as cabal test does

[jasagredo]

Running a test-suite inside a cabal project executes it in the current directory. However, testing a test-suite first changes working directory to that of the package and then runs the test-suite.

This is somewhat annoying because if a test depends on reading some files from the file-system, it might not find then (thus failing) if cabal run'ed but work if cabal test'ed.

I am proposing that cabal run a test-suite executable should emulate what cabal test does, this is: moving to the appropriate directory before running the executable.

An example of the (at least to me) annoying behaviour:

➜ tree -L 3
.
├── aa
│   ├── aa.cabal
│   ├── CHANGELOG.md
│   └── test
│       └── Main.hs
├── bb
│   ├── bb.cabal
│   ├── CHANGELOG.md
│   └── test
│       └── Main.hs
└── cabal.project

➜ cat ../aa/test/Main.hs
module Main (main) where

main :: IO ()
main = readFile "CHANGELOG.md" >>= putStr

➜ tail aa/CHANGELOG.md bb/CHANGELOG.md
==> aa/CHANGELOG.md <==
AAAAA

==> bb/CHANGELOG.md <==
BBBB

~\code\aa via λ 9.8.2
➜ cd bb

code\aa\bb via λ 9.8.2
➜ cabal test aa:aa-test --test-show-details=direct
Build profile: -w ghc-9.8.2 -O1
In order, the following will be built (use -v for more details):
 - aa-0.1.0.0 (test:aa-test) (ephemeral targets)
Preprocessing test suite 'aa-test' for aa-0.1.0.0..
Building test suite 'aa-test' for aa-0.1.0.0..
Running 1 test suites...
Test suite aa-test: RUNNING...
AAAAA
Test suite aa-test: PASS
Test suite logged to:
C:\Users\Javier\code\aa\dist-newstyle\build\x86_64-windows\ghc-9.8.2\aa-0.1.0.0\t\aa-test\test\aa-0.1.0.0-aa-test.log
1 of 1 test suites (1 of 1 test cases) passed.

code\aa\bb via λ 9.8.2
➜ cabal run aa:aa-test
Build profile: -w ghc-9.8.2 -O1
In order, the following will be built (use -v for more details):
 - aa-0.1.0.0 (test:aa-test) (additional components to build)
Preprocessing test suite 'aa-test' for aa-0.1.0.0..
Building test suite 'aa-test' for aa-0.1.0.0..
BBBB

[philderbeast]

#9923 may be related, closed in anticipation of the imminent cabal-install-3.12.* release that will be able to cabal run from the Cabal-tests subdirectory without a workaround.

@jasagredo is there a change in behaviour between cabal-install-3.10.3.0 and cabal-install built from HEAD, #9923 (comment)?

[mpickering]

It seems quite ad-hoc to me to do something different in cabal run because something happens to be a test executable.

[jasagredo]

@philderbeast No change with a cabal from master as of this Sunday (e1f73a4).

@mpickering Fair point. What if I propose that cabal run also moves to the directory of the package in the project in which the executable was defined, be it a executable, a benchmark or a test-suite, so not only for test-suites?

[geekosaur]

That's likely to be fairly user-visible, and I'd want to hear input from cabal-install users before making that change.

[michaelpj]

What if I propose that cabal run also moves to the directory of the package in the project in which the executable was defined, be it a executable, a benchmark or a test-suite, so not only for test-suites?

What about executables? There it would be quite odd to change directory, and quite probably not what the user wants at all. Consider, e.g. an executable that defines a code generator - you are quiet plausibly going to want to run that somewhere else, not in it's own package!

I wonder if we could ask a different question: why do people cabal run their test suites? I can think of a couple of reasons why I've done this in the past:

1. Cabal won't intercept the executable output in the way it does for test suites, which can be annoying
2. It's much easier to pass a bunch of arguments to the test executable using cabal run's --- trailing arguments syntax, versus cabal test's --test-options option (quoting!)
   • This is maybe a pretty easy fix: we could just add the trailing argument syntax to cabal test and cabal bench (but what about testing multiple components?)
3. More?

Perhaps if we can just fix those issues we can leave cabal run doing the straightforward thing.

[geekosaur]

Your 1 and 2 are the only reasons I've heard for cabal run-ing tests.

[michaelpj]

I think test-show-details: direct already fixes 1, anyway?

[geekosaur]

It does, but apparently people are annoyed by needing to say cabal test someTest --test-show-details=direct. I gather they don't want to put it in cabal.project because for most things, e.g. CI, they want the normal behavior; it's when trying to figure out why a test is failing that they need the extra noise.

Maybe a short option for specifically that option is the way to go. (I haven't checked; maybe -D is available?)

[michaelpj]

It's the default these days anyway: #8942

[jasagredo]

Indeed I use cabal run precisely because of the test arguments and how annoying is to get the quoting right. If cabal test would have the trailing slashes arguments, that would be awesome

[mpickering]

I agree with @michaelpj that I expect benchmarks and test to support -- like cabal run.

[jasagredo]

I'm fine with that. Let's close this issue and open a proper one that explicitly requests that feature.

[jasagredo]

FTR #10131