Thanks for filing this issue. Could one of you take a look please, @mattonem and @theseion?

@Rinzwind Why is your expectation that the message is only shown once? Resources are set up and torn down for every test suite and Dr. Test constructs one suite per test class. Ergo, you should see the message as many times as there are test classes that use the resource.

Because I expected it to use a single suite like #runClasses:named: for HDTestReport does do (at least in Pharo 13 build 225). That ignores overrides of #suiteClass though. In #runClasses:spec: for SCIPharoTestRunner the resources could be combined so that they only need to be set up once, while still keeping each class’s #suiteClass into account, by replacing the ‘do’ over the classes with the following:

resources := OrderedCollection new.
classes do: [ :class |
	| classSuite |
	classSuite := class suite.
	resources addAll: (classSuite resources reject: [ :resource | resources includes: resource ]).
	classSuite resources: #().
	suite addTest: classSuite ].
suite resources: resources.

That will prevent overrides of #suiteClass from working. I don't feel that this is an issue with SmalltalkCI but with SUnit in Pharo. TestSuite>>#tearDown sends #resetResources to TestResource, which resets all the resources of the suite. This completely defeats the idea that suites can be nested. IMO, nested suites must know that they are nested and leave tear down to the outer suite (this already works for #setUp, because TestResource>>#isAvailable returns true after the outer suite has run initialization).

I don’t quite follow how it would prevent overrides of #suiteClass from working. I tried replacing the ‘do’ in #runClasses:spec: for SCIPharoTestRunner. That allowed #testCustomTestSuite to still pass while causing the message of the test resource in my example to be shown only once instead of five times. I do agree though the framework should probably better take into account that suites can be composed when it comes to resources.

You're right, it does work with the code you posted, sorry. I had simply patched the code to mimic what HDTestReport does, which is to construct a single suite. I'll look into patching SmalltalkCI. Would you mind raising the issue with Pharo? Or would you rather I do that?

I opened Pharo issue #17113 about it.

I've opened a PR (#662). I'd appreciate your feedback @Rinzwind.

I added my feedback to the pull request.