XUnit Test Fails When Run with Others, Passes When Run Alone. Arb.register() being Lost?

[WalkerCodeRanger]

I am using XUnit from C# with Visual Studio and the Visual Studio Test Runner. I was experiencing a very weird issue where it seemed a test inexplicably failed intermittently. I think I've pinned down the steps. I can now reproduce it consistently with a small test program on my machine.

Using the code below, running just ProblemTest alone causes the test to pass. Running all tests in the Example class causes it to fail. I've set the Replay value so it isn't just random variation. It appears the effect of Arb.Register<Example>(); is being lost. When it fails, the output shows that a CustomType created by another means is being passed (I assume the default generator?). See output below code. The debugger shows that the Arb.Register<Example>(); is still executed before the test. It seems to be very sensitive to changes. Changing the other test from [Theory] to fact makes the problem go away. Changing the namespace can make the problem go away. I have removed all other projects from the solution and all other code from this project.

The docs about Arb.register() don't make it clear where it should be called. I assumed registering once globally would be sufficient. The XUnit.CSharpExamples don't use Arb.register().

using FsCheck;
using FsCheck.Xunit;
using Xunit;

namespace Adamant.Tools
{
    public class CustomType
    {
        public const string MagicValue = "My Magic Value";
        public readonly string Value;

        public CustomType(string value)
        {
            Value = value;
        }
    }

    public class Example
    {
        static Example()
        {
            Arb.Register<Example>();
        }

        public static Arbitrary<CustomType> ArbitraryCustomType()
        {
            return Arb.From(Gen.Constant(new CustomType(CustomType.MagicValue)));
        }

        [Property(MaxTest = 1, Replay = "1318340931,296507323")]
        public Property ProblemTest(CustomType c)
        {
            return (CustomType.MagicValue == c.Value)
                .Label($"Value =\"{c.Value}\"");
        }

        // If this test is removed or is [Fact] the problem goes away
        [Theory]
        [InlineData("value")]
        public void PassingTest(string _)
        {
        }
    }
}

Message: FsCheck.Xunit.PropertyFailedException : 
Falsifiable, after 1 test (0 shrinks) (StdGen (1318340931,296507323)):
Label of failing property: Value ="|�5!_ND�2wm9[ �8�"
Original:
Adamant.Tools.Compiler.Bootstrap.Syntax.Tests.CustomType

[kurtschelfthout]

Relying on static initialisation in test runs is almost never a good idea, because most test runners load up a dll via reflection. In doing so, the "normal" static initialisation rules (which are already very complicated) do not always hold and so you get seemingly random behaviour. In your case it looks like the Arb.Register is not called early enough,

To make this more reliable, I suggest you pass the type with the Arbitrary on it explicitly in the attribute:

[Property(MaxTest = 1, Replay = "1318340931,296507323", Arbitrary = new[] { typeof(Example) }) ]

If you want to do this for all the tests in a class, you can use PropertiesAttribute instead on the class level. It has the same arguments, just wider scope.

[WalkerCodeRanger]

Given this fairly big pitfall, it seems like it would be better to not have Arb.Register. It seems like this has already been discussed some with @ploeh in #334 and #198. Consider this my vote for getting rid of the mutating registration.