TestRandomChains fails randomly

[paulirwin]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

This has been split out from #271.

TestRandomChains' tests will rarely, randomly fail even with a fixed seed, indicating that there is a likely concurrency issue with (at least) one of the token stream components generated by MockRandomAnalyzer. The failure does not seem to be related to the root cause of #271.

Need to investigate and create reproducible, isolated test scenarios and fix the underlying failures. Adding a [Repeat] attribute to the tests can sometimes help reproduce.

Expected Behavior

The tests reliably pass.

Steps To Reproduce

No response

Exceptions (if any)

No response

Lucene.NET Version

4.8.0-beta00017

.NET Version

No response

Operating System

No response

Anything else?

No response

[NightOwl888]

I got this to repeat reliably. But, currently the [Repeat] attribute has not been customized to reset the Randomizer instance for each repeated test execution. It continues using the same instance from the last execution. So, if it takes 323 repeats to cause a failure on one run with a given seed, it will continue to take 323 repeats to execute if that same seed is reapplied.

Here is a case that fails reliably with [Repeat(1000)]:

Option 1:

 Apply the following assembly-level attributes:

[assembly: Lucene.Net.Util.RandomSeed("0xb78a94b3d0d07ab5")]
[assembly: NUnit.Framework.SetCulture("ca")]

Option 2:

 Use the following .runsettings file:

<RunSettings>
  <TestRunParameters>
    <Parameter name="tests:seed" value="0xb78a94b3d0d07ab5" />
    <Parameter name="tests:culture" value="ca" />
  </TestRunParameters>
</RunSettings>

Option 3:

 Create the following lucene.testsettings.json file somewhere between the test assembly and the root of your drive:

{
  "tests": {
     "seed": "0xb78a94b3d0d07ab5",
     "culture": "ca"
  }
}

After about 5-6 minutes, it will always fail with the same stack trace:

  Stack Trace: 
OffsetAttribute.SetOffset(Int32 startOffset, Int32 endOffset) line 50
ShingleFilter.IncrementToken() line 369
ValidatingTokenFilter.IncrementToken() line 86
BaseTokenStreamTestCase.CheckAnalysisConsistency(Random random, Analyzer a, Boolean useCharFilter, String text, Boolean offsetsAreCorrect, Field field) line 956
BaseTokenStreamTestCase.CheckRandomData(Random random, Analyzer a, Int32 iterations, Int32 maxWordLength, Boolean useCharFilter, Boolean simple, Boolean offsetsAreCorrect, RandomIndexWriter iw) line 840
BaseTokenStreamTestCase.CheckRandomData(Random random, Analyzer a, Int32 iterations, Int32 maxWordLength, Boolean simple, Boolean offsetsAreCorrect) line 694
TestRandomChains.TestRandomChains_() line 1171
InvokeStub_TestRandomChains.TestRandomChains_(Object, Object, IntPtr*)
MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

So, a better way to repeat (for now) is to run the tests in CI without a [Repeat] attribute. When a failure happens there, it will be queued up to fail the first time without the [Repeat] attribute.

I will check to see whether making a [Repeat] and/or a [FindFirstFailingSeed] attribute that resets the Random instance with a new seed will be easy to achieve. The Randomizer class already allows the Seed to be set without creating a new instance.

[NightOwl888]

I should point out that I got it to fail on the https://github.com/paulirwin/lucene.net/tree/issue/271 branch.