Tests should ALWAYS be deterministic
I always debate when I get into a new team about this. The biggest argument for random data is that when you use it, you can say you are doing exploratory testing, but in the real world, if people get used to a test that randomly is failing, they will ignore it. In reality, you can’t trust this kind of test. It’s the same with test results that are hard to reproduce; they transform the test into an almost useless piece of code, that only produces noise in your test suite.
You can argue that if you have 5 parameters (you shouldn’t have so many), and each one is an object (you might do something wrong over there), you can’t test each combination. You are right, but do you need to do that? Is that method so important to get a very very high (more than 85%) code coverage?
As you guessed by reading the first 2 paragraphs (if you ignored the title), I really think all tests should be deterministic (basically have the same input and expected output, every time, on any kind of machine — if you run them locally, remote, in a pipeline, anywhere)!
I can give you two examples with non-deterministic tests, that I can remember (I could create some fictional problems, but it’s not the purpose of this article)
Example 1: I had a strange big algorithm which gave me a specific number. The test had random input. Sometimes, the test was failing, but due to the complexity of the algorithm, I couldn’t figure out why. After a few months, I managed to get some time to investigate the problem. At first, I thought it was something related to the pipeline, after that something related to the agent, after that with the OS of the agent, but in the end, after 4 days of trying to reproduce I managed to do that 9 times. I pushed it in the background, continued working on other things, but kept a closer look at it. After a few weeks, I managed to understand what was going on: if the random number (the input) was a number that could be divided into 3,5, and 23, then the test was red, if not, it was green. I was happy because, in the end, I caught a bug; the test data was hard to reproduce, but in the end, it did something good. Well…it didn’t. After I lost another 2 days with that data, I understood that even if the input got divided into those numbers, the algorithm did in fact give the right result, and the result of the test was just a false positive. (The strange big algorithm was meant to split high-resolution images into smaller ones, and it calculated the space needed for an image if that resulted image needed to be split again and so on).
Example 2: For the sake of simplicity let’s say I had some handmade tasks with some attributes that could expire. The condition of the expiration was something like if the expiration date is today throw an exception. Oh well, of course, it was used DateTime.Now everywhere; Who wrote the tests didn’t know that Date time can be mocked (as I described in a previous article that can be found here), so it is just used as input data, DateTime.Now.AddMonths(-5); At first look, nothing wrong. The test in theory was green all the time; Now, imagine if you are on the interval 29–31of July 2022. If you add -5 months in this interval and check if today’s day is equal to that result date, what will you get? A BIG false! In this interval of days, -5 months will give you 28 of February (since in 2022, February doesn’t have 29 days; and even if it had, you’d have gotten the same false response, on 30–31 of July).
A lot of more examples can be found (for instance with New Guid, which if you add it as a Correlation Id in orchestration in a durable function, you’ll make that part of the code execute an infinite number of times), but what I pinpoint is that, if you want accurate results out of your tests, and if you really want to rely on them, write deterministic tests, and stop fooling around.
