6 ms·

Just skimmed the PR, I'm sure the author knows more than I - but why hard code a date at all? Why not do something like `today + 1 year`?

• whynotmaybe 5mo ago
  Because it should be `today + 1 year + randomInt(1,42) days`. Always include some randomness in test values.
  • andai 5mo ago
    Interesting, haven't heard this before (I don't know much about testing). Is this kind of like fuzzing?
    • whynotmaybe 5mo ago
      I recently had race condition that made tests randomly fail because one test created "data_1" and another test also created "data_1". - Test 1 -> set data_1 with value 1 - Test 1 -> `do some magic` - Test 1 -> assert value 1 + magic = expected value - Test 2 -> set data_1 with value 2 But this can fail if `do some magic` is slow and Test 2 starts before Test 1 asserts. So I can either stop parallelism, but in real life parallelism exists, or ensure that each test as random id, just like it would happen in real life.
  • devin 5mo ago
    Are you joking? This is the kind of thing that leads to flaky tests. I was always counseled against the use of randomness in my tests, unless we're talking generative testing like quickcheck.
    • whynotmaybe 5mo ago
      `today` is random.
      • Eldt 5mo ago
        It's dynamic, but it certainly isn't random, considering it follows a consistent sequence
      • InsideOutSanta 5mo ago
        If "today" were random, our universe would be pretty fricken weird.
        • devin 5mo ago
          What is today right now in Australia? How about where you live? You have not thought enough about what you’re saying and are probably not aware of all the weird time issues we have in our world.
          • InsideOutSanta 5mo ago
            That's not what "random" means.
    • dathinab 5mo ago
      or, maybe, there is something hugely wrong with your code, review pipeline or tests if adding randomness to unit test values makes your tests flaky and this is a good way to find it
      • devin 5mo ago
        or, maybe, it signals insufficient thought about the boundary conditions that should or shouldn't trigger test failures. doing random things to hopefully get a failure is fine if there's an actual purpose to it, but putting random values all over the place in the hopes it reveals a problem in your CI pipeline or something seems like a real weak reason to do it.
        • tomjakubowski 5mo ago
          I don't think anyone is advocating for random application of randomness.
  • rcxdude 5mo ago
    Not a good idea for CI tests. It will just make things flaky and gum up your PR/release process. Randomness or any form of nondeterminism should be in a different set of fuzzing tests (if you must use an RNG, a deterministic one is fine for CI).
    • whynotmaybe 5mo ago
      That's why it's "randomInt(1,42)", not "randomLong()".
    • dathinab 5mo ago
      if it makes thing flaky then it actually is a huge success because it found a bug you overlooked in both impl. and tests at least iff we speak about unit tests
      • jstanley 5mo ago
        Only if it becomes obvious why it is flaky. If it's just sometimes broken but really hard to reproduce then it just gets piled on to the background level of flakiness and never gets fixed.
        • nomel 5mo ago
          To get around this, I have it log the relevant inputs, so it can be reproduced. The whole concept of allowing a flaky unit test to exist is wild and dangerous to me. It makes a culture of ignoring real failures in what, should be, deterministic code.
          • marcosdumay 5mo ago
            Well, if people can't reproduce the failures, people won't fix them. So, yes, logging the inputs is extremely important. So is minimizing any IO dependency in your tests. But then that runs against another important rule, that integration tests should test the entire system, IO included. So, your error handling must always log very clearly the cause of any IO error it finds.
  • CoastalCoder 5mo ago
    > Always include some randomness in test values. If this isn't a joke, I'd be very interested in the reasoning behind that statement, and whether or not there are some qualifications on when it applies.
    • whynotmaybe 5mo ago
      Must be some Mandela effect about some TDD documentation I read a long time ago. If you test math_add(1,2) and it returns 3, you don't know if the code does `return 3` or `return x+y`. It seems I might need to revise my view.
      • Izkata 5mo ago
        I vaguely remember the same advice, it's pretty old. How you use the randomness is test specific, for example in math_add() it'd be something like: jitter = random(5) assertEqual(3 + jitter, math_add(1, 2 + jitter)) If it was math_multiply(), then adding the jitter would fail - that would have to be multiplied in. Nowadays I think this would be done with fuzzing/constraint tests, where you define "this relation must hold true" in a more structured way so the framework can choose random values, test more at once, and give better failure messages.
        • whynotmaybe 5mo ago
          > it's pretty old. Damn, must be why only white hair is growing on my head now. >Nowadays I think this would be done with fuzzing/constraint tests, where you define "this relation must hold true" in a more structured way so the framework can choose random values, test more at once, and give better failure messages. So the concept of random is still there but expressed differently ? (= Am I partially right ?)
          • Izkata 5mo ago
            Yes, the randomness is still there but less manually specified by the developer. But also I haven't actually used it myself but had seen stuff on it before, so I had the wrong term: it's "property-based testing" you want to look for. Here's an example with a python library: https://hypothesis.readthedocs.io/en/latest/tutorial/introduction.html#testing-a-sorting-algorithm https://hypothesis.readthedocs.io/en/latest/tutorial/introdu... The strategy "st.lists(st.integers())" generates a random list of integers that get passed into the test function. And also this page says by default tests would be run (up to) 100 times: https://hypothesis.readthedocs.io/en/latest/tutorial/settings.html https://hypothesis.readthedocs.io/en/latest/tutorial/setting... So I'm thinking... (not tested) @given(st.integers(), st.integers()) def test_math_add(a, b): assert a + b == math_add(a, b) ...which is of course a little silly, but math_add() is a bit of a silly function anyway.
  • zelos 5mo ago
    Generate fuzz tests using random values with a fixed seed, sure, but using random values in tests that run on CI seems like a recipe for hard-to-reproduce flaky builds unless you have really good logging.
• johanvts 5mo ago
  That introduces dependency of a clock which might be undesirable, just had a similar problem where i also went for hardcoding for that reason.
  • cogman10 5mo ago
    There's already a clock dependency. The test fails because of that.
  • rcxdude 5mo ago
    Arguably you should have a fixed start date for any given test, but time is quite hard to abstract out like that (there's enough time APIs you'd want OS support, but linux for example doesn't support clock namespaces for the realtime clock, only a few monotonic clocks)
• CodesInChaos 5mo ago
  That can easily lead to breaking tests due to time-zones, daylight saving time or the variable length of months. We experienced several of those over the years, and generally it was the test that was wrong, not the code it was testing. For example, this simplified test hits several of those pitfalls: var expected = start.AddMonths(1); var actual = start.ToLocal().AddMonths(1).ToUtc(); Assert(expected == actual);
  • hn_throwaway_99 5mo ago
    I mean, sure, that can happen, but that obviously depends on what the test is testing, it's not like it's bad in all cases to say "now plus 1 year". In the case in question it's really just "cookie is far enough in the future so it hasn't expired", so "expire X years in the future from now" is fine.