6 ms·

With unit tests you gotta be careful though, oftentimes LLMs skip implementations with mockups that just say "not implemented yet" or similar and then the unit

With unit tests you gotta be careful though, oftentimes LLMs skip implementations with mockups that just say "not implemented yet" or similar and then the unit tests become pointless because they start to only test internal structures for being set / not default values. For me it helped a lot to try to make containerized end-to-end tests and a custom TestMain for this, where I am using podman to run the integration tests. This way the end-to-end tests are forced to be on network level, and you can test protocol and API quirks much easier with LLMs. Also, never forget to write a bootstrapping docs/ folder so that you don't have to re-explain these things all the time.

aktau 1mo ago
+1, I like to test in a similar way. For example if I'm making a CLI, the test will spawn the CLI for each test-case, instead of invoking the code directly. This provides a more realistic flow, and makes it clear which user journeys one is supporting. Faking responses I often do with environment variables, like: ts := httptest.NewServer(...) cmd := exec.Command(...) cmd.Env = append(os.Environ(), fmt.Sprintf("MYCLI_PROD_ADDR=%s", ts.URL)) Of course, it's better still to go down this turtle stack (e.g. by spawning a local instance of your backend server instead of some faked handlers), but that adds more cost. I find the trade-off OK here.
- cookiengineer 1mo ago
  I actually had to use a similar hack there due to the limitation that go test compilates cannot spawn themselves where I needed to have an environment variable with the actual binary prebuilt before the tests run. Took me a while to understand that TestMain doesn't cover that use case...
  - aktau 1mo ago
    On Linux, "self" is /proc/self/exe. In general, I've seen people use `os.Args[0]`. But on checking again, I see there's even `os.Executable()` (https://pkg.go.dev/os#Executable https://pkg.go.dev/os#Executable) for this purpose. I'm quite sure go test compilates can spawn themselves, I'm doing it on many platforms. But, the test setup I was referring to was explicitly not that: the tests are spawning the main binary, not themselves. Using bazel+runfiles this is pretty easy to do. With the pure Go build tool, I'm not sure what approach I'd use to "guarantee" that I get a binary build for the same environment as the test.
    - deleted 1mo ago
      [deleted]

aktau 1mo ago

+1, I like to test in a similar way. For example if I'm making a CLI, the test will spawn the CLI for each test-case, instead of invoking the code directly. This provides a more realistic flow, and makes it clear which user journeys one is supporting. Faking responses I often do with environment variables, like: ts := httptest.NewServer(...) cmd := exec.Command(...) cmd.Env = append(os.Environ(), fmt.Sprintf("MYCLI_PROD_ADDR=%s", ts.URL)) Of course, it's better still to go down this turtle stack (e.g. by spawning a local instance of your backend server instead of some faked handlers), but that adds more cost. I find the trade-off OK here.

cookiengineer 1mo ago
I actually had to use a similar hack there due to the limitation that go test compilates cannot spawn themselves where I needed to have an environment variable with the actual binary prebuilt before the tests run. Took me a while to understand that TestMain doesn't cover that use case...
- aktau 1mo ago
  On Linux, "self" is /proc/self/exe. In general, I've seen people use `os.Args[0]`. But on checking again, I see there's even `os.Executable()` (https://pkg.go.dev/os#Executable https://pkg.go.dev/os#Executable) for this purpose. I'm quite sure go test compilates can spawn themselves, I'm doing it on many platforms. But, the test setup I was referring to was explicitly not that: the tests are spawning the main binary, not themselves. Using bazel+runfiles this is pretty easy to do. With the pure Go build tool, I'm not sure what approach I'd use to "guarantee" that I get a binary build for the same environment as the test.
  - deleted 1mo ago
    [deleted]