33 ms·

SafeTest: A novel approach to front end testing

79 points by kolodny 3y ago

Danielben 3y ago
[flagged]
greatjack613 3y ago
Not sure why this is fundamentally different from existing e2e testers with bootstrapping
- kolodny 3y ago
  Can you clarify what you mean? Usually, e2e testers don't have a bootstrapping stage for app-level changes, only for things that can be done via the browser automation APIs.
foolswisdom 3y ago
My initial impression of this is that it enables controlling just how much of the app you want to render for your test, while more traditional solutions force you towards either rendering the whole app or testing just a component. Is that accurate?
- kolodny 3y ago
  Yes, that's correct. You get all the benefits of react-testing-library mounting and the ability to test the entire app as a unit.
mdaniel 3y ago
how many billions of dollars and they still host their blog on a nagware site SMH It's not like moving would be high drama since they already have a custom domain for it https://scribe.rip/introducing-safetest-a-novel-approach-to-front-end-testing-37f9f88c152d https://scribe.rip/introducing-safetest-a-novel-approach-to-...
- mdaniel 3y ago
  it also has no license file, with https://github.com/kolodny/safetest/blob/main/package.json#L29 https://github.com/kolodny/safetest/blob/main/package.json#L... saying "MIT" but as far as I know it says that a lot and I doubt gravely that any court would consider a json file a legitimate license declaration
  - kolodny 3y ago
    Sorry about that. I just added an official LICENSE file https://github.com/kolodny/safetest/blob/main/LICENSE https://github.com/kolodny/safetest/blob/main/LICENSE
- joks 3y ago
  Is hitting the (x) on the banner that difficult?
  - mdaniel 3y ago
    Is having nagware for another company a good look for an engineering blog? > How we GraphQL all the things, after this brief message from our ... sponsor?
    - phantomathkg 3y ago
      Not OP, but apparently many companies still hosted their engineering blog on medium. So the answer to your question could be, yes and no, it depends. Pinterest: https://medium.com/pinterest-engineering https://medium.com/pinterest-engineering Workable: https://engineering.workable.com/ https://engineering.workable.com/ Skyscanner: https://medium.com/@SkyscannerEng https://medium.com/@SkyscannerEng Agoda: https://medium.com/@agoda.eng https://medium.com/@agoda.eng
- MajimasEyepatch 3y ago
  From the Hacker News Guidelines: "Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
zie 3y ago
I've always wanted to build a UX tester that uses accessibility tech. We get to test both at the same time, and it might make the accessibility tech saner and easier to use also.
drewcoo 3y ago
> Conversely, using integration testing tools like Cypress or Playwright provides control over the page, but sacrifices the ability to instrument the bootstrapping code for the app. These tools operate by remotely controlling a browser to visit a URL and interact with the page. I don't think the author has used Cypress or Playwright. Their real value is that they do not drive browsers from the outside like slow, flakey WebDriver. They also allow injecting test doubles. And to override app functionality (not having to wait for a "normal" 60 second timeout like the author hypothetically suggests). This test framework seems to solve non-problems. Is Netflix not quite so FAANG these days?
- kolodny 3y ago
  Author here. Can you show an example of how Playwright would progress a timer on a page? For example, how would you make this component pass faster than 60 seconds? export const Countdown = () => { const [time, setTime] = React.useState(60); React.useEffect(() => { const interval = setInterval(() => { setTime(t => t - 1); if (t === 1) clearInterval(interval); }, 1000); return () => clearInterval(interval) }, []); return time > 0 ? <>Time left {time}</> : <>Done</> }; Short of using Playwright Component Tests, this isn't possible. I also found an issue where the only solution was to inject sinon into the page which I wouldn't consider a great option: https://github.com/microsoft/playwright/issues/10230 https://github.com/microsoft/playwright/issues/10230
  - drewcoo 3y ago
    If you want to do it with the whole page and talk only to the local code, then yes, I'd recommend Sinon. I think that's a much simpler solution than . . . creating an all new NIH framework! I'd also recommend refactoring to a more mock-friendly way to do that countdown if you don't want to cover up all the internal logic. If the timeout interval is loaded remotely from some API (and it probably is if you have reasonably configurable nag popups), then you can always mock that API call.
    - kolodny 3y ago
bryanrasmussen 3y ago
Are there potential performance issues to consider with overrides - I'm supposing const FetchPerson = createOverride(fetchPerson); must be less performant by some amount than just using fetchPerson? How many over-rides do you find a reasonably complex component needs?
- kolodny 3y ago
  React is pretty performant when context isn't changing. We haven't done any benchmarking but I doubt there's any real world perf hit. For large applications the number of overrides tend to be under 20. Overrides are opt-in so you can just expose any overridable value as a prop and run a isolated component test on it.
  - bryanrasmussen 3y ago
    cool any common mistakes you see people make in writing overrides or is it too early to make conclusions?
willio58 3y ago
Lots of negativity in the comments but I think the solutions outlined really interesting and valuable! There are some idiosyncrasies around unit testing ui that are just hard to work around. And the issues with e2e tests in the context of testing individual UI pieces are totally valid too. The one thing that at least initially rubs me the wrong way is how the overrides work. Like I get why that's a solution to how we can inject some data, but I don't like the idea of writing test-specific code in a component just to enable tests with this tool. That being said, I've done similar things in the past when I've run out of options and this looks pretty clean, I just wonder if there's another way.
thom 3y ago
I think this is overselling itself a little. It seems genuinely useful to be able to do more expansive integration tests of components, and combinations of components, especially running in real browsers. But that’s only really an incremental gain over component unit tests with mocks. In the case where you own a very complex SPA with multiple back end dependencies you don’t control, then yeah, maybe this is all you need. But I don’t see how this can replace real end to end functional tests in most apps.
somewhereoutth 3y ago
I've said it before, and I'll say it again: Interfaces used by humans must be tested by humans. These can either be your staff, or your customers. Yes that does mean wading through each interaction pathway every time you make a release, there really is no substitute.
- technion 3y ago
  I think netflix is at a scale where you can question what I'm saying below, but every frontend test suit I've seen is so full of mocks you basically only test your own test suite. And then common front end bugs like "renders off the screen" or "doesnt work in safari" aren't caught anyway. I hugely support tests and I write a lot more of them than most people. I just don't think it usually works on the frontend.
  - nosefurhairdo 3y ago
    I work on fairly complex, dashboard-y, web app UI used by many fortune 100 companies. I can count on one hand the number of times the huge test suite I inherited has prevented bugs entering production. I've spent at least 40 hours debugging false positives and config issues, and about half of our deployment time is spent running these tests. From what I can tell, there are two types of tests that are worthwhile in UI: unit tests on functions (not UI elements), and basic integration tests. I believe it's possible to write valuable tests that don't fit into this framework, but from what I can tell every other UI engineer feels compelled to write tests that just regurgitate component implementation details.
s4i 3y ago
Needing to change the application code to mock things rubs me the wrong way. I 100% do agree with the frontend testing challenges they lay out in the beginning of the post though.
- 5Qn8mNbc2FNCiVV 3y ago
  No different than any other dependency injection

79 points by kolodny 3y ago

Danielben 3y ago

[flagged]

greatjack613 3y ago

Not sure why this is fundamentally different from existing e2e testers with bootstrapping

kolodny 3y ago
Can you clarify what you mean? Usually, e2e testers don't have a bootstrapping stage for app-level changes, only for things that can be done via the browser automation APIs.

foolswisdom 3y ago

My initial impression of this is that it enables controlling just how much of the app you want to render for your test, while more traditional solutions force you towards either rendering the whole app or testing just a component. Is that accurate?

kolodny 3y ago
Yes, that's correct. You get all the benefits of react-testing-library mounting and the ability to test the entire app as a unit.

mdaniel 3y ago

how many billions of dollars and they still host their blog on a nagware site SMH It's not like moving would be high drama since they already have a custom domain for it https://scribe.rip/introducing-safetest-a-novel-approach-to-front-end-testing-37f9f88c152d https://scribe.rip/introducing-safetest-a-novel-approach-to-...

mdaniel 3y ago
it also has no license file, with https://github.com/kolodny/safetest/blob/main/package.json#L29 https://github.com/kolodny/safetest/blob/main/package.json#L... saying "MIT" but as far as I know it says that a lot and I doubt gravely that any court would consider a json file a legitimate license declaration
- kolodny 3y ago
  Sorry about that. I just added an official LICENSE file https://github.com/kolodny/safetest/blob/main/LICENSE https://github.com/kolodny/safetest/blob/main/LICENSE
joks 3y ago
Is hitting the (x) on the banner that difficult?
- mdaniel 3y ago
  Is having nagware for another company a good look for an engineering blog? > How we GraphQL all the things, after this brief message from our ... sponsor?
  - phantomathkg 3y ago
    Not OP, but apparently many companies still hosted their engineering blog on medium. So the answer to your question could be, yes and no, it depends. Pinterest: https://medium.com/pinterest-engineering https://medium.com/pinterest-engineering Workable: https://engineering.workable.com/ https://engineering.workable.com/ Skyscanner: https://medium.com/@SkyscannerEng https://medium.com/@SkyscannerEng Agoda: https://medium.com/@agoda.eng https://medium.com/@agoda.eng
MajimasEyepatch 3y ago
From the Hacker News Guidelines: "Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

zie 3y ago

I've always wanted to build a UX tester that uses accessibility tech. We get to test both at the same time, and it might make the accessibility tech saner and easier to use also.

drewcoo 3y ago

> Conversely, using integration testing tools like Cypress or Playwright provides control over the page, but sacrifices the ability to instrument the bootstrapping code for the app. These tools operate by remotely controlling a browser to visit a URL and interact with the page. I don't think the author has used Cypress or Playwright. Their real value is that they do not drive browsers from the outside like slow, flakey WebDriver. They also allow injecting test doubles. And to override app functionality (not having to wait for a "normal" 60 second timeout like the author hypothetically suggests). This test framework seems to solve non-problems. Is Netflix not quite so FAANG these days?

kolodny 3y ago
Author here. Can you show an example of how Playwright would progress a timer on a page? For example, how would you make this component pass faster than 60 seconds? export const Countdown = () => { const [time, setTime] = React.useState(60); React.useEffect(() => { const interval = setInterval(() => { setTime(t => t - 1); if (t === 1) clearInterval(interval); }, 1000); return () => clearInterval(interval) }, []); return time > 0 ? <>Time left {time}</> : <>Done</> }; Short of using Playwright Component Tests, this isn't possible. I also found an issue where the only solution was to inject sinon into the page which I wouldn't consider a great option: https://github.com/microsoft/playwright/issues/10230 https://github.com/microsoft/playwright/issues/10230
- drewcoo 3y ago
  If you want to do it with the whole page and talk only to the local code, then yes, I'd recommend Sinon. I think that's a much simpler solution than . . . creating an all new NIH framework! I'd also recommend refactoring to a more mock-friendly way to do that countdown if you don't want to cover up all the internal logic. If the timeout interval is loaded remotely from some API (and it probably is if you have reasonably configurable nag popups), then you can always mock that API call.
  - kolodny 3y ago

bryanrasmussen 3y ago

Are there potential performance issues to consider with overrides - I'm supposing const FetchPerson = createOverride(fetchPerson); must be less performant by some amount than just using fetchPerson? How many over-rides do you find a reasonably complex component needs?

kolodny 3y ago
React is pretty performant when context isn't changing. We haven't done any benchmarking but I doubt there's any real world perf hit. For large applications the number of overrides tend to be under 20. Overrides are opt-in so you can just expose any overridable value as a prop and run a isolated component test on it.
- bryanrasmussen 3y ago
  cool any common mistakes you see people make in writing overrides or is it too early to make conclusions?

willio58 3y ago

Lots of negativity in the comments but I think the solutions outlined really interesting and valuable! There are some idiosyncrasies around unit testing ui that are just hard to work around. And the issues with e2e tests in the context of testing individual UI pieces are totally valid too. The one thing that at least initially rubs me the wrong way is how the overrides work. Like I get why that's a solution to how we can inject some data, but I don't like the idea of writing test-specific code in a component just to enable tests with this tool. That being said, I've done similar things in the past when I've run out of options and this looks pretty clean, I just wonder if there's another way.

thom 3y ago

I think this is overselling itself a little. It seems genuinely useful to be able to do more expansive integration tests of components, and combinations of components, especially running in real browsers. But that’s only really an incremental gain over component unit tests with mocks. In the case where you own a very complex SPA with multiple back end dependencies you don’t control, then yeah, maybe this is all you need. But I don’t see how this can replace real end to end functional tests in most apps.

somewhereoutth 3y ago

I've said it before, and I'll say it again: Interfaces used by humans must be tested by humans. These can either be your staff, or your customers. Yes that does mean wading through each interaction pathway every time you make a release, there really is no substitute.

technion 3y ago
I think netflix is at a scale where you can question what I'm saying below, but every frontend test suit I've seen is so full of mocks you basically only test your own test suite. And then common front end bugs like "renders off the screen" or "doesnt work in safari" aren't caught anyway. I hugely support tests and I write a lot more of them than most people. I just don't think it usually works on the frontend.
- nosefurhairdo 3y ago
  I work on fairly complex, dashboard-y, web app UI used by many fortune 100 companies. I can count on one hand the number of times the huge test suite I inherited has prevented bugs entering production. I've spent at least 40 hours debugging false positives and config issues, and about half of our deployment time is spent running these tests. From what I can tell, there are two types of tests that are worthwhile in UI: unit tests on functions (not UI elements), and basic integration tests. I believe it's possible to write valuable tests that don't fit into this framework, but from what I can tell every other UI engineer feels compelled to write tests that just regurgitate component implementation details.

s4i 3y ago

Needing to change the application code to mock things rubs me the wrong way. I 100% do agree with the frontend testing challenges they lay out in the beginning of the post though.

5Qn8mNbc2FNCiVV 3y ago
No different than any other dependency injection