14 ms·

The weirdest bug I've seen yet

619 points by jevans 3y ago

c0ndu17 3y ago
Honourable mention, “we can’t send emails more than 500 miles” https://web.mit.edu/jemorris/humor/500-miles https://web.mit.edu/jemorris/humor/500-miles
devilsAdv0cate 3y ago
[dead]
wobblyasp 3y ago
Such a tease. At least upload the gif so people can poke at it!
- yuck39 3y ago
  Agreed! I have absolutely no idea how a combination of grammarly and a specific gif would cause a browser crash though… Anyone here use the grammarly desktop app? Any additional clues?
  - hyperhello 3y ago
    I would guess the gif triggers a specific edge case that would crash, and grammarly is just a common enough addition to chrome with a lot of edge case triggers that it was identified. I’d guess this is on the Chrome team to fix soon, but yeah, at least post the spinner file for us!
  - fifafu 3y ago
    I‘d guess it’s because Grammarly enables full accessibility support in Chrome to be able to access the browser elements similar to e.g. a screen reader. This is off by default and has caused me various issues in the past when enabled (e.g. https://bugs.chromium.org/p/chromium/issues/detail?id=136448 https://bugs.chromium.org/p/chromium/issues/detail?id=136448... ). However it’s probably good that the Accessibility functions get more exposure due to this.
- deleted 3y ago
  [deleted]
12_throw_away 3y ago
Well, this is a fascinating murder mystery that establishes 3 compelling suspects - Grammarly, Chrome, and a gif - and then just ... ends, right before the big reveal.
- 0xNotMyAccount 3y ago
  I have a friend who worked at Gusto, and my wife tried using Gusto for her small business (they handle payroll for small business, got a big boost from the pandemic). The lack of technical resolution here is so Gusto, it hurts.
  - Andrex 3y ago
    They nuked my account after trying to charge an expired card three times. The customer response team was extremely quick and responsive telling me their hands were tied. Fuck Gusto.
    - Kognito 3y ago
      Hate that. “Sorry, the system says no” Had similar situations with PayPal and Uber recently where their support have absolutely no information or ability to take a decision. Support essentially becomes a glorified text-to-speech system.
    - deleted 3y ago
      [deleted]
  - robocat 3y ago
    That's unfair: isn't this is exactly how most strange bugs get "fixed" by most companies? It is an abnormal developer and an even more abnormal business that actually spends enough time to find the root cause of outré glitches. Especially when you start having to debug complex third party systems to debug them properly - requires skills and motivation plus a company that will encourage a developer to do that. The story is not specific to Gusto - it is the story of every developers life. I have chased down bugs in my OS and my browser - it is rarely well rewarded! Fixing a compiler bug should be on my bucket list! A long time ago I worked around a compiler bug by inserting a label: (I think the label prevented certain optimisations where the label was put).
azlev 3y ago
Hash collision?
- sapiogram 3y ago
  What?
tru3_power 3y ago
Is there a copy of the gif available? That’s interesting
- digging 3y ago
  I wouldn't even know how to look for something unusual in a gif's source code but I also feel this is the most compelling part. I wish they'd uploaded it.
  - guessmyname 3y ago
    > I wouldn't even know how to look for something unusual in a gif's source code but I also feel this is the most compelling part. I wish they'd uploaded it. GIF stands out as a widely understood file format [1][2]. To kick things off, delve into the GIF file using a hexadecimal editor. HexFiend [3], for instance, offers a template for visualizing GIF file structures [4]. Another excellent option is Synalyze It! [5], which comes pre-loaded with an extensive list of file formats, encompassing GIF among others. These visualizations serve as a guide to pinpoint any irregular byte clusters that might pose issues when loading the file into an application with an image reader lacking support for that specific byte group or its arrangement. Once you've identified such a cluster, consider it the bug. [1] https://en.wikipedia.org/wiki/GIF#Example_GIF_file https://en.wikipedia.org/wiki/GIF#Example_GIF_file [2] https://www.w3.org/Graphics/GIF/spec-gif89a.txt https://www.w3.org/Graphics/GIF/spec-gif89a.txt [3] http://hexfiend.com http://hexfiend.com [4] https://github.com/HexFiend/HexFiend/blob/master/templates/Images/GIF.tcl https://github.com/HexFiend/HexFiend/blob/master/templates/I... [5] https://www.synalysis.net https://www.synalysis.net
    - spuz 3y ago
      Another good option is ImHex which is an open source hex editor that supports file patterns. The gif pattern is one of the patterns already available: https://github.com/WerWolv/ImHex https://github.com/WerWolv/ImHex
    - tchebb 3y ago
      Kaitai Struct[1] is my preferred tool for parsing binary files. It's open-source (unlike Synalize It! and 010 Editor) and cross-platform (unlike HexFiend). Not sure how it compares to ImHex's pattern language, though—I've been put off by ImHex's UI the few times I've tried it. My favorite thing about the Kaitai Web IDE is that it instantly updates the parsed tree as you make changes to the format specification, which makes it viable for reversing unknown formats instead of just specifying known ones. [1] https://ide.kaitai.io/ https://ide.kaitai.io/
dblitt 3y ago
> For security reasons, we do not have Chrome crash reporting enabled. > We also confirmed with many of our affected users that they had Grammarly installed on their computers. Ironic.
- roozbeh18 3y ago
  haha, that was questionable for me as well. It's ok for Grammarly to read your stuff, but crash metadata is a no no.
  - JohnMakin 3y ago
    Welcome to security in 2023 :)
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
madeofpalk 3y ago
> Unfortunately, with access to neither the Chrome source code I mean, you basically do! You can just go check out the chromium source.
- flutas 3y ago
  > I mean, you basically do! You can just go check out the chromium source. It's mentioned in the bullet points in the "trouble reproducing the bug" section that chromium wasn't affected. > Using open-source Chromium instead of Chrome did not cause crashes, so we couldn’t see what Chrome code was failing either.
  - j1mmie 3y ago
    I wonder what version of Chromium they used. If it was a nightly, it could be weeks before the fix makes it into Chrome. They might've tested at a time when latest Chrome had the bug and Chromium didn't.
  - orbv 3y ago
    Google provides symbols for Chrome release builds, including source indexing so source code should be available. See sections symbol server and source indexing at https://www.chromium.org/developers/how-tos/debugging-on-windows/ https://www.chromium.org/developers/how-tos/debugging-on-win...
    - trelliscoded 3y ago
      I wish this was upvoted more. This is the correct way to troubleshoot the bug, full stop. You can get symbolic stack traces with full arguments and source code on Windows in about 5-10 minutes for any Chrome crash by following these instructions. I always have a last change exception handler that fires up a WinDBG script on Windows for our chromium-based test runners, which reduces troubleshooting time to just a couple minutes in order to find the symbol in their bug database. Playing blackbox what-if games like the Gusto team is a waste of time and doesn't contribute any situation-specific knowledge to bugs.chromium.org.
    - 3y ago
simion314 3y ago
I am using chromium to print web pages to pdf, and I have some images that will crash chrome's to pdf process, I found nothing wrong with this images, the metadata is fine (nothing weird in it). The other bad thing it does not reproduce n my dev machine only on the production server , so nothing I can do, in rare cases an image will always crash crhomium, I find it, open it and re-export it and then it works.
- LgWoodenBadger 3y ago
  This sounds more like a hardware fault than something wrong with the software, especially since it doesn't seem to be deterministic. But stranger things have happened, and given the enormous surface area of a modern computer (hardware, software, drivers, state, etc.) can anything truly be deterministic?
  - simion314 3y ago
    It happens with that image no matter what. I can have a html with 100 images and one bad image, I make one new html only with that image and it still has the problem. My guess is that probably a bug in a low level image decoder. My local machine has different kernel, different libs, plus I have different cpu,gpu and X11 on top so too much difference and I do not have the expertese to do aremote debug(or local)
    - saagarjha 3y ago
      If you can grab a crash log I am sure the Chrome team would be happy to look at it.
  - saagarjha 3y ago
    Most bugs are software issues, though.
lloydatkinson 3y ago
Discord suffered (suffers?) from a similar thing with gifs. It is or was common for people to post specifically crafted gifs in channels, anyone viewing the channel immediately had their client crash. Discord client uses Electron, which is in turn Chromium.
j1mmie 3y ago
What an interesting conflux of tech to create this bug. That's the web in 2023. I would love to know if it was a Chromium bug that got resolved, but navigating this is tough: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=gif%20status%3AFixed https://bugs.chromium.org/p/chromium/issues/list?can=1&q=gif... Also, I am fully here for Gusto posting this to say "wasn't our fault" and to throw some shade at Grammarly in the process
- thenoblesunfish 3y ago
  Seems like they're posting it because it was a fun story, and it's free advertising - this wasn't externally visible so I don't see where fault comes into it.
- nonethewiser 3y ago
  They should definitely publish the gif
- j16sdiz 3y ago
  Something like this https://bugs.chromium.org/p/chromium/issues/detail?id=1297706&q=&can=1 https://bugs.chromium.org/p/chromium/issues/detail?id=129770...
computerfriend 3y ago
If they can figure it out, they're sitting on potentially a very valuable exploit.
- jdminhbg 3y ago
  One that's been patched already, though, as they say that in current versions of Chrome and Grammarly it doesn't crash.
- chatmasta 3y ago
  Yes, I too would like to read more details about this. It's a great writeup from an engineer who got stuck debugging this. But I hope some experts in security or reverse-engineering can replicate it and take a closer look. There's definitely a more interesting story here, probably regarding the localhost bridge between Grammarly extension and desktop. (Grammarly has a bug bounty btw... and their chrome extension has quite a large surface area...) If OP is here: can you provide the raw .gif file? (And if you're feeling generous, maybe even a minimal ruby example that replicates that templating setup, although it sounds like that wasn't required to reproduce it in the end.) P.S. "For security reasons, we do not have Chrome crash reporting enabled" - maybe consider disabling Grammarly extension for the same reasons ;)
  - saulpw 3y ago
    It wasn't the Grammarly extension, it was the desktop app.
    - chatmasta 3y ago
      I guess I just assumed the extension was installed too, and communicating with the desktop app. But now I see the post doesn't mention the extension. If it was triggered even without the presence of the extension then that's quite strange, and even more suspicious - is that gif triggering a call to a localhost endpoint? Is the grammarly desktop app interacting with browser elements without using the extension? (IIRC the grammarly app uses some accessibility privileges to inject into textareas across all apps) Grammarly is honestly insane, I can't believe corporations allow it to run on employee machines.
Modified3019 3y ago
The true bug in the photo is a "candy-striped leafhopper", Graphocephala coccinea, which is tiny but has very striking coloration. The Larvae of leafhoppers are commonly known as spittlebugs, which create protective bubble nests while feeding on plant stems
- deleted 3y ago
  [deleted]
barbegal 3y ago
A disappointing ending to the tale. I really want to know how Grammarly desktop works now. It must have interfered with the Chrome process in some bad way.
- deleted 3y ago
  [deleted]
whirlwin 3y ago
Shared library like libgif used by both Chrome and Grammarly but different versions?
JohnMakin 3y ago
I have a saying that isn't perfectly true but often will apply to "fixes" like this - If you don't know why the fix worked, you may not have actually fixed it.
- bicijay 3y ago
  But you may have
  - lbhdc 3y ago
    Ahhh, Schrodinger's patch.
    - HPsquared 3y ago
      Schrödinger's bug?
- gwbas1c 3y ago
  But they couldn't fix the bug: The bug was in another product that they couldn't access source code or submit patches. The best they could do was work around it. Sometimes workarounds are the best you can do until your vendor provides a real fix.
  - JohnMakin 3y ago
    I didn't mean literally fix the underlying bug. They also don't really know why their workaround worked, which means it could not really be fixed at all.
- tetha 3y ago
  Someone once said, there is accidental function, and deliberate function. If your system doesn't work, and you just plonk around at values, until, very surprisingly, the system starts behaving well and you the call it working... well it might be working now. But it's just accidental correctness. As soon as something causes the system to bank left, something's gonna break and no one knows how to fix it - and you're back to square one. On the other hand, as hard as it is, if you can clearly tell why your fix will restore function to the system without even applying it, you have deliberate correctness and function. If done right, it is very boring, because exactly and only the expected thing will happen. You should know about the unknowns and plan around those as well, so even if an unknown bites you, it's a known and handled unknown. This can be exhausting to make happen, because it is much harder, but those systems will just work. But this is a fight I have with some development teams probably forever. "But we poked at the values, and that stopped the flames. It is fixed!" "but why?" "Dunny. But no fire anymore. All good." And then 2 weeks pass, and there is more fire and everyone is like "Oh but why would this happen? How should we have known for this to happen again"
djbusby 3y ago
Thought this was going to be about Gusto "remember this device" which keeps failing. Reported like 2 years ago
- deleted 3y ago
  [deleted]
sonicanatidae 3y ago
Twirly prompts have sucked, for weird reasons, since the WWIV BBS days. I guess some things never change.
zwieback 3y ago
Left me hanging, would not post something like this from my engineering blog. They don't have deeep debugging skills at gusto?
- golergka 3y ago
  Sounds like they have good time management and prioritisation skills. They found the source of the problem and fixed it.
  - KerrAvon 3y ago
    No, they didn't. They figured out a workaround. Since they didn't find the root cause of the problem themselves and apparently didn't take it up with the Chrome or Grammarly development teams, they don't actually know what happened or when it might bite their customers again.
    - jdminhbg 3y ago
      They say it no longer reproduces on current Chrome/Grammarly, so taking it up with them is fruitless. They're not going to investigate crashers from old versions.
  - dhritzkiv 3y ago
    :( I would've tried to determine the cause of the crash with that specific file in my off time, provided that I could isolate the code in the Grammarly extension in Chrome. The main reason –other than curiousity– is to ensure that a future regression (in Chrome/Grammarly) wouldn't lead to it again.
- deleted 3y ago
  [deleted]
cyco130 3y ago
The weirdest one I saw was this: User claims that the wording of the info they enter into a certain form changes when they save. At first I suspected someone else editing the same form at the same time unbeknownst to each other but it wasn't the case according to the logs. And I saw the correct wording on my own computer. Then I noticed in their screenshots that some of the menus had weird wording too. Turns out they had Chrome's "Translate this page" option on. Problem went away when we showed them how to properly switch languages in the app.
- bee_rider 3y ago
  What was translating from English to American or something?
  - cyco130 3y ago
    From English to Turkish but Google won't just leave the parts that are already in Turkish alone and subtly reword them instead for some reason.
- robocat 3y ago
  I added <meta name="google" content="notranslate"> to all pages in a single-page-web-app after discovering some bug or other with Chrome screwing up the page. Apparently the new incantation to fix an app (can be applied to an element) is (ugly: I presume it isn't CSS to avoid supporting dynamically changing it): <html translate="no"> https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes/translate https://developer.mozilla.org/en-US/docs/Web/HTML/Global_att... Every now and then I would look at the meta tags for a major single page app and discover some new horror when searching for the reason for the meta tag!
  - michaelcampbell 3y ago
    I know what you mean, but this caused me a second of "wait, what?" > all pages in a single-page-web-app
    - 3y ago
jrockway 3y ago
Did they open a bug against Chrome with the image file? I feel like any crash on user-provided data is a big deal, always a correctness problem, but potentially a security problem. "We deleted the image so the problem is fixed for us" is OK (I wouldn't waste time writing a blog post about it personally), but I think that Chrome needs to fix this bug.
- tedivm 3y ago
  Was it actually a Chrome bug though? It only happened when the Grammerly desktop app was installed. My guess is grammerly is doing something sketchy.
  - bayindirh 3y ago
    Maybe adding Grammarly created enough of a lag causing the GIF file to be shown?
    - nonethewiser 3y ago
      That doesnt hold when you consider they opened the gif in the browser with and without grammarly and it only crashed with. So its not simply a bad gif and chrome.
  - AnimalMuppet 3y ago
    Does Grammerly hook something in Chrome? If not, then it's still probably a Chrome bug, even if some second-order effect of Grammerly is necessary to trigger it.
    - majormunky 3y ago
      It looks like the desktop Grammerly app hooks into all sorts of things, "An all-in-one writing assistant that works on your desktop and in your browser. Use it in apps, word processors, email clients, and more."
    - MattDaEskimo 3y ago
user3939382 3y ago
Reminds me of the story I read where the guy's car wouldn't start depending on what flavor of ice cream he picked and when investigated he was right. Some kind of evaporation/vacuum leak or something that was dependent on time and some flavors were farther away in the store and took more time to buy.
- nudgeee 3y ago
  Vapor lock. Legend here: https://www.snopes.com/fact-check/cone-of-silence/ https://www.snopes.com/fact-check/cone-of-silence/
  - RajT88 3y ago
    I heard a similar tale in high school. A friend of mine had an aunt who passed away, and so he ended up inheriting her car. The car came with a petrified apple pie in the back. He was insistent that the car would not start without the pie in the back window. Several of his friends who he played in a punk band with confirmed this, that they had tested it. Take the pie out, car won't start. Put the pie back in, the car starts. I don't think anyone ever figured out what was going on, I graduated a couple of months after hearing the story, and fell out of touch. But - timing and vapor lock makes sense, if they were always testing it by first starting the car, removing the pie, and then putting the pie back in. As an aside, the aunt who had passed away was one Aunt Martha (after which the car was also named), which in honor of the strange car and its strange pie was what their garage punk band was named after. There's some totally unrelated band now called Aunt Martha - any evidence of their band is not on the internet.
- superfrank 3y ago
  Up there with the "I can't send an email over 500 miles" story https://web.mit.edu/jemorris/humor/500-miles https://web.mit.edu/jemorris/humor/500-miles
  - suzzer99 3y ago
    I love this one so much.
mgaunard 3y ago
The main reason for tabs to crash is running out of RAM. Never do you see the guy investigating memory usage, which is weird.
- marcellus23 3y ago
  Yeah, that would have been a good first step, but he does admit to not really knowing much about browsers: > This was fairly far outside the usual scope of our on-call issues. Our team is generally well-insulated by other teams from issues like browser compatibility, so I didn’t know the first thing about browser debugging.
  - whimsicalpedal 3y ago
    [flagged]
- gwbas1c 3y ago
  If that were the case, I think the bug would be much easier to reproduce; and be a lot more widespread.
NotYourLawyer 3y ago
[flagged]
neilv 3y ago
Why would they disable crash reporting for security reasons (which might actually help solve the root cause of their availability problem, which they never did solve)... yet run Grammarly (which I'd guess, security-wise, is less trustworthy than Google, in how they secure data themselves once they've inevitably stolen it from the customer)?
- Zetobal 3y ago
  Maybe they have the enterprise licence with grammarlys pinky swear that they won't train on your data.
- deleted 3y ago
  [deleted]
hilux 3y ago
What a cool mystery-solving post! I wish all technology writing were this clear and explanatory. For another fun debugging tale, google: Mazda radio Seattle NPR bug
- EdwardDiego 3y ago
  Holy hell lol.
realmike33 3y ago
This reminds me of similar issues I've encountered as a software engineer. I first ran into this issue about a decade ago, albeit not because of Grammerly,but due to some specific gif causing web app to crash. Both times the gifs were animated. Happened years apart and at different companies. I see some comments highlighting RAM, which could totally have been the issue. Totally looking forward to a follow up to this later down the road, I am sure this isn't going to be the last time we hear of this.
neilv 3y ago
> Using open-source Chromium instead of Chrome did not cause crashes, so we couldn’t see what Chrome code was failing either. They don't address why they didn't just run Chromium. Or Firefox. (This is potentially better than the 'solution' they much later ended up with, in which they probably only relieved a symptom of an underlying problem that can exhibit again, and in the meantime is a zero-day exploit waiting to happen. At least, with a different browser, there's a chance that the vulnerability doesn't actually exist, when it's known to exist in their Chrome configuration.)
- Etheryte 3y ago
  What makes this doubly frustrating is that they also didn't report the bug to Chrome. It's super easy to do, plus they're very responsive if you have a repro case which in this case they do. I think I'm now up to three or four Chrome bugs reported that their team has subsequently fixed.
  - ncann 3y ago
    They said it wasn't reproducible anymore though. So if they make a bug report now and say "this used to cause a crash in an old version of Chrome while also having an old version of another software installed, but is no longer reproducible in latest builds", most people would probably just ignore it.
    - masto 3y ago
      It was reproducible at the time they found it, and trivially so: install Grammarly, drag this GIF into Chrome, and it crashes. I understood everything up to the point where they just changed the GIF and moved on without ever telling the Chrome or Grammarly folks about it.
- gwbas1c 3y ago
  > They don't address why they didn't just run Chromium. Or Firefox. The article implies that Gusto's employees can whatever browser they want. And, honestly, telling your employees to run a browser that only techies have heard of sounds like a really dumb idea.
1123581321 3y ago
How could you write all this and not post the gif so we can try it?
- deleted 3y ago
  [deleted]
andrewfromx 3y ago
oh and there's this one: https://web.mit.edu/jemorris/humor/500-miles https://web.mit.edu/jemorris/humor/500-miles
tgsovlerkhgsel 3y ago
Extremely disappointing that they seem to have neither investigated nor enabled others to investigate (e.g. by filing a bug against Chrome). This smells like a potential security vulnerability.
- ljm 3y ago
  I was hoping for a bit more of a payoff. Like, if the gif was broken, why did the grammarly extension trigger it reliably?
- deleted 3y ago
  [deleted]
deleted 3y ago
[deleted]
jiveturkey 3y ago
I'm guessing this is the webp bug. The auto conversion to webp on the backend, signaled by chrome, resulted in a bad image that crashes the browser due to grammarly parsing of said bad webp. Safari doesn't tell the server it does webp and so it downloads the actual gif, and doesn't crash.
- saagarjha 3y ago
  GIFs don’t get automatically converted to webp files.
  - jiveturkey 3y ago
    says who? any web server (nginx, whatever) can be told to do this
    - saagarjha 3y ago
      Yes, but why would it?
      - jiveturkey 3y ago
        because it's super convenient, and improves performance?
romanhn 3y ago
Another favorite bug investigation of you're into this sort of thing: https://www.pagerduty.com/blog/the-discovery-of-apache-zookeepers-poison-packet/ https://www.pagerduty.com/blog/the-discovery-of-apache-zooke...
ecshafer 3y ago
I don't think I would believe myself if I found this being a specific gif. This is a great amount of coincidences in code to cause this. Grammarly is an application that I don't get. The fact that people are installing, basically spyware, on their computers just to get grammar suggestions to make their writing more boring and add a spellchecker (which is already inside web browsers) is pretty astounding to me. The fact companies allow employees to have it, despite obvious security issues of sending everything one types to a saas, is even more wild.
- eichin 3y ago
  I worked with someone who really needed it, but we had the usual "keep sales users as far from the actual product as possible" organizational isolation so it worked out in practice. (For engineering, it was on the "don't install this in particular" list.)
- _jal 3y ago
  > Grammarly is an application that I don't get. You write like a native speaker, so I'm not surprised. But imagine having a few years of school-German, and then taking a German language job. I'd bet there would be times you'd want a writing assistant, too. There are also plenty of native English speakers who for whatever reason got a crappy education, and didn't get a lot of writing feedback. As far as corporate security goes, you are correct, and we ban it. But I get why people want it.
  - ryandrake 3y ago
    > As far as corporate security goes, you are correct, and we ban it. But I get why people want it. That is what stuck out to me: Installing rando applications on your corporate computer that has access to internal stuff... Whoooaaaa Baby! That's just a security disaster waiting to happen. It's stuff like this that eventually leads to draconian and crappy "Nobody gets admin access to their machines" corporate policies coming down. Most TechCorp places I worked, if someone installed something like that on their corporate device, they'd get at least a stern talking-to and probably sent back to security training.
  - 3y ago
hartator 3y ago
Maybe the name was odd `loader-spinner.gif`
FrankWilhoit 3y ago
Error code 5, on Windows, means that the code tried to dereference a null pointer. I'm guessing that the GIF content is corrupt, containing some 0x00 bytes where they shouldn't be. Then the question becomes, whose responsibility is it to program defensively against things like that? If, as may well be, Chrome is using some third-party library nested several layers deep to render GIFs, then would there be any action that the chrome devs could take, aside from replacing that library with a better one and adding a malformed-GIF test case? (Why don't they already have a malformed-GIF test case...?)
- deleted 3y ago
  [deleted]
- londons_explore 3y ago
  Lets be honest, it was probably some grammarly .dll that had been injected...
- saagarjha 3y ago
  Considering they’re trying Safari I suspect they’re on macOS.
Night_Thastus 3y ago
I've actually seen something like this in the wild myself. For awhile there were some GIFs that if placed in Discord, would cause it to crash for everyone who was looking at the chat. Admins had a fun day when that was found!
sethammons 3y ago
> The code for our main navigation bar has a fair amount of metaprogramming, and chasing down threads here was often more confusing than not. One more point for Don't Be Clever. As Brian Kernighan put it: “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”
- deleted 3y ago
  [deleted]
ja5087 3y ago
We used to develop software that used the Windows Accessibility APIs (UI Automation). On certain versions of Excel with some files it would crash the client application with a null pointer exception once you try to read the window name/class. It would be interesting to see the cause of the crash e.g. a core dump/user-mode dump/event viewer log.
bluesmoon 3y ago
Reminds me of the time back in 2010 when a piece of CSS on the Yahoo Search page would cause a complete desktop crash on Red Hat Linux: https://tech.bluesmoon.info/2010/04/can-website-crash-your-rhel5-desktop.html https://tech.bluesmoon.info/2010/04/can-website-crash-your-r... To the author, did you ever consider contacting the Chrome dev team about this? They're pretty responsive to bug reports.
rvanlaar 3y ago
Ah, Chrome and slow spinners. Python tests were taking ages on VSCode due to an SVG spinner: https://bugs.chromium.org/p/chromium/issues/detail?id=1036262 https://bugs.chromium.org/p/chromium/issues/detail?id=103626... https://github.com/microsoft/vscode-python/issues/9216 https://github.com/microsoft/vscode-python/issues/9216
- deleted 3y ago
  [deleted]
boringuser2 3y ago
The blink debug logs would probably be pretty useful for the engineers involved...
rootsudo 3y ago
For security reasons, your organization disables Chrome crash reports but allows the use of Grammarly, an app that essentially functions as a keylogger. Consented or not, it’s a keylogger. https://support.grammarly.com/hc/en-us/articles/360003816032-Is-Grammarly-a-keylogger- https://support.grammarly.com/hc/en-us/articles/360003816032...
crazygringo 3y ago
tl;dr: a certain GIF would crash a Chrome tab, but only when the desktop version of Grammarly was installed. (Not a Chrome extension.) That's insane! Can anyone think by what possible mechanism the installation of Grammarly could affect whether a .gif file would crash Chrome? The company seems to be on Macs since they report that the problem doesn't surface in Safari. Is there some kind of dynamically-linked GIF decoding library used by macOS that Chrome relies upon, and Grammarly somehow installs one that takes precedence for all applications? I didn't think this would be possible -- I thought image decoding was done natively in the browser and not outsourced to the OS, for security reasons.
- saagarjha 3y ago
  This would be strange since macOS should not let Chrome load libraries that aren’t signed by Google.
- masswerk 3y ago
  Maybe, the GIF contains a text comment and Grammarly not only accesses this, but also alters it (for whatever reason), thus overwriting the file in a way in memory that offsets in the GIF file or its in-memory representation are violated?
- mike_hearn 3y ago
  It's not an uncommon problem unfortunately. On Windows the OS lets programs inject DLLs into each other's address spaces, and this is actually a very common programming technique for various reasons. Apps don't expect it to happen of course, and so this is a fertile field of bugs. Chrome does in fact attempt to block some apps from doing this because they're known to cause crashes. You can see the blacklist here: https://source.chromium.org/chromium/chromium/src/+/main:chrome/chrome_elf/third_party_dlls/hardcoded_blocklist.cc;l=19?q=inject%20blocklist&ss=chromium%2Fchromium%2Fsrc https://source.chromium.org/chromium/chromium/src/+/main:chr... They don't like apps doing it but you can apply for an exception: https://sites.google.com/a/chromium.org/dev/Home/third-party-developers https://sites.google.com/a/chromium.org/dev/Home/third-party... So my guess is that Grammarly is injecting code into other processes, probably to try and gain access to their text controls so it can Grammarlyze stuff, and somewhere along the line corrupts the heap or stack in some way. The GIF aspect might be a red herring, as the crash was IIUC never reliably reproducible. Random crashes that happen sometimes but not others is strongly indicative of something being randomly corrupted. Or it could be something else. Chrome moved to blocking third party injects by default some time ago, so maybe that's also a red herring.
hcrean 3y ago
I wonder if they checked for exploit code in the image the they likely originally found somewhere on Google.
toddmorey 3y ago
Once my college professor was working on her research paper and told me she was struggling get text to stay underlined. Assuming a simple user error, I expected to help her out in 5 minutes. Over three hours later, we discovered that the combo of her specific video card driver version along with her specific printer driver version would keep text from printing out underlined.
- generationP 3y ago
  Huh? How does a video card affect printing?
  - jfoutz 3y ago
    A lot of rendering will go through the video card if available, like the jvm does this as an optimization.
    - toddmorey 3y ago
      Ah interesting. That makes sense. I had no idea.
  - shever73 3y ago
    It often did, particularly on older versions of Windows. I helped uncover a bug in Epson printer drivers ~20 years ago that was caused by a specific graphics card.
    - shermantanktop 3y ago
      20+ years ago I was in tech support and had to help someone figure out an issue where her document wouldn't print on a Brother printer. Turns out a section divider line would block the entire doc from printing (by crashing the app) if the line's end-cap style was set to square rather than rounded.
  - to11mtm 3y ago
    One of the ways to print things (especially on windows) is Via GDI. [0] Basically using the OS's rendering to make a raster that is then sent to the printer. The main thing the printer's driver does in this case is know how to take the bitmap and tell the printer to print the bitmap (i.e. chunking data and/or sliding the right commands into the bitmap stream) Contrast to, say, PostScript which allow for more compact and better scaling definition of what to print. This obviously works better for quality, however for a long time the issue was you then had to have sufficient processing power on the printer itself to handle it. [0] - Search for 'GDI Printer' for a little more info.
nialv7 3y ago
I feel the author might have missed out on a multi-thousand dollar bug bounty.
aaaronic 3y ago
This is _so_ familiar! I have seen accessibility tools in Chrome lead to this kind of issue in the past with a dropdown menu -- to the point where it could be replicated with a miniscule amount of HTML. The particular bug I hit 2 years ago was in Chromium-Edge, but the symptoms and cause were very similar. Grammarly almost certainly leans on some of the accessibility tools in Chrome. These tools are somewhat different in the various Chromium flavors (Edge, Brave, Chrome, etc.).
- nonethewiser 3y ago
  So the theory would be that grammarly desktop sees the gif (what? How?) and calls some browser accessibility function on it (or?) which chrome cant handle and it crashes?
  - saagarjha 3y ago
    Perhaps it has an extension that it installs that does this?
    - codethief 3y ago
      They did try disabling browser extensions, according to the blog post.
  - aaaronic 3y ago
    With the bug I saw years ago, just having certain accessibility features of the browser enabled _at all_ caused the bug (we were able to temporarily mitigate by disabling some obscure Edge accessibility feature via a launch parameter). So, my theory here is Grammarly is just enabling an optional accessibility feature in Chrome that has this bug when trying to "read" the gif.
gelatocar 3y ago
As I was reading this I was thinking to myself "I wonder if it is grammarly related" because I experienced a bug some time ago that presented itself in a similar way. It was impossible to reproduce but affecting lots of people internally within certain departments. Eventually we figured out the thing they had in common was that they had the Grammarly extension installed. The other key thing was that the bug only appeared on our staging preview urls, not on the live website. It turned out it was because of a bad regex in the grammarly extension that caused the page to hang if the domain name was more than about 100 characters. Our staging domains were pretty long, I think they contained a few subdomains and had a job number or something in there. This one is more crazy though if it is really caused by the desktop app - that's pretty scary!
- codethief 3y ago
  > It turned out it was because of a bad regex in the grammarly extension that caused the page to hang if the domain name was more than about 100 characters. Just today I debugged a regex that would DoS our backend whenever the user enters the wrong thing in a form. Now I'm reading up on regex engines: https://swtch.com/%7Ersc/regexp/regexp1.html https://swtch.com/%7Ersc/regexp/regexp1.html
  - dev_slash_null 3y ago
    Just in case you haven't seen the postmortem of the Cloudflare outage which also was caused by a regex based DoS: https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ https://blog.cloudflare.com/details-of-the-cloudflare-outage...
    - radiojosh 3y ago
      That was a great read, but there was one thing I didn't understand: Why would the regex string have "." twice in a row? What does ".." find that "." doesn't find? Does that just mean "at least two characters"?
      - 3y ago
aquafox 3y ago
I randomly had the issue that after booting up Linux, I didn't have any sound. Turns out it was related to my Windows dual-boot setup! When restarting from Windows, Windows doesn't shut down my realtek audio device, but only puts it to sleep and Linux fails to start it. Only solution is to always do a shutdown from Windows and then hitting the power button. The issue is still there: https://askubuntu.com/questions/1032543/no-sound-in-ubuntu-18-04-after-rebooting-from-windows-10/1078134#1078134 https://askubuntu.com/questions/1032543/no-sound-in-ubuntu-1...
- nightfly 3y ago
  I've seen the opposite-ish: Someone who was dual booting windows and linux only had working wifi if they booted into windows and rebooted into linux. The linux install didn't have the firmware package for their wifi card installed, so when rebooting from windows into linux it was still all primed and ready to go but not when cold booting into linux directly.
  - sodality2 3y ago
    This is my exact problem with a HP envy X360. HP refuses to fix the bug in BIOS. :/ https://bugzilla.redhat.com/show_bug.cgi?id=2107845 https://bugzilla.redhat.com/show_bug.cgi?id=2107845
    - 6510 3y ago
      Every other day I read some fascinating HP adventure. I've done some questing myself, cant really tell if I did well, are they new multiple issues or the good old same? Your comment had me wonder if a kickstarter for an rpg game called HP printers would take off. The potential fan base is HUGE, I imagine even people who would never play a game would want to see it. Let the GUI be like little computer people (only an office) https://youtu.be/SkTgX1mGmDg?t=17 https://youtu.be/SkTgX1mGmDg?t=17 Then the player has to go though rituals depending on the HP printer type, swap old for new then new for old then old for new ink cartridges after finding the instant ink package in storage, if any spawned of course. If you try to go to fast the hero starts destroying the office, starting with the printer. Then a new different model is delivered. Ready for setup.... or maybe not? In some laps of consciousness I purchase one more some years ago. The hours of configuring it made me feel dumber than the wrath of the HP dungeon master. The ISP has a wifi router fixed on channel 11, half the neighborhood sits on channel 11, range is about 4 meters, the printer doesn't reconnect unless unplugged first. But there is wifi direct, this doesn't have 50 cm of range, it wants to use channel 11. in the game our hero at least at first doesn't know about channels. The next printer arrives: Something seems off, there is no usb cable in the box? Do you want to A) order a cable B) we will use wifi! Of course if you order the cable the boss will complaint that it takes to long to set up the printer but when you eventually get the cable the boss wants to print from his phone... https://freesound.org/people/InspectorJ/sounds/402095/ https://freesound.org/people/InspectorJ/sounds/402095/
sfink 3y ago
Next time this happens, I recommend just letting people use a different browser. Firefox in particular has gotten much better at importing bookmarks, passwords, etc. from Chrome. It was a Sign from the universe that it was time to make the switch. Who are we to reject Signs? (Full disclosure: I'm an engineer on Firefox. But that has nothing whatsoever to do with my advice here, no siree Bob, not in the least.)
- mplewis 3y ago
  This is explicitly stated as a workaround in the linked article.
- OsrsNeedsf2P 3y ago
  As an engineer, yes Firefox is a good solution As a PM, we spent 4 months making the onboard easier, and now you want people to install a new browser?
  - noizejoy 3y ago
    Having multiple browsers as a standard installation and employee training is an investment in business continuity. And business continuity investment is like insurance: A waste until you need it. Too much monoculture is short sighted and long term expensive.
    - hirsin 3y ago
      Do you ship them extra computers too? Dual boot? There's gotta be a limit somewhere, and browser choice is a fair one. For engineers, absolutely, two browsers because some are liable to hit (self or team induced) bugs. But for sales? Support? Nah.
  - Shaanie 3y ago
    I run into Firefox compatibility issues waaay more frequently than chrome compatibility issues, for obvious reasons. Even extremely obvious issues, like 2fa login not working on mobile Firefox, are surprisingly common to find, so I'd never install Firefox as a primary browser for non-techies nowadays. I use Firefox myself for ideological reasons, though.
denton-scratch 3y ago
Awww. I was really enjoying that; I like detective stories and rabbit-holes. Then we get to the punchline: "Uh, we fixed the bug, but sorry folks; we didn't solve the puzzle". So I guess we'll never know why that particular anigif crashed Crome but only Chrome, and only if Grammarly was installed (or had been installed during the same session). I hope Amy Lai lets us know if the story ever gets an ending!
- deleted 3y ago
  [deleted]
lantry 3y ago
> Turbo is a gem we added to speed up our Rails application, but it turned out to be a red herring: it was only introduced after the bug had already been reported, and we learned that the engineer who introduced it had actually been experiencing these Chrome crashes for months prior to Turbo being introduced, and months prior to the bug being escalated to us. lolol I can just imagine the engineer working on this: "huh, it keeps crashing, that's weird. Oh well, it was like that when I got here. ship it"
CGamesPlay 3y ago
I love the corporate security policy that disables Chrome crash reporting for security reasons, but allows employees to install Grammarly.
NautilusWave 3y ago
This reminds me of an old Chrome bug we ran up against involving the auto fill feature. All day, our software users are entering patient data, including address information. Chrome would grind to a halt on certain pages trying to provide all their previous entries as auto fill options to the form fields. Our sysadmins disabled the Chrome feature for all of our internal users and the issue disappeared.
numitus 3y ago
Interesting advertisment, but not really a tech article.
- EdwardDiego 3y ago
  Wrong article?
resonious 3y ago
I gotta say, the ending was pretty disappointing. I was so hyped to learn why Grammarly crashes when Chrome loads a particular gif. What about the gif was different? What part of Grammarly made that happen? I hate to say it but this is modern web dev in a nutshell. Remove stuff until you find where the bug is, tweak it randomly until it's fixed, done. No root cause analysis or anything deep. This is a great post anyway. Well written and still quite intriguing right up until the end. And it seems lots of comments on here seem to also know about this problem, so I think I can still satisfy my curiosity.
- deleted 3y ago
  [deleted]
- eyelidlessness 3y ago
  > I hate to say it but this is modern web dev in a nutshell. Remove stuff until you find where the bug is, tweak it randomly until it's fixed, done. I’ve alternated between frontend, “full stack”, and backend roles for over 20 years. It’s my experience that what you’re describing is “dev in a nutshell”—neither “modern” nor “web” in the sense you seem to mean. And in any case it’s highly situational and variable depending on the dev and their team.
- randshift 3y ago
  They never root caused the bug! They know a symptom (Grammarly and this gif cause things to crash) but still no root cause. I was pretty disappointed at the end.
  - sanderjd 3y ago
    It was like reading a Sherlock Holmes story where they just decide to put Moriarty in jail and that fixes everything, but Holmes doesn't actually figure out what was going on.
- karim79 3y ago
  > When I removed loader-spinner.gif, the placeholder we display while the menu options load, the page stopped crashing. Eureka! It’s the gif! We swapped in a different gif and the page did not crash. I would love to have the original and the un-crashy gifs mentioned. It's super easy, generally, and even without an extensive knowledge of image formats to get a grasp of what might be going on and then going down some really exciting rabbit-hole of image encoding/decoding issues. Just take the two gifs and run them (one by one) through ImageMagick or GraphicsMagick to print out the details of what's in them, and look for differences. Assuming ImageMagick is installed (or GraphicsMagick) installed, something like: #imagemagick $ diff <(identify -verbose loader-spinner-CRASHY.gif) <(identify -verbose loader-spinner.gif) #graphicsmagick $ diff <(gm identify -verbose loader-spinner-CRASHY.gif) <(gm identify -verbose loader-spinner.gif) ...and rabbit-hole away EDIT: formatting
userbinator 3y ago
I agree with the others here about the ending being a total letdown. Unfortunately, with access to neither the Chrome source code nor the Grammarly source code, we can only guess. Is this what the "open source" movement has created --- developers who are totally lost without source code and refuse to dig deeper? Of course the corporate interests who don't want us to know the truth, because their profit depends on it, would absolutely adore such propaganda... I still remember a time when a lot of people would disassemble, understand, and patch programs without source --- and many of them weren't even career developers; it was just a way to get software to do what one wanted, and driven by that motivation, one would naturally learn enough to do so. The article also touches on another point worth mentioning: the amount of complexity in the whole stack is insane. Seeing all the frameworks upon frameworks being name-dropped, I can't help but feel like a lot of this is self-inflicted. When I removed loader-spinner.gif, the placeholder we display while the menu options load, the page stopped crashing. Do menu options take long enough to load that they need a loading animation?
- hmry 3y ago
  Presumably they make a network request to get the options. So it makes sense to have a loading spinner just in case it takes a long time, even if it's usually near-instant.
- windowsworkstoo 3y ago
  Very well said - even with open source there seems to be a general lack of willingness to actually read code, let alone crack open the disassembler or attach a debugger - the skill is apparently not taught anymore
- cyclotron3k 3y ago
  Expecting every developer to be able to patch a binary is, to put it politely, unrealistic. The world of software engineering is enormous, and knowing all parts of the software stack hasn't been remotely feasible for a very long time. Also consider the fact that software engineering is a journey of learning, and every one of us is on a different part of that journey.
andershaig 3y ago
I can't remember the details anymore but the craziest bug I ever found was one that would crash the page whenever Chrome's Developer Tools were opened. I ended up having to use an embedded JS-based dev tool to figure out what was going on (I was a pretty junior developer and didn't have any better ideas at the time).
DaleCurtis 3y ago
Even if you have crash reporting disabled there should be a .dmp generated somewhere in the user profile directory. Manually uploading that to a bug at https://crbug.com/new https://crbug.com/new would allow a Chrome developer to debug it. If you can't share the dump for similar reasons to why you have crash reporting disabled, you can build minidump_stackwalk from Chromium and use it to generate an unsymbolized stack trace that you can post to the bug. A Chrome developer can then symbolize it. https://www.chromium.org/developers/decoding-crash-dumps/ https://www.chromium.org/developers/decoding-crash-dumps/ has some more details.
LASR 3y ago
I discovered a bug like this with GIFs too. It turns out some gifs can be encoded such that it is animated and a single frame long. Can’t remember the exact event name now, but the browser fires a play event when gifs loop. This particular gif was issuing too many of these play events that made the app super slow and freeze since it was doing some work in those handlers. We had a bunch of crash reports and bug reports. None of those showed the actual gifs our customers were using. When we asked for the actual gif, we immediately spotted the problem.
1letterunixname 3y ago
Spoiler alert But if you read the article... ;@) Improper animated GIF decoding is potentially a bug and potentially a security vulnerability. The reality is this leads to duck-tape workarounds and greater tech debt on the production side of the web. Sigh. There are standards, there are expectations, there quirks across N implementations, and MxN layers of duck tape in M consuming implementations. Just one implementation intentionally or unintentionally being different causes M headaches. Also fun Chromium-derived browserisms: - Updating while open is allowed and leads to silent and not-so-silent breakage. - MSFT MDE causes unexpected breakage in fun ways every now and then, including the cause of a crash while dragging a tab on Windows. - Some flavors of Chromium browsers are broken with IPv6 enabled, leading to an ERR_NETWORK_CHANGED on every nth page visit.
slavboj 3y ago
Grammarly is an ipso facto security hole that is extremely likely to be pwned or controlled outright by a number of state security services. It should be treated like malware.
- jtokoph 3y ago
  It’s hard blocked on any corporate devices where I work.
- fijiaarone 3y ago
  Ah, Tsutomo, my learned disciple!
jongjong 3y ago
Grammarly is awful. I remember adding custom grammarly attributes to sensitive password fields so that Grammarly would not read and store our users' passwords inside their service. So nasty. This company would sure make a great asset for the FBI, NSA and CIA given that they're so interested in snooping in on foreign language speakers; which happen to be grammarly's main demographic. The thing is malware.
- jongjong 3y ago
  Update: After writing this comment, me and my wife started getting Grammarly ads. Awful.
deleted 3y ago
[deleted]
socketcluster 3y ago
The weirdest bug I've seen yet was a concurrency race condition which led to 'spooky action at a distance'. Every time I tried to console.log() to observe some value to see the cause of the bug, the bug would go away. But alas, I hadn't discovered some new quantum effect... It turns out to have been because the race condition was close within maybe 1 millisecond and adding the console.log() statement there meant that one part of the code would take a bit longer to execute and so the race condition would not occur.
- KevinMS 3y ago
  https://en.wikipedia.org/wiki/Heisenbug https://en.wikipedia.org/wiki/Heisenbug
- ghosty141 3y ago
  In Qt QML using print can lead to different behavior very very quickly since their engine creates bindings on whatever it comes by so using two variables in a print statement can lead to different behavior than just using one. QML in general is just awful.
hrtk 3y ago
Can you share the `loader-spinner.gif` ? Preferably over a service that does not recompress it.
6510 3y ago
alert(banana++) Should lock things up and not continue until OK is clicked. If it is just you and your application you can just spray paint from the hip the alerts from top to bottom. After each line of html you can have one more <script>alert(banana++)</script>, in the middle of your css </style><script>alert(banana++)</script><style> etc If there are uhh thousands(?) of people actively using the page you put just one alert some place in the middle. Everyone will have to click on OK, the page crashes. You ask the crashee: Did it crash before or after the "make America great again" alert message. (call it something they would remember) Now you know the issue is in the top or bottom half. You move the alert to half way the half with the issue in it. If you can get some sort of reasonably consistent crashing you will find it pretty quick even in production. Hope this helps, or maybe it is a stupid idea and you could explain why.
deleted 3y ago
[deleted]
jspaetzel 3y ago
I've run into something similar to this a few years ago where chrome randomly introduced a rendering bug that resulted in a similar crash and only affected our application in seemingly random fashion. I don't remember the full context but I think it was something involving layers & transparency. We finally figured it out when a coworker couldn't replicate and we noticed they were a version of chrome behind. We were able to track down the specific commits in chrome that broke & fixed it in our case which was pretty cool to see at the time. If you have any more time I recommend reading through the commit log and see if you can find the changes that broke/fixed this for you. I'd bet on another rendering bug.
cristeigabriel 3y ago
If Chrome is able to generate a crash-report, nothing would prevent you to intercept it. > Unfortunately, with access to neither the Chrome source code [...] I mean, it's still very possible to debug (especially with the fact in mind that Chromium is open-source, and for me has been a very useful source when reverse-engineering and debugging Chrome), but I understand why web developers would not be trained in reverse-engineering techniques.
bradley13 3y ago
It is so satisfying to finally identify the root cause of such an obscure bug. Of course, along the way, you will shed tears. Sometimes literally. I had a funny one a couple of weeks ago: Text on a button would sometimes not display. After much experimentation, it turned out that the text would display unless the character 'D' was present. It turned out that the sizing of the buttons was just barely large enough for the font-size selected, and apparently the 'D' in the selected font was microscopically taller than other characters. FWIW I have to thank ChatGPT for helping solve that one. I explained the problem, and it gave me a list of things to try...
jon-wood 3y ago
The thing that struck me here is that Chrome crash reporting is disabled for security reasons, presumably not wanting to risk customer information being sent to a third party, but Grammerly is apparently fine? How is a tool that sends literally everything you type to a third party ok, but one that maybe sends something if the browser crashes isn't?
- jaggirs 3y ago
  Because doing that is a core part of Grammarly, i.e. they can argue in court that the user should obviously expect grammarly to send the data to their servers. Sending crash report data is not so, as it is not essential to using the service.
SonOfLilit 3y ago
When I reached the "Stroke of Good Luck" title, I put down my phone and tried to think what it could be based on what I know before the reveal. It was really hard to generate thoughts of the form "subsystem x causes the crash" and not of the form "try to do experiment x next", but here's what I came up with: - a headache trying to think about it - if it's not an extension, it's likely another piece of software that has code running in the browser, like PDF plugins used to before Chrome started handling PDFs itself - the crash is in one of the browser subsystems, so either in the JS engine, the GPU code, the layout engine, the parser, some format plugin (video? pdf?) or the network code. Some format plugin is most likely of these to only crash for tech support and remote developers but not devs Seems I was pretty close. After doing this exercise I would probaply have compared lists of installed software between staff who see the bug and those who don't, which would have solved it. My takeaway is that Zen debugging (just thinking what possible causes the seen behavior could have, without atking any new actions or measurements) is useful even when it seems completely useless and causes a headache.
butz 3y ago
And that's yet another reason that we must have several different browser engines.
Fanmade 3y ago
Ah yes, the great fun of bug-hunting. Probably most people in working IT have some stories to tell about this. One thing that comes to my mind happened in a company where they used a simple software to create medical reports. One clerk suddenly started creating "broken" PDF files. I debugged this for hours and could not find anything. At some point, I just copied the full raw text content that the clerk had written into one of those files out of the database into Sublime Text and I suddenly saw a small box appear between two characters of one word. It turned out that this clerk had created a Word document with text templates. Some of those texts were copied (at least partially) from other documents or websites. Apparently, either something went wrong while copying some text, or there was some control-character embedded for whatever reason in that text. I don't even remember what we did to prevent this from happening anymore, but one single "invisible" character can really cause a lot of trouble. Another point is of course if you have strange side-effects just by combining specific software. Back in the day when Crysis 1 was new and popular, I was a little involved in the modding-scene. At some point, my CryEngine editor would refuse to start up. I reinstalled it and tried to debug it, but I could not find anything myself. So I turned to the forums and found a thread where a small group of people had the exact same problem. We all shared our setups and - basically by accident - found out that we all had Wacom tablets plugged-in to our computers. I don't remember if it was enough to unplug the devices or if we also had to uninstall the Wacom drivers, but one of those things fixed the issue and we could all continue to build beautiful maps in the CryEngine. Btw, I am now suddenly constantly aware of my Grammarly plugin ...

619 points by jevans 3y ago

Honourable mention, “we can’t send emails more than 500 miles” https://web.mit.edu/jemorris/humor/500-miles https://web.mit.edu/jemorris/humor/500-miles

devilsAdv0cate 3y ago

[dead]

wobblyasp 3y ago

Such a tease. At least upload the gif so people can poke at it!

yuck39 3y ago
Agreed! I have absolutely no idea how a combination of grammarly and a specific gif would cause a browser crash though… Anyone here use the grammarly desktop app? Any additional clues?
- hyperhello 3y ago
  I would guess the gif triggers a specific edge case that would crash, and grammarly is just a common enough addition to chrome with a lot of edge case triggers that it was identified. I’d guess this is on the Chrome team to fix soon, but yeah, at least post the spinner file for us!
- fifafu 3y ago
  I‘d guess it’s because Grammarly enables full accessibility support in Chrome to be able to access the browser elements similar to e.g. a screen reader. This is off by default and has caused me various issues in the past when enabled (e.g. https://bugs.chromium.org/p/chromium/issues/detail?id=136448 https://bugs.chromium.org/p/chromium/issues/detail?id=136448... ). However it’s probably good that the Accessibility functions get more exposure due to this.
deleted 3y ago
[deleted]

12_throw_away 3y ago

Well, this is a fascinating murder mystery that establishes 3 compelling suspects - Grammarly, Chrome, and a gif - and then just ... ends, right before the big reveal.

0xNotMyAccount 3y ago
I have a friend who worked at Gusto, and my wife tried using Gusto for her small business (they handle payroll for small business, got a big boost from the pandemic). The lack of technical resolution here is so Gusto, it hurts.
- Andrex 3y ago
  They nuked my account after trying to charge an expired card three times. The customer response team was extremely quick and responsive telling me their hands were tied. Fuck Gusto.
  - Kognito 3y ago
    Hate that. “Sorry, the system says no” Had similar situations with PayPal and Uber recently where their support have absolutely no information or ability to take a decision. Support essentially becomes a glorified text-to-speech system.
  - deleted 3y ago
    [deleted]
- robocat 3y ago
  That's unfair: isn't this is exactly how most strange bugs get "fixed" by most companies? It is an abnormal developer and an even more abnormal business that actually spends enough time to find the root cause of outré glitches. Especially when you start having to debug complex third party systems to debug them properly - requires skills and motivation plus a company that will encourage a developer to do that. The story is not specific to Gusto - it is the story of every developers life. I have chased down bugs in my OS and my browser - it is rarely well rewarded! Fixing a compiler bug should be on my bucket list! A long time ago I worked around a compiler bug by inserting a label: (I think the label prevented certain optimisations where the label was put).

azlev 3y ago

Hash collision?

sapiogram 3y ago
What?

tru3_power 3y ago

Is there a copy of the gif available? That’s interesting

digging 3y ago
I wouldn't even know how to look for something unusual in a gif's source code but I also feel this is the most compelling part. I wish they'd uploaded it.
- guessmyname 3y ago
  > I wouldn't even know how to look for something unusual in a gif's source code but I also feel this is the most compelling part. I wish they'd uploaded it. GIF stands out as a widely understood file format [1][2]. To kick things off, delve into the GIF file using a hexadecimal editor. HexFiend [3], for instance, offers a template for visualizing GIF file structures [4]. Another excellent option is Synalyze It! [5], which comes pre-loaded with an extensive list of file formats, encompassing GIF among others. These visualizations serve as a guide to pinpoint any irregular byte clusters that might pose issues when loading the file into an application with an image reader lacking support for that specific byte group or its arrangement. Once you've identified such a cluster, consider it the bug. [1] https://en.wikipedia.org/wiki/GIF#Example_GIF_file https://en.wikipedia.org/wiki/GIF#Example_GIF_file [2] https://www.w3.org/Graphics/GIF/spec-gif89a.txt https://www.w3.org/Graphics/GIF/spec-gif89a.txt [3] http://hexfiend.com http://hexfiend.com [4] https://github.com/HexFiend/HexFiend/blob/master/templates/Images/GIF.tcl https://github.com/HexFiend/HexFiend/blob/master/templates/I... [5] https://www.synalysis.net https://www.synalysis.net
  - spuz 3y ago
    Another good option is ImHex which is an open source hex editor that supports file patterns. The gif pattern is one of the patterns already available: https://github.com/WerWolv/ImHex https://github.com/WerWolv/ImHex
  - tchebb 3y ago
    Kaitai Struct[1] is my preferred tool for parsing binary files. It's open-source (unlike Synalize It! and 010 Editor) and cross-platform (unlike HexFiend). Not sure how it compares to ImHex's pattern language, though—I've been put off by ImHex's UI the few times I've tried it. My favorite thing about the Kaitai Web IDE is that it instantly updates the parsed tree as you make changes to the format specification, which makes it viable for reversing unknown formats instead of just specifying known ones. [1] https://ide.kaitai.io/ https://ide.kaitai.io/

dblitt 3y ago

> For security reasons, we do not have Chrome crash reporting enabled. > We also confirmed with many of our affected users that they had Grammarly installed on their computers. Ironic.

roozbeh18 3y ago
haha, that was questionable for me as well. It's ok for Grammarly to read your stuff, but crash metadata is a no no.
- JohnMakin 3y ago
  Welcome to security in 2023 :)
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]

madeofpalk 3y ago

> Unfortunately, with access to neither the Chrome source code I mean, you basically do! You can just go check out the chromium source.

flutas 3y ago
> I mean, you basically do! You can just go check out the chromium source. It's mentioned in the bullet points in the "trouble reproducing the bug" section that chromium wasn't affected. > Using open-source Chromium instead of Chrome did not cause crashes, so we couldn’t see what Chrome code was failing either.
- j1mmie 3y ago
  I wonder what version of Chromium they used. If it was a nightly, it could be weeks before the fix makes it into Chrome. They might've tested at a time when latest Chrome had the bug and Chromium didn't.
- orbv 3y ago
  Google provides symbols for Chrome release builds, including source indexing so source code should be available. See sections symbol server and source indexing at https://www.chromium.org/developers/how-tos/debugging-on-windows/ https://www.chromium.org/developers/how-tos/debugging-on-win...
  - trelliscoded 3y ago
    I wish this was upvoted more. This is the correct way to troubleshoot the bug, full stop. You can get symbolic stack traces with full arguments and source code on Windows in about 5-10 minutes for any Chrome crash by following these instructions. I always have a last change exception handler that fires up a WinDBG script on Windows for our chromium-based test runners, which reduces troubleshooting time to just a couple minutes in order to find the symbol in their bug database. Playing blackbox what-if games like the Gusto team is a waste of time and doesn't contribute any situation-specific knowledge to bugs.chromium.org.
  - 3y ago

simion314 3y ago

I am using chromium to print web pages to pdf, and I have some images that will crash chrome's to pdf process, I found nothing wrong with this images, the metadata is fine (nothing weird in it). The other bad thing it does not reproduce n my dev machine only on the production server , so nothing I can do, in rare cases an image will always crash crhomium, I find it, open it and re-export it and then it works.

LgWoodenBadger 3y ago
This sounds more like a hardware fault than something wrong with the software, especially since it doesn't seem to be deterministic. But stranger things have happened, and given the enormous surface area of a modern computer (hardware, software, drivers, state, etc.) can anything truly be deterministic?
- simion314 3y ago
  It happens with that image no matter what. I can have a html with 100 images and one bad image, I make one new html only with that image and it still has the problem. My guess is that probably a bug in a low level image decoder. My local machine has different kernel, different libs, plus I have different cpu,gpu and X11 on top so too much difference and I do not have the expertese to do aremote debug(or local)
  - saagarjha 3y ago
    If you can grab a crash log I am sure the Chrome team would be happy to look at it.
- saagarjha 3y ago
  Most bugs are software issues, though.

lloydatkinson 3y ago

Discord suffered (suffers?) from a similar thing with gifs. It is or was common for people to post specifically crafted gifs in channels, anyone viewing the channel immediately had their client crash. Discord client uses Electron, which is in turn Chromium.

j1mmie 3y ago

What an interesting conflux of tech to create this bug. That's the web in 2023. I would love to know if it was a Chromium bug that got resolved, but navigating this is tough: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=gif%20status%3AFixed https://bugs.chromium.org/p/chromium/issues/list?can=1&q=gif... Also, I am fully here for Gusto posting this to say "wasn't our fault" and to throw some shade at Grammarly in the process

thenoblesunfish 3y ago
Seems like they're posting it because it was a fun story, and it's free advertising - this wasn't externally visible so I don't see where fault comes into it.
nonethewiser 3y ago
They should definitely publish the gif
j16sdiz 3y ago
Something like this https://bugs.chromium.org/p/chromium/issues/detail?id=1297706&q=&can=1 https://bugs.chromium.org/p/chromium/issues/detail?id=129770...

computerfriend 3y ago

If they can figure it out, they're sitting on potentially a very valuable exploit.

jdminhbg 3y ago
One that's been patched already, though, as they say that in current versions of Chrome and Grammarly it doesn't crash.
chatmasta 3y ago
Yes, I too would like to read more details about this. It's a great writeup from an engineer who got stuck debugging this. But I hope some experts in security or reverse-engineering can replicate it and take a closer look. There's definitely a more interesting story here, probably regarding the localhost bridge between Grammarly extension and desktop. (Grammarly has a bug bounty btw... and their chrome extension has quite a large surface area...) If OP is here: can you provide the raw .gif file? (And if you're feeling generous, maybe even a minimal ruby example that replicates that templating setup, although it sounds like that wasn't required to reproduce it in the end.) P.S. "For security reasons, we do not have Chrome crash reporting enabled" - maybe consider disabling Grammarly extension for the same reasons ;)
- saulpw 3y ago
  It wasn't the Grammarly extension, it was the desktop app.
  - chatmasta 3y ago
    I guess I just assumed the extension was installed too, and communicating with the desktop app. But now I see the post doesn't mention the extension. If it was triggered even without the presence of the extension then that's quite strange, and even more suspicious - is that gif triggering a call to a localhost endpoint? Is the grammarly desktop app interacting with browser elements without using the extension? (IIRC the grammarly app uses some accessibility privileges to inject into textareas across all apps) Grammarly is honestly insane, I can't believe corporations allow it to run on employee machines.

Modified3019 3y ago

The true bug in the photo is a "candy-striped leafhopper", Graphocephala coccinea, which is tiny but has very striking coloration. The Larvae of leafhoppers are commonly known as spittlebugs, which create protective bubble nests while feeding on plant stems

deleted 3y ago
[deleted]

barbegal 3y ago

A disappointing ending to the tale. I really want to know how Grammarly desktop works now. It must have interfered with the Chrome process in some bad way.

deleted 3y ago
[deleted]

whirlwin 3y ago

Shared library like libgif used by both Chrome and Grammarly but different versions?

JohnMakin 3y ago

I have a saying that isn't perfectly true but often will apply to "fixes" like this - If you don't know why the fix worked, you may not have actually fixed it.

bicijay 3y ago
But you may have
- lbhdc 3y ago
  Ahhh, Schrodinger's patch.
  - HPsquared 3y ago
    Schrödinger's bug?
gwbas1c 3y ago
But they couldn't fix the bug: The bug was in another product that they couldn't access source code or submit patches. The best they could do was work around it. Sometimes workarounds are the best you can do until your vendor provides a real fix.
- JohnMakin 3y ago
  I didn't mean literally fix the underlying bug. They also don't really know why their workaround worked, which means it could not really be fixed at all.
tetha 3y ago
Someone once said, there is accidental function, and deliberate function. If your system doesn't work, and you just plonk around at values, until, very surprisingly, the system starts behaving well and you the call it working... well it might be working now. But it's just accidental correctness. As soon as something causes the system to bank left, something's gonna break and no one knows how to fix it - and you're back to square one. On the other hand, as hard as it is, if you can clearly tell why your fix will restore function to the system without even applying it, you have deliberate correctness and function. If done right, it is very boring, because exactly and only the expected thing will happen. You should know about the unknowns and plan around those as well, so even if an unknown bites you, it's a known and handled unknown. This can be exhausting to make happen, because it is much harder, but those systems will just work. But this is a fight I have with some development teams probably forever. "But we poked at the values, and that stopped the flames. It is fixed!" "but why?" "Dunny. But no fire anymore. All good." And then 2 weeks pass, and there is more fire and everyone is like "Oh but why would this happen? How should we have known for this to happen again"

djbusby 3y ago

Thought this was going to be about Gusto "remember this device" which keeps failing. Reported like 2 years ago

deleted 3y ago
[deleted]

sonicanatidae 3y ago

Twirly prompts have sucked, for weird reasons, since the WWIV BBS days. I guess some things never change.

zwieback 3y ago

Left me hanging, would not post something like this from my engineering blog. They don't have deeep debugging skills at gusto?

golergka 3y ago
Sounds like they have good time management and prioritisation skills. They found the source of the problem and fixed it.
- KerrAvon 3y ago
  No, they didn't. They figured out a workaround. Since they didn't find the root cause of the problem themselves and apparently didn't take it up with the Chrome or Grammarly development teams, they don't actually know what happened or when it might bite their customers again.
  - jdminhbg 3y ago
    They say it no longer reproduces on current Chrome/Grammarly, so taking it up with them is fruitless. They're not going to investigate crashers from old versions.
- dhritzkiv 3y ago
  :( I would've tried to determine the cause of the crash with that specific file in my off time, provided that I could isolate the code in the Grammarly extension in Chrome. The main reason –other than curiousity– is to ensure that a future regression (in Chrome/Grammarly) wouldn't lead to it again.
deleted 3y ago
[deleted]

cyco130 3y ago

The weirdest one I saw was this: User claims that the wording of the info they enter into a certain form changes when they save. At first I suspected someone else editing the same form at the same time unbeknownst to each other but it wasn't the case according to the logs. And I saw the correct wording on my own computer. Then I noticed in their screenshots that some of the menus had weird wording too. Turns out they had Chrome's "Translate this page" option on. Problem went away when we showed them how to properly switch languages in the app.

bee_rider 3y ago
What was translating from English to American or something?
- cyco130 3y ago
  From English to Turkish but Google won't just leave the parts that are already in Turkish alone and subtly reword them instead for some reason.
robocat 3y ago
I added <meta name="google" content="notranslate"> to all pages in a single-page-web-app after discovering some bug or other with Chrome screwing up the page. Apparently the new incantation to fix an app (can be applied to an element) is (ugly: I presume it isn't CSS to avoid supporting dynamically changing it): <html translate="no"> https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes/translate https://developer.mozilla.org/en-US/docs/Web/HTML/Global_att... Every now and then I would look at the meta tags for a major single page app and discover some new horror when searching for the reason for the meta tag!
- michaelcampbell 3y ago
  I know what you mean, but this caused me a second of "wait, what?" > all pages in a single-page-web-app
  - 3y ago

jrockway 3y ago

Did they open a bug against Chrome with the image file? I feel like any crash on user-provided data is a big deal, always a correctness problem, but potentially a security problem. "We deleted the image so the problem is fixed for us" is OK (I wouldn't waste time writing a blog post about it personally), but I think that Chrome needs to fix this bug.

tedivm 3y ago
Was it actually a Chrome bug though? It only happened when the Grammerly desktop app was installed. My guess is grammerly is doing something sketchy.
- bayindirh 3y ago
  Maybe adding Grammarly created enough of a lag causing the GIF file to be shown?
  - nonethewiser 3y ago
    That doesnt hold when you consider they opened the gif in the browser with and without grammarly and it only crashed with. So its not simply a bad gif and chrome.
- AnimalMuppet 3y ago
  Does Grammerly hook something in Chrome? If not, then it's still probably a Chrome bug, even if some second-order effect of Grammerly is necessary to trigger it.
  - majormunky 3y ago
    It looks like the desktop Grammerly app hooks into all sorts of things, "An all-in-one writing assistant that works on your desktop and in your browser. Use it in apps, word processors, email clients, and more."
  - MattDaEskimo 3y ago

user3939382 3y ago

Reminds me of the story I read where the guy's car wouldn't start depending on what flavor of ice cream he picked and when investigated he was right. Some kind of evaporation/vacuum leak or something that was dependent on time and some flavors were farther away in the store and took more time to buy.

nudgeee 3y ago
Vapor lock. Legend here: https://www.snopes.com/fact-check/cone-of-silence/ https://www.snopes.com/fact-check/cone-of-silence/
- RajT88 3y ago
  I heard a similar tale in high school. A friend of mine had an aunt who passed away, and so he ended up inheriting her car. The car came with a petrified apple pie in the back. He was insistent that the car would not start without the pie in the back window. Several of his friends who he played in a punk band with confirmed this, that they had tested it. Take the pie out, car won't start. Put the pie back in, the car starts. I don't think anyone ever figured out what was going on, I graduated a couple of months after hearing the story, and fell out of touch. But - timing and vapor lock makes sense, if they were always testing it by first starting the car, removing the pie, and then putting the pie back in. As an aside, the aunt who had passed away was one Aunt Martha (after which the car was also named), which in honor of the strange car and its strange pie was what their garage punk band was named after. There's some totally unrelated band now called Aunt Martha - any evidence of their band is not on the internet.
superfrank 3y ago
Up there with the "I can't send an email over 500 miles" story https://web.mit.edu/jemorris/humor/500-miles https://web.mit.edu/jemorris/humor/500-miles
- suzzer99 3y ago
  I love this one so much.

mgaunard 3y ago

The main reason for tabs to crash is running out of RAM. Never do you see the guy investigating memory usage, which is weird.

marcellus23 3y ago
Yeah, that would have been a good first step, but he does admit to not really knowing much about browsers: > This was fairly far outside the usual scope of our on-call issues. Our team is generally well-insulated by other teams from issues like browser compatibility, so I didn’t know the first thing about browser debugging.
- whimsicalpedal 3y ago
  [flagged]
gwbas1c 3y ago
If that were the case, I think the bug would be much easier to reproduce; and be a lot more widespread.

NotYourLawyer 3y ago

[flagged]

neilv 3y ago

Why would they disable crash reporting for security reasons (which might actually help solve the root cause of their availability problem, which they never did solve)... yet run Grammarly (which I'd guess, security-wise, is less trustworthy than Google, in how they secure data themselves once they've inevitably stolen it from the customer)?

Zetobal 3y ago
Maybe they have the enterprise licence with grammarlys pinky swear that they won't train on your data.
deleted 3y ago
[deleted]

hilux 3y ago

What a cool mystery-solving post! I wish all technology writing were this clear and explanatory. For another fun debugging tale, google: Mazda radio Seattle NPR bug

EdwardDiego 3y ago
Holy hell lol.

realmike33 3y ago

This reminds me of similar issues I've encountered as a software engineer. I first ran into this issue about a decade ago, albeit not because of Grammerly,but due to some specific gif causing web app to crash. Both times the gifs were animated. Happened years apart and at different companies. I see some comments highlighting RAM, which could totally have been the issue. Totally looking forward to a follow up to this later down the road, I am sure this isn't going to be the last time we hear of this.

neilv 3y ago

> Using open-source Chromium instead of Chrome did not cause crashes, so we couldn’t see what Chrome code was failing either. They don't address why they didn't just run Chromium. Or Firefox. (This is potentially better than the 'solution' they much later ended up with, in which they probably only relieved a symptom of an underlying problem that can exhibit again, and in the meantime is a zero-day exploit waiting to happen. At least, with a different browser, there's a chance that the vulnerability doesn't actually exist, when it's known to exist in their Chrome configuration.)

Etheryte 3y ago
What makes this doubly frustrating is that they also didn't report the bug to Chrome. It's super easy to do, plus they're very responsive if you have a repro case which in this case they do. I think I'm now up to three or four Chrome bugs reported that their team has subsequently fixed.
- ncann 3y ago
  They said it wasn't reproducible anymore though. So if they make a bug report now and say "this used to cause a crash in an old version of Chrome while also having an old version of another software installed, but is no longer reproducible in latest builds", most people would probably just ignore it.
  - masto 3y ago
    It was reproducible at the time they found it, and trivially so: install Grammarly, drag this GIF into Chrome, and it crashes. I understood everything up to the point where they just changed the GIF and moved on without ever telling the Chrome or Grammarly folks about it.
gwbas1c 3y ago
> They don't address why they didn't just run Chromium. Or Firefox. The article implies that Gusto's employees can whatever browser they want. And, honestly, telling your employees to run a browser that only techies have heard of sounds like a really dumb idea.

1123581321 3y ago

How could you write all this and not post the gif so we can try it?

deleted 3y ago
[deleted]

andrewfromx 3y ago

oh and there's this one: https://web.mit.edu/jemorris/humor/500-miles https://web.mit.edu/jemorris/humor/500-miles

tgsovlerkhgsel 3y ago

Extremely disappointing that they seem to have neither investigated nor enabled others to investigate (e.g. by filing a bug against Chrome). This smells like a potential security vulnerability.

ljm 3y ago
I was hoping for a bit more of a payoff. Like, if the gif was broken, why did the grammarly extension trigger it reliably?
deleted 3y ago
[deleted]

deleted 3y ago

[deleted]

jiveturkey 3y ago

I'm guessing this is the webp bug. The auto conversion to webp on the backend, signaled by chrome, resulted in a bad image that crashes the browser due to grammarly parsing of said bad webp. Safari doesn't tell the server it does webp and so it downloads the actual gif, and doesn't crash.

saagarjha 3y ago
GIFs don’t get automatically converted to webp files.
- jiveturkey 3y ago
  says who? any web server (nginx, whatever) can be told to do this
  - saagarjha 3y ago
    Yes, but why would it?
    - jiveturkey 3y ago
      because it's super convenient, and improves performance?

romanhn 3y ago

Another favorite bug investigation of you're into this sort of thing: https://www.pagerduty.com/blog/the-discovery-of-apache-zookeepers-poison-packet/ https://www.pagerduty.com/blog/the-discovery-of-apache-zooke...

ecshafer 3y ago

I don't think I would believe myself if I found this being a specific gif. This is a great amount of coincidences in code to cause this. Grammarly is an application that I don't get. The fact that people are installing, basically spyware, on their computers just to get grammar suggestions to make their writing more boring and add a spellchecker (which is already inside web browsers) is pretty astounding to me. The fact companies allow employees to have it, despite obvious security issues of sending everything one types to a saas, is even more wild.

eichin 3y ago
I worked with someone who really needed it, but we had the usual "keep sales users as far from the actual product as possible" organizational isolation so it worked out in practice. (For engineering, it was on the "don't install this in particular" list.)
_jal 3y ago
> Grammarly is an application that I don't get. You write like a native speaker, so I'm not surprised. But imagine having a few years of school-German, and then taking a German language job. I'd bet there would be times you'd want a writing assistant, too. There are also plenty of native English speakers who for whatever reason got a crappy education, and didn't get a lot of writing feedback. As far as corporate security goes, you are correct, and we ban it. But I get why people want it.
- ryandrake 3y ago
  > As far as corporate security goes, you are correct, and we ban it. But I get why people want it. That is what stuck out to me: Installing rando applications on your corporate computer that has access to internal stuff... Whoooaaaa Baby! That's just a security disaster waiting to happen. It's stuff like this that eventually leads to draconian and crappy "Nobody gets admin access to their machines" corporate policies coming down. Most TechCorp places I worked, if someone installed something like that on their corporate device, they'd get at least a stern talking-to and probably sent back to security training.
- 3y ago

hartator 3y ago

Maybe the name was odd `loader-spinner.gif`

FrankWilhoit 3y ago

Error code 5, on Windows, means that the code tried to dereference a null pointer. I'm guessing that the GIF content is corrupt, containing some 0x00 bytes where they shouldn't be. Then the question becomes, whose responsibility is it to program defensively against things like that? If, as may well be, Chrome is using some third-party library nested several layers deep to render GIFs, then would there be any action that the chrome devs could take, aside from replacing that library with a better one and adding a malformed-GIF test case? (Why don't they already have a malformed-GIF test case...?)

deleted 3y ago
[deleted]
londons_explore 3y ago
Lets be honest, it was probably some grammarly .dll that had been injected...
saagarjha 3y ago
Considering they’re trying Safari I suspect they’re on macOS.

Night_Thastus 3y ago

I've actually seen something like this in the wild myself. For awhile there were some GIFs that if placed in Discord, would cause it to crash for everyone who was looking at the chat. Admins had a fun day when that was found!

sethammons 3y ago

> The code for our main navigation bar has a fair amount of metaprogramming, and chasing down threads here was often more confusing than not. One more point for Don't Be Clever. As Brian Kernighan put it: “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”

deleted 3y ago
[deleted]

ja5087 3y ago

We used to develop software that used the Windows Accessibility APIs (UI Automation). On certain versions of Excel with some files it would crash the client application with a null pointer exception once you try to read the window name/class. It would be interesting to see the cause of the crash e.g. a core dump/user-mode dump/event viewer log.

bluesmoon 3y ago

Reminds me of the time back in 2010 when a piece of CSS on the Yahoo Search page would cause a complete desktop crash on Red Hat Linux: https://tech.bluesmoon.info/2010/04/can-website-crash-your-rhel5-desktop.html https://tech.bluesmoon.info/2010/04/can-website-crash-your-r... To the author, did you ever consider contacting the Chrome dev team about this? They're pretty responsive to bug reports.

rvanlaar 3y ago

Ah, Chrome and slow spinners. Python tests were taking ages on VSCode due to an SVG spinner: https://bugs.chromium.org/p/chromium/issues/detail?id=1036262 https://bugs.chromium.org/p/chromium/issues/detail?id=103626... https://github.com/microsoft/vscode-python/issues/9216 https://github.com/microsoft/vscode-python/issues/9216

deleted 3y ago
[deleted]

boringuser2 3y ago

The blink debug logs would probably be pretty useful for the engineers involved...

rootsudo 3y ago

For security reasons, your organization disables Chrome crash reports but allows the use of Grammarly, an app that essentially functions as a keylogger. Consented or not, it’s a keylogger. https://support.grammarly.com/hc/en-us/articles/360003816032-Is-Grammarly-a-keylogger- https://support.grammarly.com/hc/en-us/articles/360003816032...

crazygringo 3y ago

tl;dr: a certain GIF would crash a Chrome tab, but only when the desktop version of Grammarly was installed. (Not a Chrome extension.) That's insane! Can anyone think by what possible mechanism the installation of Grammarly could affect whether a .gif file would crash Chrome? The company seems to be on Macs since they report that the problem doesn't surface in Safari. Is there some kind of dynamically-linked GIF decoding library used by macOS that Chrome relies upon, and Grammarly somehow installs one that takes precedence for all applications? I didn't think this would be possible -- I thought image decoding was done natively in the browser and not outsourced to the OS, for security reasons.

saagarjha 3y ago
This would be strange since macOS should not let Chrome load libraries that aren’t signed by Google.
masswerk 3y ago
Maybe, the GIF contains a text comment and Grammarly not only accesses this, but also alters it (for whatever reason), thus overwriting the file in a way in memory that offsets in the GIF file or its in-memory representation are violated?
mike_hearn 3y ago
It's not an uncommon problem unfortunately. On Windows the OS lets programs inject DLLs into each other's address spaces, and this is actually a very common programming technique for various reasons. Apps don't expect it to happen of course, and so this is a fertile field of bugs. Chrome does in fact attempt to block some apps from doing this because they're known to cause crashes. You can see the blacklist here: https://source.chromium.org/chromium/chromium/src/+/main:chrome/chrome_elf/third_party_dlls/hardcoded_blocklist.cc;l=19?q=inject%20blocklist&ss=chromium%2Fchromium%2Fsrc https://source.chromium.org/chromium/chromium/src/+/main:chr... They don't like apps doing it but you can apply for an exception: https://sites.google.com/a/chromium.org/dev/Home/third-party-developers https://sites.google.com/a/chromium.org/dev/Home/third-party... So my guess is that Grammarly is injecting code into other processes, probably to try and gain access to their text controls so it can Grammarlyze stuff, and somewhere along the line corrupts the heap or stack in some way. The GIF aspect might be a red herring, as the crash was IIUC never reliably reproducible. Random crashes that happen sometimes but not others is strongly indicative of something being randomly corrupted. Or it could be something else. Chrome moved to blocking third party injects by default some time ago, so maybe that's also a red herring.

hcrean 3y ago

I wonder if they checked for exploit code in the image the they likely originally found somewhere on Google.

toddmorey 3y ago

Once my college professor was working on her research paper and told me she was struggling get text to stay underlined. Assuming a simple user error, I expected to help her out in 5 minutes. Over three hours later, we discovered that the combo of her specific video card driver version along with her specific printer driver version would keep text from printing out underlined.

generationP 3y ago
Huh? How does a video card affect printing?
- jfoutz 3y ago
  A lot of rendering will go through the video card if available, like the jvm does this as an optimization.
  - toddmorey 3y ago
    Ah interesting. That makes sense. I had no idea.
- shever73 3y ago
  It often did, particularly on older versions of Windows. I helped uncover a bug in Epson printer drivers ~20 years ago that was caused by a specific graphics card.
  - shermantanktop 3y ago
    20+ years ago I was in tech support and had to help someone figure out an issue where her document wouldn't print on a Brother printer. Turns out a section divider line would block the entire doc from printing (by crashing the app) if the line's end-cap style was set to square rather than rounded.
- to11mtm 3y ago
  One of the ways to print things (especially on windows) is Via GDI. [0] Basically using the OS's rendering to make a raster that is then sent to the printer. The main thing the printer's driver does in this case is know how to take the bitmap and tell the printer to print the bitmap (i.e. chunking data and/or sliding the right commands into the bitmap stream) Contrast to, say, PostScript which allow for more compact and better scaling definition of what to print. This obviously works better for quality, however for a long time the issue was you then had to have sufficient processing power on the printer itself to handle it. [0] - Search for 'GDI Printer' for a little more info.

nialv7 3y ago

I feel the author might have missed out on a multi-thousand dollar bug bounty.

aaaronic 3y ago

This is _so_ familiar! I have seen accessibility tools in Chrome lead to this kind of issue in the past with a dropdown menu -- to the point where it could be replicated with a miniscule amount of HTML. The particular bug I hit 2 years ago was in Chromium-Edge, but the symptoms and cause were very similar. Grammarly almost certainly leans on some of the accessibility tools in Chrome. These tools are somewhat different in the various Chromium flavors (Edge, Brave, Chrome, etc.).

nonethewiser 3y ago
So the theory would be that grammarly desktop sees the gif (what? How?) and calls some browser accessibility function on it (or?) which chrome cant handle and it crashes?
- saagarjha 3y ago
  Perhaps it has an extension that it installs that does this?
  - codethief 3y ago
    They did try disabling browser extensions, according to the blog post.
- aaaronic 3y ago
  With the bug I saw years ago, just having certain accessibility features of the browser enabled _at all_ caused the bug (we were able to temporarily mitigate by disabling some obscure Edge accessibility feature via a launch parameter). So, my theory here is Grammarly is just enabling an optional accessibility feature in Chrome that has this bug when trying to "read" the gif.

gelatocar 3y ago

As I was reading this I was thinking to myself "I wonder if it is grammarly related" because I experienced a bug some time ago that presented itself in a similar way. It was impossible to reproduce but affecting lots of people internally within certain departments. Eventually we figured out the thing they had in common was that they had the Grammarly extension installed. The other key thing was that the bug only appeared on our staging preview urls, not on the live website. It turned out it was because of a bad regex in the grammarly extension that caused the page to hang if the domain name was more than about 100 characters. Our staging domains were pretty long, I think they contained a few subdomains and had a job number or something in there. This one is more crazy though if it is really caused by the desktop app - that's pretty scary!

codethief 3y ago
> It turned out it was because of a bad regex in the grammarly extension that caused the page to hang if the domain name was more than about 100 characters. Just today I debugged a regex that would DoS our backend whenever the user enters the wrong thing in a form. Now I'm reading up on regex engines: https://swtch.com/%7Ersc/regexp/regexp1.html https://swtch.com/%7Ersc/regexp/regexp1.html
- dev_slash_null 3y ago
  Just in case you haven't seen the postmortem of the Cloudflare outage which also was caused by a regex based DoS: https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ https://blog.cloudflare.com/details-of-the-cloudflare-outage...
  - radiojosh 3y ago
    That was a great read, but there was one thing I didn't understand: Why would the regex string have "." twice in a row? What does ".." find that "." doesn't find? Does that just mean "at least two characters"?
    - 3y ago

aquafox 3y ago

I randomly had the issue that after booting up Linux, I didn't have any sound. Turns out it was related to my Windows dual-boot setup! When restarting from Windows, Windows doesn't shut down my realtek audio device, but only puts it to sleep and Linux fails to start it. Only solution is to always do a shutdown from Windows and then hitting the power button. The issue is still there: https://askubuntu.com/questions/1032543/no-sound-in-ubuntu-18-04-after-rebooting-from-windows-10/1078134#1078134 https://askubuntu.com/questions/1032543/no-sound-in-ubuntu-1...

nightfly 3y ago
I've seen the opposite-ish: Someone who was dual booting windows and linux only had working wifi if they booted into windows and rebooted into linux. The linux install didn't have the firmware package for their wifi card installed, so when rebooting from windows into linux it was still all primed and ready to go but not when cold booting into linux directly.
- sodality2 3y ago
  This is my exact problem with a HP envy X360. HP refuses to fix the bug in BIOS. :/ https://bugzilla.redhat.com/show_bug.cgi?id=2107845 https://bugzilla.redhat.com/show_bug.cgi?id=2107845
  - 6510 3y ago
    Every other day I read some fascinating HP adventure. I've done some questing myself, cant really tell if I did well, are they new multiple issues or the good old same? Your comment had me wonder if a kickstarter for an rpg game called HP printers would take off. The potential fan base is HUGE, I imagine even people who would never play a game would want to see it. Let the GUI be like little computer people (only an office) https://youtu.be/SkTgX1mGmDg?t=17 https://youtu.be/SkTgX1mGmDg?t=17 Then the player has to go though rituals depending on the HP printer type, swap old for new then new for old then old for new ink cartridges after finding the instant ink package in storage, if any spawned of course. If you try to go to fast the hero starts destroying the office, starting with the printer. Then a new different model is delivered. Ready for setup.... or maybe not? In some laps of consciousness I purchase one more some years ago. The hours of configuring it made me feel dumber than the wrath of the HP dungeon master. The ISP has a wifi router fixed on channel 11, half the neighborhood sits on channel 11, range is about 4 meters, the printer doesn't reconnect unless unplugged first. But there is wifi direct, this doesn't have 50 cm of range, it wants to use channel 11. in the game our hero at least at first doesn't know about channels. The next printer arrives: Something seems off, there is no usb cable in the box? Do you want to A) order a cable B) we will use wifi! Of course if you order the cable the boss will complaint that it takes to long to set up the printer but when you eventually get the cable the boss wants to print from his phone... https://freesound.org/people/InspectorJ/sounds/402095/ https://freesound.org/people/InspectorJ/sounds/402095/

sfink 3y ago

Next time this happens, I recommend just letting people use a different browser. Firefox in particular has gotten much better at importing bookmarks, passwords, etc. from Chrome. It was a Sign from the universe that it was time to make the switch. Who are we to reject Signs? (Full disclosure: I'm an engineer on Firefox. But that has nothing whatsoever to do with my advice here, no siree Bob, not in the least.)

mplewis 3y ago
This is explicitly stated as a workaround in the linked article.
OsrsNeedsf2P 3y ago
As an engineer, yes Firefox is a good solution As a PM, we spent 4 months making the onboard easier, and now you want people to install a new browser?
- noizejoy 3y ago
  Having multiple browsers as a standard installation and employee training is an investment in business continuity. And business continuity investment is like insurance: A waste until you need it. Too much monoculture is short sighted and long term expensive.
  - hirsin 3y ago
    Do you ship them extra computers too? Dual boot? There's gotta be a limit somewhere, and browser choice is a fair one. For engineers, absolutely, two browsers because some are liable to hit (self or team induced) bugs. But for sales? Support? Nah.
- Shaanie 3y ago
  I run into Firefox compatibility issues waaay more frequently than chrome compatibility issues, for obvious reasons. Even extremely obvious issues, like 2fa login not working on mobile Firefox, are surprisingly common to find, so I'd never install Firefox as a primary browser for non-techies nowadays. I use Firefox myself for ideological reasons, though.

denton-scratch 3y ago

Awww. I was really enjoying that; I like detective stories and rabbit-holes. Then we get to the punchline: "Uh, we fixed the bug, but sorry folks; we didn't solve the puzzle". So I guess we'll never know why that particular anigif crashed Crome but only Chrome, and only if Grammarly was installed (or had been installed during the same session). I hope Amy Lai lets us know if the story ever gets an ending!

deleted 3y ago
[deleted]

lantry 3y ago

> Turbo is a gem we added to speed up our Rails application, but it turned out to be a red herring: it was only introduced after the bug had already been reported, and we learned that the engineer who introduced it had actually been experiencing these Chrome crashes for months prior to Turbo being introduced, and months prior to the bug being escalated to us. lolol I can just imagine the engineer working on this: "huh, it keeps crashing, that's weird. Oh well, it was like that when I got here. ship it"

CGamesPlay 3y ago

I love the corporate security policy that disables Chrome crash reporting for security reasons, but allows employees to install Grammarly.

NautilusWave 3y ago

This reminds me of an old Chrome bug we ran up against involving the auto fill feature. All day, our software users are entering patient data, including address information. Chrome would grind to a halt on certain pages trying to provide all their previous entries as auto fill options to the form fields. Our sysadmins disabled the Chrome feature for all of our internal users and the issue disappeared.

numitus 3y ago

Interesting advertisment, but not really a tech article.

EdwardDiego 3y ago
Wrong article?

resonious 3y ago

I gotta say, the ending was pretty disappointing. I was so hyped to learn why Grammarly crashes when Chrome loads a particular gif. What about the gif was different? What part of Grammarly made that happen? I hate to say it but this is modern web dev in a nutshell. Remove stuff until you find where the bug is, tweak it randomly until it's fixed, done. No root cause analysis or anything deep. This is a great post anyway. Well written and still quite intriguing right up until the end. And it seems lots of comments on here seem to also know about this problem, so I think I can still satisfy my curiosity.

deleted 3y ago
[deleted]
eyelidlessness 3y ago
> I hate to say it but this is modern web dev in a nutshell. Remove stuff until you find where the bug is, tweak it randomly until it's fixed, done. I’ve alternated between frontend, “full stack”, and backend roles for over 20 years. It’s my experience that what you’re describing is “dev in a nutshell”—neither “modern” nor “web” in the sense you seem to mean. And in any case it’s highly situational and variable depending on the dev and their team.
randshift 3y ago
They never root caused the bug! They know a symptom (Grammarly and this gif cause things to crash) but still no root cause. I was pretty disappointed at the end.
- sanderjd 3y ago
  It was like reading a Sherlock Holmes story where they just decide to put Moriarty in jail and that fixes everything, but Holmes doesn't actually figure out what was going on.
karim79 3y ago
> When I removed loader-spinner.gif, the placeholder we display while the menu options load, the page stopped crashing. Eureka! It’s the gif! We swapped in a different gif and the page did not crash. I would love to have the original and the un-crashy gifs mentioned. It's super easy, generally, and even without an extensive knowledge of image formats to get a grasp of what might be going on and then going down some really exciting rabbit-hole of image encoding/decoding issues. Just take the two gifs and run them (one by one) through ImageMagick or GraphicsMagick to print out the details of what's in them, and look for differences. Assuming ImageMagick is installed (or GraphicsMagick) installed, something like: #imagemagick $ diff <(identify -verbose loader-spinner-CRASHY.gif) <(identify -verbose loader-spinner.gif) #graphicsmagick $ diff <(gm identify -verbose loader-spinner-CRASHY.gif) <(gm identify -verbose loader-spinner.gif) ...and rabbit-hole away EDIT: formatting

userbinator 3y ago

I agree with the others here about the ending being a total letdown. Unfortunately, with access to neither the Chrome source code nor the Grammarly source code, we can only guess. Is this what the "open source" movement has created --- developers who are totally lost without source code and refuse to dig deeper? Of course the corporate interests who don't want us to know the truth, because their profit depends on it, would absolutely adore such propaganda... I still remember a time when a lot of people would disassemble, understand, and patch programs without source --- and many of them weren't even career developers; it was just a way to get software to do what one wanted, and driven by that motivation, one would naturally learn enough to do so. The article also touches on another point worth mentioning: the amount of complexity in the whole stack is insane. Seeing all the frameworks upon frameworks being name-dropped, I can't help but feel like a lot of this is self-inflicted. When I removed loader-spinner.gif, the placeholder we display while the menu options load, the page stopped crashing. Do menu options take long enough to load that they need a loading animation?

hmry 3y ago
Presumably they make a network request to get the options. So it makes sense to have a loading spinner just in case it takes a long time, even if it's usually near-instant.
windowsworkstoo 3y ago
Very well said - even with open source there seems to be a general lack of willingness to actually read code, let alone crack open the disassembler or attach a debugger - the skill is apparently not taught anymore
cyclotron3k 3y ago
Expecting every developer to be able to patch a binary is, to put it politely, unrealistic. The world of software engineering is enormous, and knowing all parts of the software stack hasn't been remotely feasible for a very long time. Also consider the fact that software engineering is a journey of learning, and every one of us is on a different part of that journey.

andershaig 3y ago

I can't remember the details anymore but the craziest bug I ever found was one that would crash the page whenever Chrome's Developer Tools were opened. I ended up having to use an embedded JS-based dev tool to figure out what was going on (I was a pretty junior developer and didn't have any better ideas at the time).

DaleCurtis 3y ago

Even if you have crash reporting disabled there should be a .dmp generated somewhere in the user profile directory. Manually uploading that to a bug at https://crbug.com/new https://crbug.com/new would allow a Chrome developer to debug it. If you can't share the dump for similar reasons to why you have crash reporting disabled, you can build minidump_stackwalk from Chromium and use it to generate an unsymbolized stack trace that you can post to the bug. A Chrome developer can then symbolize it. https://www.chromium.org/developers/decoding-crash-dumps/ https://www.chromium.org/developers/decoding-crash-dumps/ has some more details.

LASR 3y ago

I discovered a bug like this with GIFs too. It turns out some gifs can be encoded such that it is animated and a single frame long. Can’t remember the exact event name now, but the browser fires a play event when gifs loop. This particular gif was issuing too many of these play events that made the app super slow and freeze since it was doing some work in those handlers. We had a bunch of crash reports and bug reports. None of those showed the actual gifs our customers were using. When we asked for the actual gif, we immediately spotted the problem.

1letterunixname 3y ago

Spoiler alert But if you read the article... ;@) Improper animated GIF decoding is potentially a bug and potentially a security vulnerability. The reality is this leads to duck-tape workarounds and greater tech debt on the production side of the web. Sigh. There are standards, there are expectations, there quirks across N implementations, and MxN layers of duck tape in M consuming implementations. Just one implementation intentionally or unintentionally being different causes M headaches. Also fun Chromium-derived browserisms: - Updating while open is allowed and leads to silent and not-so-silent breakage. - MSFT MDE causes unexpected breakage in fun ways every now and then, including the cause of a crash while dragging a tab on Windows. - Some flavors of Chromium browsers are broken with IPv6 enabled, leading to an ERR_NETWORK_CHANGED on every nth page visit.

slavboj 3y ago

Grammarly is an ipso facto security hole that is extremely likely to be pwned or controlled outright by a number of state security services. It should be treated like malware.

jtokoph 3y ago
It’s hard blocked on any corporate devices where I work.
fijiaarone 3y ago
Ah, Tsutomo, my learned disciple!

jongjong 3y ago

Grammarly is awful. I remember adding custom grammarly attributes to sensitive password fields so that Grammarly would not read and store our users' passwords inside their service. So nasty. This company would sure make a great asset for the FBI, NSA and CIA given that they're so interested in snooping in on foreign language speakers; which happen to be grammarly's main demographic. The thing is malware.

jongjong 3y ago
Update: After writing this comment, me and my wife started getting Grammarly ads. Awful.

deleted 3y ago

[deleted]

socketcluster 3y ago

The weirdest bug I've seen yet was a concurrency race condition which led to 'spooky action at a distance'. Every time I tried to console.log() to observe some value to see the cause of the bug, the bug would go away. But alas, I hadn't discovered some new quantum effect... It turns out to have been because the race condition was close within maybe 1 millisecond and adding the console.log() statement there meant that one part of the code would take a bit longer to execute and so the race condition would not occur.

KevinMS 3y ago
https://en.wikipedia.org/wiki/Heisenbug https://en.wikipedia.org/wiki/Heisenbug
ghosty141 3y ago
In Qt QML using print can lead to different behavior very very quickly since their engine creates bindings on whatever it comes by so using two variables in a print statement can lead to different behavior than just using one. QML in general is just awful.

hrtk 3y ago

Can you share the `loader-spinner.gif` ? Preferably over a service that does not recompress it.

6510 3y ago

alert(banana++) Should lock things up and not continue until OK is clicked. If it is just you and your application you can just spray paint from the hip the alerts from top to bottom. After each line of html you can have one more <script>alert(banana++)</script>, in the middle of your css </style><script>alert(banana++)</script><style> etc If there are uhh thousands(?) of people actively using the page you put just one alert some place in the middle. Everyone will have to click on OK, the page crashes. You ask the crashee: Did it crash before or after the "make America great again" alert message. (call it something they would remember) Now you know the issue is in the top or bottom half. You move the alert to half way the half with the issue in it. If you can get some sort of reasonably consistent crashing you will find it pretty quick even in production. Hope this helps, or maybe it is a stupid idea and you could explain why.

deleted 3y ago

[deleted]

jspaetzel 3y ago

I've run into something similar to this a few years ago where chrome randomly introduced a rendering bug that resulted in a similar crash and only affected our application in seemingly random fashion. I don't remember the full context but I think it was something involving layers & transparency. We finally figured it out when a coworker couldn't replicate and we noticed they were a version of chrome behind. We were able to track down the specific commits in chrome that broke & fixed it in our case which was pretty cool to see at the time. If you have any more time I recommend reading through the commit log and see if you can find the changes that broke/fixed this for you. I'd bet on another rendering bug.

cristeigabriel 3y ago

If Chrome is able to generate a crash-report, nothing would prevent you to intercept it. > Unfortunately, with access to neither the Chrome source code [...] I mean, it's still very possible to debug (especially with the fact in mind that Chromium is open-source, and for me has been a very useful source when reverse-engineering and debugging Chrome), but I understand why web developers would not be trained in reverse-engineering techniques.

bradley13 3y ago

It is so satisfying to finally identify the root cause of such an obscure bug. Of course, along the way, you will shed tears. Sometimes literally. I had a funny one a couple of weeks ago: Text on a button would sometimes not display. After much experimentation, it turned out that the text would display unless the character 'D' was present. It turned out that the sizing of the buttons was just barely large enough for the font-size selected, and apparently the 'D' in the selected font was microscopically taller than other characters. FWIW I have to thank ChatGPT for helping solve that one. I explained the problem, and it gave me a list of things to try...

jon-wood 3y ago

The thing that struck me here is that Chrome crash reporting is disabled for security reasons, presumably not wanting to risk customer information being sent to a third party, but Grammerly is apparently fine? How is a tool that sends literally everything you type to a third party ok, but one that maybe sends something if the browser crashes isn't?

jaggirs 3y ago
Because doing that is a core part of Grammarly, i.e. they can argue in court that the user should obviously expect grammarly to send the data to their servers. Sending crash report data is not so, as it is not essential to using the service.

SonOfLilit 3y ago

When I reached the "Stroke of Good Luck" title, I put down my phone and tried to think what it could be based on what I know before the reveal. It was really hard to generate thoughts of the form "subsystem x causes the crash" and not of the form "try to do experiment x next", but here's what I came up with: - a headache trying to think about it - if it's not an extension, it's likely another piece of software that has code running in the browser, like PDF plugins used to before Chrome started handling PDFs itself - the crash is in one of the browser subsystems, so either in the JS engine, the GPU code, the layout engine, the parser, some format plugin (video? pdf?) or the network code. Some format plugin is most likely of these to only crash for tech support and remote developers but not devs Seems I was pretty close. After doing this exercise I would probaply have compared lists of installed software between staff who see the bug and those who don't, which would have solved it. My takeaway is that Zen debugging (just thinking what possible causes the seen behavior could have, without atking any new actions or measurements) is useful even when it seems completely useless and causes a headache.

butz 3y ago

And that's yet another reason that we must have several different browser engines.

Fanmade 3y ago

Ah yes, the great fun of bug-hunting. Probably most people in working IT have some stories to tell about this. One thing that comes to my mind happened in a company where they used a simple software to create medical reports. One clerk suddenly started creating "broken" PDF files. I debugged this for hours and could not find anything. At some point, I just copied the full raw text content that the clerk had written into one of those files out of the database into Sublime Text and I suddenly saw a small box appear between two characters of one word. It turned out that this clerk had created a Word document with text templates. Some of those texts were copied (at least partially) from other documents or websites. Apparently, either something went wrong while copying some text, or there was some control-character embedded for whatever reason in that text. I don't even remember what we did to prevent this from happening anymore, but one single "invisible" character can really cause a lot of trouble. Another point is of course if you have strange side-effects just by combining specific software. Back in the day when Crysis 1 was new and popular, I was a little involved in the modding-scene. At some point, my CryEngine editor would refuse to start up. I reinstalled it and tried to debug it, but I could not find anything myself. So I turned to the forums and found a thread where a small group of people had the exact same problem. We all shared our setups and - basically by accident - found out that we all had Wacom tablets plugged-in to our computers. I don't remember if it was enough to unplug the devices or if we also had to uninstall the Wacom drivers, but one of those things fixed the issue and we could all continue to build beautiful maps in the CryEngine. Btw, I am now suddenly constantly aware of my Grammarly plugin ...