9 ms·

Every time this conversation comes up, I'm reminded of my team at Dropbox, where it was a rite of passage for new engineers to introduce a segfault in our Go se

• commandersaki 1y ago
  To put things in perspective, I posit to you, how many memory unsafe things can you do in Go that isn’t a variant of the same thing? Or put another way what is the likelihood that a go program is memory unsafe?
• junebash 1y ago
  Swift is in the process of fixing this, but it’s a slow and painful transition; there’s an awful lot of unsafe code in the wild that wasn’t unsafe until recently.
  • cosmic_cheese 1y ago
    One of the biggest hurdles is just getting all the iOS/macOS/etc APIs up to speed with the thread safety improvements. It won’t make refactoring all that application code any easier, but as things stand even if you’ve done that, you’re going to run into problems anywhere your code makes contact with UI code because there’s a lot of AppKit and UIKit that have yet to make the transition.
  • RetpolineDrama 1y ago
    Swift 6 is only painful if you wrote a ton of terrible Swift 5, and even then Swift 5 has had modes where you could gracefully adopt the Swift 6 safety mechanisms for a long time (years?) ~130k LoC Swift app was converted from 5 -> 6 for us in about 3 days.
    • jamil7 1y ago
      Yes and no, our app is considerably larger than 130k LoC. While we’ve migrated some modules there are some parts that do a lot of multithreaded work that we probably will never migrate because they’d need to essentially be rewritten and the tradeoff isn’t really worth it for us.
    • isodev 1y ago
      It's also painful if you wrote good Swift 5 code but now suddenly you need to closely follow Apple's progress on porting their own frameworks, filling your code base with #if and control flow just to make the compiler happy.
  • ardit33 1y ago
    It is still incomplete and a mess. I don't think they thought through the actual main cases Swift is used for (ios apps), and built a hypothetical generic way which is failing on most clients. Hence lots of workarounds, and ways to get around it (The actor system). The isolated/nonisolated types are a bit contrived and causing real productivity loss, when the old way was really just 'everything ui in main thread, everything that takes time, use a dispatch queue, and call main when done'. Swift is strating to look more like old java beans. (if you are old enough to remember this, most swift developers are too young). Doing some of the same mistakes. Anways https://forums.swift.org/t/has-swifts-concurrency-model-gone-too-far/77468 https://forums.swift.org/t/has-swifts-concurrency-model-gone... Common problems all devs face: https://www.massicotte.org/problematic-patterns https://www.massicotte.org/problematic-patterns Anyways, they are trying to reinvent 'safe concurrency' while almost throwing the baby with the bathwater, and making swift even more complex and harder to get into. There is ways to go. For simple apps, the new concurrency is easy to adopt. But for anything that is less than trivial, it becomes a lot of work, to the point that it might not make it worth it.
    • pjmlp 1y ago
      Their goal was always to be able to evolve to the point of being able fully replace C, Objective-C and C++ with Swift, it has been on their documentation and plenty of WWDC sessions since the early days.
    • isodev 1y ago
      You're getting downvoted but I fully agree. The problem with Swift's safety has now moved to the tooling. While your code doesn't fail so often at runtime (still does, because the underlying system SDKs are not all migrated), the compiler itself often fails. Even the latest developer snapshot with Swift 6.2 it's quite easy to make it panic with just... "weird syntax". A much bigger problem I think are the way concurrency settings are provided via flags. It's no longer possible to know what a piece of code does without knowing the exact build settings. For example, depending on Xcode project flags, a snippet may always run on the main loop, or not at all or on a dedicated actor all together. A piece of code in a library (SPM) can build just fine in one project but fail to build in another project due to concurrency settings. The amount of overhead makes this very much unusable in a production / high pressure environment.
• shadowgovt 1y ago
  Mostly because it was a remarkable improvement over what came before (and what came before was hilariously fragile).
  • pjmlp 1y ago
    Only for those not paying attention outside mainstream, or too young to remember former languages.
    • shadowgovt 1y ago
      I'm certainly not disagreeing, but I will note that by definition, most people are in the mainstream, so something being a remarkable improvement over what came before (in the mainstream) is a remarkable improvement (for most people).
    • the_plus_one 1y ago
      > or too young to remember former languages. Do you have any good examples? Not trying to argue, just genuinely curious as someone who hasn't been in this field for decades.
      • LtWorf 1y ago
        Basically go was designed ignoring all the research and progress that had been made in programming languages until then. It was designed with contempt for developers, for example disallowing developers to create generic data structures, or lacking a decent way of error checking that is not extremely error prone and verbose.
      • deleted 1y ago
        [deleted]
  • int_19h 1y ago
    It was certainly not a remarkable improvement in the sense of being memory safe even in the face of race conditions. As the article points out, Java and C# both managed to do that, and both predate Go.
• Mawr 1y ago
  Safety isn't binary, so your comment makes no sense.
  • kstrauser 1y ago
    I’d argue that unsafety is binary. If a normal eng doing normal things can break it without going out of their way to deliberately fool the compiler or runtime, I’d call it unsafe.
    • Calavar 1y ago
      By that definition Rust also counts as unsafe. Even managed languages like C# and Java would be unsafe.
      • kstrauser 1y ago
        My impression of the Rust devs is that they’d agree with you about any easy-to-trigger calamities. So would Java contributors. C# might not because MS is institutionally not good about admitting mistakes, but I bet the individual devs would agree over a beer.
      • dcminter 1y ago
        What kinds of breakage do you have in mind though? The number of times I've segfaulted the JVM is tiny.
      • ackfoobar 1y ago
        Do you have some examples? I think JDK developers make a lot of effort to make sure users bugs will not corrupt the runtime.
      • gpm 1y ago
        There's a reason why rust devs qualify it as "memory safe" so frequently, we tend to agree that rust is, like virtually every current programming language, unsafe in other ways. Memory safety is just the source of bugs that we've figured out how to eliminate. It's a significant source of really bad (hard to debug due to action at a distance, high impact, etc) bugs so that's worth a lot, but it's not perfect. And even then we have a more frequently used escape hatch to the memory-unsafe world than would be ideal from a safety perspective for practical reasons. A more complete version of safety would be achieved with a language that proves code correct to arbitrary specifications. We aren't there yet for there being such a language that is practical for every day use. Personally I'm increasingly optimistic we'll get there sooner rather than later (say, within 20 years). Even then there will probably be specification level bugs that prevent a claim of complete safety...
• pjmlp 1y ago
  It is kind of wild that for a 21st century programming language, the amount of stuff in Go that should have been but never was, but hey Docker and Kubernetes.
  • 9rx 1y ago
    On the flip side, what would be the point? There are already a million other languages that have everything and the kitchen sink. Not going down the same road is the only reason it didn't end up on the pile of obscure languages nobody uses.
    • pjmlp 1y ago
      The only reason it didn't end on pile of obscure languages nobody uses, it called Google, followed by luck with Docker and Kubernetes adoption on the market, after they decided to rewrite from Python and Java respectively into Go, after Go heads joined their teams. Case in point, Limbo and Oberon-2, the languages that influenced its design, and authors were involved with.
      • 9rx 1y ago
        > The only reason it didn't end on pile of obscure languages nobody uses, it called Google Dart ended up on the pile of languages nobody uses. And Carbon? What's Carbon? Exactly! > Case in point, Limbo and Oberon-2, the languages that influenced its design Agreed. Limbo and Oberon-2, as primitive as they may look now, had the kitchen sinks of their time. Why wouldn't they have ended up on the pile of languages nobody uses?
        • pjmlp 1y ago
          People love to bring those as counter examples, without actually knowing a single fact about them. Dart was a victim of internal politics between the Chrome team, Dart team, AdWords moving away from GWT wanting AngularDart (see Angular documentary), and the Web in general. Had Chrome team kept pushing DartVM, it might have been quite different story. Carbon, good example of failure to actually know what the team purposes are. It is officially a research project for Google themselves, where the team is the first to advise using Rust or another MSL. One just needs to actually spend like a couple of minutes on their wiki, but I guess that is asking too much on modern times. Limbo and Oberon-2 were definitely not kitchen sinks of their time, their failure was that neither Bell Labs in 1996, nor ETHZ in 1992, were that relevant for the programming language community in the industry.
• potato-peeler 1y ago
  I am curious. Generally basic structures like map are not thread safe and care has to be taken while modifying it. This is pretty well documented in go spec. In your case in dropbox, what was essentially going on?
  • maxlybbert 1y ago
    I thought the same thing. Maybe the point of the story isn’t “we were surprised to learn you had to synchronize access” but instead “we all thought we were careful, but each of us made this mistake no matter how careful we tried to be.”
  • nine_k 1y ago
    In Java, there are separate synchronized collections, because acquiring a lock takes time. Normally one uses thread-unsafe collections. Java also gives a very ergonomic way to run any fragment under a lock (the `synchronized` operator). Rust avoids all this entirely, by using its type system.
    • noisem4ker 1y ago
      Golang has a synchronized map: https://pkg.go.dev/sync#Map https://pkg.go.dev/sync#Map
    • layer8 1y ago
      Java has separate synchronized collections only because that was initially the default, until people realized that it doesn’t help for the common cases of check-and-modify operations or of having consistency invariants with state outside a single collections (besides the performance impact). In practice, synchronized collections are rarely useful, and instead accesses are synchronized externally.
  • tsimionescu 1y ago
    I think the surprise here is that failing to synchronize writes leads to a SEGFAULT, not a panic or an error. This is the point GP was making, that Go is not fully memory safe in the presence of unsynchronized concurrent writes. By contrast, in Java or C#, unsynchronized writes will either throw an exception (if you're lucky and they get detected) or let the program continue with some unexpected values (possibly ones that violate some invariants). Getting a SEGFAULT can only happen if you're explicitly using native code, raw memory access APIs, or found a bug in the runtime.
• adamwk 1y ago
  Crashing on shared access is the safe thing to do
  • mirashii 1y ago
    An intentional exit by a runtime is a safe crash. A segfault is not, and is here a clear sign that memory safety has been violated.
    • adamwk 1y ago
      I guess I was thinking specifically of the swift case where values have exclusive access enforcement. Normally caught by a compiler, they will safely crash if the compiler didn’t catch it. I think the only way to segfault would be by using Unsafe*Pointer types, which are explicitly marked unsafe
    • Gibbon1 1y ago
      Yeah it's not the segfault that's bad, it's when it's when the write to address 0x20001854 succeeds and now some hapless postal clerk is going to jail.
  • LtWorf 1y ago
    a segfault is completely unintentional. Had the kernel been older it could be used to execute code.
    • zbentley 1y ago
      > a segfault is completely unintentional Usually, but not always! https://jcdav.is/2015/10/06/SIGSEGV-as-control-flow/ https://jcdav.is/2015/10/06/SIGSEGV-as-control-flow/
      • gowld 1y ago
        > Faulted trying to access 0x10 - the offset in the string we were trying to read from :) Is guaranteed that every offset you can try to read is guaranteed to create a segfault?
• deleted 1y ago
  [deleted]
• tptacek 1y ago
  Right, the issue here is that the "Rust and Java sense" of memory safety is not the actual meaning of the term. People talk as if "memory safety" was a PLT axiom. It's not; it's a software security term of art. This is just two groups of people talking past each other. It's not as if Go programmers are unaware of the distinction you're talking about. It's literally the premise of the language; it's the basis for "share by communicating, don't communicate by sharing". Obviously, that didn't work out, and modern Go does a lot of sharing and needs a lot of synchronization. But: everybody understands that.
  • Ygg2 1y ago
    > People talk as if "memory safety" was a PLT axiom. It's not; it's a software security term of art. It's been in usage for PLT for at least twenty years[1]. You are at least two decades late to the party. Software is memory-safe if (a) it never references a memory location outside the address space allocated by or that entity, and (b) it never executes intstruction outside code area created by the compiler and linker within that address space. [1]https://llvm.org/pubs/2003-05-05-LCTES03-CodeSafety.pdf https://llvm.org/pubs/2003-05-05-LCTES03-CodeSafety.pdf
    • zbentley 1y ago
      Not GP, but that definition seems not to be the one in use when describing languages like Rust--or even tools like valgrind. Those tools value a definition of "memory safety" that is a superset (a big one) of the definition referenced in that paper: safety as preventing incorrect memory accesses within a program, regardless of whether those accesses are out of bounds/segmentation violations.
      • adgjlsfhk1 1y ago
        it's not, but for a very subtle reason. To prove memory safety, you need to know that the program never encounters UB (since at that point you have nothing known about the program)
    • Wowfunhappy 1y ago
• gok 1y ago
  Java is not memory-safe in the Rust sense.
  • ferreiratb 1y ago
    Can you elaborate on that?
    • wpollock 1y ago
      A Java program can share mutable state between threads without synchronization, and it will compile and run. In Rust, such a program will not compile.
      • int_19h 1y ago
        Yes, but even so you will never see e.g. an invalid pointer value as the result of a torn memory write. Basically, no matter what you do with threads in Java, it will not segfault. TFA's point is that (safe) Rust is also like that, but achieves it by restricting all cases where a torn write could be observed through its type system instead of VM's memory model.
        • dontlaugh 1y ago
          More specifically, Rust prevents data races.
          • BlackFly 1y ago
            No, rust forces you to use a mutex but nothing will prevent you from making the mutex too small and creating tearing in your own data structures by sequentially modifying things covered by mutexes so that in between acquisition of the locks you are violating invariants. The borrow checker certainly helps however, but not without cost that was finally minimized when the scoped threads api came along. Java has a very specific memory model, so the behavior of variables across threads is quite well defined. Basic variables can tear however (a 64bit long on a 32bit architecture) without the volatile keyword and that is quite different than rust.
• tapirl 1y ago
  Listens your team had not sufficient review capacity at that time.
• nosefrog 1y ago
  Hi Chad!
• CJefferson 1y ago
  Before Rust, I'd reached the personal conclusion that large-scale thread-safe software was almost impossible -- certainly it required the highest levels of software engineering. Multi-process code was a much more reasonable option for mere mortals. Rust on the other hand solves that. There is code you can't write easily in Rust, but just yesterday I took a rust iteration, changed 'iter()' to 'par_iter()', and given it compiled I had high confidence it was going to work (which it did).
• rowanG077 1y ago
  I'm pretty surprised by some other comments in this thread saying this is a rare occurrence in go. In my experience it's not rare at all.
• Thaxll 1y ago
  I have a hard time believing that it's common to create SEGFAULT in Go, I worked with the language for a very long time and don't remember a single time where I've seen that. ( and i've seen many data race ) Not synchronizing writes on most data structure does not create a SEGFAULT, you have to be in a very specific condition to create one, those conditions are extremely rares and un-usual ( from the programmer perspective). In OP blog to triggers one he's doing one of those condition in an infinite loop. https://research.swtch.com/gorace https://research.swtch.com/gorace
  • commandersaki 1y ago
    You really have to go hunting for a segfault in Go. The critical sentence in OP article is: in practice, of course, safety is not binary, it is a spectrum, and on that spectrum Go is much closer to a typical safe language than to C. OP just has a vested interest in proving safety of languages and is making a big deal where in practice there is none. People are not making loads of unsafe programs in Go nor deploying as such because it would be pretty quickly detected. This is much different to C and C++.