9 ms·

Show HN: Uscope, a new Linux debugger written from scratch

262 points by jcalabro 2y ago

Hi! I've been building a debugger on my nights and weekends because it's fun, and I personally need a better debugger for my work. GDB and LLDB pain me greatly; we can and will do better! As explained in the README, it's still very early-days and it's not ready for use yet, but check back often because it's improving all the time! Check out https://calabro.io/uscope https://calabro.io/uscope for a more detailed explanation. Thanks for taking a look!

rickydroll 2y ago
Make it debug systemd config problems and I'd be thrilled.
adolph 2y ago
To clarify “from scratch” as in from basic materials and not “in scratch” as in the MIT graphical programming g system[0]. It appears to be written in Zig[1], which I find interesting for analysis. 0. https://scratch.mit.edu/ https://scratch.mit.edu/ 1. https://github.com/jcalabro/uscope https://github.com/jcalabro/uscope
- TZubiri 2y ago
  lol, I think no one expected a linux debugger to be written in the visual programming language for kids.
  - duskwuff 2y ago
    I'd be very impressed if it were, though.
- AdmiralAsshat 2y ago
  "written from scratch" is one of those phrases that seems to cause any submission to instantly hit HN frontpage, along with "for fun and profit", "considered harmful", and a couple others.
lsllc 2y ago
Looks promising -- it's about time, I haven't been satisfied with any debugger since the days of Periscope! https://www.os2museum.com/files/docs/periscope/periscope-manual-m54.pdf https://www.os2museum.com/files/docs/periscope/periscope-man...
- malkia 2y ago
  Somehow this brought the memory of Numega's awesome SoftICE!
  - zwieback 2y ago
    Oh yea, we used both SoftICE and Periscope. I loved the pushbutton to fire off an NMI directly on the ISA bus. It was sad when your device driver crashed but it was a fun experience to push that button.
    - malkia 2y ago
      I've used only few times back then, but it was impressive (previous experience was with Borland's / Turbo debuggers which was nice also, but Numega was just another class)
tux1968 2y ago
This is great to see. It'd be lovely if Linux gets a decent debugger. Another project to keep an eye on is https://github.com/EpicGamesExt/raddebugger https://github.com/EpicGamesExt/raddebugger Although, they haven't expanded much beyond Windows, yet.
- mesbahamin 2y ago
  They fully plan to support Linux.
quotemstr 2y ago
It's not clear to me why we'd need a new debugger instead of GDB enhancements. GDB has an extensible, modern C++ codebase with literal decades of hard earned knowledge baked into it. How are we better off rewriting it, especially if the rewrite isn't memory safe?
- ziddoap 2y ago
  Consider this part of the post: >I've been building a debugger on my nights and weeks because it's fun
  - dima55 2y ago
    I'm considering the part that says "GDB and LLDB pain me greatly". This project both says "I'm doing this for fun" and "the state of the art is bad, and I'm trying to fix it", so this is a fair question. GDB is great. He says it crashes a lot, and cannot interpret some of his types. Maybe that's his experience, but I haven't had those issues after literally decades of using gdb. Some bug reports would be nice.
    - Agingcoder 2y ago
      Decades for me as well, and we’ve had very different experiences : gdb is very powerful , but the user experience is atrocious and it does crash a lot. To be fair, in a number of cases the type issues I had were not gdb’s fault but bad DWARF produced by the compiler.
    - marssaxman 2y ago
      After literally decades of occasionally dipping into GDB only to quickly remember why I would rather use no debugger at all than that, it's hard to imagine what you could mean by "great". What are you comparing it to?
- chc4 2y ago
  > GDB has an extensible, modern C++ codebase with literal decades of hard earned knowledge baked into it. I literally laughed outloud at this. Have you ever actually opened a GDB source file? It is, as a whole, extremely poor quality code. Almost all of it is doing C string manipulation and raw pointer arithmetic; almost none of it uses C++ smart pointers, nevermind the rest of "modern C++"; the vast majority is completely uncommented, and "literal decades of hard earned knowledge" can be better translated as "literal decades of historical baggage, cruft, and hacks". I regularly cause GDB to segfault in normal, mundane operations.
MaximilianEmel 2y ago
How does it compare to gf? https://github.com/nakst/gf https://github.com/nakst/gf
- oguz-ismail 2y ago
  gf is a frontend to gdb. This is supposed to be a brand new debugger
  - Brian_K_White 2y ago
    orthogonal to the question
godelski 2y ago
> Build as a library so other people can build other interesting things as well I LOVE this! I firmly believe so much tech has gone to shit because things are no longer hackable. We say "move fast and break things" but we try so hard to prevent that that when we do break things we just create bigger messes and debt, so no one cleans it up. It seems slower to write hackable code but that's in the moment. In the long run it is much faster. Not to mention you get added benefits of others being able to contribute more easily with a lower likelihood of actually breaking shit.
- Analemma_ 2y ago
  Is gdb another thing like gcc where the un-hackability and un-extendability was a deliberate choice by rms to ensure nobody would ever build proprietary toolchains on top of it?
  - dooglius 2y ago
    No, GDB has a pretty good Python extension framework
    - junon 2y ago
      Yes, this. I write a lot of my own GDB tooling for debugging my kernel.
    - mort96 2y ago
      That's not comparable. That lets you extend GDB with your own additions. Having the debugger work as a library means being able to incorporate the debugger's functionality into other systems.
  - debatem1 2y ago
    I don't know if it was deliberate, but writing code that interfaces with GDB is unpleasant enough that I opted to build our debugger-like tooling in eBPF + pyelftools instead.
  - godelski 2y ago
koek67 2y ago
Nice work! I remember meeting you at Systems Distributed in NYC where you mentioned this project, so cool to see the progress. Well done!
meitham 2y ago
Very happy to see the increasing momentum in the zig community
ribs 2y ago
TotalView (which I rep), Linaro DDT - thoughts? (Yes, they’re both proprietary)
- cintellis 2y ago
  how much do those cost approx? for an individual hobbyist developer
feelamee 2y ago
> The available Linux debuggers are gdb and lldb. They both suck. They crash all the time, don't understand the data types I care about, and they don't make the data I need available at my fingertips as quickly as possible. quote from https://calabro.io/uscope https://calabro.io/uscope Of course gdb, lldb have their problems (e.g. smashing tui with app output, what can be easily fixed, or very very very very long tab-tab completion, and crashing of course), but I dont know anything better. I am forced to use visual studio at work and its debugger really sucks - it can't even compare strings in conditional breakpoint, it cant even skip all std::function / std::bind garbage to let me step in callback, it can't watch evaluated expressions. Probably it can evaluate exprs (immediate window?), but there are very little guides about this. So, gdb is winner for me now. rr (record-repeat)[0] also looks very nice, but it require hardware support((( [0] https://rr-project.org/ https://rr-project.org/
- AlotOfReading 2y ago
  If you're willing to deal with enterprise pricing, Undo [0] implements something reasonably comparable to rr without the hardware timer requirements. [0] https://undo.io/ https://undo.io/
  - feelamee 2y ago
    eh, thanks > How can I persuade my boss to pay for Undo? fun quote.. but my boss will never pay for it because I am the only one who use gdb in our company, unfortunately)
    - mark_undoio 2y ago
      For what it's worth we do offer a more printf-y interface for people who don't like a debugger - https://docs.undo.io/PostFailureLogging.html https://docs.undo.io/PostFailureLogging.html And CLion / VS Code for people who prefer an IDE interface. But a lot of people do really want to stick with their printf debugging. If your boss won't buy you an Undo you can still use https://rr-project.org/ https://rr-project.org/ - or on Windows the built in time travel debug of WinDbg.
mplemay 2y ago
This opinion is not backed by facts, any insight about linux (or even the languages in question), or even related to this post. Nevertheless, I wonder if it was a good idea to allow rust contributions to the linux project. From all of the bits and pieces I read about zig (including this project), I feel like it would have been better aligned (than rust) to pick up where the mainly C codebase left off.
- renox 2y ago
  Zig isn't 1.0 so..
- whytevuhuni 2y ago
  Not really. Zig's answer to safety is mostly based on runtime panics, and the kernel really, really hates panics. As such, the only thing left is better ergonomics, and that's not really worth the effort to switch. Rust isn't being adopted because it's an easier language to code in, and in fact it's being adopted in spite of the fact that it's harder to code in. And that's because to some kernel devs, the promise of better security and fewer vulnerabilities is worth that much. On the other hand, Zig is great for user-space applications. The stuff to replace GNU's coreutils with.
  - defen 2y ago
    Rust also has runtime panics (e.g. indexing outside the bounds of a slice) - how does kernel Rust handle that?
    - nine_k 2y ago
      Static checks remove many more potential sources of panic. I suspect that with certain restraint one can write Rust code that is statically guaranteed to not panic. Also, Rust's panics may be recoverable, it's not necessarily fatal.
    - whytevuhuni 2y ago
      It does not, for slice indexing Rust is just as bad. My best guess is that this will likely blow up into such a big issue that the Rust devs are going to be forced to implement a feature to disable indexing (and leave just the safe .get()), or the kernel devs will fork the core library. But Rust prevents a myriad of other things that would be panics in Zig or undefined behavior in C. It has a really strong type system, capable of reducing a large amount of invalid states and keeping many invariants throughout very large codebases.
jcranmer 2y ago
This is definitely an ambitious project, and I worry that you are biting off more than you can chew in doing so. (I've attempted my fair share of debugger projects in the past). At a low level, one of the main problems is that Linux's kernel interfaces for debugging are just absolute trash. (I see you have multithreaded support mentioned as a future task item, and that's one of the areas where you discover just how bad it really is). And of course ptrace composes somewhat poorly with, well, anything else, so if you want to use perf_event or eBPF to help drive the low-level stuff, well, combining everything into a single event loop is just painful. (Oh, and the documentation for all of this stuff sucks.) At the medium level, everything is hampered by the number of secret handshakes that go on between the compiler, the linker, the runtime loader, the debugger. And the chronic issue that, in the debugger, you have to be prepared for the possibility that everything is just catastrophically wrong: your stack might be garbage (or downright nonexistent), the thread-local storage register might be a garbage value, variables may have values that don't make sense. After all, you're resorting to a debugger when things don't work, and part of the reason why it might not be working is because you've accidentally corrupted everything. And at the high level, there's the UI which, as someone who doesn't work on UI, I find terrifying in its own right. Trying to solve not one of these issues, but all of them, at once, in a new project is ambitious. I'd personally prefer a lot more a project that only tried to tackle one slice of the stack, not all of it.
- HexDecOctBin 2y ago
  Yup, Linux is nowhere the Unix successor we deserved. I have started multiple systems projects in Linux and gave always given up due to how shoddy the foundations are. It seems like the kernel just copies whatever cool feature they find in Solaris, BSD, Plan 9 or even NT without paying any attention to their composability, while the user space people are just a clique trying to force their way on everyone (why is the loader part of libc again?).
  - jcranmer 2y ago
    There are a few things where Linux has been the innovator (eBPF comes to mind). But process control is not one of them, and almost any other operating system manages to have a more sane interface. Personally, if I were writing a debugger, I think the OS with the sanest kernel interface is probably Fuchsia, partially because everything is handle-based and partially because pretty much every syscall takes in a handle to the process to operate on, so you can do things like manipulate memory maps of another process without driving yourself insane.
breatheoften 2y ago
Super supportive as I really value debuggers as tools even tho they are horror shows to use! > Similarly, the following features are non-goals of the project: > Supporting non-native languages (i.e. Java, Python, etc.) But I think that position is likely a mistake in terms of leaving killer features on the table and baking in architecture decisions that might continue to make these kinds of features impossible / very low-class experiences. Properly integrating with the python interpreter to be able to debug python + c/cpp extensions running in the same process is a huge missing whole in the debugger market. I don't know how other people do it but I 'solve' this problem by attaching either a python debugger or lldb to the python process -- meaning I can debug either python or the cpp but not both at the same time. The experience is very lacking. Even just seeing the python call-stack while paused at a cpp breakpoint would be huge.
- mark_undoio 2y ago
  We've got a prototype of debugging in python and C/C++ in GDB - https://undo.io/resources/how-i-debug-python-code-with-a-time-travel-debugger/ https://undo.io/resources/how-i-debug-python-code-with-a-tim... If you'd like to try it please get in touch, feedback is always useful.
titzer 2y ago
Nice project! One killer feature would be the ability to connect to the debugger via a socket and control it. Gdb has this interface and for some use cases it's great. As one of those long-tail "native" languages, Virgil might benefit from this. So far, I've had a student build a DWARF backend, and the experience from that is that DWARF is way too complicated and consequently implementations are broken and crappy in many ways. I think DWARF draws the wrong dividing line here. Control of the machine and customizing the source-level support to the language is probably better.
- sroussey 2y ago
  Have it use the WebKit debugger protocol and we can use browser dev tools as the UI. :p
- mindcrime 2y ago
  Speaking as a Java guy, remote debugging is SO useful (at times anyway). Thankfully the JVM has built in support for a remote debugging protocol[1] and there are good debuggers that can connect to a running Java system (if it was started with the correct command line flags) and do symbolic debugging over the network. There can be certain situations where the network latency can make things difficult, but generally speaking I find it an incredibly useful facility to have. [1]: https://docs.oracle.com/javase/8/docs/technotes/guides/jpda/jdwp-spec.html https://docs.oracle.com/javase/8/docs/technotes/guides/jpda/...
  - funcDropShadow 2y ago
    Which debuggers do you like the most?
- peterfirefly 2y ago
  > the experience from that is that DWARF is way too complicated The representation and the compression are far too intertwined :(
  - peterfirefly 2y ago
gregthelaw 2y ago
Congrats on this work -- writing a debugger from scratch is a big job. I have cloned the repo, will take a proper look this w/e.
IAmLiterallyAB 2y ago
Yay this is awesome! GDB is a buggy (https://sourceware.org/bugzilla/show_bug.cgi?id=18772 https://sourceware.org/bugzilla/show_bug.cgi?id=18772 https://sourceware.org/bugzilla/show_bug.cgi?id=9425 https://sourceware.org/bugzilla/show_bug.cgi?id=9425) mess and rough code quality, I've wanted a do something like this for a while. To be 100% clear, it's not using gdb/gdbserver under the hood right? The bugs I linked above are over a decade old, and I have to patch them every time I compile GDB server. Ultimately (IIRC) GDB needs to rework how it handles signals (to their credit, ptrace is a horribly stupid API, especially before PTRACE_SEIZE, so I don't blame them for having issues)
- mort96 2y ago
  Interesting, LLDB also has a serious bug related to ^c! In LLDB, if you run 'lldb -o run /path/to/executable', it won't pause on SIGINT. As a result, you can't have an ergonomic setup where you have e.g a make target to run your software under LLDB, you always need to manually type 'run' for ^c to work.
drewg123 2y ago
How entwined is it with Linux specific APIs? Eg, how hard would it be to port to FreeBSD?
bieganski 2y ago
> The available Linux debuggers are gdb and lldb. They both suck. They crash all the time, don't understand the data types I care about, and they don't make the data I need available at my fingertips as quickly as possible. Okay, so this is the author's answer to the most important question: "why?" For me this is a serious issue, making strong statements without any single backing example. If you experience crashes, please report to the maintainers - i guarantee that you won't be ignored. You say that it's missing some data that you need? Fine, but be precise - what data? Otherwise it sounds like a "yeah, let's make a project for fun and pretend that it's serious by finding sort-of-justification/use case" (i'm not telling that it is the case for you - i'm telling that it sounds like it, based on your own description). Also, would you feel nice if i put in my project's README a note that the project of you, the one that you put your effort to, "sucks"?
- omnicognate 2y ago
  I'd rather the time taken enumerating the deficiencies of existing debuggers be spent building something better instead. I doubt many people that have used gdb/lldb need convincing that that's possible. If it is needed, Microsoft's windbg (far from what's possible but light years ahead of gdb/lldb) is a proof by construction.
  - ethin 2y ago
    Bit of a possibly unpopular take but I very much disagree. The syntax of windbg/cdb is just... Really bad. I feel like I'm writing assembly and the mnemonics don't actually align with what the command does. So it's difficult for me to get confident with it. At least the commands in gdb make sense (and I wish gdb supported windows executables, but alas...)
  - matu3ba 2y ago
    windbg has no sane library api, which makes it by construction unsuitable for automation and inspection. I'd personally prefer, if we would have the options of multiple time-travel debugging sessions based on synchronization points (if needed) being overlapped to single out bugs in interesting areas (not covered by static or dynamic analysis). Debugger would be essentially a virtual program slicer being able to hide unnecessary program parts or reduce it. However, so far we neither have the options for piece-wise time-accurate multi-threading recording, nor slicing overlays nor in-memory reducers (ideally also reducing AST). I may be mistaken on "piece-wise time-accurate multi-threading", since I've not checked what is possible with hardware yet. Heck, we dont even have overview data for common bug classes and cost for verification or runtime-checking in academia etc.
zwieback 2y ago
Everyone here seems to thing GDB is awful. It's been a while for me but I remember using front-ends like Eclipse's CDT or similar and didn't find that experience so bad. Do most people use GDB straight-up? I haven't done that in probably 15 years although it's nice to have a lightweight command line on small embedded systems.
- jcelerier 2y ago
  example from today, timed with a stopwatch, on a 5950X 5GHz CPU: from the moment I click on "debug" in my IDE, - lldb takes 40 seconds to reach "main" - 2 minutes 36 total to reach the GUI and start being able to reproduce things - gdb coredumps as soon as I want to put a breakpoint so it's unuseable $ lldb --version lldb version 19.1.7 $ gdb --version GNU gdb (GDB) 16.1
- boricj 2y ago
  I do use GDB straight up for native programs and microcontrollers, mostly out of laziness. I haven't set up VS Code debugging at my current day job yet and it's been 6 months. Out of the box GDB is kinda awful, especially for C++ codebases. I should probably look into scripting at some point, but meh. Even then, as far as I know I'm the best at using a debugger at work by a wide margin, but I attribute that more to my knowledge of low-level programming than my ability to use most of the basic GDB commands. Also, it tends to crash once a month or so. If I needed to debug a userland program on a small Linux embedded system, I'd probably whip out gdbserver and attach gdb to the target remotely. I haven't done that in a while though.
- jcelerier 2y ago
  > Everyone here seems to thing GDB is awful. It's been a while for me but I remember using front-ends like Eclipse's CDT or similar and didn't find that experience so bad. Do most people use GDB straight-up? I haven't done that in probably 15 years although it's nice to have a lightweight command line on small embedded systems. the issue is not with gdb's UX - you can build as many cool UI / UX on top of it as you want. The problem is that gdb will sometimes take 5 minutes to start debugging (and that's without debuginfod), straight up crash, be unable to resolve obvious symbols etc. which are all back-end bugs. For me developing a medium-sized C++ app, it's really hell to use and usually printf debugging is MUCH faster in the sense that I have the time to find and fix my problem through an iterative recompile cycle sometimes before gdb has even finished parsing my binary
AndyKelley 2y ago
He's live on twitch right now demoing this on zig showtime: https://www.twitch.tv/kristoff_it https://www.twitch.tv/kristoff_it
- AndyKelley 2y ago
  recording: https://www.youtube.com/watch?v=stWBTv6grBc https://www.youtube.com/watch?v=stWBTv6grBc
tux1968 2y ago
Author just did a podcast about Uscope a few minutes ago, where they mention this HN post: https://youtu.be/stWBTv6grBc https://youtu.be/stWBTv6grBc Mentioned at : https://youtu.be/stWBTv6grBc?t=456 https://youtu.be/stWBTv6grBc?t=456
mamcx 2y ago
One thing that as far as I know no single debuggers has (except FoxPro from DOS) is the ability to show a large list of things, but well In general, if something is large (text, arrays, etc) all bets are off. Basically, something like this: https://github.com/okbob/pspg https://github.com/okbob/pspg --- The second thing that is very hard with debuggers, is to tell them what to skip. In the normal sense of 'I don't wanna debug Rust std' but also 'What the heck, why is stepping into #[Debug]???'
armitron 2y ago
GDB (I don't use LLDB) is fully programmable and has multiple user interfaces (TUI, Emacs, DDD, graphical frontends and various .gdbinit configurations that emulate SoftICE L&F). It's a power tool and takes some time and effort to learn and master. Superficially dismissing it / wasting your time with something that in all likelihood will end up going nowhere while lacking most features that make GDB great, does not a good recommendation make. At least you'll probably learn a thing or two while you implement it, but I'd rather not waste my time using it.
s3graham 2y ago
Keep at it! raddbg is one worth watching too (currently only Windows x64) https://github.com/EpicGamesExt/raddebugger https://github.com/EpicGamesExt/raddebugger
mort96 2y ago
This is exciting! I use GDB and LLDB, but their TUIs/CLIs make certain things difficult (I feel like debugging seriously benefits from a proper GUI) and graphical front-ends I've tried are pretty janky. Looking forward to trying it out, but I'm excited to see something happening in this space.

262 points by jcalabro 2y ago

rickydroll 2y ago

Make it debug systemd config problems and I'd be thrilled.

adolph 2y ago

To clarify “from scratch” as in from basic materials and not “in scratch” as in the MIT graphical programming g system[0]. It appears to be written in Zig[1], which I find interesting for analysis. 0. https://scratch.mit.edu/ https://scratch.mit.edu/ 1. https://github.com/jcalabro/uscope https://github.com/jcalabro/uscope

TZubiri 2y ago
lol, I think no one expected a linux debugger to be written in the visual programming language for kids.
- duskwuff 2y ago
  I'd be very impressed if it were, though.
AdmiralAsshat 2y ago
"written from scratch" is one of those phrases that seems to cause any submission to instantly hit HN frontpage, along with "for fun and profit", "considered harmful", and a couple others.

lsllc 2y ago

Looks promising -- it's about time, I haven't been satisfied with any debugger since the days of Periscope! https://www.os2museum.com/files/docs/periscope/periscope-manual-m54.pdf https://www.os2museum.com/files/docs/periscope/periscope-man...

malkia 2y ago
Somehow this brought the memory of Numega's awesome SoftICE!
- zwieback 2y ago
  Oh yea, we used both SoftICE and Periscope. I loved the pushbutton to fire off an NMI directly on the ISA bus. It was sad when your device driver crashed but it was a fun experience to push that button.
  - malkia 2y ago
    I've used only few times back then, but it was impressive (previous experience was with Borland's / Turbo debuggers which was nice also, but Numega was just another class)

tux1968 2y ago

This is great to see. It'd be lovely if Linux gets a decent debugger. Another project to keep an eye on is https://github.com/EpicGamesExt/raddebugger https://github.com/EpicGamesExt/raddebugger Although, they haven't expanded much beyond Windows, yet.

mesbahamin 2y ago
They fully plan to support Linux.

quotemstr 2y ago

It's not clear to me why we'd need a new debugger instead of GDB enhancements. GDB has an extensible, modern C++ codebase with literal decades of hard earned knowledge baked into it. How are we better off rewriting it, especially if the rewrite isn't memory safe?

ziddoap 2y ago
Consider this part of the post: >I've been building a debugger on my nights and weeks because it's fun
- dima55 2y ago
  I'm considering the part that says "GDB and LLDB pain me greatly". This project both says "I'm doing this for fun" and "the state of the art is bad, and I'm trying to fix it", so this is a fair question. GDB is great. He says it crashes a lot, and cannot interpret some of his types. Maybe that's his experience, but I haven't had those issues after literally decades of using gdb. Some bug reports would be nice.
  - Agingcoder 2y ago
    Decades for me as well, and we’ve had very different experiences : gdb is very powerful , but the user experience is atrocious and it does crash a lot. To be fair, in a number of cases the type issues I had were not gdb’s fault but bad DWARF produced by the compiler.
  - marssaxman 2y ago
    After literally decades of occasionally dipping into GDB only to quickly remember why I would rather use no debugger at all than that, it's hard to imagine what you could mean by "great". What are you comparing it to?
chc4 2y ago
> GDB has an extensible, modern C++ codebase with literal decades of hard earned knowledge baked into it. I literally laughed outloud at this. Have you ever actually opened a GDB source file? It is, as a whole, extremely poor quality code. Almost all of it is doing C string manipulation and raw pointer arithmetic; almost none of it uses C++ smart pointers, nevermind the rest of "modern C++"; the vast majority is completely uncommented, and "literal decades of hard earned knowledge" can be better translated as "literal decades of historical baggage, cruft, and hacks". I regularly cause GDB to segfault in normal, mundane operations.

MaximilianEmel 2y ago

How does it compare to gf? https://github.com/nakst/gf https://github.com/nakst/gf

oguz-ismail 2y ago
gf is a frontend to gdb. This is supposed to be a brand new debugger
- Brian_K_White 2y ago
  orthogonal to the question

godelski 2y ago

> Build as a library so other people can build other interesting things as well I LOVE this! I firmly believe so much tech has gone to shit because things are no longer hackable. We say "move fast and break things" but we try so hard to prevent that that when we do break things we just create bigger messes and debt, so no one cleans it up. It seems slower to write hackable code but that's in the moment. In the long run it is much faster. Not to mention you get added benefits of others being able to contribute more easily with a lower likelihood of actually breaking shit.

Analemma_ 2y ago
Is gdb another thing like gcc where the un-hackability and un-extendability was a deliberate choice by rms to ensure nobody would ever build proprietary toolchains on top of it?
- dooglius 2y ago
  No, GDB has a pretty good Python extension framework
  - junon 2y ago
    Yes, this. I write a lot of my own GDB tooling for debugging my kernel.
  - mort96 2y ago
    That's not comparable. That lets you extend GDB with your own additions. Having the debugger work as a library means being able to incorporate the debugger's functionality into other systems.
- debatem1 2y ago
  I don't know if it was deliberate, but writing code that interfaces with GDB is unpleasant enough that I opted to build our debugger-like tooling in eBPF + pyelftools instead.
- godelski 2y ago

koek67 2y ago

Nice work! I remember meeting you at Systems Distributed in NYC where you mentioned this project, so cool to see the progress. Well done!

meitham 2y ago

Very happy to see the increasing momentum in the zig community

ribs 2y ago

TotalView (which I rep), Linaro DDT - thoughts? (Yes, they’re both proprietary)

cintellis 2y ago
how much do those cost approx? for an individual hobbyist developer

feelamee 2y ago

> The available Linux debuggers are gdb and lldb. They both suck. They crash all the time, don't understand the data types I care about, and they don't make the data I need available at my fingertips as quickly as possible. quote from https://calabro.io/uscope https://calabro.io/uscope Of course gdb, lldb have their problems (e.g. smashing tui with app output, what can be easily fixed, or very very very very long tab-tab completion, and crashing of course), but I dont know anything better. I am forced to use visual studio at work and its debugger really sucks - it can't even compare strings in conditional breakpoint, it cant even skip all std::function / std::bind garbage to let me step in callback, it can't watch evaluated expressions. Probably it can evaluate exprs (immediate window?), but there are very little guides about this. So, gdb is winner for me now. rr (record-repeat)[0] also looks very nice, but it require hardware support((( [0] https://rr-project.org/ https://rr-project.org/

AlotOfReading 2y ago
If you're willing to deal with enterprise pricing, Undo [0] implements something reasonably comparable to rr without the hardware timer requirements. [0] https://undo.io/ https://undo.io/
- feelamee 2y ago
  eh, thanks > How can I persuade my boss to pay for Undo? fun quote.. but my boss will never pay for it because I am the only one who use gdb in our company, unfortunately)
  - mark_undoio 2y ago
    For what it's worth we do offer a more printf-y interface for people who don't like a debugger - https://docs.undo.io/PostFailureLogging.html https://docs.undo.io/PostFailureLogging.html And CLion / VS Code for people who prefer an IDE interface. But a lot of people do really want to stick with their printf debugging. If your boss won't buy you an Undo you can still use https://rr-project.org/ https://rr-project.org/ - or on Windows the built in time travel debug of WinDbg.

mplemay 2y ago

This opinion is not backed by facts, any insight about linux (or even the languages in question), or even related to this post. Nevertheless, I wonder if it was a good idea to allow rust contributions to the linux project. From all of the bits and pieces I read about zig (including this project), I feel like it would have been better aligned (than rust) to pick up where the mainly C codebase left off.

renox 2y ago
Zig isn't 1.0 so..
whytevuhuni 2y ago
Not really. Zig's answer to safety is mostly based on runtime panics, and the kernel really, really hates panics. As such, the only thing left is better ergonomics, and that's not really worth the effort to switch. Rust isn't being adopted because it's an easier language to code in, and in fact it's being adopted in spite of the fact that it's harder to code in. And that's because to some kernel devs, the promise of better security and fewer vulnerabilities is worth that much. On the other hand, Zig is great for user-space applications. The stuff to replace GNU's coreutils with.
- defen 2y ago
  Rust also has runtime panics (e.g. indexing outside the bounds of a slice) - how does kernel Rust handle that?
  - nine_k 2y ago
    Static checks remove many more potential sources of panic. I suspect that with certain restraint one can write Rust code that is statically guaranteed to not panic. Also, Rust's panics may be recoverable, it's not necessarily fatal.
  - whytevuhuni 2y ago
    It does not, for slice indexing Rust is just as bad. My best guess is that this will likely blow up into such a big issue that the Rust devs are going to be forced to implement a feature to disable indexing (and leave just the safe .get()), or the kernel devs will fork the core library. But Rust prevents a myriad of other things that would be panics in Zig or undefined behavior in C. It has a really strong type system, capable of reducing a large amount of invalid states and keeping many invariants throughout very large codebases.

jcranmer 2y ago

This is definitely an ambitious project, and I worry that you are biting off more than you can chew in doing so. (I've attempted my fair share of debugger projects in the past). At a low level, one of the main problems is that Linux's kernel interfaces for debugging are just absolute trash. (I see you have multithreaded support mentioned as a future task item, and that's one of the areas where you discover just how bad it really is). And of course ptrace composes somewhat poorly with, well, anything else, so if you want to use perf_event or eBPF to help drive the low-level stuff, well, combining everything into a single event loop is just painful. (Oh, and the documentation for all of this stuff sucks.) At the medium level, everything is hampered by the number of secret handshakes that go on between the compiler, the linker, the runtime loader, the debugger. And the chronic issue that, in the debugger, you have to be prepared for the possibility that everything is just catastrophically wrong: your stack might be garbage (or downright nonexistent), the thread-local storage register might be a garbage value, variables may have values that don't make sense. After all, you're resorting to a debugger when things don't work, and part of the reason why it might not be working is because you've accidentally corrupted everything. And at the high level, there's the UI which, as someone who doesn't work on UI, I find terrifying in its own right. Trying to solve not one of these issues, but all of them, at once, in a new project is ambitious. I'd personally prefer a lot more a project that only tried to tackle one slice of the stack, not all of it.

HexDecOctBin 2y ago
Yup, Linux is nowhere the Unix successor we deserved. I have started multiple systems projects in Linux and gave always given up due to how shoddy the foundations are. It seems like the kernel just copies whatever cool feature they find in Solaris, BSD, Plan 9 or even NT without paying any attention to their composability, while the user space people are just a clique trying to force their way on everyone (why is the loader part of libc again?).
- jcranmer 2y ago
  There are a few things where Linux has been the innovator (eBPF comes to mind). But process control is not one of them, and almost any other operating system manages to have a more sane interface. Personally, if I were writing a debugger, I think the OS with the sanest kernel interface is probably Fuchsia, partially because everything is handle-based and partially because pretty much every syscall takes in a handle to the process to operate on, so you can do things like manipulate memory maps of another process without driving yourself insane.

breatheoften 2y ago

Super supportive as I really value debuggers as tools even tho they are horror shows to use! > Similarly, the following features are non-goals of the project: > Supporting non-native languages (i.e. Java, Python, etc.) But I think that position is likely a mistake in terms of leaving killer features on the table and baking in architecture decisions that might continue to make these kinds of features impossible / very low-class experiences. Properly integrating with the python interpreter to be able to debug python + c/cpp extensions running in the same process is a huge missing whole in the debugger market. I don't know how other people do it but I 'solve' this problem by attaching either a python debugger or lldb to the python process -- meaning I can debug either python or the cpp but not both at the same time. The experience is very lacking. Even just seeing the python call-stack while paused at a cpp breakpoint would be huge.

mark_undoio 2y ago
We've got a prototype of debugging in python and C/C++ in GDB - https://undo.io/resources/how-i-debug-python-code-with-a-time-travel-debugger/ https://undo.io/resources/how-i-debug-python-code-with-a-tim... If you'd like to try it please get in touch, feedback is always useful.

titzer 2y ago

Nice project! One killer feature would be the ability to connect to the debugger via a socket and control it. Gdb has this interface and for some use cases it's great. As one of those long-tail "native" languages, Virgil might benefit from this. So far, I've had a student build a DWARF backend, and the experience from that is that DWARF is way too complicated and consequently implementations are broken and crappy in many ways. I think DWARF draws the wrong dividing line here. Control of the machine and customizing the source-level support to the language is probably better.

sroussey 2y ago
Have it use the WebKit debugger protocol and we can use browser dev tools as the UI. :p
mindcrime 2y ago
Speaking as a Java guy, remote debugging is SO useful (at times anyway). Thankfully the JVM has built in support for a remote debugging protocol[1] and there are good debuggers that can connect to a running Java system (if it was started with the correct command line flags) and do symbolic debugging over the network. There can be certain situations where the network latency can make things difficult, but generally speaking I find it an incredibly useful facility to have. [1]: https://docs.oracle.com/javase/8/docs/technotes/guides/jpda/jdwp-spec.html https://docs.oracle.com/javase/8/docs/technotes/guides/jpda/...
- funcDropShadow 2y ago
  Which debuggers do you like the most?
peterfirefly 2y ago
> the experience from that is that DWARF is way too complicated The representation and the compression are far too intertwined :(
- peterfirefly 2y ago

gregthelaw 2y ago

Congrats on this work -- writing a debugger from scratch is a big job. I have cloned the repo, will take a proper look this w/e.

IAmLiterallyAB 2y ago

Yay this is awesome! GDB is a buggy (https://sourceware.org/bugzilla/show_bug.cgi?id=18772 https://sourceware.org/bugzilla/show_bug.cgi?id=18772 https://sourceware.org/bugzilla/show_bug.cgi?id=9425 https://sourceware.org/bugzilla/show_bug.cgi?id=9425) mess and rough code quality, I've wanted a do something like this for a while. To be 100% clear, it's not using gdb/gdbserver under the hood right? The bugs I linked above are over a decade old, and I have to patch them every time I compile GDB server. Ultimately (IIRC) GDB needs to rework how it handles signals (to their credit, ptrace is a horribly stupid API, especially before PTRACE_SEIZE, so I don't blame them for having issues)

mort96 2y ago
Interesting, LLDB also has a serious bug related to ^c! In LLDB, if you run 'lldb -o run /path/to/executable', it won't pause on SIGINT. As a result, you can't have an ergonomic setup where you have e.g a make target to run your software under LLDB, you always need to manually type 'run' for ^c to work.

drewg123 2y ago

How entwined is it with Linux specific APIs? Eg, how hard would it be to port to FreeBSD?

bieganski 2y ago

> The available Linux debuggers are gdb and lldb. They both suck. They crash all the time, don't understand the data types I care about, and they don't make the data I need available at my fingertips as quickly as possible. Okay, so this is the author's answer to the most important question: "why?" For me this is a serious issue, making strong statements without any single backing example. If you experience crashes, please report to the maintainers - i guarantee that you won't be ignored. You say that it's missing some data that you need? Fine, but be precise - what data? Otherwise it sounds like a "yeah, let's make a project for fun and pretend that it's serious by finding sort-of-justification/use case" (i'm not telling that it is the case for you - i'm telling that it sounds like it, based on your own description). Also, would you feel nice if i put in my project's README a note that the project of you, the one that you put your effort to, "sucks"?

omnicognate 2y ago
I'd rather the time taken enumerating the deficiencies of existing debuggers be spent building something better instead. I doubt many people that have used gdb/lldb need convincing that that's possible. If it is needed, Microsoft's windbg (far from what's possible but light years ahead of gdb/lldb) is a proof by construction.
- ethin 2y ago
  Bit of a possibly unpopular take but I very much disagree. The syntax of windbg/cdb is just... Really bad. I feel like I'm writing assembly and the mnemonics don't actually align with what the command does. So it's difficult for me to get confident with it. At least the commands in gdb make sense (and I wish gdb supported windows executables, but alas...)
- matu3ba 2y ago
  windbg has no sane library api, which makes it by construction unsuitable for automation and inspection. I'd personally prefer, if we would have the options of multiple time-travel debugging sessions based on synchronization points (if needed) being overlapped to single out bugs in interesting areas (not covered by static or dynamic analysis). Debugger would be essentially a virtual program slicer being able to hide unnecessary program parts or reduce it. However, so far we neither have the options for piece-wise time-accurate multi-threading recording, nor slicing overlays nor in-memory reducers (ideally also reducing AST). I may be mistaken on "piece-wise time-accurate multi-threading", since I've not checked what is possible with hardware yet. Heck, we dont even have overview data for common bug classes and cost for verification or runtime-checking in academia etc.

zwieback 2y ago

Everyone here seems to thing GDB is awful. It's been a while for me but I remember using front-ends like Eclipse's CDT or similar and didn't find that experience so bad. Do most people use GDB straight-up? I haven't done that in probably 15 years although it's nice to have a lightweight command line on small embedded systems.

jcelerier 2y ago
example from today, timed with a stopwatch, on a 5950X 5GHz CPU: from the moment I click on "debug" in my IDE, - lldb takes 40 seconds to reach "main" - 2 minutes 36 total to reach the GUI and start being able to reproduce things - gdb coredumps as soon as I want to put a breakpoint so it's unuseable $ lldb --version lldb version 19.1.7 $ gdb --version GNU gdb (GDB) 16.1
boricj 2y ago
I do use GDB straight up for native programs and microcontrollers, mostly out of laziness. I haven't set up VS Code debugging at my current day job yet and it's been 6 months. Out of the box GDB is kinda awful, especially for C++ codebases. I should probably look into scripting at some point, but meh. Even then, as far as I know I'm the best at using a debugger at work by a wide margin, but I attribute that more to my knowledge of low-level programming than my ability to use most of the basic GDB commands. Also, it tends to crash once a month or so. If I needed to debug a userland program on a small Linux embedded system, I'd probably whip out gdbserver and attach gdb to the target remotely. I haven't done that in a while though.
jcelerier 2y ago
> Everyone here seems to thing GDB is awful. It's been a while for me but I remember using front-ends like Eclipse's CDT or similar and didn't find that experience so bad. Do most people use GDB straight-up? I haven't done that in probably 15 years although it's nice to have a lightweight command line on small embedded systems. the issue is not with gdb's UX - you can build as many cool UI / UX on top of it as you want. The problem is that gdb will sometimes take 5 minutes to start debugging (and that's without debuginfod), straight up crash, be unable to resolve obvious symbols etc. which are all back-end bugs. For me developing a medium-sized C++ app, it's really hell to use and usually printf debugging is MUCH faster in the sense that I have the time to find and fix my problem through an iterative recompile cycle sometimes before gdb has even finished parsing my binary

AndyKelley 2y ago

He's live on twitch right now demoing this on zig showtime: https://www.twitch.tv/kristoff_it https://www.twitch.tv/kristoff_it

AndyKelley 2y ago
recording: https://www.youtube.com/watch?v=stWBTv6grBc https://www.youtube.com/watch?v=stWBTv6grBc

tux1968 2y ago

Author just did a podcast about Uscope a few minutes ago, where they mention this HN post: https://youtu.be/stWBTv6grBc https://youtu.be/stWBTv6grBc Mentioned at : https://youtu.be/stWBTv6grBc?t=456 https://youtu.be/stWBTv6grBc?t=456

mamcx 2y ago

One thing that as far as I know no single debuggers has (except FoxPro from DOS) is the ability to show a large list of things, but well In general, if something is large (text, arrays, etc) all bets are off. Basically, something like this: https://github.com/okbob/pspg https://github.com/okbob/pspg --- The second thing that is very hard with debuggers, is to tell them what to skip. In the normal sense of 'I don't wanna debug Rust std' but also 'What the heck, why is stepping into #[Debug]???'

armitron 2y ago

GDB (I don't use LLDB) is fully programmable and has multiple user interfaces (TUI, Emacs, DDD, graphical frontends and various .gdbinit configurations that emulate SoftICE L&F). It's a power tool and takes some time and effort to learn and master. Superficially dismissing it / wasting your time with something that in all likelihood will end up going nowhere while lacking most features that make GDB great, does not a good recommendation make. At least you'll probably learn a thing or two while you implement it, but I'd rather not waste my time using it.

s3graham 2y ago

Keep at it! raddbg is one worth watching too (currently only Windows x64) https://github.com/EpicGamesExt/raddebugger https://github.com/EpicGamesExt/raddebugger

mort96 2y ago

This is exciting! I use GDB and LLDB, but their TUIs/CLIs make certain things difficult (I feel like debugging seriously benefits from a proper GUI) and graphical front-ends I've tried are pretty janky. Looking forward to trying it out, but I'm excited to see something happening in this space.