5 ms·

UndoDB – The interactive time travel debugger for Linux C/C++ for debugging

109 points by droideqa 1y ago

schaefer 1y ago
Let me save you a click: Pricing & Licensing A UDB floating license costs $7,900 per year.
- dima55 1y ago
  rr is awesome and is free and open and all that. How much better could this possibly be?
  - Veserv 1y ago
    Well, if you have a Google L5 making ~365k [1] then it would need to make them ~2.2% more productive overall to be worth it when just considering direct pay. If we consider a Google L3 at ~187k then it would need to make them ~4.2% more productive overall. This, of course, ignores employee benefits and overhead which usually amount to ~100% extra costs over direct pay. So that is now ~1.1% and ~2.1%, respectively. And that ignores the fact that you need to pay people less than they produce to be profitable which probably drops us down to ~0.5% and ~1.0%, respectively. [1] https://www.levels.fyi/companies/google/salaries/software-engineer?country=254 https://www.levels.fyi/companies/google/salaries/software-en... edit: Incorrectly linked to product designer instead of software engineer levels.
    - dima55 1y ago
      OK... Most of us don't know what a "google l5" is, so I guess we can safely ignore this. Heh.
    - esafak 1y ago
      No-one is going to spend $8K out of pocket to A/B test this on themselves. Of all the things you could be doing to improve your productivity, this is some high hanging fruit.
      - Veserv 1y ago
        If you have a US employer who is unwilling to spend 8 k$ on software engineering productivity then they are pennywise, pound foolish. It literally costs 10x that for a single junior engineer. And, as I pointed out, the net productivity improvement you need to see to justify that expense is miniscule. If your employer really is skeptical, then they can run a A/B test over a small group of engineers to prove out changes in productivity. But not even being willing to run that test when it is so cheap is just management incompetence. Engineers are ridiculously expensive. In electrical engineering, where the engineers are generally less well-paid than in software, employers routinely spend multiple hundreds of thousands of dollars per engineer per year in tooling. Not being willing to spend 8 k$ on a test of well known technology and attempting to identify mere single digit percentage improvements is just stupid.
        deleted 1y ago
        [deleted]
        ranger_danger 1y ago
        Not everyone is Google. Some people work for themselves, or have very small teams, or live in a developing country, and don't have lots of spare cash laying around. Please try to understand that the world is not as simple and black and white as you'd like.
        more-nitor 1y ago
        this if someone's bringing "google payscale" for comparison... well that's not some average joe... why not just bring Bill Gates and say "everything -- including private jet -- is dirt cheap" ? $8k per year simply doesn't make sense for 95% of the programmers. For a lot of developing countries, that's more than a well-paying programmer's annual salary...
        Veserv 1y ago
        I presented a conditional calculation using numbers contextually-relevant to a forum run by a company that funds US tech startups. IF you are such a company, THEN you only need very minor productivity improvements to justify such a price difference. IF you are such a company, THEN such costs are well within the cost of doing business. IF you are NOT then you are free to apply your own numbers to your situation. However, the overwhelming majority of companies that fall into the applicable category outright refuse to even consider the possibility of a 8 k$ capital expense on software development tooling out of the absolutely foolish belief that such expenditure CAN NOT be justified as such a cost could not possibly be recouped in benefits or out of the absolutely foolish belief that a product that costs 100$ must be 100x better than a product that costs 1$ to be justified, and thus a product that costs money must be infinitely better than a product that is free to be justified. It is absolutely management incompetence for US employers to just shut down upon hearing about a 8 k$ capital expenditure instead of doing a proper cost-benefit analysis to determine if such expenditures would be justified.
    - eviks 1y ago
      The major fail of such "just a 1% / cup of coffee" is that there is an infinite number of things you could pay for with the same potential productivity promise without any hard data on whether those are true, so just the fact the you can use a calculator and divide to get to a low % doesn't help you much if at all
  - AlotOfReading 1y ago
    They have a comparison page: https://undo.io/resources/undo-vs-rr/ https://undo.io/resources/undo-vs-rr/ I was in talks with them recently because I kept running into limitations with rr. The main advantages for my use case were that undo doesn't have the same dependency on hardware timers, which means the ARM support is much better, you can run it in a VM (e.g. a cloud machine) and you can do replays on different systems.
    - dzaima 1y ago
      A couple minor notes: - If your program is very light on syscalls (i.e. basically entirely in-memory computation), rr can go to a basically 1.0x slowdown. In particular this means you can run benchmarks in it at full capacity, provided that I/O is outside of the repeated part (e.g. if sometimes the bench is noticably slower, you can replay and see if some important loads/stores crossed a cacheline/page). You can even "perf record" / "perf stat" a replay if you want to! (none of this is too useful, but it's fun! Gathering repeated stats over the same execution for more resolution might be useful with proper tooling though) - rr does have an in-memory buffer of recording data. - rr recordings should be portable within the architecture, as long as the replay hardware has the extensions the recorder did (or if replayer-unsupported features are disabled at record-time).
      - AlotOfReading 1y ago
        I regularly deal with 3 different architectures. I can go and spin up a cloud instance every time I want to run rr (and in fact that's the solution I've been working with), but it's just annoying enough to justify spending a couple hours in sales calls.
  - gregthelaw 1y ago
    Undo co-founder here. rr is indeed awesome. If it works for your use-case, you should use it! Undo is mostly used by companies whose world is complex enough that rr doesn't work for them, and they understand how powerful time travel debugging is. There has now been a LOT of engineering invested by a lot of very smart people into Undo, so it does also have a lot of polish and nice features. But honestly, if rr is working for you, that's great. I'm just glad you're not doing printf debugging the whole time :)
- db48x 1y ago
  If you’re going to spend money, then you would be better off using rr and paying for Pernosco. Pernosco is amazing.
  - rurban 1y ago
    Thanks for the Pernosco tip. It really looks amazing and you can try it for free as github user
    - db48x 1y ago
      You’re welcome.
- deleted 1y ago
  [deleted]
ognarb 1y ago
What's the difference with RR?
- leni536 1y ago
  AFAIK it records multithreaded applications on multiple threads and CPU, rr records them on a single OS thread, AFAIK. Not sure about replay. Never used undo though, so not sure how much better it is.
  - dzaima 1y ago
    rr does support multithreaded and multi-process applications, via, like Undo[1], allowing only a single thread to run at a time. (edit note - that's only about multithreading; Undo might have parallel multi-process recording) [1]: https://undo.io/resources/undo-performance-benchmarks/ https://undo.io/resources/undo-performance-benchmarks/ - "Undo serializes their execution"
    - leni536 1y ago
      I stand corrected, not sure where I heard this then.
      - dzaima 1y ago
        https://undo.io/resources/undo-vs-rr/ https://undo.io/resources/undo-vs-rr/ does note parallel recording for multi-process (not multi-threaded), so perhaps that.
- kristopolous 1y ago
  [flagged]
  - gregthelaw 1y ago
    Undo founder here. I just got a slack from one of our marketing team who is thrilled you appreciate their work. :-) Our customers are top tier tech companies and our users are among the smartest engineers on the planet. I'm proud of our marketing, but no-one is going to spend a bunch of money with us just because of that. So there must be something else in it I guess ;-)
ranger_danger 1y ago
FOSS alternative: https://rr-project.org/ https://rr-project.org/
delta_p_delta_x 1y ago
Free Windows equivalent: WinDbg Time Travel Debugging (https://learn.microsoft.com/en-gb/windows-hardware/drivers/debuggercmds/time-travel-debugging-overview https://learn.microsoft.com/en-gb/windows-hardware/drivers/d...).
- mark_undoio 1y ago
  WinDbg's time travel debug is really cool and more people should know about it. I'm always a little sad that it's not (so far!) officially integrated in something like VS Code. Before it was released publicly I believe Microsoft had been using it internally to share recordings on bug reports against massive pieces of software like Office. So it's a serious piece of tech.
  - senderista 1y ago
    I used it (iDNA) on the Windows team starting around 2006 or so and we were able to resolve bugs in minutes that had been open for years. It was absolute magic.
rixtox 1y ago
Another more powerful option: Intel SDE / PinPlay
- deleted 1y ago
  [deleted]
minamoto625 1y ago
gdb should already have a similar feature?
- mark_undoio 1y ago
  GDB does: https://sourceware.org/gdb/current/onlinedocs/gdb.html/Process-Record-and-Replay.html https://sourceware.org/gdb/current/onlinedocs/gdb.html/Proce... But it's limited. It's really cool that it's integrated by default but it doesn't scale to big applications / workloads. RR and Undo both use GDB as a user interface, though, so any skills you have there will carry over.
  - ho_schi 1y ago
    A lot people don't know and don't use GDBs reverse debugging. It is an awesome and hidden feature, which more developers should know :) All these Oh wait. I missed it...debugging sessions. and these What exactly changed over there? are answerable.
- db48x 1y ago
  It does, but it is really sad by comparison with rr and UndoDB. You could use it to record a few function calls or perhaps if you’re lucky a whole frame of your game but not a whole program.
drawnwren 1y ago
if you could get this working on embedded arm cpus, I think you'd be surprised how many customers there would be.
- mark_undoio 1y ago
  What's limiting us is that Undo does need a Linux kernel - so traditional embedded programming wouldn't be a fit. Embedded Linux could work and we do support ARM64. I've thought I bit about how you might support time travel on bare metal embedded - but actually there are hardware-assisted solutions (Lauterbach's Trace32 was one we came across) there sometimes.
- Veserv 1y ago
  Time travel debugging on embedded ARM has been available for over 20 years via trace probes [1]. The category namer of time-travel debugging, TimeMachine, (hence time-travel debugging in contrast to other attempted names such as reversible, bidirectional, record-replay, etc.) was available in 2003 and supports/supported the ARM7 [2]. Note, that is not ARMv7 architecture, that is the ARM7 chip [3] in use from 1993-2001. From what I know, the ARM7 was one of the first ARM designs implementing the Embedded Trace Macrocell (ETM) which could output the instruction and data trace data used to support trace probe-based time travel debugging. [1] https://jakob.engbloms.se/archives/1564 https://jakob.engbloms.se/archives/1564 [2] https://www.ghs.com/products/probe.html https://www.ghs.com/products/probe.html [3] https://en.m.wikipedia.org/wiki/ARM7 https://en.m.wikipedia.org/wiki/ARM7

109 points by droideqa 1y ago

schaefer 1y ago

Let me save you a click: Pricing & Licensing A UDB floating license costs $7,900 per year.

dima55 1y ago
rr is awesome and is free and open and all that. How much better could this possibly be?
- Veserv 1y ago
  Well, if you have a Google L5 making ~365k [1] then it would need to make them ~2.2% more productive overall to be worth it when just considering direct pay. If we consider a Google L3 at ~187k then it would need to make them ~4.2% more productive overall. This, of course, ignores employee benefits and overhead which usually amount to ~100% extra costs over direct pay. So that is now ~1.1% and ~2.1%, respectively. And that ignores the fact that you need to pay people less than they produce to be profitable which probably drops us down to ~0.5% and ~1.0%, respectively. [1] https://www.levels.fyi/companies/google/salaries/software-engineer?country=254 https://www.levels.fyi/companies/google/salaries/software-en... edit: Incorrectly linked to product designer instead of software engineer levels.
  - dima55 1y ago
    OK... Most of us don't know what a "google l5" is, so I guess we can safely ignore this. Heh.
  - esafak 1y ago
    No-one is going to spend $8K out of pocket to A/B test this on themselves. Of all the things you could be doing to improve your productivity, this is some high hanging fruit.
    - Veserv 1y ago
      If you have a US employer who is unwilling to spend 8 k$ on software engineering productivity then they are pennywise, pound foolish. It literally costs 10x that for a single junior engineer. And, as I pointed out, the net productivity improvement you need to see to justify that expense is miniscule. If your employer really is skeptical, then they can run a A/B test over a small group of engineers to prove out changes in productivity. But not even being willing to run that test when it is so cheap is just management incompetence. Engineers are ridiculously expensive. In electrical engineering, where the engineers are generally less well-paid than in software, employers routinely spend multiple hundreds of thousands of dollars per engineer per year in tooling. Not being willing to spend 8 k$ on a test of well known technology and attempting to identify mere single digit percentage improvements is just stupid.
      - deleted 1y ago
        [deleted]
      - ranger_danger 1y ago
        Not everyone is Google. Some people work for themselves, or have very small teams, or live in a developing country, and don't have lots of spare cash laying around. Please try to understand that the world is not as simple and black and white as you'd like.
        more-nitor 1y ago
        this if someone's bringing "google payscale" for comparison... well that's not some average joe... why not just bring Bill Gates and say "everything -- including private jet -- is dirt cheap" ? $8k per year simply doesn't make sense for 95% of the programmers. For a lot of developing countries, that's more than a well-paying programmer's annual salary...
        Veserv 1y ago
        I presented a conditional calculation using numbers contextually-relevant to a forum run by a company that funds US tech startups. IF you are such a company, THEN you only need very minor productivity improvements to justify such a price difference. IF you are such a company, THEN such costs are well within the cost of doing business. IF you are NOT then you are free to apply your own numbers to your situation. However, the overwhelming majority of companies that fall into the applicable category outright refuse to even consider the possibility of a 8 k$ capital expense on software development tooling out of the absolutely foolish belief that such expenditure CAN NOT be justified as such a cost could not possibly be recouped in benefits or out of the absolutely foolish belief that a product that costs 100$ must be 100x better than a product that costs 1$ to be justified, and thus a product that costs money must be infinitely better than a product that is free to be justified. It is absolutely management incompetence for US employers to just shut down upon hearing about a 8 k$ capital expenditure instead of doing a proper cost-benefit analysis to determine if such expenditures would be justified.
  - eviks 1y ago
    The major fail of such "just a 1% / cup of coffee" is that there is an infinite number of things you could pay for with the same potential productivity promise without any hard data on whether those are true, so just the fact the you can use a calculator and divide to get to a low % doesn't help you much if at all
- AlotOfReading 1y ago
  They have a comparison page: https://undo.io/resources/undo-vs-rr/ https://undo.io/resources/undo-vs-rr/ I was in talks with them recently because I kept running into limitations with rr. The main advantages for my use case were that undo doesn't have the same dependency on hardware timers, which means the ARM support is much better, you can run it in a VM (e.g. a cloud machine) and you can do replays on different systems.
  - dzaima 1y ago
    A couple minor notes: - If your program is very light on syscalls (i.e. basically entirely in-memory computation), rr can go to a basically 1.0x slowdown. In particular this means you can run benchmarks in it at full capacity, provided that I/O is outside of the repeated part (e.g. if sometimes the bench is noticably slower, you can replay and see if some important loads/stores crossed a cacheline/page). You can even "perf record" / "perf stat" a replay if you want to! (none of this is too useful, but it's fun! Gathering repeated stats over the same execution for more resolution might be useful with proper tooling though) - rr does have an in-memory buffer of recording data. - rr recordings should be portable within the architecture, as long as the replay hardware has the extensions the recorder did (or if replayer-unsupported features are disabled at record-time).
    - AlotOfReading 1y ago
      I regularly deal with 3 different architectures. I can go and spin up a cloud instance every time I want to run rr (and in fact that's the solution I've been working with), but it's just annoying enough to justify spending a couple hours in sales calls.
- gregthelaw 1y ago
  Undo co-founder here. rr is indeed awesome. If it works for your use-case, you should use it! Undo is mostly used by companies whose world is complex enough that rr doesn't work for them, and they understand how powerful time travel debugging is. There has now been a LOT of engineering invested by a lot of very smart people into Undo, so it does also have a lot of polish and nice features. But honestly, if rr is working for you, that's great. I'm just glad you're not doing printf debugging the whole time :)
db48x 1y ago
If you’re going to spend money, then you would be better off using rr and paying for Pernosco. Pernosco is amazing.
- rurban 1y ago
  Thanks for the Pernosco tip. It really looks amazing and you can try it for free as github user
  - db48x 1y ago
    You’re welcome.
deleted 1y ago
[deleted]

ognarb 1y ago

What's the difference with RR?

leni536 1y ago
AFAIK it records multithreaded applications on multiple threads and CPU, rr records them on a single OS thread, AFAIK. Not sure about replay. Never used undo though, so not sure how much better it is.
- dzaima 1y ago
  rr does support multithreaded and multi-process applications, via, like Undo[1], allowing only a single thread to run at a time. (edit note - that's only about multithreading; Undo might have parallel multi-process recording) [1]: https://undo.io/resources/undo-performance-benchmarks/ https://undo.io/resources/undo-performance-benchmarks/ - "Undo serializes their execution"
  - leni536 1y ago
    I stand corrected, not sure where I heard this then.
    - dzaima 1y ago
      https://undo.io/resources/undo-vs-rr/ https://undo.io/resources/undo-vs-rr/ does note parallel recording for multi-process (not multi-threaded), so perhaps that.
kristopolous 1y ago
[flagged]
- gregthelaw 1y ago
  Undo founder here. I just got a slack from one of our marketing team who is thrilled you appreciate their work. :-) Our customers are top tier tech companies and our users are among the smartest engineers on the planet. I'm proud of our marketing, but no-one is going to spend a bunch of money with us just because of that. So there must be something else in it I guess ;-)

ranger_danger 1y ago

FOSS alternative: https://rr-project.org/ https://rr-project.org/

delta_p_delta_x 1y ago

Free Windows equivalent: WinDbg Time Travel Debugging (https://learn.microsoft.com/en-gb/windows-hardware/drivers/debuggercmds/time-travel-debugging-overview https://learn.microsoft.com/en-gb/windows-hardware/drivers/d...).

mark_undoio 1y ago
WinDbg's time travel debug is really cool and more people should know about it. I'm always a little sad that it's not (so far!) officially integrated in something like VS Code. Before it was released publicly I believe Microsoft had been using it internally to share recordings on bug reports against massive pieces of software like Office. So it's a serious piece of tech.
- senderista 1y ago
  I used it (iDNA) on the Windows team starting around 2006 or so and we were able to resolve bugs in minutes that had been open for years. It was absolute magic.

rixtox 1y ago

Another more powerful option: Intel SDE / PinPlay

deleted 1y ago
[deleted]

minamoto625 1y ago

gdb should already have a similar feature?

mark_undoio 1y ago
GDB does: https://sourceware.org/gdb/current/onlinedocs/gdb.html/Process-Record-and-Replay.html https://sourceware.org/gdb/current/onlinedocs/gdb.html/Proce... But it's limited. It's really cool that it's integrated by default but it doesn't scale to big applications / workloads. RR and Undo both use GDB as a user interface, though, so any skills you have there will carry over.
- ho_schi 1y ago
  A lot people don't know and don't use GDBs reverse debugging. It is an awesome and hidden feature, which more developers should know :) All these Oh wait. I missed it...debugging sessions. and these What exactly changed over there? are answerable.
db48x 1y ago
It does, but it is really sad by comparison with rr and UndoDB. You could use it to record a few function calls or perhaps if you’re lucky a whole frame of your game but not a whole program.

drawnwren 1y ago

if you could get this working on embedded arm cpus, I think you'd be surprised how many customers there would be.

mark_undoio 1y ago
What's limiting us is that Undo does need a Linux kernel - so traditional embedded programming wouldn't be a fit. Embedded Linux could work and we do support ARM64. I've thought I bit about how you might support time travel on bare metal embedded - but actually there are hardware-assisted solutions (Lauterbach's Trace32 was one we came across) there sometimes.
Veserv 1y ago
Time travel debugging on embedded ARM has been available for over 20 years via trace probes [1]. The category namer of time-travel debugging, TimeMachine, (hence time-travel debugging in contrast to other attempted names such as reversible, bidirectional, record-replay, etc.) was available in 2003 and supports/supported the ARM7 [2]. Note, that is not ARMv7 architecture, that is the ARM7 chip [3] in use from 1993-2001. From what I know, the ARM7 was one of the first ARM designs implementing the Embedded Trace Macrocell (ETM) which could output the instruction and data trace data used to support trace probe-based time travel debugging. [1] https://jakob.engbloms.se/archives/1564 https://jakob.engbloms.se/archives/1564 [2] https://www.ghs.com/products/probe.html https://www.ghs.com/products/probe.html [3] https://en.m.wikipedia.org/wiki/ARM7 https://en.m.wikipedia.org/wiki/ARM7