9 ms·

Debugging a Mixed Python and C Language Stack

152 points by quasiben 3y ago

fmajid 3y ago
GDB can decode Python stack traces as Python code, as long as you have have python-gdb.py from the Python source distribution in the same place as your Python executable.
- winter_blue 3y ago
  Wow, that's pretty amazing. I wonder how it's implemented, and if there are any tutorials on implementing something similar, for programming language designers/creators.
  - bobbyi 3y ago
    Theres also Py Spy, a profiling tool that can generate flame charts containing a mix of python and C (or C++) calls. https://github.com/benfred/py-spy https://github.com/benfred/py-spy It's worked really well for my needs
    - misnome 3y ago
      py-spy also makes it very easy to scrape from existing processes, which is invaluable if you are e.g. ever trying to diagnose a deadlock
  - vmiklos 3y ago
    I think it uses gdb's frame filter API: https://sourceware.org/gdb/onlinedocs/gdb/Writing-a-Frame-Filter.html https://sourceware.org/gdb/onlinedocs/gdb/Writing-a-Frame-Fi...
  - fmajid 3y ago
    GDB itself is fully scriptable in Python: https://sourceware.org/gdb/current/onlinedocs/gdb.html/Python.html#Python https://sourceware.org/gdb/current/onlinedocs/gdb.html/Pytho...
- deleted 3y ago
  [deleted]
- westurner 3y ago
  From https://github.com/jupyterlab/debugger/issues/284 https://github.com/jupyterlab/debugger/issues/284 en ENH: Mixed Python/C debugging (GDB,) #284 > "Users can do [mixed mode] debugging with GDB (and/or other debuggers) and log the session in a notebook; in particular in order to teach. > Your question is specifically about IDEs with support for mixed-mode debugging (with gdb), so I went looking for an answer: > https://wiki.python.org/moin/DebuggingWithGdb https://wiki.python.org/moin/DebuggingWithGdb (which is not responsive and almost unreadable on a mobile device) links to https://fedoraproject.org/wiki/Features/EasierPythonDebugging https://fedoraproject.org/wiki/Features/EasierPythonDebuggin... , which mentions the py-list, py-up and py-down, py-bt, py-print, and py-locals GDB commands that are also described in * https://devguide.python.org/gdb/ https://devguide.python.org/gdb/ * > https://wiki.python.org/moin/PythonDebuggingTools https://wiki.python.org/moin/PythonDebuggingTools Ctrl-F "gdb" mentions: DDD, pyclewn (vim), trepan3k (which is gdb-like and supports breaking at c-line and also handles bytecode disassembly) > Apparently, GHIDRA does not have a debugger but there is a plugin for following along with gdb in ghidra called https://github.com/Comsecuris/gdbghidra https://github.com/Comsecuris/gdbghidra [...] https://github.com/Comsecuris/gdbghidra/blob/master/data/gdb_ghidra_bridge_client.py https://github.com/Comsecuris/gdbghidra/blob/master/data/gdb... (zero dependencies) > https://reverseengineering.stackexchange.com/questions/1392/decent-gui-for-gdb https://reverseengineering.stackexchange.com/questions/1392/... lists a number of GUIs for GDB; including voltronnn: >> There's Voltron, which is an extensible Python debugger UI that supports LLDB, GDB, VDB, and WinDbg/CDB (via PyKD) and runs on macOS, Linux and Windows. For the first three it supports x86, x86_64, and arm with even arm64 support for lldb while adding even powerpc support for gdb. https://github.com/snare/voltron https://github.com/snare/voltron > "The GDB Python API" https://developers.redhat.com/blog/2017/11/10/gdb-python-api/ https://developers.redhat.com/blog/2017/11/10/gdb-python-api... describes the GDB Python API. > https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_in_ide.html#writing-a-c-function-to-call-any-python-unit-test https://pythonextensionpatterns.readthedocs.io/en/latest/deb... may be helpful [for writing-a-c-function-to-call-any-python-unit-test] > The GDB Python API docs: https://sourceware.org/gdb/onlinedocs/gdb/Python-API.html https://sourceware.org/gdb/onlinedocs/gdb/Python-API.html > The devguide gdb page may be the place to list IDEs with support for mixed-mode debugging of Python and C/C++/Cython specifically with gdb? These days, we have CFFI and Apache Arrow for C+Python etc.
- planede 3y ago
  What I would love is also to be able to set python breakpoints from gdb. And integration with rr, so I could reverse-continue to said breakpoints. And ponies.
- mort96 3y ago
  GDB does so much cool. It just sucks that it's missing a high quality front-end. I know there are a bunch out there, but they all seem pretty janky and get confused a lot; I just want a really serious group of people (some Red Hat people for example) to make a high quality native debugging app which properly speaks the GDB protocol and handles all the edge cases. (And no, I don't want it integrated into an IDE.)
- ttyprintk 3y ago
  This has been packaged as Tools/gdb/libpython.py lately.
PaulHoule 3y ago
That reminds me of the time I debugged some code (also neural network) that was in both Java and C++ and I was able to attach both gdb and jdb to the same process but had to disable the segfault trap on gdb because the jvm segfaults all the time in normal operation.
- kelahcim 3y ago
  Which reminds me that with IntelliJ you can get a really nice combo for debugging Java/C/C++ code: https://youtu.be/8Cjeq4l5COU https://youtu.be/8Cjeq4l5COU
- MereInterest 3y ago
  As a C++ developer, I'm struggling to understand how and why segfaults would ever be part of normal operation. In my mental model, the presence of a segfault means that a program has gone so far off the expected path that no guarantees can be made about its state whatsoever, so the only safe thing to do is to let the program crash. Is there a reason why the JVM regularly segfaults?
  - aa-jv 3y ago
    A pretty good investigation into this matter - "segfaults in Java (redux)": https://www.lukeshu.com/blog/java-segfault-redux.html https://www.lukeshu.com/blog/java-segfault-redux.html Read the prior article for some context, too ..
  - planede 3y ago
    Segfaults are unexpected and shouldn't happen in a conforming C++ program that doesn't evaluate operations with undefined behavior. But in non-portable C or C++ program doing manual memory management with OS primitives targeting a specific OS the conditions for segfault are well documented, and you can also rely on the programs behavior when that happens (the OS raises a signal for you, that you can handle). There are not many programs that should be written this way, but I assume the JVM might fall into this category. I'm still not sure if handling page faults this way in regular operation is the best strategy, but I would worry about performance more than correctness.
zdevito 3y ago
When developing PyTorch, we also run into a lot of mixed Python/C++ language situations. We've recently been experimenting with in-process 'combined' Python/C++/PyTorch 2.0 stack traces to make it easier to understand where code is executing (https://dev-discuss.pytorch.org/t/fast-combined-c-python-torchscript-inductor-tracebacks/1158 https://dev-discuss.pytorch.org/t/fast-combined-c-python-tor...).
- voz_ 3y ago
  Funny, I was going to post this, but the horse's mouth is even better ;)
nightpool 3y ago
Awesome to see some more Excalidraw diagrams on the front page, feels like it's really taking off.
jaza 3y ago
An inspiring, albeit daunting, write-up for someone like me, who has been a python coder for over 10 years, but who has never professionally coded in C or C++. I'd be pleasantly surprised with myself if I could one day debug like that. Also somewhat depressing that, yet again, the GIL was to blame, and that after all that impressive investigatory work, the fix was (spoiler alert!) "rewrite the offending function in C".
- jebarker 3y ago
  I predominantly develop in Python but somewhat frequently peer into the C++ and CUDA underlying Pytorch. I am not a competent C or C++ programmer but often am able to debug and fix bugs that originate in the lower levels of the stack. That ability developed just through exposure and focused prodding when I found issues, but I don't think it has much relation or translates to being a professional coder in C/C++/CUDA. Coding and debugging are different skill sets IMO.
vaughan 3y ago
It's unfortunate that no one thinks about the debugging experience with anything they build today. It's always a "strap yourself in" experience like in the post. I dream of an IDE that is one-click, full-stack, local debug. I dropped some ideas here: https://news.ycombinator.com/item?id=35650253 https://news.ycombinator.com/item?id=35650253

152 points by quasiben 3y ago

GDB can decode Python stack traces as Python code, as long as you have have python-gdb.py from the Python source distribution in the same place as your Python executable.

winter_blue 3y ago
Wow, that's pretty amazing. I wonder how it's implemented, and if there are any tutorials on implementing something similar, for programming language designers/creators.
- bobbyi 3y ago
  Theres also Py Spy, a profiling tool that can generate flame charts containing a mix of python and C (or C++) calls. https://github.com/benfred/py-spy https://github.com/benfred/py-spy It's worked really well for my needs
  - misnome 3y ago
    py-spy also makes it very easy to scrape from existing processes, which is invaluable if you are e.g. ever trying to diagnose a deadlock
- vmiklos 3y ago
  I think it uses gdb's frame filter API: https://sourceware.org/gdb/onlinedocs/gdb/Writing-a-Frame-Filter.html https://sourceware.org/gdb/onlinedocs/gdb/Writing-a-Frame-Fi...
- fmajid 3y ago
  GDB itself is fully scriptable in Python: https://sourceware.org/gdb/current/onlinedocs/gdb.html/Python.html#Python https://sourceware.org/gdb/current/onlinedocs/gdb.html/Pytho...
deleted 3y ago
[deleted]
westurner 3y ago
From https://github.com/jupyterlab/debugger/issues/284 https://github.com/jupyterlab/debugger/issues/284 en ENH: Mixed Python/C debugging (GDB,) #284 > "Users can do [mixed mode] debugging with GDB (and/or other debuggers) and log the session in a notebook; in particular in order to teach. > Your question is specifically about IDEs with support for mixed-mode debugging (with gdb), so I went looking for an answer: > https://wiki.python.org/moin/DebuggingWithGdb https://wiki.python.org/moin/DebuggingWithGdb (which is not responsive and almost unreadable on a mobile device) links to https://fedoraproject.org/wiki/Features/EasierPythonDebugging https://fedoraproject.org/wiki/Features/EasierPythonDebuggin... , which mentions the py-list, py-up and py-down, py-bt, py-print, and py-locals GDB commands that are also described in * https://devguide.python.org/gdb/ https://devguide.python.org/gdb/ * > https://wiki.python.org/moin/PythonDebuggingTools https://wiki.python.org/moin/PythonDebuggingTools Ctrl-F "gdb" mentions: DDD, pyclewn (vim), trepan3k (which is gdb-like and supports breaking at c-line and also handles bytecode disassembly) > Apparently, GHIDRA does not have a debugger but there is a plugin for following along with gdb in ghidra called https://github.com/Comsecuris/gdbghidra https://github.com/Comsecuris/gdbghidra [...] https://github.com/Comsecuris/gdbghidra/blob/master/data/gdb_ghidra_bridge_client.py https://github.com/Comsecuris/gdbghidra/blob/master/data/gdb... (zero dependencies) > https://reverseengineering.stackexchange.com/questions/1392/decent-gui-for-gdb https://reverseengineering.stackexchange.com/questions/1392/... lists a number of GUIs for GDB; including voltronnn: >> There's Voltron, which is an extensible Python debugger UI that supports LLDB, GDB, VDB, and WinDbg/CDB (via PyKD) and runs on macOS, Linux and Windows. For the first three it supports x86, x86_64, and arm with even arm64 support for lldb while adding even powerpc support for gdb. https://github.com/snare/voltron https://github.com/snare/voltron > "The GDB Python API" https://developers.redhat.com/blog/2017/11/10/gdb-python-api/ https://developers.redhat.com/blog/2017/11/10/gdb-python-api... describes the GDB Python API. > https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_in_ide.html#writing-a-c-function-to-call-any-python-unit-test https://pythonextensionpatterns.readthedocs.io/en/latest/deb... may be helpful [for writing-a-c-function-to-call-any-python-unit-test] > The GDB Python API docs: https://sourceware.org/gdb/onlinedocs/gdb/Python-API.html https://sourceware.org/gdb/onlinedocs/gdb/Python-API.html > The devguide gdb page may be the place to list IDEs with support for mixed-mode debugging of Python and C/C++/Cython specifically with gdb? These days, we have CFFI and Apache Arrow for C+Python etc.
planede 3y ago
What I would love is also to be able to set python breakpoints from gdb. And integration with rr, so I could reverse-continue to said breakpoints. And ponies.
mort96 3y ago
GDB does so much cool. It just sucks that it's missing a high quality front-end. I know there are a bunch out there, but they all seem pretty janky and get confused a lot; I just want a really serious group of people (some Red Hat people for example) to make a high quality native debugging app which properly speaks the GDB protocol and handles all the edge cases. (And no, I don't want it integrated into an IDE.)
ttyprintk 3y ago
This has been packaged as Tools/gdb/libpython.py lately.

PaulHoule 3y ago

That reminds me of the time I debugged some code (also neural network) that was in both Java and C++ and I was able to attach both gdb and jdb to the same process but had to disable the segfault trap on gdb because the jvm segfaults all the time in normal operation.

kelahcim 3y ago
Which reminds me that with IntelliJ you can get a really nice combo for debugging Java/C/C++ code: https://youtu.be/8Cjeq4l5COU https://youtu.be/8Cjeq4l5COU
MereInterest 3y ago
As a C++ developer, I'm struggling to understand how and why segfaults would ever be part of normal operation. In my mental model, the presence of a segfault means that a program has gone so far off the expected path that no guarantees can be made about its state whatsoever, so the only safe thing to do is to let the program crash. Is there a reason why the JVM regularly segfaults?
- aa-jv 3y ago
  A pretty good investigation into this matter - "segfaults in Java (redux)": https://www.lukeshu.com/blog/java-segfault-redux.html https://www.lukeshu.com/blog/java-segfault-redux.html Read the prior article for some context, too ..
- planede 3y ago
  Segfaults are unexpected and shouldn't happen in a conforming C++ program that doesn't evaluate operations with undefined behavior. But in non-portable C or C++ program doing manual memory management with OS primitives targeting a specific OS the conditions for segfault are well documented, and you can also rely on the programs behavior when that happens (the OS raises a signal for you, that you can handle). There are not many programs that should be written this way, but I assume the JVM might fall into this category. I'm still not sure if handling page faults this way in regular operation is the best strategy, but I would worry about performance more than correctness.

zdevito 3y ago

When developing PyTorch, we also run into a lot of mixed Python/C++ language situations. We've recently been experimenting with in-process 'combined' Python/C++/PyTorch 2.0 stack traces to make it easier to understand where code is executing (https://dev-discuss.pytorch.org/t/fast-combined-c-python-torchscript-inductor-tracebacks/1158 https://dev-discuss.pytorch.org/t/fast-combined-c-python-tor...).

voz_ 3y ago
Funny, I was going to post this, but the horse's mouth is even better ;)

nightpool 3y ago

Awesome to see some more Excalidraw diagrams on the front page, feels like it's really taking off.

jaza 3y ago

An inspiring, albeit daunting, write-up for someone like me, who has been a python coder for over 10 years, but who has never professionally coded in C or C++. I'd be pleasantly surprised with myself if I could one day debug like that. Also somewhat depressing that, yet again, the GIL was to blame, and that after all that impressive investigatory work, the fix was (spoiler alert!) "rewrite the offending function in C".

jebarker 3y ago
I predominantly develop in Python but somewhat frequently peer into the C++ and CUDA underlying Pytorch. I am not a competent C or C++ programmer but often am able to debug and fix bugs that originate in the lower levels of the stack. That ability developed just through exposure and focused prodding when I found issues, but I don't think it has much relation or translates to being a professional coder in C/C++/CUDA. Coding and debugging are different skill sets IMO.

vaughan 3y ago

It's unfortunate that no one thinks about the debugging experience with anything they build today. It's always a "strap yourself in" experience like in the post. I dream of an IDE that is one-click, full-stack, local debug. I dropped some ideas here: https://news.ycombinator.com/item?id=35650253 https://news.ycombinator.com/item?id=35650253