6 ms·

Had to write a fairly substantial native extension to Python a couple years ago and one of the things I enjoyed was that the details were not easily "Googleable

Had to write a fairly substantial native extension to Python a couple years ago and one of the things I enjoyed was that the details were not easily "Googleable" because implementation results were swamped by language level results. It took me back to the old days of source diving and accumulated knowledge that you carried around in your head. https://www.dave.org/posts/20220806_python/ https://www.dave.org/posts/20220806_python/

davepeck 8mo ago
I made some small contributions to cpython during the 3.14 cycle. The codebase is an interesting mix of modern and “90s style” C code. I found that agentic coding tools were quite good at answering my architectural questions; even when their answers were only half correct, they usually pointed me in the right direction. (I didn’t use AI to write code and I wonder if agentic tools would struggle with certain aspects of the codebase like, for instance, the Cambrian explosion of utility macros used throughout.)
- squeedles 8mo ago
  This was around 2021 so AI code tools had not yet eaten everyone. One of the most interesting challenges was finding the right value judgements when blending multiple type systems. I doubt any agentic coding tool could do it today. I blended the python type system with a large low-level type system (STEP AIM low level types) and a smaller set of higher-level types (STEP ARM, similar to a database view). I already was familiar with STEP, so I needed to really grok what Python was doing under the covers because I needed to virtualize the STEP ARM and AIM access while making it look like "normal" Python.
  - davepeck 8mo ago
    Oh, that's very interesting work. And, yes, I'd also be surprised if (today's) agentic tools were at all helpful for that: it's way outside of distribution, and conceptual correctness truly matters.
EdwardDiego 8mo ago
Great write up, thank you for sharing it! Quick question though, in your first code example (dynamic enum with a metaclass) what is "m" in this line towards the start? Py_DECREF(m) Is it the metaclass?
- lozenge 8mo ago
  That's a standard error clause. In the case PyImport_ImportModule threw a Python exception, you need to Py_DECREF any C local variables which are new references(not borrowed references) and return -1. From the later call PyModule_AddObject, it's clear this code has come from the PyInit_ module initialisation function. This code is running on import of the C extension to initialise the "FruitEnum" module attribute. https://docs.python.org/3/c-api/extension-modules.html#c.PyInit_modulename https://docs.python.org/3/c-api/extension-modules.html#c.PyI...
  - squeedles 8mo ago
    Exactly so. I didn't notice that missing def when I put together the blog post, but you are right to call it out. In this case that decref was copypasta from some other code -- I don't decref on the other error returns. I combined code that was in several places and omitted the decref for mod_enum too! The module init function is where you would normally create the module object (PyModule_Create) and decref it if an error occurs. The blog example is utility code that you would call within the module init function to add an enum. Someone should really create a blog post compiler to catch these sorts of things :-)
throwaway81523 8mo ago
There's a file on docs.python.org explaining the C api. The rest is pretty straightforward, at least until recently when free threading was introduced (IDK about now). Main hassle is manually having to track reference borrowing etc. Understandable in Python 2, but another tragedy in Python 3.

davepeck 8mo ago

I made some small contributions to cpython during the 3.14 cycle. The codebase is an interesting mix of modern and “90s style” C code. I found that agentic coding tools were quite good at answering my architectural questions; even when their answers were only half correct, they usually pointed me in the right direction. (I didn’t use AI to write code and I wonder if agentic tools would struggle with certain aspects of the codebase like, for instance, the Cambrian explosion of utility macros used throughout.)

squeedles 8mo ago
This was around 2021 so AI code tools had not yet eaten everyone. One of the most interesting challenges was finding the right value judgements when blending multiple type systems. I doubt any agentic coding tool could do it today. I blended the python type system with a large low-level type system (STEP AIM low level types) and a smaller set of higher-level types (STEP ARM, similar to a database view). I already was familiar with STEP, so I needed to really grok what Python was doing under the covers because I needed to virtualize the STEP ARM and AIM access while making it look like "normal" Python.
- davepeck 8mo ago
  Oh, that's very interesting work. And, yes, I'd also be surprised if (today's) agentic tools were at all helpful for that: it's way outside of distribution, and conceptual correctness truly matters.

EdwardDiego 8mo ago

Great write up, thank you for sharing it! Quick question though, in your first code example (dynamic enum with a metaclass) what is "m" in this line towards the start? Py_DECREF(m) Is it the metaclass?

lozenge 8mo ago
That's a standard error clause. In the case PyImport_ImportModule threw a Python exception, you need to Py_DECREF any C local variables which are new references(not borrowed references) and return -1. From the later call PyModule_AddObject, it's clear this code has come from the PyInit_ module initialisation function. This code is running on import of the C extension to initialise the "FruitEnum" module attribute. https://docs.python.org/3/c-api/extension-modules.html#c.PyInit_modulename https://docs.python.org/3/c-api/extension-modules.html#c.PyI...
- squeedles 8mo ago
  Exactly so. I didn't notice that missing def when I put together the blog post, but you are right to call it out. In this case that decref was copypasta from some other code -- I don't decref on the other error returns. I combined code that was in several places and omitted the decref for mod_enum too! The module init function is where you would normally create the module object (PyModule_Create) and decref it if an error occurs. The blog example is utility code that you would call within the module init function to add an enum. Someone should really create a blog post compiler to catch these sorts of things :-)

throwaway81523 8mo ago

There's a file on docs.python.org explaining the C api. The rest is pretty straightforward, at least until recently when free threading was introduced (IDK about now). Main hassle is manually having to track reference borrowing etc. Understandable in Python 2, but another tragedy in Python 3.