7 ms·

The Trouble with all

33 points by the1024 2y ago

InfoSecErik 2y ago
The author seems to expect someone to be patrolling imports with a gun rather than a strongly-worded "we're not liable if you hurt yourself" sign.
- klyrs 2y ago
  Tell me you've never violated pep8 in public without saying pep8
kstrauser 2y ago
Eh. Python does indicate public and private APIs. "Name" is public. "_Name" is internal, but you can mess with it if you need to. "__Name" is private, and if you go there anyway and the thing explodes and gives you a bad haircut, you were warned. Python's position is that we're all adults here. Don't touch "_Name". Really don't touch "__Name". You can if you're an expert and willing to take responsibility for your actions. Addendum: And in this ModuleWrapper thing, I can still access `core._module.PrivateApi` so we're back to square one.
- 0x63_Problems 2y ago
  In my experience, Python devs rarely use the underscore. Package authors are pretty good about it. But when the only people using your code are your coworkers, people don't seem to put as much thought into having a clean interface.
  - kstrauser 2y ago
    That’s a reasonable decision if no one else will be using the code.
  - polotics 2y ago
    The notion that developers are not to be treated as adults and must have access forbidden to them, or else tight coupling will develop and rot the code, is super strange to me. If you cannot trust your coworkers to respect the _meaning of __names, then how can you trust them with much harder concerns like algorithms and data-structures? The article contains this amazing quote: """I’ve seen a unicorn startup dump their existing codebase and start over because the modules within it became so tightly coupled together, it was impossible to effectively develop within it or break them apart.""" This quote definitely needs flesh on its bones. I don't think this was Instagram, and I am not sure who else? Could the author chime in maybe?
    - eddd-ddde 2y ago
      Well, I don't trust my coworkers, neither do I trust my self. That's why code review is a thing. It's easy to miss an internal api access, so why not make it just impossible?
    - MrJohz 2y ago
      For me it's less about trusting coworkers, and more about trusting myself. When I see these sorts of discussions through those eyes, I can understand much more why people want guard rails when they're programming. I fancy myself a fairly competent dev, but I've seen code that I've written on a bad day where I can barely even understand what I was trying to achieve, let alone what I actually did achieve. Ideally, yes, there's code review and a second pair of eyes to make sure everything does make sense, and that my publics are public and privates are private. But sometimes a reviewer misses things as well, or the reviewer isn't as familiar with that bit of the codebase, or whatever other problem - and mistakes happen. Obviously mistakes can also happen with guard rails enabled. In a language with explicit exports or private-by-default attributes, you can still make mistakes and expose more implementation details than you wanted. But in my experience, that happens a lot less often than it does in Python, where privacy is just a matter of convention that's easy to forget. So in that regard, yes I trust myself and my coworkers, even with the difficult parts, but I also assume that we'll all occasionally make mistakes, and if I can find ways to avoid those mistakes, I'm all for it.
    - the1024 2y ago
- VeejayRampay 2y ago
  "we're all adults here" is self-justification for the fact that the language is unable to enforce anything
  - kstrauser 2y ago
    The false assumption that a language can enforce something is the root of 10,000 CVEs. For instance, in C, you have read/write access to the whole program's memory space. Want to work around some security mechanism? A well-aimed pointer will do the right trick. Python doesn't prevent you from ever touching _internal or __private names. If you're reviewing a Python PR that does that, you should be as suspicious as if you were reviewing C code that stomps all over memory. In either case you almost certainly should not be doing that. But sometimes, only sometimes, you might be the right person in the right use case to do it. Then you can ignore the linters screaming at you and concentrate on convincing your coworker this is a good idea.
matsz 2y ago
Python's imports are the worst I've seen in any mainstream programming language by far. Relative path imports are unnecessarily difficult, especially if you want to import something from a parent directory. There's no explicit way to define what you'd like to export either. The syntax is inconsistent, too: from X import Y import Z vs. (modern JS) import { Y } from 'X'; import * as Z from 'Z'; Even C/C++ make more sense here.
- simonw 2y ago
  It took me far longer than I'd like to admit for "import *" in modern JavaScript to make sense to me. Python imports made sense straight away - but then I've been a Python programmer for a long time so I'm far removed from a beginner's perspective.
  - jnpnj 2y ago
    Relative imports are fuzzy if you don't dig deep, that said with a project scaffolded for you, it's rarely a headache (like testing in the old days), you can rapidly try various amount of dots.
    - klyrs 2y ago
      I still haven't bothered to learn why relative imports require you to be in a package. It's a major headache if, like me, you do a lot of one-off work that doesn't warrant that project scaffolding. Well, not a major headache. I can always revert to the py2 way: symlinks.
      - Doxin 2y ago
        > It's a major headache if, like me, you do a lot of one-off work that doesn't warrant that project scaffolding. Instead of creating "myproject.py", create a myproject folder with a __main__.py file. Run it as "python -m myproject". Hardly any scaffolding and your code is in a package. It's one of the things I like about python, it's quite easy to move "up the ladder" scaffolding-wise if you need to.
aatarax 2y ago
Don't know about using __all__ for introspection, but I have found it immensely useful for organizing, reading, and communicating code. When a package has a bunch of files inside of it, but only a handful of names exposed in __all__ it helps a lot with orienting yourself around the package.
zokier 2y ago
If you think this sort of hack is going to keep your python codebase clean I got some bad news for ya. Also: https://docs.astral.sh/ruff/rules/import-private-name/ https://docs.astral.sh/ruff/rules/import-private-name/
- anamexis 2y ago
  That's the point of the article.
trainfromkansas 2y ago
__all__ is only relevant for * imports. And please, just don't use * imports. It really doesn't save you much time at the cost of implicit untraceable behavior. If you don't worry about * imports, you don't need to add the __all__ boilerplate to every module. This article is more about advertising a package called tach, that I suppose tries to add "true" private classes to Python. But it doesn't actually enforce anything, because you still need to run their tool that checks anything. You could just easily configure your standard linter to enforce this type of thing rather than use a super specialized tool.
- claxo 2y ago
  A direct benefit of using `__all__` at module A is better intellisense while editing files that imports A, if A has a small intended public API and many internal usage symbols.
  - trainfromkansas 2y ago
    I just tried it and at least the autocomplete in IPython appears to ignore __all__ when suggesting possible imports. I haven't tried any other tools' autocompletes. If module A has a small intended public API, you can structure it no matter how you want to achieve that. You can put those internal symbols behind their own object/class/module if you prefer. Using `__all__` has one functional consequence, which is `from A import *`. Again, I would avoid * imports entirely, but if you want to try to curb possible downstream problems from users who do indeed use * imports, I would also prefer not defining `__all__` because it's extra boilerplate you have to maintain and can very easily be missed on future updates.
  - ErikBjare 2y ago
    This is why I still (sometimes) bother with __all__. Makes autodoc better too.
nuttingd 2y ago
import my_module This is compatible with `__all__` if you have your code broken down into smaller sub-modules and collect them in the main module as follows: # my_module/__init__.py from .submodule import * from .another_submodule import *
- jph00 2y ago
  Yup that's what I do too. Works great. And nbdev auto-generates __all__ so I don't have to think about anything -- it all just works. (Besides which, when you do have __all__ just regular wildcard imports work well anyway. I've been using them for >10 years without trouble. I think people just repeat the claim that they're a problem without having 1st-hand experience of using them with __all__ correctly.)
VeejayRampay 2y ago
once again, python being the absolute worst amongst all widely used languages
plasticeagle 2y ago
Python is just plain unsuitable for any project larger than a couple of files. Every Python project contains a hidden and deadly complexity that will grow over time - and will eventually destroy it. There's no way around it, it creeps in no matter what you do. The imports situation is only part of it - it wasn't what killed our simulation tools, or our build scripts, or our test framework - and required that we rewrite them all in different and more suitable languages. Python's performance, global modules, whitespace, untyped-by-default code are all killers. You pretty much have to use virtual environments to permit isolation between the multiple differently-versioned sets of dependent packages that you'll need for any project of any complexity, which are a cumbersome and painful solution to a problem that simply shouldn't exist. It may technically be possible to write clean and maintainable code in Python if you try hard enough, but you're always skating so close to the edge that eventually somebody is going to get in there tip the whole fragile mess into the abyss.
- nick238 2y ago
  What's the one true language we should all be using?
  - valiant55 2y ago
    I don't think such a thing exists or should exist. Languages should be different and those differences will naturally be better at a variety of problems. But also it's definitely C#.
- Daishiman 2y ago
  Looks like somebody isn't running a linter on CI.
- sevensor 2y ago
  I’ve led the maintenance and development of a medium sized (100s of kloc) Python codebase over a period of five years. As it matured, the code became significantly less fragile and more maintainable. Using private members of modules (or any other object) turned into way less of a problem as we improved our design. Turns out the problems were mostly around I/O and mutable state. We fixed those problems by pushing side effects and mutation as far out to the edge of the codebase as possible. If your core data processing routines don’t have side effects and don’t mutate global state, who cares who the caller is?
gorgoiler 2y ago
It would be interesting to compare this with an alternative based on static analysis. The Python ecosystem has many standard tools nowadays to enforce consistent style, including how modules import each other. The ast and libcst modules are very fast and can quickly identify any imported symbols beginning with an underscore: from a import _naughty It’s also quite possible to build a list of symbols that were imported and ensure that their underscore-prefixed attributes are not accessed: import a a._naughty() You could get creative I suppose… import a f = next( getattr(a, f“_{s}”) for s in synonyms(“cheeky”) ) f() …but at that point one hopes that, as a last resort, ones reviewers would cry foul.
efilife 2y ago
This is a problem with the language

33 points by the1024 2y ago

The author seems to expect someone to be patrolling imports with a gun rather than a strongly-worded "we're not liable if you hurt yourself" sign.

klyrs 2y ago
Tell me you've never violated pep8 in public without saying pep8

kstrauser 2y ago

Eh. Python does indicate public and private APIs. "Name" is public. "_Name" is internal, but you can mess with it if you need to. "__Name" is private, and if you go there anyway and the thing explodes and gives you a bad haircut, you were warned. Python's position is that we're all adults here. Don't touch "_Name". Really don't touch "__Name". You can if you're an expert and willing to take responsibility for your actions. Addendum: And in this ModuleWrapper thing, I can still access `core._module.PrivateApi` so we're back to square one.

0x63_Problems 2y ago
In my experience, Python devs rarely use the underscore. Package authors are pretty good about it. But when the only people using your code are your coworkers, people don't seem to put as much thought into having a clean interface.
- kstrauser 2y ago
  That’s a reasonable decision if no one else will be using the code.
- polotics 2y ago
  The notion that developers are not to be treated as adults and must have access forbidden to them, or else tight coupling will develop and rot the code, is super strange to me. If you cannot trust your coworkers to respect the _meaning of __names, then how can you trust them with much harder concerns like algorithms and data-structures? The article contains this amazing quote: """I’ve seen a unicorn startup dump their existing codebase and start over because the modules within it became so tightly coupled together, it was impossible to effectively develop within it or break them apart.""" This quote definitely needs flesh on its bones. I don't think this was Instagram, and I am not sure who else? Could the author chime in maybe?
  - eddd-ddde 2y ago
    Well, I don't trust my coworkers, neither do I trust my self. That's why code review is a thing. It's easy to miss an internal api access, so why not make it just impossible?
  - MrJohz 2y ago
    For me it's less about trusting coworkers, and more about trusting myself. When I see these sorts of discussions through those eyes, I can understand much more why people want guard rails when they're programming. I fancy myself a fairly competent dev, but I've seen code that I've written on a bad day where I can barely even understand what I was trying to achieve, let alone what I actually did achieve. Ideally, yes, there's code review and a second pair of eyes to make sure everything does make sense, and that my publics are public and privates are private. But sometimes a reviewer misses things as well, or the reviewer isn't as familiar with that bit of the codebase, or whatever other problem - and mistakes happen. Obviously mistakes can also happen with guard rails enabled. In a language with explicit exports or private-by-default attributes, you can still make mistakes and expose more implementation details than you wanted. But in my experience, that happens a lot less often than it does in Python, where privacy is just a matter of convention that's easy to forget. So in that regard, yes I trust myself and my coworkers, even with the difficult parts, but I also assume that we'll all occasionally make mistakes, and if I can find ways to avoid those mistakes, I'm all for it.
  - the1024 2y ago
VeejayRampay 2y ago
"we're all adults here" is self-justification for the fact that the language is unable to enforce anything
- kstrauser 2y ago
  The false assumption that a language can enforce something is the root of 10,000 CVEs. For instance, in C, you have read/write access to the whole program's memory space. Want to work around some security mechanism? A well-aimed pointer will do the right trick. Python doesn't prevent you from ever touching _internal or __private names. If you're reviewing a Python PR that does that, you should be as suspicious as if you were reviewing C code that stomps all over memory. In either case you almost certainly should not be doing that. But sometimes, only sometimes, you might be the right person in the right use case to do it. Then you can ignore the linters screaming at you and concentrate on convincing your coworker this is a good idea.

matsz 2y ago

Python's imports are the worst I've seen in any mainstream programming language by far. Relative path imports are unnecessarily difficult, especially if you want to import something from a parent directory. There's no explicit way to define what you'd like to export either. The syntax is inconsistent, too: from X import Y import Z vs. (modern JS) import { Y } from 'X'; import * as Z from 'Z'; Even C/C++ make more sense here.

simonw 2y ago
It took me far longer than I'd like to admit for "import *" in modern JavaScript to make sense to me. Python imports made sense straight away - but then I've been a Python programmer for a long time so I'm far removed from a beginner's perspective.
- jnpnj 2y ago
  Relative imports are fuzzy if you don't dig deep, that said with a project scaffolded for you, it's rarely a headache (like testing in the old days), you can rapidly try various amount of dots.
  - klyrs 2y ago
    I still haven't bothered to learn why relative imports require you to be in a package. It's a major headache if, like me, you do a lot of one-off work that doesn't warrant that project scaffolding. Well, not a major headache. I can always revert to the py2 way: symlinks.
    - Doxin 2y ago
      > It's a major headache if, like me, you do a lot of one-off work that doesn't warrant that project scaffolding. Instead of creating "myproject.py", create a myproject folder with a __main__.py file. Run it as "python -m myproject". Hardly any scaffolding and your code is in a package. It's one of the things I like about python, it's quite easy to move "up the ladder" scaffolding-wise if you need to.

aatarax 2y ago

Don't know about using __all__ for introspection, but I have found it immensely useful for organizing, reading, and communicating code. When a package has a bunch of files inside of it, but only a handful of names exposed in __all__ it helps a lot with orienting yourself around the package.

zokier 2y ago

If you think this sort of hack is going to keep your python codebase clean I got some bad news for ya. Also: https://docs.astral.sh/ruff/rules/import-private-name/ https://docs.astral.sh/ruff/rules/import-private-name/

anamexis 2y ago
That's the point of the article.

trainfromkansas 2y ago

__all__ is only relevant for * imports. And please, just don't use * imports. It really doesn't save you much time at the cost of implicit untraceable behavior. If you don't worry about * imports, you don't need to add the __all__ boilerplate to every module. This article is more about advertising a package called tach, that I suppose tries to add "true" private classes to Python. But it doesn't actually enforce anything, because you still need to run their tool that checks anything. You could just easily configure your standard linter to enforce this type of thing rather than use a super specialized tool.

claxo 2y ago
A direct benefit of using `__all__` at module A is better intellisense while editing files that imports A, if A has a small intended public API and many internal usage symbols.
- trainfromkansas 2y ago
  I just tried it and at least the autocomplete in IPython appears to ignore __all__ when suggesting possible imports. I haven't tried any other tools' autocompletes. If module A has a small intended public API, you can structure it no matter how you want to achieve that. You can put those internal symbols behind their own object/class/module if you prefer. Using `__all__` has one functional consequence, which is `from A import *`. Again, I would avoid * imports entirely, but if you want to try to curb possible downstream problems from users who do indeed use * imports, I would also prefer not defining `__all__` because it's extra boilerplate you have to maintain and can very easily be missed on future updates.
- ErikBjare 2y ago
  This is why I still (sometimes) bother with __all__. Makes autodoc better too.

nuttingd 2y ago

import my_module This is compatible with `__all__` if you have your code broken down into smaller sub-modules and collect them in the main module as follows: # my_module/__init__.py from .submodule import * from .another_submodule import *

jph00 2y ago
Yup that's what I do too. Works great. And nbdev auto-generates __all__ so I don't have to think about anything -- it all just works. (Besides which, when you do have __all__ just regular wildcard imports work well anyway. I've been using them for >10 years without trouble. I think people just repeat the claim that they're a problem without having 1st-hand experience of using them with __all__ correctly.)

VeejayRampay 2y ago

once again, python being the absolute worst amongst all widely used languages

plasticeagle 2y ago

Python is just plain unsuitable for any project larger than a couple of files. Every Python project contains a hidden and deadly complexity that will grow over time - and will eventually destroy it. There's no way around it, it creeps in no matter what you do. The imports situation is only part of it - it wasn't what killed our simulation tools, or our build scripts, or our test framework - and required that we rewrite them all in different and more suitable languages. Python's performance, global modules, whitespace, untyped-by-default code are all killers. You pretty much have to use virtual environments to permit isolation between the multiple differently-versioned sets of dependent packages that you'll need for any project of any complexity, which are a cumbersome and painful solution to a problem that simply shouldn't exist. It may technically be possible to write clean and maintainable code in Python if you try hard enough, but you're always skating so close to the edge that eventually somebody is going to get in there tip the whole fragile mess into the abyss.

nick238 2y ago
What's the one true language we should all be using?
- valiant55 2y ago
  I don't think such a thing exists or should exist. Languages should be different and those differences will naturally be better at a variety of problems. But also it's definitely C#.
Daishiman 2y ago
Looks like somebody isn't running a linter on CI.
sevensor 2y ago
I’ve led the maintenance and development of a medium sized (100s of kloc) Python codebase over a period of five years. As it matured, the code became significantly less fragile and more maintainable. Using private members of modules (or any other object) turned into way less of a problem as we improved our design. Turns out the problems were mostly around I/O and mutable state. We fixed those problems by pushing side effects and mutation as far out to the edge of the codebase as possible. If your core data processing routines don’t have side effects and don’t mutate global state, who cares who the caller is?

gorgoiler 2y ago

It would be interesting to compare this with an alternative based on static analysis. The Python ecosystem has many standard tools nowadays to enforce consistent style, including how modules import each other. The ast and libcst modules are very fast and can quickly identify any imported symbols beginning with an underscore: from a import _naughty It’s also quite possible to build a list of symbols that were imported and ensure that their underscore-prefixed attributes are not accessed: import a a._naughty() You could get creative I suppose… import a f = next( getattr(a, f“_{s}”) for s in synonyms(“cheeky”) ) f() …but at that point one hopes that, as a last resort, ones reviewers would cry foul.

efilife 2y ago

This is a problem with the language

The Trouble with __all__

The Trouble with __all__

The Trouble with all

The Trouble with all