5 ms·

Doubtful lazy imports would’ve helped at all… I joined a project where almost every import statement has side effects, some of which took multiple minutes to re

Doubtful lazy imports would’ve helped at all… I joined a project where almost every import statement has side effects, some of which took multiple minutes to read things into memory etc. Tried some poor-man’s debugging and never hit a breakpoint on the first significant line of code… took a while to figure out as it was my first Python project. It almost feels like Python needs a scripting and non-scripting mode, or some kind of warning logging “you did everything wrong”

deepsun 4y ago
Python is a very good scripting language, so good that people sometimes mistake it for application language.
- KerrAvon 4y ago
  All of Python, Perl (!), Ruby, Tcl have been application languages at one time or another. Python evangelists, reel it in a bit, please.
- solarkraft 4y ago
  I have yet to understand how to clearly separate scripts from applications.
  - melony 4y ago
    For me it is AOT compilation and packaging. An application language should be able to pull in most of its dependencies and runtime and be packaged into a reasonably sized and performant binary. The reason why scripting languages (that are often dynamic) struggle with this is that too many of their dependencies have hotpaths written in third party languages like C which makes it hard to do easy cross compilation and linking. Some programming languages come from a culture that eschews AOT compilation. It is difficult to compile large dynamic languages with Lisp heritages (CL, Julia, Ruby etc., Scheme is an exception to this however). These languages always resort to using a stateful VM/interpreter snapshot in lieu of true AOT packaging (or even Java/.NET style lowered bytecode binaries). The existence of eval makes many things trivial but at the same time complicates deployment tremendously. Personally I am betting on JavaScript being the first dynamic and JITted (or incrementally compiled) language that can attain the AOT compilation without massively bloated binaries.
    - deepsun 4y ago
      Re. AOT, as I said in my sibling comment, there's a counterexample -- Julia. It's AOT compiled, but dynamically typed. And it falls in scripting camp, IMHO.
      - melony 4y ago
        Julia only does partial AOT, it still bundles the sysimage (~100MB+ of runtime, what I refer to in my above comment as the VM snapshot) c.f. https://docs.juliahub.com/PackageCompiler/MMV8C/1.2.7/devdocs/sysimages_part_1.html https://docs.juliahub.com/PackageCompiler/MMV8C/1.2.7/devdoc... Traditional AOT like you see in C/C++/Rust/Go/Zig would be able to treeshake and eliminate redundant codepaths, the binaries are all fairly small with minimal startup overhead.
        deepsun 4y ago
        Makes sense, thanks, I don't know much about Julia. Another counterexample to AOT compilation being a feature of application languages is Java and C# -- both are clearly application languages, with strong focus and presence there, but both are interpreted. Although I can argue that it depends on the terminology regarding "compile", whether transpilation to .class bytecode can be called compilation (and everyone do call it "compilation", but if it really was, there would be no point in real AOT compilers for Java/C#).
    - int_19h 4y ago
      So wait, then C# wasn't an application language before there was a way to package AOT binaries to deploy directly to users?
      - melony 4y ago
        It is, because the runtime is bundled with the OS (similar its counterpart the JVM is also extremely widely deployed) and its output is significantly lowered. C.f. > (or even Java/.NET style lowered bytecode binaries).
  - deepsun 4y ago
    Mostly it's original focus of the language and ecosystem developers, IMO. Although I found one feature to very strongly correlate -- static typing. For example, Julia is AOT compiled, but still dynamically typed. And coincidentally, it was designed for scientists, who need scripting much more. Another observation is how easy it is to write unit-tests (for canonically written code). Overloading "import" statement doesn't help there at all.
striking 4y ago
I added lazy loading by default to my workplace's monolith in Node.js. We relied on a couple of imports with side effects, which no longer worked because no one was accessing the modules that ran the side effects, and that was the trigger for actually loading the module. Our fix for this was to relegate stateful imports to files with "bootstrap" in the name, which the lazy loader stub would allow to be eagerly imported. Moreover, any imports listed in such "bootstrap" files would then be eagerly imported. But that's it (at least wrt the code belonging to our codebase); no one else, not even the children of the bootstrapping modules, were exempted from lazy loading. This allowed for a centralization of all the stateful import effects. If you tried to write stateful code outside of bootstrapping, it simply wouldn't run. (You could hypothetically hack around this. But it would be rather obvious, and I'm perfectly ready to revert pull requests that try.) Maybe you folks could try some variation of this in Python.
sdenton4 4y ago
Import side effects are the devil and should be killed with fire wherever they appear.
- jonnycomputer 4y ago
  Amen
- Spivak 4y ago
  The problem is that side effects are really broad and include things that are idiomatic Python. # jobs.py @scheduler.register_job(when=“daily”) def batch_job(): … This adds the job to the global scheduler. # models.py class Model(DBModel): def on_change(): # do stuff This adds the model to the ORM’s list of modules via a metaclasss and registers its hooks. # plugin.py class SomePlugin(PluginBase): … Same thing. The code to load plugins like this is just importing the module and letting the metaclass do the work.

deepsun 4y ago

Python is a very good scripting language, so good that people sometimes mistake it for application language.

KerrAvon 4y ago
All of Python, Perl (!), Ruby, Tcl have been application languages at one time or another. Python evangelists, reel it in a bit, please.
solarkraft 4y ago
I have yet to understand how to clearly separate scripts from applications.
- melony 4y ago
  For me it is AOT compilation and packaging. An application language should be able to pull in most of its dependencies and runtime and be packaged into a reasonably sized and performant binary. The reason why scripting languages (that are often dynamic) struggle with this is that too many of their dependencies have hotpaths written in third party languages like C which makes it hard to do easy cross compilation and linking. Some programming languages come from a culture that eschews AOT compilation. It is difficult to compile large dynamic languages with Lisp heritages (CL, Julia, Ruby etc., Scheme is an exception to this however). These languages always resort to using a stateful VM/interpreter snapshot in lieu of true AOT packaging (or even Java/.NET style lowered bytecode binaries). The existence of eval makes many things trivial but at the same time complicates deployment tremendously. Personally I am betting on JavaScript being the first dynamic and JITted (or incrementally compiled) language that can attain the AOT compilation without massively bloated binaries.
  - deepsun 4y ago
    Re. AOT, as I said in my sibling comment, there's a counterexample -- Julia. It's AOT compiled, but dynamically typed. And it falls in scripting camp, IMHO.
    - melony 4y ago
      Julia only does partial AOT, it still bundles the sysimage (~100MB+ of runtime, what I refer to in my above comment as the VM snapshot) c.f. https://docs.juliahub.com/PackageCompiler/MMV8C/1.2.7/devdocs/sysimages_part_1.html https://docs.juliahub.com/PackageCompiler/MMV8C/1.2.7/devdoc... Traditional AOT like you see in C/C++/Rust/Go/Zig would be able to treeshake and eliminate redundant codepaths, the binaries are all fairly small with minimal startup overhead.
      - deepsun 4y ago
        Makes sense, thanks, I don't know much about Julia. Another counterexample to AOT compilation being a feature of application languages is Java and C# -- both are clearly application languages, with strong focus and presence there, but both are interpreted. Although I can argue that it depends on the terminology regarding "compile", whether transpilation to .class bytecode can be called compilation (and everyone do call it "compilation", but if it really was, there would be no point in real AOT compilers for Java/C#).
  - int_19h 4y ago
    So wait, then C# wasn't an application language before there was a way to package AOT binaries to deploy directly to users?
    - melony 4y ago
      It is, because the runtime is bundled with the OS (similar its counterpart the JVM is also extremely widely deployed) and its output is significantly lowered. C.f. > (or even Java/.NET style lowered bytecode binaries).
- deepsun 4y ago
  Mostly it's original focus of the language and ecosystem developers, IMO. Although I found one feature to very strongly correlate -- static typing. For example, Julia is AOT compiled, but still dynamically typed. And coincidentally, it was designed for scientists, who need scripting much more. Another observation is how easy it is to write unit-tests (for canonically written code). Overloading "import" statement doesn't help there at all.

striking 4y ago

I added lazy loading by default to my workplace's monolith in Node.js. We relied on a couple of imports with side effects, which no longer worked because no one was accessing the modules that ran the side effects, and that was the trigger for actually loading the module. Our fix for this was to relegate stateful imports to files with "bootstrap" in the name, which the lazy loader stub would allow to be eagerly imported. Moreover, any imports listed in such "bootstrap" files would then be eagerly imported. But that's it (at least wrt the code belonging to our codebase); no one else, not even the children of the bootstrapping modules, were exempted from lazy loading. This allowed for a centralization of all the stateful import effects. If you tried to write stateful code outside of bootstrapping, it simply wouldn't run. (You could hypothetically hack around this. But it would be rather obvious, and I'm perfectly ready to revert pull requests that try.) Maybe you folks could try some variation of this in Python.

sdenton4 4y ago

Import side effects are the devil and should be killed with fire wherever they appear.

jonnycomputer 4y ago
Amen
Spivak 4y ago
The problem is that side effects are really broad and include things that are idiomatic Python. # jobs.py @scheduler.register_job(when=“daily”) def batch_job(): … This adds the job to the global scheduler. # models.py class Model(DBModel): def on_change(): # do stuff This adds the model to the ORM’s list of modules via a metaclasss and registers its hooks. # plugin.py class SomePlugin(PluginBase): … Same thing. The code to load plugins like this is just importing the module and letting the metaclass do the work.