7 ms·

This helps highlights the main issue I have with python today, and that's running python apps. Having to ship a compiler to a host|container to build python wi

• hprotagonist 6y ago
  > Then there's the rigmarole around getting one of these environments to play nice with cron... You pass cron the full path to your entry point. Where’s the rigamarole?
  • jalons 6y ago
    You've also got to include the pyenv shim instantiation. So now you've got something like 0 0 * * * /path/to/bash/script/to/init/pyenv && /my/path/to/poetry run /my/path/to/python.py -arg1
    • cbcoutinho 6y ago
      If the path to your executable is fixed, just put it in the shebang and you're done - makes everything way more explicit at the cost of some dynamic behavior. An anecdote: Homebrew uses this method for shipping python executables.
    • Chiron1991 6y ago
      Hm, haven't tried it, but doing this should be much easier: 0 0 * * * /path/to/interpreter/created/by/poetry/bin/python myscript.py
      • hprotagonist 6y ago
        this is precisely correct.
      • EE84M3i 6y ago
        Does bin/python in a virtualenv set PYTHONHOME correctly..?
        • remlov 6y ago
          Yes. And the answer by Chiron1991 is the proper way to do this since pretty much forever.
          • EE84M3i 6y ago
            Strange, this does seem to work with python3.8 on ubuntu 20.04 (the site-packages shows up in sys.path), but for me in a virtualenv bin/python is a symlink to the system python, so how does python 'know' what path to use? Is there logic baked into the interpreter? I seem to recall that with python2.7 that calling bin/python in a virtualenv without activating the virtualenv did not used to "work" (i.e. it would use the system packages). Did this change at some point or is my memory just wrong?
          • jalons 6y ago
            Yes, if the python interpreter the poetry environment utilizes is in your $PATH.
    • im3w1l 6y ago
      The "production version" of your script should be running in your system environment with system packages. pyenv and friends should be used for testing with different versions and making sure you don't accidentally depend on idiosyncrasies of your box. The exception is if your python thingy is "the main thing" running on a server, i.e. your customer facing webapp. My $.10 anyway
      • jalons 6y ago
        I tend to agree with you that pyenv|pipenv|etc shouldn't be used for actual production usage. This of course leads to other issues to solve, now that your development environment doesn't actually mirror production.
      • ses1984 6y ago
        How do you package your entire pyenv as one or more system packages?
        • hprotagonist 6y ago
          pyinstaller, shiv, pex, docker: depending on use case, any of these may be appropriate.
        • im3w1l 6y ago
          This isn't exactly what you are asking, but https://askubuntu.com/questions/90764/how-do-i-create-a-deb-package-for-a-single-python-script https://askubuntu.com/questions/90764/how-do-i-create-a-deb-...
• jtdev 6y ago
  Containerizing Python applications can really simplify things in that regard.
  • linkdd 6y ago
    No, the problem is just moved inside the container.
    • londt8 6y ago
      docker with pip-tools is great combination, you get deterministic builds easily
    • jbergknoff 6y ago
      In other words, it becomes the concern of the person shipping the code, rather than the concern of the person trying to run the code. That's exactly how it should be.
      • throwaway894345 6y ago
        In other languages it's a problem for neither, which I think is the parent's point.
      • linkdd 6y ago
        Still, the person trying to run the code has to setup Docker. And if you're on Windows, your Docker host is in a virtual machine, so networking and volumes are not so simple anymore. Replacing one kind of complexity by another is not a solution, it is a trade-off.
        • jtdev 6y ago
          Are people still suffering through hosting Docker containers on Windows? Why would anyone do that at this point other than to comply with outdated, arbitrary IT policies?
          • 6y ago
• foobarbecue 6y ago
  Why a whole OS? Can't you just install Conda or miniconda?
  • jalons 6y ago
    It was a tongue in cheek joke that conda is complex enough to be it's own OS.
• traverseda 6y ago
  On linux I normally just use pipx. As long as the package uses proper entry-points it just works.
• BiteCode_dev 6y ago
  Most of the time, you don't need all that, since Python has zipapps. You defined deps, you zip it, you ship to any same os with the same python version. It embeds everything, and just run. We even how have a nice tool to automate the bundling for you: https://pypi.org/project/shiv/ https://pypi.org/project/shiv/ Of course you still have to figure out how to get a Python installed on the final machine, that's the price to pay to be an interpreted language. We don't have yet a story to ship a beautiful exe/dmg/deb/rpm that embeds the zipapp and libpython in an easy way.
  • michaelcampbell 6y ago
    > you ship to any same os with the same python version This is a non-trivial thing to handwave away.
    • BiteCode_dev 6y ago
      I agree, but it's still way easier than the original story, which is the one you also have with PHP, Ruby, JS, etc. Using an interpretted language always leads to this. I know no popular interpretted language with a seamless experience to ship a standalone exe. In fact, Python is probably the one with the best story here, since it has nuitka (https://nuitka.net/ https://nuitka.net/), which allows to compile Python code into a fully standalone exe. But then you need to install a compiler, headers, etc. And no cross compilation of course. Not to mention on Linux, you have to ensure you target the lowest version of libc you can. You are still very far from Go or Rust, and I'm hoping one day that RustPython will succeed because that would mean an amazing deployment story. Meanwhile, you trade the ease of deployment of compiled languages for the ease of development of interpretted ones. I think it's a fare trade for most people: you dev the program much more often that you deploy it. That doesn't mean we shouldn't work, as a community, to improve the deployment story. It's a serious hindrance. That's the raison d'être of the Briefcase project (https://beeware.org/project/projects/tools/briefcase/ https://beeware.org/project/projects/tools/briefcase/). It's still in progress, but the last prez I saw on it was quite impressive already.
      • 6y ago
• squaresmile 6y ago
  I probably haven't bought into all of poetry yet but for deployment, I have been using "poetry export" to get the pinned requirements.txt, commit it to the repo and install to a virtualenv. A bit of work to keep it in sync with the poetry dependency file but that's ok. For PATH with cron or others, I use the full path to the virtualenv such as /path/to/project/.venv/bin/python. The path can be extracted by "which" or "Get-Command" when the venv is active. Using a python version different from the system python version is probably the messiest part but well, targeting 3.6 is alright. I do agree it could be better and it's not quite as streamlined as other ecosystems.
  • BiteCode_dev 6y ago
    Honestly, pip freeze includes the whole content of a venv site-packages, and the exact versions. For most projects, that's equivalent to all the dependancies recursively pins with peotry, although you don't have the clean pyproject-dev-prod/lock file separation. So a huge number of cases can be handled with just that. It will be "reproducible" enought for a lot of people.
    • squaresmile 6y ago
      > although you don't have the clean pyproject-dev-prod/lock file separation That's why I use "poetry export -f requirements.txt > requirements.txt" instead of pip freeze. It only exports prod requirements from the poettry lock file.
      • BiteCode_dev 6y ago
        Nice trick, and very useful to make sure people never have to know about poetry if your team can't deal with it.
• moreaccountspls 6y ago
  This is why I've switched to writing "quick wins" in shell [or Go]. It's just so much nonsense that has nothing to do with actually programming. Posix shell can be a bit baroque, but you know that it's not ever going to change and because of that, it's pretty easy to ship to any *nix. There is the question of the dependencies of a shell script, but I find in practice just checking for deps like `curl` at the beginning leads to be a better user experience. It's unlikely that there is going to be a ton of tools you require, and the tools you do require are probably going to be good about backwards compatibility [curl again as an example].
  • dijit 6y ago
    I generally agree with your sentiment here, but be careful with assuming bash==bash There are differences between versions. I can't even remember what they are off the top of my head like I used to, which makes them all the more aggravating to discover again. But I would recommend sticking to a subset of bash, not any of the new fancy features like 'globstar' which allows recursively globbing. There are tools to manage these kinds of tests, like bashenv. But you're in the same problem scope at that point.
    • moreaccountspls 6y ago
      That's why I said "Posix shell" and not bash.
    • GolDDranks 6y ago
      I much agree with the sister comment and I write my shells for /bin/sh also. There is this wonderful tool called ShellCheck ( https://www.shellcheck.net/ https://www.shellcheck.net/ ) that checks that your script is actually POSIX-compliant if it starts with #!/bin/sh
  • Dobbs 6y ago
    Except it does all the time. There are innumerable differences between the OSX, BSD, GNU, and other versions of common command line tools. There are plenty of cases where `jq` will or will not be available. Finally there are differences in how `/bin/sh` will interpret things (which there shouldn't be) depending upon underlying shell is running ksh, zsh, bash, dash, etc.
• brown9-2 6y ago
  > Or you use requirements files, and don't get deterministic results. What about pip freeze?
  • easterncalculus 6y ago
    I was also confused, pip freeze defines version numbers so it should be pretty clear-cut.
• nojito 6y ago
  I solved all of my python deployment concerns by using lxd. I wonder how many others are doing the same but are keeping it close to their chest because it’s such an amazing advantage.
  • levi_n 6y ago
    I'd love to more about how you are using lxd
• Mikhail_K 6y ago
  The best modern Python practice is, don't use Python.
• doorstar 6y ago
  I decided to drag myself kicking-and-screaming to the 21st century and start writing my handy-dandy utility scripts in python instead of bash. All was well and good until I made them available to the rest of my team, and suddenly I'm in python dependency hell. I search the internet and there are a lot of different solutions but all have their problems and there's no standard answer. I decided "to heck with it" and went back to bash. There's no built-in JSON parser but I can use 'grep' and 'cut' as well as anyone so the end result is the same. I push it to our repo, I tell coworkers to run it, and I wash my hands of the thing.
  • zxexz 6y ago
    jq has been a lifesaver for me parsing json in bash. Of course, it's an external utility not present by default in most systems. Another thing to consider is more of a middle-ground approach. Most systems do have a python interpreter, so you can use a lot of base python without worrying about dependency hell. I use inline python in bash all the time, e.g. ls | python -c 'import sys,json;lines=sys.stdin.read();print(json.dumps(list(filter(bool,lines.split("\n"))),sort_keys=True,indent=2))' You can even use variable substitution, if you surround the python code in double quotes. Even mix f-strings and bash substitution python -c "print(f'Congrats, ${USER}, you are visitor number ${RANDOM}. This is {__name__}, running in $(pwd)')"
    • moreaccountspls 6y ago
      Great trick with using the python standard lib! Thanks for posting that. edit: You probably already know this, but for anyone reading along, piping `ls` is unsafe if you plan to use the paths for anything except for printing them out. A path on linux can contain any byte except for NULL, so when `ls` prints them out, you can get broken behavior if you try to break on newlines.
      • deleted 6y ago
        [deleted]
    • kbenson 6y ago
      Or use a heredoc to not worry about competing quote chars: # python << EOPYTHON print("Congrats, ${USER}") print("You are visitor ${RANDOM}") print("This is {__name__}, running in ${pwd}") print("It's a heredoc to allow both quote characters") EOPYTHON
• alexhutcheson 6y ago
  You can use Bazel to build a self-contained Python binary that bundles the interpreter and all its dependencies by using a py_runtime rule[1]. It's fairly straightforward and doesn't require much Bazel knowledge - there are simple examples on GitHub[2]. There are a couple other tools that take the same approach, including PyOxidizer[3], which was written by a Mercurial maintainer. [1] https://docs.bazel.build/versions/master/be/python.html#py_runtime https://docs.bazel.build/versions/master/be/python.html#py_r... [2] https://github.com/erain/bazel-python-example https://github.com/erain/bazel-python-example [3] https://pyoxidizer.readthedocs.io/en/stable/overview.html https://pyoxidizer.readthedocs.io/en/stable/overview.html
  • ed25519FUUU 6y ago
    Can you really make it self contained? For example, does the host need a tz package? What about libssl or libcrypto? As far as I know the only language making static binaries easily is Go, but it was a first class language design principle. For everybody else it’s a jenky 1000 line Makefile. And don’t get me started on cross-compiling!
    • toby 6y ago
      Rust?
      • ed25519FUUU 6y ago
        Rust can definitely do it, but there still are a lot of gotchas. Many languages can do it, but there are so many pitfalls. For example, a host tz package.
        • jzoch 6y ago
          I would argue rust does it much better than go. When you have to resort to hacks like cgo that subtly change the performance and functional characteristics of your program i wouldnt call it "first class". Its good, dont get me wrong, I like how go cross-compiles most things. I wouldn't say its the gold-standard as long as cgo continues to be a thing Edit: I mention cgo as many who want to cross-compile a statically linked binary may want to interface with other libs via FFI and this is a huge gotcha. It is a bit tangential to strict "static linking binary building".
• deleted 6y ago
  [deleted]
• radus 6y ago
  For a trivial-ish command-line tool, I've enjoyed using pyinstaller with --onefile to put out a single file executable. Using GitHub Actions, it was also relatively easy to create cross-plaform releases.
• ebg13 6y ago
  I once threw a relatively complex Python application with background server/client processes at Cython and the generated .exe literally just worked without any special effort. I don't know how transferable that is, but N=1 it's not always as hard as what you're thinking.
• potta_coffee 6y ago
  That's why I switched to Go for a lot of things. I don't care that much for Go as a language but compiling and shipping code is just darn easy.
  • overcast 6y ago
    Let me know when Django is rewritten in Go.
    • moreaccountspls 6y ago
      That's not a "quick win" use case though.
    • potta_coffee 6y ago
      I like Django too. I'm building tools on top of AWS, and small REST apis in Go. I don't use it for everything.
• rufugee 6y ago
  What about PyInstaller? https://realpython.com/pyinstaller-python/ https://realpython.com/pyinstaller-python/
• _bxg1 6y ago
  It may be unpopular to say here, but I see Node as the best option. - Runtime comes with a package manager - Dependencies (not just imports, but tooling) are fully manifested in a project-local file - Installs dependencies in a project-local directory - Can specify exact package versions if you want maximum stability - Left-pad can't happen again due to policy changes: https://docs.npmjs.com/cli/unpublish#description https://docs.npmjs.com/cli/unpublish#description - Doesn't require any build steps or extra hoops if you're fine with skipping static types In general it just does a really great job isolating from the environment. No messing with environment variables, most things even run fine on Windows out of the box. All you need is node itself installed and you're off to the races, whether you're starting a new project or running one you checked out from github.
• erlend_sh 6y ago
  RustPython might paint a prettier picture for a better future in this regard. https://github.com/RustPython/RustPython https://github.com/RustPython/RustPython
• antb123 6y ago
  The minimal interpreter is pretty small when compressed - we shipped it to thousands of windows pcs as one exe file.
• DreamScatter 6y ago
  Recommend you try out Julia language!