7 ms·
Show HN: Vex, a new way to run things in Python's virtualenvs
- waitingkuo 12y agoCool! That's the tool I always need!
- sashahart 12y agoThis is based on my personal experience with tools for using virtualenv. virtualenvwrapper is very powerful with all kinds of hooks but I just never used all that power and needed things like speed more. virtualenvwrapper provided a lazy loader but that would have various problems from time to time as you'll see on the issue tracker. So then for a long time I just ran 'virtualenv' and source wherever/bin/activate and found that I really didn't miss virtualenvwrapper all that much, which was a surprise because everyone finds 'workon' invaluable, a position I do understand. Later my wife taught herself virtualenv and then wrote like 5 lines of shell to do the same thing as virtualenvwrapper (the classic environment-modifying thing) which I found hilarious, and I wrote a version of that in zsh with a little more error checking to make me happy and just jammed it in my dotfiles. Then after some months of using that I decided this is nice enough to publish so I worked on rewriting it as portable shell functions with more error checking and was going to put it on PyPI. Toward the end of writing that I was feeling pretty finished, when some last detail that I forget made me hacked off about modifying the current shell environment, so on a lark I wrote a prototype of the new idea of only modifying a spawned process's environment in about an hour (most of which was learning further about portable shell) and once it was working I looked at implementing more error checking for all the weird conditions that may arise in shell and I said, you know what, I have no reason not to do this in Python and totally cut the dependency on which shell is being used, who is going to use this without Python anyway? Then I ended up figuring out how to do prompts and completion and pretty much had a publishable project, which was surprising. It's still faster to directly run the virtualenv's Python or to use your own small shell function but while YMMV, for my uses virtualenvwrapper is a lot of code I will never use and I use vex every day. Highlights of my todos: pydoc still doesn't work (the reasons for this are lame) and probably it will be nice to implement --make/--remove to substitute mkvirtualenv/rmvirtualenv (which would also let me dump a pydoc script in virtualenv's bin/ or Script\ dir to fix that problem...)
- burntsushi 12y ago> pydoc still doesn't work Are you using `pydoc` on the command line? If so, try pdoc: https://github.com/BurntSushi/pdoc https://github.com/BurntSushi/pdoc. I wrote it as a tool to replace epydoc (strictly automatic documentation), but it works on the command line just like pydoc. (It can also generate HTML documentation with cross module linking that isn't crap. See an example here: http://pdoc.burntsushi.net/nfldb http://pdoc.burntsushi.net/nfldb)
- sashahart 12y agoI do sometimes, more as a matter of muscle memory than volition. I'm going to make a try to use your tool as a replacement for a while and see if I can generate any issues for you :) However, I still would like to eventually unbreak pydoc for users who might have bad habits like I do. The issue is that virtualenv itself had an opportunity to fix this (with multiple patches submitted IIRC) and they decided to punt by implementing it as yet another shell function. So for me to unpunt without coupling to the shell, I pretty much have to dump a script into bin/. I guess maybe not that many people use pydoc anyway?
- burntsushi 12y ago> I guess maybe not that many people use pydoc anyway? That would be my guess. IMO, pydoc isn't very discriminating with what it produces. The HTML is pretty ugly and the command line version just kind of vomits everything in the module to stdout. My idea with pdoc was to try and discover a module's public interface, and then display that.
- runiq 12y ago> everyone finds 'workon' invaluable, a position I do understand. At first I wanted to praise virtualfish [1] and its ability to automatically activate a virtualenv when I cd into it, but then I realized that vex's way does indeed make more sense. What do we do in a virtualenv? We basically just run pip to do various things, or run the program (in case it's an application and not a library) or test suite. So yeah, I think I'm gonna switch. The added portability is a nice bonus. [1] https://github.com/adambrenecki/virtualfish https://github.com/adambrenecki/virtualfish
- acjohnson55 12y agoHa. I always thought something was unnecessarily awkward about virtualenv, but I couldn't quite put my finger on it. Virtualenvwrapper was an improvement in many ways, but its lack of portability could be a pain. Reading the docs, Vex makes perfect sense. We should use virtualenv basically the same way we sudo. It's kind of amazing it took this long for someone to do it. Thanks!
- ProblemFactory 12y agoVex seems very convenient, but it is also possible to avoid the "magic" with just standard virtualenv. If you provide the full path to the python binary in the virtualenv, it executes with the virtualenv loaded: /home/acjohnson/virtualenvs/foo/bin/python myscript.py This is useful for crontabs, system service config, etc. which don't load a regular user's bash setup files.
- shuzchen 12y agoSimilarly, anything you've installed in that virtualenv that installs itself in a bin folder (pip, mercurial, django-admin.py, etc) will be in that bin folder, and calling them from that directory will execute them in the context of that virtualenv as well.
- sashahart 12y agoFully agreed, and the README for vex suggests this if you are worried about overhead from the wrapper especially for non-interactive purposes where you are repeatedly incurring python startup.
- Deejahll 12y agoI'm very happy more people are coming around to this idea, despite multiple independent discoveries. :) Here's another project that attempts something similar: https://github.com/berdario/invewrapper https://github.com/berdario/invewrapper Here's a sortof-blog-post and some more discussion about the superiority of modifying a subshell rather than sourcing a bunch of janky shell-functions that modify your current shell: https://gist.github.com/datagrok/2199506 https://gist.github.com/datagrok/2199506
- sashahart 12y agoI did not know about invewrapper, thanks. pew is another decent name. edit: after looking at the doc, my impression is that invewrapper seems to follow virtualenvwrapper's design more closely with a large number of subcommands with names similar to virtualenvwrapper's, most of virtualenvwrapper's options and features (except hooks and maybe some of the project stuff). I would describe the options as "comprehensive." I specifically wanted something with a dramatically simpler interface and feature set. invewrapper also seems primarily or only intended to run a shell. I get personal use out of running arbitrary commands under vex as if it were sudo or something. It is an improvement not to modify the current environment and couple tightly to specific shells either way, though. One thing I do envy is the PowerShell prompt ;)
- berdario 12y ago(invewrapper/pew author here) I just discovered your project, so I haven't tried it yet, but since the approach seems similar, it should work just fine on Windows (maybe with some tweaks)... and you can just add the Powershell prompt to your docs, no need to envy it :) I agree that pew is intended as a virtualenvwrapper replacement, obviously you can get something similar to `vex env cmd` with `pew in env cmd`... Unfortunately I've neglected a little bit the project lately: the most serious thing I want to do is rewrite the test suite, to be able to run it on windows. Since you mention PowerShell maybe you use it quite often? I'd like to get some feedback on Windows (I added Windows support out of completeness, but I seldom use that OS), since I don't know anyone who uses pew on windows :) BTW, I discovered some new tools to manage environments in the last year: pyenv (I actually already knew rbenv), modules[1] and I even started to use Nix These have not the same scope, and in fact I appreciate the latter 2 idea of using the same system to manage dependencies and versions for all different kind of tools. I don't have an opinion yet on what's the correct way to do these things so, in the meanwhile, even tools that are python-specific (like ours) have a niche to fill [1] https://archive.fosdem.org/2014/schedule/event/hpc_devroom_environment/ https://archive.fosdem.org/2014/schedule/event/hpc_devroom_e...
- mmelin 12y agoThis looks great. Any way to have a system-wide .vexrc?
- sashahart 12y agoYes, there CAN be a way :) but would you do me a favor and write a spec for how you want that to work and jam that in a github issue? https://github.com/sashahart/vex/issues https://github.com/sashahart/vex/issues
- tudborg 12y agoLooks good. I currently use http://tudb.org/articles/2014/03/31/vpython/ http://tudb.org/articles/2014/03/31/vpython/ for pretty much all of my python projects (also, i am the author of vpython).
- sashahart 12y agoAh, discovering it from the directory. I considered this in analogy with the way vagrant works.
- tudborg 12y agoDownside is, ofc, that compared to just running something inside a virtualenv shell, vpython takes forever (up to 100ms-ish on my machine) to find the virtualenv, and set the correct env variables. Most of my projects are long running though (webservers, number crunchers, etc), so the extra miliseconds it takes to load is really not that important.
- deathanatos 12y agoWell huh. > Vpython is a tiny(-ish) bash script to help with your day to day virtualenv needs. > * You don't have to worry about sourcing the activate script. > * You don't have to point to your virtualenv path. > just use vpython instead of python to invoke your scripts. That's basically why I wrote vpython: I'm the author of this vpython[1], which isn't the same as yours, nor what the article is. (I suspect mine is a little more immature, as I've not worked on it for as long.) I guess similar needs beget similar utilities. [1]: https://github.com/thanatos/vpython
- tudborg 12y agoHa. How fun :) We take different aproaches though. Repo link for reference: https://github.com/tbug/vpython https://github.com/tbug/vpython Mine expects you to know virtualenv and how it works, and also that you explicitly install it. Same goes for dependencies. My needs where to avoid sourcing the virtualenv, so i wrote the tool to fix just that. Also, i built mine to resolve symlinks and be usable as the shebang in your python scripts, which is pretty neat when writing command line tools.
- schwuk 12y agoI've never bothered with virtualenvwrapper - I create virtualenvs alongside my projects, and call `$virtualenv/bin/python` or `$virtualenv/bin/pip` etc. directly. Sometimes I activate the virtualenv, sometimes I don't.
- StavrosK 12y agoThat's what I do too. I mostly activate them by hand nowadays, just to avoid calling python. Fish is smart enough that it knows that, if I type a period in a directory that contains a virtualenv, I want to activate it.
- selectnull 12y agoThis is cool, I like tools that focus on one thing. Just like the author, I never used all virtualenvwrapper's features and was always happy to create and delete VEs on my own. The only thing missing from standard virtualenv workflow is easy way to activate the VE; so I wrote that on my own and been using it since ever. It's trivially small shell script but it does the work well, you can check it out at pypi: https://pypi.python.org/pypi/ave/ https://pypi.python.org/pypi/ave/
- sashahart 12y agoWhat is truly horrible is that I wrote my shell function, named it ave, thought it was a great name and then months later (edit: after I released vex) realized you had written a package named ave. Oh well.. :) This is also worth checking out if you just want 'workon' type functionality without the fuss.
- selectnull 12y agoI guess I was faster picking ave :) but vex is good too.
- crdoconnor 12y agoI usually do this just by cd'ing to the project directory, which has the virtualenv inside it and running './venv/bin/python'. That eliminates the possibility that I'll accidentally run the wrong version of python - e.g. a system python or a python from another environment that didn't properly deactivate. Most other issues are easy to deal with (e.g. non-existent virtualenv, being in the wrong folder), but the issue of accidentally using the wrong environment is awful because it can manifest itself with subtle bugs or weird exceptions rather than an obvious error.
- e12e 12y agoI mostly do this, along with a little hack for certain commands I need, such as mercurial: I have a virtualenv under ~/opt/venv/py-2.7-misc, and make a symbolic link from ~/opt/venv/py-2.7-misc/bin to ~/opt/pybin -- and then my .bashrc checks for a ~/pybin and adds it to the PATH if it exists ( [[ -d "${HOME}/pybin" ]] ...). So I can just go "pip install -u mercurial", or "pip install <some useful utility>" without worrying too much. And (my) "hg" is in my PATH -- and I can still run stuff in custom virtualenvs on a project basis, just as you say, by specifying the full path (./proj/venv/bin/python ...). I've never understand why people use anything more complicated. I do see occasions where activate is useful -- certain packages, I think scipy and pygame are among them -- are not only rather difficult to install via pip, but also tend to be very picky about their environment. Normally I get by by just installing those (and other packages with strong c/c++ library dependencies) via the package manager though.
- neotrinity 12y agoone of the things I do in virtualenvwrapper is cdsitepackages to do some patch/workaround on the installed libraries. Not a lot though but its a nifty command to have.
- isbadawi 12y agoI like this. I've used both virtualenvwrapper and pew before, but they both have lots of features I don't use. Using this plus some small shell functions like the below to create, list and delete virtualenvs is enough for me. function mkvirtualenv { virtualenv "$HOME/.virtualenvs/$1" if [ $# -eq 2 ]; then vex "$1" pip install -r "$2" fi } function lsvirtualenv { ls "$HOME/.virtualenvs" } function rmvirtualenv { rm -rf "$HOME/.virtualenvs/$1" }
- sashahart 12y agoIt looks like there is a lot of demand for --make/--remove options to vex so I will do those (and that should nearly approach the end of the new features I will do on vex). Probably good to do some error checking on the arguments to mkvirtualenv and rmvirtualenv (sleepily hitting enter too early, etc.)