ole_tange

49 ms·

1.
by ole_tange 7y ago
Examples simulated with GNU Parallel: simple_gpu_scheduler --gpus 0 1 2 < gpu_commands.txt parallel -j3 --shuf CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}' < gpu_commands.txt simple_hypersearch "pyth
2.
GNU Parallel Cheat Sheet [pdf](gnu.org)
119 points by ole_tange 8y ago|63 comments
3.
by ole_tange 9y ago
Fun fact: GNU Parallel used to be a wrapper for making Makefiles: https://www.gnu.org/software/parallel/history.html Also see this if you have a system without GNU Parallel: http://oletange.blogspot.dk&
4.
by ole_tange 9y ago
--line-buffer may or may not give additional speed up.
5.
by ole_tange 9y ago
--pipe is well know for being slow. Try --pipe-part instead: parallel -a ngrams.tsv --pipe-part --block -1 awk -f map.awk | awk -f reduce.awk
6.
by ole_tange 10y ago
> Granted, it's always possible that the parallel on $PATH might not be the one you're thinking of, but... Whatevs. For exactly this reason, GNU Parallel has the option --minversion. So in your script you put: parallel --minver
7.
by ole_tange 10y ago
> Same syntax applies. Except that is not entirely true: For instance according to the author the '{= perl expression =}' will probably never be supported. (Full disclosure: I am the author of GNU Parallel. I fully support buil
8.
by ole_tange 10y ago
Rust-parallel _is_ fast, and there is clearly a niche here, that GNU Parallel is unlikely to fill: By design GNU Parallel will never require a compiler; this is so you can use GNU Parallel on old systems with no compilers (Think an old, dus
9.
by ole_tange 10y ago
The page with differences to other tools will be moved to 'man parallel_aternatives' in next version. It made sense when the section was short, but it has grown so big that it can make it harder to navigate. Thanks for input. (The
10.
by ole_tange 10y ago
Except for GNU Parallel --pipe. When you use '<' what you typically mean is '--pipepart -a file'.
11.
by ole_tange 10y ago
Sure do. Makes you feel that the future is now!
12.
by ole_tange 10y ago
Did you start by reading the "Reader's guide" which is presented before the first option is introduced in the man page?
13.
by ole_tange 10y ago
Do you have an example of "piping next to process invocation"? It sounds interesting.
14.
by ole_tange 10y ago
Just be aware that Stderr will contain stuff you did not ask for: https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B... The syntax difference is small for simple tasks.
15.
by ole_tange 10y ago
A good start is looking at the section 'Remote execution' in 'man parallel_tutorial'. Spoiler: You can have GNU Parallel kick the machines if the fail, and rerun the job on another machine.
16.
by ole_tange 10y ago
The options section is no doubt for people, who know the tool. But the huge section with examples is not. Did you read the "Reader's guide" which is presented before the first option in the man page?
17.
by ole_tange 10y ago
A lot of effort has gone into the documentation of GNU Parallel: Intro videos, a one hour tutorial, tons of examples, a man page on the design decisions behind the code, and even a "Reader's guide" in the man-page. Can you be
18.
by ole_tange 10y ago
Can I ask to to run these and then comment? LESS=+/EXAMPLE: man parallel LESS=+/Reader.s man parallel man parallel_tutorial
19.
by ole_tange 10y ago
Fun fact: GNU Parallel was originally a wrapper script that used `make` for the parallelization: https://www.gnu.org/software/parallel/history.html
20.
by ole_tange 10y ago
"inside" = temporary files - not RAM. See comparison on https://www.gnu.org/software/parallel/parallel_design.html#B...
21.
by ole_tange 10y ago
I hope you wont take this the wrong way, but can you elaborate on why you feel Tollef's parallel (from moreutils) is better than GNU Parallel?
22.
by ole_tange 10y ago
Using Makefiles, however, depends on the results being files. If that is not the case, make cannot tell how far you got. GNU Parallel has --joblog and can continue from where it left off or retry all failed jobs again.
23.
by ole_tange 10y ago
Bigger list: https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B...
24.
by ole_tange 10y ago
From: https://www.gnu.org/software/parallel/history.html > parallel dates back to around the same time. It was originally a wrapper that generated a makefile and used make -j to do the parallelization.
25.
by ole_tange 10y ago
Hitachi already made your theme song: https://www.youtube.com/watch?v=xb_PyKuI7II :)
26.
by ole_tange 10y ago
> a version in C that I think was first released before GNU parallel While this is technically correct, it is misleading: GNU Parallel existed before it became GNU. See details on: https://www.gnu.org/software/parall
27.
by ole_tange 10y ago
And it is exactly the hacking part that GNU Parallel tries to help with: A lot of the helper functions in GNU Parallel could be done by expert users (--nice, --tmux, --pipepart, env_parallel, --compress, --fifo, --cat, --transfer, --return,
28.
by ole_tange 10y ago
One of the reasons for the high line number count is the decision not to depend on modules that are not part of the core of Perl 5.8. Quite a few of the lines are to deal with different flavours of operating systems. The benefit of this is
29.
by ole_tange 10y ago
It does: Rsync.
30.
by ole_tange 10y ago
GNU Parallel is extremely sluggish because it does all sort of different things behind your back: It buffers output on disk (so the from different jobs are not mixed and you are not limited by the amount of RAM - it will even compress the b

More ›