6 ms·

Thanks for your feedback. We do mention parallelization in the designdoc, it's just not implemented yet. It's quite easy to add though. We have a lot of feature

Thanks for your feedback. We do mention parallelization in the designdoc, it's just not implemented yet. It's quite easy to add though. We have a lot of features spec'ed out, but not implemented. I would appreciate if you elaborated on separating step definitions from dependency definitions. In my mind, they are the same thing. If you mean that steps might not be connected by input-output relationship, but still have dependencies, Drake fully supports that via tags. If you mean that steps might be connected through input-output files, but not depend upon each other, I don't frankly see how it's possible. And if you mean some other syntax which more clearly separates the two, Drake supports methods which achieves exactly that. If you mean something else, I would love to see an example. Thanks!

zmmmmm 14y ago
> I would appreciate if you elaborated on separating step definitions from dependency definitions As I said, I only very quickly skimmed since I'm busy, I might have overlooked information, and apologies in that case. But take the example from the front page: evergreens.csv <- contracts.csv grep Evergreen $INPUT > $OUTPUT So now suppose a new requirement comes along - Evergreen is also called "Neverbrown" sometimes. It's decided the best way is to convert all references at input so nothing else gets confused downstream. So I need an extra step, now renamed.csv <- contracts.csv: sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT evergreens.csv <- renamed.csv: grep Evergreen $INPUT > $OUTPUT Adding this step forced me to modify the declaration of the original command, even though what I added had nothing to do with that command. With Bpipe, for example, you say extract_evergreens = { exec "grep Evergreen $input > $output" } fix_names = { exec "sed 's/Neverbrown/Evergreen/g' $input > $output" } Then you define your pipeline order separately - run { fix_names + extract_evergreens } If I get contracts from a different source that don't need the renaming, I can still run my old version and I'm not changing the definition of anything: run { extract_evergreens } Hope this explains what I mean, and again apologies if this is all clearly explained in your docs and I just jumped to conclusions from the simple examples!
- aboytsov 14y ago
  I see. Thank you very much. I think this is very cool. I can see several problems with this approach, and I would greatly appreciate it if you could comment on that. After all, I don't know Bpipe. The fundamental issue is why do you have to repeat the filename, and I did give it some thought. 1. What your example does is allows to allocate dependencies based on positions. It's pretty cool. This seems to be easily reproducible in Drake, if we add a special symbol that would just mean "a temporary file" for the output, and "last temporary output" for the input (by the way, you don't need colons): _ <- contracts.csv sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT evergreens.csv <- _ grep Evergreen $INPUT > $OUTPUT or even: <<- contracts.csv sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT evergreens.csv <<- grep Evergreen $INPUT > $OUTPUT 2. One of the problems, as you can see, that it only works if you don't care about the filenames, i.e. you use a temporary file. Similarly, your Bpipe expression: run { fix_names + extract_evergreens } doesn't care about filenames as well. How do you add it there? What if you need this file for debugging purposes, or if it's an input to some further step down the road? In this case, you'd have to do what you want to avoid doing (i.e. modify the original step). 3. I'm even more concerned with multiple inputs and multiple outputs. As long as your workflow is simple, you can get away with a + b. But when it's more complicated, you would have to do something like: run { (((fix_names + extract_evergreens) * and_some_otheroutput) + some_other_step) * some_other_output } (I used * as an operator that puts two outputs together to create an input with two files for the next command. Mathematically, + is better for that and * is for what + is used in your examples. :)) As you can see, it gets unreadable so fast, that you'd want to use some sort of identifiers to specify dependencies, and would end up with a scheme pretty much equivalent to filenames. The fact that some file might be a temporary is a related, but parallel problem. 4. Now even worse, I'm not quite sure how this syntax could accomodate multiple outputs. If fix_name creates several outputs, and extract_evergreens uses only one, you can't get around it without some weird syntax and specifying a numeric position. It also gets out of hand pretty quickly and you're back to using some sort of identifiers, be it filenames or not. 5. Speaking of identifiers, you can use variables in Drake instead of filenames, so you can abstract filenames away. But it seems to me there's a more fundamental problem in play. 6. If you're concerned with coupling implementation and input and output names, Drake has methods for this: fix_names() sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT extract_evergreens() grep Evergreen $INPUT > $OUTPUT renamed.csv <- contracts.csv [method:fix_names] evergreens.csv <- renamed.csv [method:extract_evergreens] or even, as discussed above: <<- contracts.csv [method:fix_names] evergreens.csv <<- [method:extract_evergreens] To summarize, I think your example is cool, but it seems to only be practical for rather simple workflows. And I can also see how Drake can easily be extended to support such syntactic sugar. For more complicated dependencies though, I don't really see a better approach. I would love to hear your further thoughts on the matter, and whether you'd like to see something similar to what I proposed in Drake. Or something else. Artem.

zmmmmm 14y ago

> I would appreciate if you elaborated on separating step definitions from dependency definitions As I said, I only very quickly skimmed since I'm busy, I might have overlooked information, and apologies in that case. But take the example from the front page: evergreens.csv <- contracts.csv grep Evergreen $INPUT > $OUTPUT So now suppose a new requirement comes along - Evergreen is also called "Neverbrown" sometimes. It's decided the best way is to convert all references at input so nothing else gets confused downstream. So I need an extra step, now renamed.csv <- contracts.csv: sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT evergreens.csv <- renamed.csv: grep Evergreen $INPUT > $OUTPUT Adding this step forced me to modify the declaration of the original command, even though what I added had nothing to do with that command. With Bpipe, for example, you say extract_evergreens = { exec "grep Evergreen $input > $output" } fix_names = { exec "sed 's/Neverbrown/Evergreen/g' $input > $output" } Then you define your pipeline order separately - run { fix_names + extract_evergreens } If I get contracts from a different source that don't need the renaming, I can still run my old version and I'm not changing the definition of anything: run { extract_evergreens } Hope this explains what I mean, and again apologies if this is all clearly explained in your docs and I just jumped to conclusions from the simple examples!

aboytsov 14y ago
I see. Thank you very much. I think this is very cool. I can see several problems with this approach, and I would greatly appreciate it if you could comment on that. After all, I don't know Bpipe. The fundamental issue is why do you have to repeat the filename, and I did give it some thought. 1. What your example does is allows to allocate dependencies based on positions. It's pretty cool. This seems to be easily reproducible in Drake, if we add a special symbol that would just mean "a temporary file" for the output, and "last temporary output" for the input (by the way, you don't need colons): _ <- contracts.csv sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT evergreens.csv <- _ grep Evergreen $INPUT > $OUTPUT or even: <<- contracts.csv sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT evergreens.csv <<- grep Evergreen $INPUT > $OUTPUT 2. One of the problems, as you can see, that it only works if you don't care about the filenames, i.e. you use a temporary file. Similarly, your Bpipe expression: run { fix_names + extract_evergreens } doesn't care about filenames as well. How do you add it there? What if you need this file for debugging purposes, or if it's an input to some further step down the road? In this case, you'd have to do what you want to avoid doing (i.e. modify the original step). 3. I'm even more concerned with multiple inputs and multiple outputs. As long as your workflow is simple, you can get away with a + b. But when it's more complicated, you would have to do something like: run { (((fix_names + extract_evergreens) * and_some_otheroutput) + some_other_step) * some_other_output } (I used * as an operator that puts two outputs together to create an input with two files for the next command. Mathematically, + is better for that and * is for what + is used in your examples. :)) As you can see, it gets unreadable so fast, that you'd want to use some sort of identifiers to specify dependencies, and would end up with a scheme pretty much equivalent to filenames. The fact that some file might be a temporary is a related, but parallel problem. 4. Now even worse, I'm not quite sure how this syntax could accomodate multiple outputs. If fix_name creates several outputs, and extract_evergreens uses only one, you can't get around it without some weird syntax and specifying a numeric position. It also gets out of hand pretty quickly and you're back to using some sort of identifiers, be it filenames or not. 5. Speaking of identifiers, you can use variables in Drake instead of filenames, so you can abstract filenames away. But it seems to me there's a more fundamental problem in play. 6. If you're concerned with coupling implementation and input and output names, Drake has methods for this: fix_names() sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT extract_evergreens() grep Evergreen $INPUT > $OUTPUT renamed.csv <- contracts.csv [method:fix_names] evergreens.csv <- renamed.csv [method:extract_evergreens] or even, as discussed above: <<- contracts.csv [method:fix_names] evergreens.csv <<- [method:extract_evergreens] To summarize, I think your example is cool, but it seems to only be practical for rather simple workflows. And I can also see how Drake can easily be extended to support such syntactic sugar. For more complicated dependencies though, I don't really see a better approach. I would love to hear your further thoughts on the matter, and whether you'd like to see something similar to what I proposed in Drake. Or something else. Artem.