7 ms·

Thank you very much for your response. Actually, I don't think there are any philosophical differences, and I'll try to make my case. > Bpipe absolutely says

Thank you very much for your response. Actually, I don't think there are any philosophical differences, and I'll try to make my case. > Bpipe absolutely says you don't want to manage the file names. I think this is too strong a statement as I try to show below. > So, for example, after coming through the 'fix_names' stage, 'input.csv' will be called 'input.fix_names.csv'. fix_names is the identifier in this case. There's really not much of a difference whether you use identifiers to come up with filenames, or you use filenames to come up with identifiers. If anything, I think filenames are preferable, because the user doesn't have to be aware of the scheme the tool uses to convert identifiers to filenames. The fact that identifiers are just a little bit shorter (e.g. don't have .txt extension or something) does not overweigh the inconvenience of knowing where the files are. The problem with this approach is because figuring out where the files are requires knowledge of the tool inner workings, that can only be acquired from reading the code or documentation. There's another problem with these naming conventions, is that if you use the same code in multiple steps, things can become quite confusing. How will BPipe name them? Or is the only way to handle it is to copy-and-paste the code and create another rule? It seems like not clear enough separation between the code and the filenames can be a source of problems... Please correct me if I'm wrong. When I compare: _ <- contracts.csv sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT evergreens.csv <- _ grep Evergreen $INPUT > $OUTPUT with contracts: sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT.csv evergreens: grep Evergreen $INPUT.csv > $OUTPUT.csv contracts + evergreens I strongly prefer the first option, because there's less implicit things going on, and the code is separated clearer from the file naming. Besides, it's even shorter. > Similarly if there are a lot of inputs and you need the one ending with ".txt" you will write "$input.txt", if you want the second input ending with ".txt" you will write "$input2.txt", and so on. This can work for very simple workflows with maybe several cases of multiple inputs and outputs, but it's unmanageable when complexity grows. Imagine a step which takes 3 inputs - one separate, one which is output #2 of a previous step, and one which is output #6 of yet another step. You can't use numbers to resolve that. You will end up coming up with some sort of semantic identifiers, which will almost completely replace BPipe's naming convention. And what's worse, they will be hard-coded in your step's commands, which means you'll have to edit the code if you want to change the filenames, or re-use this step's implementation somewhere else. > When you start having hundreds or thousands of outputs naming them quickly goes from being something you want to do to a chore that drives you completely crazy and you want a tool to help you with. I'm not sure I agree here. Here's how I see it: Instead of naming hundreds of files, you have to name hundreds of methods (commands). Yes, you don't have to repeat the filenames to create dependencies, but you have to repeat the method names (in "contracts + evergreens"), and in a way which quickly breaches the boundaries of readability. This doesn't work for complicated workflows, and for simple ones, I would prefer positional linking rather than comping up with names, like in the example I provided above. There's nothing that prevents Drake from coming up with filenames from more abstract identifiers. We could come up with some syntax where you'd just give an identifier (say, "~contracts"), and we'll take care of the file location and name, just like BPipe does. The major difference is not this. The major difference is that we think you need to identify inputs and outputs to build the graph, and the method name is insignificant until you want code re-use, and BPipe seems to take the opposite position - that you need to give method names, and then use a separate expression to build the graph. I think I provided at least a few strong arguments why BPipe is wrong on this one. I would really love to hear your further thoughts. > As I touch on above, it's really not too hard. Bpipe gives you ways to query for inputs in a flexible manner to get the ones you want. I'm sorry I didn't understand neither this nor the example you provided. Could you please elaborate? In the example you provided you identify different outputs by adding a number to their names. Is that how subsequent steps are supposed to refer to them as inputs - by the positional output number from the step that used to generate them? > I'd argue that it's more than syntactic sugar, though - it's a different philosophy about what problems are important and what the tool should be helping you with. I appreciate your opinion. But the way I see it is this: 1) As far as different philosophies go, I find BPipe's one to be a bit problematic for complicated cases. 2) And for simple cases, it all comes down to syntactic sugar. I understand it's hard to argue an abstract, so I'll tell you what. Give me an example of a BPipe workflow that you particularly like, and I'll put it in Drake. I might need to invent some Drake features on the fly, but it's a good thing. This is what these discussions are for. I'll try to show you that there's no philosophical difference, and Drake has a more flexible approach overall. I am looking forward to this challenge, because your opinion is important to me. Thank you! Artem.

zmmmmm 14y ago
Hey, just want to say thanks for the great discussion again. I'm a bit humbled at the length & depth of thought you're putting into it. > The problem with this approach is because figuring out where the files are requires knowledge of the tool inner workings, that can only be acquired from reading the code or documentation I suppose this is true but it's really not an issue I have in practice. I run the pipeline and it produces (let's say) a .csv file as a result. I execute ls -lt *.csv And I see my result at the top. There's really not a huge inconvenience in trying to find the output. Having the pipeline tool automatically name everything instead of me having to specify it is definitely a win in my case. I suspect we're using these tools in very different contexts and that's why we feel differently about this. It sounds like you need the output to be well defined (probably because there's some other automated process that then takes the files?) You can specify the output file exactly with Bpipe, it's just not something you generally want to do. There's nothing wrong with either one - right tool for the job always wins! > if you use the same code in multiple steps, things can become quite confusing. How will BPipe name them It just keeps appending the identifiers: run { fix_names + fix_names + fix_names } will produce input.fix_names.fix_names.fix_names.csv. So there's no problem with file names stepping on each other, and it'll even be clear from the name that the file got processed 3 times. One problem is you do end up with huge file names - by the time it gets though 10 stages it's not uncommon to have gigantic 200 character file names. But after getting used to that I actually like the explicitness of it. > Imagine a step which takes 3 inputs - one separate, one which is output #2 of a previous step, and one which is output #6 of yet another step Absolutely - you can get situations like this. We're sort of into the 20% of cases that need more advanced syntax (eventually we'll explore all of Bpipes's functions this way :-) ). But basically Bpipe gives you a query language that lets you "glob" the results of the pipeline output tree (not the files in the directory) to find input files. So to get files from specific stages you could write: from(".xls", ".fix_names.csv", ".extract_evergreens.csv") { exec "combine_stuff.py $input.xls $input1.csv $input2.csv" } It doesn't solve everything, but I guess the idea is, make it work right for the majority of cases ("sensible defaults") and then offer ways to deal with harder cases ("make simple things easy, hard things possible"). And when you really get in trouble it's actually groovy code so you can write any programmatic logic you like to find and figure out the inputs if you really need to. > Instead of naming hundreds of files, you have to name hundreds of methods (commands) Not at all - if my pipeline has 15 stages then I have 15 commands to name. Those 15 stages might easily create hundreds of outputs though. > The major difference is that we think you need to identify inputs and outputs to build the graph, and the method name is insignificant until you want code re-use, and BPipe seems to take the opposite position - that you need to give method names, and then use a separate expression to build the graph Again, a really insightful comment, but I'd take it further (and this goes back to my very first comment). Bpipe isn't just not trying to build a graph up front, it really doesn't think there is a graph at all! At least, not an interesting one. The "graph" is a runtime product of the pipeline's execution. We don't actually know the graph until the pipeline finished. An individual pipeline stage can use if / then logic at runtime to decide whether to use a certain input or a different input and that will change the dependency graph. You have to go back and ask why you care about having the graph up front in the first place, and in fact it turns out you can get nearly everything you want without it. By not having the graph you lose some ability to do static analysis on the pipeline, but to have it you are giving up dynamic flexibility. So that's a tradeoff Bpipe makes (and there are downsides, it's just in the context where Bpipe shines the tradeoff is worth it). > In the example you provided you identify different outputs by adding a number to their names. Is that how subsequent steps are supposed to refer to them as inputs - by the positional output number from the step that used to generate them I think the "from" example above probably illustrates it. The simplest method is positional, but it doesn't have to be, you can filter with glob style matching to get inputs as well so if you need to pick out one then you just do so. > 1) As far as different philosophies go, I find BPipe's one to be a bit problematic for complicated cases. I can't argue with that - but that's sort of the idea: simple things easy, hard things possible. Complicated cases are complicated with every tool. I guess I would say that pipeline tools live at a level of abstraction where they aren't meant to get that complicated. > 2) And for simple cases, it all comes down to syntactic sugar. I guess I'd have to disagree with this, as I really think there are some fundamental differences in approach that go well beyond syntactic sugar. > Give me an example of a BPipe workflow that you particularly like, and I'll put it in Drake I wouldn't mind doing that - I'll need to look around and find an example I can share that would make sense (what I do is very domain specific - unless you have familiarity with bioinformatics it will probably be very hard to understand). I'll pm you when I manage to do this, but it may take me a little while (apologies). Thanks as always for the interesting discussion. I think this is a fascinating space, not least because there have been so many attempts at it - I would say there are probably dozens of tools like this going back over 20 years or so - and it seems like nobody has ever nailed it. Bpipe has problems, but so does every tool I've ever tried (I'm probably up to my 8th one or so now!).

zmmmmm 14y ago

Hey, just want to say thanks for the great discussion again. I'm a bit humbled at the length & depth of thought you're putting into it. > The problem with this approach is because figuring out where the files are requires knowledge of the tool inner workings, that can only be acquired from reading the code or documentation I suppose this is true but it's really not an issue I have in practice. I run the pipeline and it produces (let's say) a .csv file as a result. I execute ls -lt *.csv And I see my result at the top. There's really not a huge inconvenience in trying to find the output. Having the pipeline tool automatically name everything instead of me having to specify it is definitely a win in my case. I suspect we're using these tools in very different contexts and that's why we feel differently about this. It sounds like you need the output to be well defined (probably because there's some other automated process that then takes the files?) You can specify the output file exactly with Bpipe, it's just not something you generally want to do. There's nothing wrong with either one - right tool for the job always wins! > if you use the same code in multiple steps, things can become quite confusing. How will BPipe name them It just keeps appending the identifiers: run { fix_names + fix_names + fix_names } will produce input.fix_names.fix_names.fix_names.csv. So there's no problem with file names stepping on each other, and it'll even be clear from the name that the file got processed 3 times. One problem is you do end up with huge file names - by the time it gets though 10 stages it's not uncommon to have gigantic 200 character file names. But after getting used to that I actually like the explicitness of it. > Imagine a step which takes 3 inputs - one separate, one which is output #2 of a previous step, and one which is output #6 of yet another step Absolutely - you can get situations like this. We're sort of into the 20% of cases that need more advanced syntax (eventually we'll explore all of Bpipes's functions this way :-) ). But basically Bpipe gives you a query language that lets you "glob" the results of the pipeline output tree (not the files in the directory) to find input files. So to get files from specific stages you could write: from(".xls", ".fix_names.csv", ".extract_evergreens.csv") { exec "combine_stuff.py $input.xls $input1.csv $input2.csv" } It doesn't solve everything, but I guess the idea is, make it work right for the majority of cases ("sensible defaults") and then offer ways to deal with harder cases ("make simple things easy, hard things possible"). And when you really get in trouble it's actually groovy code so you can write any programmatic logic you like to find and figure out the inputs if you really need to. > Instead of naming hundreds of files, you have to name hundreds of methods (commands) Not at all - if my pipeline has 15 stages then I have 15 commands to name. Those 15 stages might easily create hundreds of outputs though. > The major difference is that we think you need to identify inputs and outputs to build the graph, and the method name is insignificant until you want code re-use, and BPipe seems to take the opposite position - that you need to give method names, and then use a separate expression to build the graph Again, a really insightful comment, but I'd take it further (and this goes back to my very first comment). Bpipe isn't just not trying to build a graph up front, it really doesn't think there is a graph at all! At least, not an interesting one. The "graph" is a runtime product of the pipeline's execution. We don't actually know the graph until the pipeline finished. An individual pipeline stage can use if / then logic at runtime to decide whether to use a certain input or a different input and that will change the dependency graph. You have to go back and ask why you care about having the graph up front in the first place, and in fact it turns out you can get nearly everything you want without it. By not having the graph you lose some ability to do static analysis on the pipeline, but to have it you are giving up dynamic flexibility. So that's a tradeoff Bpipe makes (and there are downsides, it's just in the context where Bpipe shines the tradeoff is worth it). > In the example you provided you identify different outputs by adding a number to their names. Is that how subsequent steps are supposed to refer to them as inputs - by the positional output number from the step that used to generate them I think the "from" example above probably illustrates it. The simplest method is positional, but it doesn't have to be, you can filter with glob style matching to get inputs as well so if you need to pick out one then you just do so. > 1) As far as different philosophies go, I find BPipe's one to be a bit problematic for complicated cases. I can't argue with that - but that's sort of the idea: simple things easy, hard things possible. Complicated cases are complicated with every tool. I guess I would say that pipeline tools live at a level of abstraction where they aren't meant to get that complicated. > 2) And for simple cases, it all comes down to syntactic sugar. I guess I'd have to disagree with this, as I really think there are some fundamental differences in approach that go well beyond syntactic sugar. > Give me an example of a BPipe workflow that you particularly like, and I'll put it in Drake I wouldn't mind doing that - I'll need to look around and find an example I can share that would make sense (what I do is very domain specific - unless you have familiarity with bioinformatics it will probably be very hard to understand). I'll pm you when I manage to do this, but it may take me a little while (apologies). Thanks as always for the interesting discussion. I think this is a fascinating space, not least because there have been so many attempts at it - I would say there are probably dozens of tools like this going back over 20 years or so - and it seems like nobody has ever nailed it. Bpipe has problems, but so does every tool I've ever tried (I'm probably up to my 8th one or so now!).