noamteyssier

8 ms·

1.
by noamteyssier 11mo ago
Good question! I just added that comparison and the rust uutils coreutils port is significantly faster than the standard coreutils.
2.
by noamteyssier 11mo ago
I think this could potentially really reduce the amount of memory required - especially in cases where there is a lot of repetitive prefixes. Would be interesting to try this out
3.
by noamteyssier 11mo ago
I think you'd still need to go through that if you were really optimizing both `sort` and `uniq` working with their constraints. What I'm really optimizing here is the functional equivalent of `sort | uniq -c | sort -n`
4.
by noamteyssier 11mo ago
yeah that's right - there are trade-offs in doing so as it can require much more memory. So like everything it's an application specific decision
5.
by noamteyssier 11mo ago
I've added this functionality to `hist-0.1.5` with a benchmark of other tools that do this on the CLI
6.
by noamteyssier 11mo ago
this looks very interesting and I'd love to add it to the benchmarking! I was interested in trying it but unfortunately got an installation error on my macbook where I'm running the benchmarks: ``` clang \ -g -ggdb
7.
by noamteyssier 11mo ago
I've actually added a benchmark for this specific task and added `unic` to it. It may not be the most fair comparison because with these random fastqs I'm generating the vast majority of the input is unique so it could be overload
8.
by noamteyssier 11mo ago
Shows up a lot in bioinformatics actually - trying to identify sequences with a specific subsequence (grep) and how many of each unique sequence there are. The number of lines here could be massive (order of 1-10's of GB) You don'
9.
by noamteyssier 11mo ago
I've added awk into the benchmarks also!
10.
by noamteyssier 11mo ago
Yeah I'm using it to serialize the output lines as a TSV. Rust's `println!` is notoriously slow and using `csv` to serialize the output is a nice way to boost throughput
11.
by noamteyssier 11mo ago
Yes you're right, it's not trying to do what `sort` is doing, it's trying to reproduce the output of `sort | uniq -c | sort -n` which is a more specialized but common task. But you're right - it will be limited by RAM in
12.
by noamteyssier 11mo ago
Thanks for sharing! You're right that the `cat` is unnecessary - and removing it actually had some marginal gains to the naive solution. I've updated the benchmarks to show this Cheers
13.
by noamteyssier 11mo ago
Yeah definitely, it's always a trade-off. I think in many cases where I use it especially the number of unique values is actually not crazy high (much less than the required RAM) and the number of lines is crazy high. So in those set
14.
by noamteyssier 11mo ago
I didn't know that - I've added in buffer size with a fairly large buffer to the benchmarks as well
15.
by noamteyssier 11mo ago
Cool suggestions! I definitely would be interested in exploring other hash functions for this (and other binf works) so I'll definitely take a look at your stringzilla lib.
16.
by noamteyssier 11mo ago
Totally agree it's a bit of weird benchmark - it was just the first thing that I thought of to generate a huge amount of lines to test throughput. There are definitely other benchmarks that we could try as well to test other characteri
17.
by noamteyssier 11mo ago
Nice tool!
18.
by noamteyssier 11mo ago
I think that it's a pretty common use case for text processing - I end up needing to use it a lot in bioinformatics where there is a lot of text processing. It's great when you quickly need to see what the distribution of classes
19.
Show HN: hist: An overengineered solution to `sort|uniq -c` with 25x throughput(github.com)
4 points by noamteyssier 11mo ago|3 comments
20.
An overengineered solution to `sort | uniq -c` with 25x throughput (hist)(github.com)
118 points by noamteyssier 11mo ago|109 comments