ssinchenko

4 ms·

1.
by ssinchenko 21d ago
User friendly tooling (cargo, IDE integration), user friendly error messages.
2.
by ssinchenko 2mo ago
> All of this would have easily fit in memory on any reasonable modern system. Understand me correctly. This is my research project and I only have a laptop, not a server with 256 GB of RAM. I tested my project on a 2B graph with a hard
3.
by ssinchenko 2mo ago
Yes, it is both out-of-core and multi-processor. I created this toy project to process graphs that cannot fit into memory (in CSR format) using all available cores. The multi-processing relies on DataFusion's Tokyo workers. The out-of-
4.
by ssinchenko 2mo ago
Thank you!
5.
by ssinchenko 2mo ago
I agree 100%! DataFusion is beautiful and easy to extend in any direction. For the second version of my "out-of-core" graph algorithms project, for example, I implemented my own "co-partitioning" to speed up joins and ac
6.
by ssinchenko 2mo ago
Agreed, but I didn't see anything like this in popular projects such as Networkit or IGraph. If you have an example of an implementation, I would appreciate it!
7.
by ssinchenko 2mo ago
The previous issues were in my mind, not in DataFusion. I tried using DataFusion as an in-memory tool, which was a mistake. If the graph fits in memory, Networkit, IGraph, etc. will almost always be faster. These tools cannot process anythi
8.
by ssinchenko 2mo ago
Yes, my toy tool is similar by the concept to graphchi. But I did not write the vertex-centric processing from scratch and I'm relying on DataFusion built-ins (select, join, group by, aggregate)