7 ms·
> For most of the data the sizes we often deal with it's perfect. Interested here: for me it works for out of core work. Where is the limit? On a related note:
by goerch 5mo ago
> For most of the data the sizes we often deal with it's perfect.
Interested here: for me it works for out of core work. Where is the limit? On a related note: do you need to handle concurrency restrictions?
- gigatexal 5mo agoi must be doing something wrong but if i try a huge join on a table bigger than my ram no matter the flags or the spill-to-disk modes enabled i get crashes. im sure im doing something wrong.
- goerch 5mo agoHm, only anecdotal evidence, but page rank computation for Wikipedia works on my laptop (https://github.com/idesis-gmbh/WikiExperiments https://github.com/idesis-gmbh/WikiExperiments) where `NetworkX` fails. And it uses some joins like here: https://github.com/idesis-gmbh/WikiExperiments/blob/0b108f3fe7693881ed015764efaf75b3122b92db/pr.py#L75-L90 https://github.com/idesis-gmbh/WikiExperiments/blob/0b108f3f...
- infinet 5mo agoFrom my very limited experience with duckdb, being a column based database, simple query also need loads entire columns and that uses a lots of RAM if a table has millions of rows. Perhaps there is a way to make it more memory friendly but I don't know how.