8 ms·

The killer feature for me with this app is the stupidly easy CSV loading. I deal with huge CSVs at work often. SQLite Browser is the easiest, fastest and most l

The killer feature for me with this app is the stupidly easy CSV loading. I deal with huge CSVs at work often. SQLite Browser is the easiest, fastest and most lightweight UI tool to analyze them I've tried. It can load millions of records into a table in like <3 seconds. Then obviously you can query the CSV data via SQL.

erremerre 2y ago
I tried the non-installer exe version (32 &64bits) because that csv loading sounded very interesting, but cant make it work without installation because sqlite3.dll, qt5network.dll, qt5xml.dll nor qt5printsupport.dll can't be found. I don't see the point of providing a standalone exe version that does not work? Well, back to excel for csv :/
- justinclift 2y ago
  > I don't see the point of providing a standalone exe version that does not work? Which download did you use? For windows there's really only an .msi which installs everything, and a .zip which needs to be unzipped and have the executable file run from inside the uncompressed directory. The files it's complaining about missing should next to the .exe inside the unpacked directory. If you move the .exe elsewhere, or just extract it by itself, it's not going to work.
- Stagnant 2y ago
  All of those DLLs are included in the same zip file as the portable exe. Sounds like you may be trying to run it without extracting first.
  - erremerre 2y ago
    I can see them but somehow it does not find it being next to it. Don't know if it is a windows policy I am not aware.
    - justinclift 2y ago
      How are you unzipping the .zip before running the .exe? It sort of sounds like you're double clicking on the zip? I'm not personally sure if that unzips the whole folder in the right way or not. If that's what you're doing, maybe right click on the .zip and extract everything using that first? Note that I don't personally use windows much any more, so my windows knowledge is getting rusty for some stuff. ;)
Gormo 2y ago
You might want to give VisiData a look: https://visidata.org https://visidata.org It's a TUI program, so if you want a full-featured GUI, it might not be up your alley, but it's the best tool I've ever used for doing ad-hoc work with large CSVs.
justinclift 2y ago
One of the senior devs (Martin) spent a loooot of time optimising the CSV import process. Many different approaches using sample data sets (from rough memory) of between 800MB and a few GB I think.
tracker1 2y ago
I think this feature should be included in more structured database tools. Being able to load a CSV and query against it, either to an in-memory sqlite or a temp directory (depending on size) is a great option for this kind of work.
wvh 2y ago
I've noticed throughout the years SQLite has lots of handy features built in for dealing with CSV and other input and output formats. Sometimes it's easier to get data into it and use SQL to slice and dice that data than it is to pipe a bunch of unix shell commands together that really only understand unstructured text. Discoverability and subsequently trying to remember all shell incantations is a bit of an issue though, so many people don't see SQLite as the Swiss army knife it can be.
mrtimo 2y ago
Thanks for letting me know about this! I do the same thing... DuckDB loads CSVs very quick and easy. Malloy uses duckDB and can work in VScode or vscode.dev or the vscode built into github. So, you can analyze large .csv files in any of those places. For example, go here [1]. Press "." (this loads vscode in browser) and you will be analyzing large .csv or .parquet files directly in your browser, after you click on the .malloynb file Something I made to analyze every form submitted to the Securities and Exchange Commission (SEC) (10-K etc.) since 2014. [2] Go there, press ".", install the malloy extension, and click on the .malloynb file! Do it all within your browser, or download it and use it in VScode. [1] https://github.dev/malloydata/quickstart/blob/main/README https://github.dev/malloydata/quickstart/blob/main/README [2] https://github.dev/mrtimo/isa5 https://github.dev/mrtimo/isa5
- 392 2y ago
  How do you feel about Malloy vs PRQL?
  - mrtimo 2y ago
    I think I would miss the the semantic data modeling of malloy, and Malloy is definitely easier to read and write IMHO, in part because of the semantic data modeling. I would also miss the nest feature of Malloy, and the way it handles time / dates / timestamps.

erremerre 2y ago

I tried the non-installer exe version (32 &64bits) because that csv loading sounded very interesting, but cant make it work without installation because sqlite3.dll, qt5network.dll, qt5xml.dll nor qt5printsupport.dll can't be found. I don't see the point of providing a standalone exe version that does not work? Well, back to excel for csv :/

justinclift 2y ago
> I don't see the point of providing a standalone exe version that does not work? Which download did you use? For windows there's really only an .msi which installs everything, and a .zip which needs to be unzipped and have the executable file run from inside the uncompressed directory. The files it's complaining about missing should next to the .exe inside the unpacked directory. If you move the .exe elsewhere, or just extract it by itself, it's not going to work.
Stagnant 2y ago
All of those DLLs are included in the same zip file as the portable exe. Sounds like you may be trying to run it without extracting first.
- erremerre 2y ago
  I can see them but somehow it does not find it being next to it. Don't know if it is a windows policy I am not aware.
  - justinclift 2y ago
    How are you unzipping the .zip before running the .exe? It sort of sounds like you're double clicking on the zip? I'm not personally sure if that unzips the whole folder in the right way or not. If that's what you're doing, maybe right click on the .zip and extract everything using that first? Note that I don't personally use windows much any more, so my windows knowledge is getting rusty for some stuff. ;)

Gormo 2y ago

You might want to give VisiData a look: https://visidata.org https://visidata.org It's a TUI program, so if you want a full-featured GUI, it might not be up your alley, but it's the best tool I've ever used for doing ad-hoc work with large CSVs.

justinclift 2y ago

One of the senior devs (Martin) spent a loooot of time optimising the CSV import process. Many different approaches using sample data sets (from rough memory) of between 800MB and a few GB I think.

tracker1 2y ago

I think this feature should be included in more structured database tools. Being able to load a CSV and query against it, either to an in-memory sqlite or a temp directory (depending on size) is a great option for this kind of work.

wvh 2y ago

I've noticed throughout the years SQLite has lots of handy features built in for dealing with CSV and other input and output formats. Sometimes it's easier to get data into it and use SQL to slice and dice that data than it is to pipe a bunch of unix shell commands together that really only understand unstructured text. Discoverability and subsequently trying to remember all shell incantations is a bit of an issue though, so many people don't see SQLite as the Swiss army knife it can be.

mrtimo 2y ago

Thanks for letting me know about this! I do the same thing... DuckDB loads CSVs very quick and easy. Malloy uses duckDB and can work in VScode or vscode.dev or the vscode built into github. So, you can analyze large .csv files in any of those places. For example, go here [1]. Press "." (this loads vscode in browser) and you will be analyzing large .csv or .parquet files directly in your browser, after you click on the .malloynb file Something I made to analyze every form submitted to the Securities and Exchange Commission (SEC) (10-K etc.) since 2014. [2] Go there, press ".", install the malloy extension, and click on the .malloynb file! Do it all within your browser, or download it and use it in VScode. [1] https://github.dev/malloydata/quickstart/blob/main/README https://github.dev/malloydata/quickstart/blob/main/README [2] https://github.dev/mrtimo/isa5 https://github.dev/mrtimo/isa5

392 2y ago
How do you feel about Malloy vs PRQL?
- mrtimo 2y ago
  I think I would miss the the semantic data modeling of malloy, and Malloy is definitely easier to read and write IMHO, in part because of the semantic data modeling. I would also miss the nest feature of Malloy, and the way it handles time / dates / timestamps.