10 ms·

Parquet-WASM: Rust-based WebAssembly bindings to read and write Parquet data

• indianmilf4627 2y ago
  [flagged]
• SEXMCNIGGA34914 2y ago
  [flagged]
• SEXMCNIGGA37282 2y ago
  [flagged]
• 2genders35107 2y ago
  [flagged]
• 2genders6880 2y ago
  [flagged]
• SEXMCNIGGA29177 2y ago
  [flagged]
• 2genders34139 2y ago
  [flagged]
• SEXMCNIGGA765 2y ago
  [dead]
• SEXMCNIGGA45535 2y ago
  [flagged]
• 2genders42516 2y ago
  [flagged]
• SEXMCNIGGA25023 2y ago
  [dead]
• SEXMCNIGGA47292 2y ago
  [dead]
• SEXMCNIGGA44115 2y ago
  [dead]
• SEXMCNIGGA5453 2y ago
  [dead]
• 2genders6778 2y ago
  [dead]
• 2genders16981 2y ago
  [flagged]
• 2genders11320 2y ago
  [flagged]
• 2genders6679 2y ago
  [flagged]
• 2genders3617 2y ago
  [flagged]
• 2genders2187 2y ago
  [flagged]
• SEXMCNIGGA27937 2y ago
  [dead]
• 2genders12206 2y ago
  [flagged]
• 2genders14968 2y ago
  [flagged]
• SEXMCNIGGA28931 2y ago
  [dead]
• SEXMCNIGGA46822 2y ago
  [dead]
• SEXMCNIGGA46390 2y ago
  [dead]
• sexmc11475 2y ago
  [flagged]
• sexmc20986 2y ago
  [flagged]
• SEXMCNIGGA42494 2y ago
  [flagged]
• SEXMCNIGGA15767 2y ago
  [flagged]
• SEXMCNIGGA22912 2y ago
  [flagged]
• SEXMCNIGGA23932 2y ago
  [flagged]
• 2genders42939 2y ago
  [dead]
• SEXMCNIGGA47730 2y ago
  [flagged]
• SEXMCNIGGA36639 2y ago
  [flagged]
• 2genders5809 2y ago
  [dead]
• SEXMCNIGGA39626 2y ago
  [dead]
• SEXMCNIGGA40090 2y ago
  [flagged]
• 2genders31537 2y ago
  [dead]
• 2genders46870 2y ago
  [dead]
• SEXMCNIGGA29594 2y ago
  [flagged]
• SEXMCNIGGA40025 2y ago
  [flagged]
• jasonjmcghee 2y ago
  Seeing as the popular alternative here would be DuckDB-WASM, which (last time I checked) is on the order of 50MB, this is comparatively super lightweight.
  • leeoniya 2y ago
    i think duckdb-wasm is closer to 6MB over wire, but ~36MB once decompressed. (see net panel when loading https://shell.duckdb.org/ https://shell.duckdb.org/) the decompressed size should be okay since it's not the same as parsing and JITing 36MB of JS.
• leeoniya 2y ago
  in my [albeit outdated] experience ArrowJS is quite a bit slower than using native JS types. i feel like crossing the WASM<>JS boundary is very expensive, especially for anything other than numbers/typed arrays. what are people's experiences with this?
  • ingenieroariel 2y ago
    I'll let Kyle chime in but I tested it a few months ago with millions of polygons on an M2 16GB of RAM laptop and it worked very well. There is a library by the same author called lonboard that provides the JS bits inside JupyterLab. https://github.com/developmentseed/lonboard https://github.com/developmentseed/lonboard <speculation>I think it is based on the Kepler.gl / Deck.gl data loaders that go straight to GPU from network.</speculation>
  • kylebarron 2y ago
    Arrow JS is just ArrayBuffers underneath. You do want to amortize some operations to avoid unnecessary conversions. I.e. Arrow JS stores strings as UTF-8, but native JS strings are UTF-16 I believe. Arrow is especially powerful across the WASM <--> JS boundary! In fact, I wrote a library to interpret Arrow from Wasm memory into JS without any copies [0]. (Motivating blog post [1]) [0]: https://github.com/kylebarron/arrow-js-ffi https://github.com/kylebarron/arrow-js-ffi [1]: https://observablehq.com/@kylebarron/zero-copy-apache-arrow-with-webassembly https://observablehq.com/@kylebarron/zero-copy-apache-arrow-...
    • lmeyerov 2y ago
      Yeah, we built it to essentially stream columnar record batches from server GPUs to browser GPUs with minimal touching of any of the array buffers. It was very happy-path for that kind of fast bulk columnar processing, and we donated it to the community to grow to use cases beyond that. So it sounds like the client code may have been doing more than that. For high performance code, I'd have expected overhead in %s, not Xs. And not surprised to hear slowdowns for any straying beyond that -- cool to see folks have expanded further! More recently, we've been having good experiences more recently here in Perspective <-arrow-> Loaders, enough so that we haven't had to dig deeper. Our current code is targeting < 24 FPS, as genAI data analytics is more about bigger volumes than velocity, so unsure. However, it's hard to imagine going much faster though given it's bulk typed arrays without copying, especially on real code.
• SEXMCNIGGA7013 2y ago
  [dead]
• SEXMCNIGGA36519 2y ago
  [flagged]
• SEXMCNIGGA45969 2y ago
  [dead]
• SEXMCNIGGA31909 2y ago
  [dead]
• rubenvanwyk 2y ago
  Can this read and write Parquet files to S3-compatible storage?
  • kylebarron 2y ago
    It can read from HTTP urls, but you'd need to manage signing the URLs yourself. On the writing side, it currently writes to an ArrayBuffer, which then you could upload to a server or save on the user's machine.
• SEXMCNIGGA15755 2y ago
  [dead]
• SEXMCNIGGA5390 2y ago
  [flagged]
• SEXMCNIGGA7388 2y ago
  [flagged]
• SEXMCNIGGA47717 2y ago
  [flagged]
• SEXMCNIGGA27783 2y ago
  [flagged]
• SEXMCNIGGA26056 2y ago
  [flagged]
• SEXMCNIGGA19483 2y ago
  [flagged]
• SEXMCNIGGA10337 2y ago
  [flagged]
• SEXMCNIGGA5854 2y ago
  [flagged]
• SEXMCNIGGA17936 2y ago
  [flagged]
• SEXMCNIGGA2110 2y ago
  [flagged]
• SEXMCNIGGA9736 2y ago
  [flagged]
• SEXMCNIGGA45284 2y ago
  [flagged]
• SEXMCNIGGA30999 2y ago
  [flagged]
• SEXMCNIGGA9093 2y ago
  [flagged]
• SEXMCNIGGA39559 2y ago
  [flagged]
• SEXMCNIGGA43488 2y ago
  [dead]
• SEXMCNIGGA27511 2y ago
  [dead]
• SEXMCNIGGA21924 2y ago
  [dead]
• SEXMCNIGGA10073 2y ago
  [dead]
• SEXMCNIGGA8568 2y ago
  [dead]
• SEXMCNIGGA37272 2y ago
  [dead]
• SEXMCNIGGA13676 2y ago
  [dead]
• sexmc11449 2y ago
  [flagged]
• SEXMCNIGGA18814 2y ago
  [dead]
• SEXMCNIGGA43334 2y ago
  [flagged]
• SEXMCNIGGA33412 2y ago
  [dead]
• SEXMCNIGGA20702 2y ago
  [dead]
• SEXMCNIGGA9693 2y ago
  [dead]
• SEXMCNIGGA3348 2y ago
  [dead]
• SEXMCNIGGA36880 2y ago
  [dead]
• SEXMCNIGGA16339 2y ago
  [dead]
• SEXMCNIGGA45159 2y ago
  [dead]
• SEXMCNIGGA49908 2y ago
  [dead]
• SEXMCNIGGA490 2y ago
  [dead]
• SEXMCNIGGA34085 2y ago
  [dead]
• SEXMCNIGGA45407 2y ago
  [flagged]
• SEXMCNIGGA21569 2y ago
  [dead]
• SEXMCNIGGA14603 2y ago
  [flagged]
• SEXMCNIGGA42635 2y ago
  [flagged]
• 2genders33983 2y ago
  [dead]
• SEXMCNIGGA12976 2y ago
  [dead]
• SEXMCNIGGA42279 2y ago
  [dead]
• SEXMCNIGGA17459 2y ago
  [dead]
• SEXMCNIGGA15802 2y ago
  [dead]
• SEXMCNIGGA25190 2y ago
  [dead]
• SEXMCNIGGA34796 2y ago
  [dead]
• SEXMCNIGGA11106 2y ago
  [dead]
• 2genders5983 2y ago
  [dead]
• 2genders43977 2y ago
  [dead]
• 2genders15453 2y ago
  [dead]
• 2genders35195 2y ago
  [dead]
• 2genders21072 2y ago
  [dead]
• 2genders28355 2y ago
  [dead]
• 2genders16259 2y ago
  [dead]
• 2genders44428 2y ago
  [flagged]
• 2genders14547 2y ago
  [dead]
• 2genders1767 2y ago
  [flagged]
• 2genders33247 2y ago
  [flagged]
• 2genders24636 2y ago
  [flagged]
• 2genders34697 2y ago
  [flagged]
• 2genders40102 2y ago
  [flagged]
• 2genders42255 2y ago
  [dead]
• 2genders33741 2y ago
  [flagged]
• 2genders611 2y ago
  [flagged]
• SEXMCNIGGA43154 2y ago
  [dead]
• SEXMCNIGGA8890 2y ago
  [flagged]
• SEXMCNIGGA21724 2y ago
  [dead]
• SEXMCNIGGA26734 2y ago
  [dead]
• indianmilf47317 2y ago
  [flagged]
• indianmilf49931 2y ago
  [dead]
• indianmilf4975 2y ago
  [flagged]
• sexmc31283 2y ago
  [flagged]
• sexmc41620 2y ago
  [flagged]
• FridgeSeal 2y ago
  @dang we have a mass spam incursion in this comment thread.
  • deleted 2y ago
    [deleted]
  • seanw444 2y ago
    It's site-wide.
• m_d_ 2y ago
  I'd like to point out that fastparquet has been built for wasm (pydide/pyscript) for some time and works fine, producing pandas dataframes. Unfortunately, the thread/socket/async nature of fsspec means you have to get the files yourself into the "local filesystem" (meaning: the wasm sandbox). (I am the fastparquet author)
• nickfs 2y ago
  ;8y aiu;khjbvnvxzg;o9