6 ms·

Hosting SQLite Databases on GitHub Pages (2021)

86 points by WA9ACE 11mo ago

pspeter3 11mo ago
I wonder if the author would use DuckDB WASM now?
- ngc6677 11mo ago
  is it doing HTTP range request, to not download an entire db?
- mrtimo 11mo ago
  I'm using DuckDB WASM on github pages. This will take about 10 seconds to load [1] and shows business trends in my county (Spokane County). This site is built using data-explorer [2] which uses many other open-source projects including malloy and malloy-explorer. One cool thing... if you use the UI to make a query on the data - you can share the URL with someone and they will see the same result / query (it's all embedded in the URL). [1] - https://mrtimo.github.io/spokane-co-biz/#/model/businesses/explorer/businesses?query=run:%20businesses-%3Ebig_dashboard&run=true https://mrtimo.github.io/spokane-co-biz/#/model/businesses/e... [2] - https://github.com/aszenz/data-explorer https://github.com/aszenz/data-explorer
- westurner 11mo ago
  From "Show HN: TeaTime – distributed book library powered by SQLite, IPFS and GitHub" https://news.ycombinator.com/item?id=42264274 https://news.ycombinator.com/item?id=42264274 : >> phiresky/sql.js-httpvfs: https://github.com/phiresky/sql.js-httpvfs https://github.com/phiresky/sql.js-httpvfs >> mmomtchev/sqlite-wasm-http: https://github.com/mmomtchev/sqlite-wasm-http https://github.com/mmomtchev/sqlite-wasm-http >> This project is inspired from @phiresky/sql.js-httpvfs but uses the new official SQLite WASM distribution duckdb/duckdb-wasm: https://github.com/duckdb/duckdb-wasm https://github.com/duckdb/duckdb-wasm
  - westurner 11mo ago
    "PSA: SQLite WAL checksums fail silently and may lose data" so that's probably not how to sync sqlite; https://news.ycombinator.com/item?id=44672902 https://news.ycombinator.com/item?id=44672902 electric-sql/electric: https://github.com/electric-sql/electric https://github.com/electric-sql/electric : > Specifically, Electric is a read-path sync engine for Postgres. It syncs data out of Postgres into ... anything you like. The core sync protocol is based on a low-level HTTP API. This integrates with CDNs for highly-scalable data delivery. electric-sql/pglite: https://github.com/electric-sql/pglite https://github.com/electric-sql/pglite : > Embeddable Postgres with real-time, reactive bindings. "Using Postgres for Everything" https://news.ycombinator.com/item?id=40420474 https://news.ycombinator.com/item?id=40420474
crtasm 11mo ago
(2021) >(or IPFS or any static file hoster)
deleted 11mo ago
[deleted]
misthop 11mo ago
It would be a it more compelling if every example, at least for me, didn't return `[error: NetworkError: A network error occurred.]`
- cwmma 11mo ago
  I'm assuming your using firefox, if so it's this bug[1] that basically prevents range requests from working. Basically firefox says it will accept gziped data even though it's a range request and github pages dudifly sends back an unreadable slice of a gziped file. 1. https://bugzilla.mozilla.org/show_bug.cgi?id=1874840 https://bugzilla.mozilla.org/show_bug.cgi?id=1874840
  - smithcoin 11mo ago
    It doesn't work for me on safari either.
    - cwmma 11mo ago
      works fine on safari desktop for me
  - Groxx 11mo ago
    huh. is this due to ambiguity in whether you want gzipped content vs gzipped transport (of arbitrary content), and/or which range the bytes are requesting? I can see both being useful, but I don't know what headers are available for these intentions...
    - cwmma 11mo ago
      my understanding is that technically only gzipped content is supported, not gzipped transport of arbitrary content. Due to ambiguity around the word 'append' [1] in the spec, firefox adds 'identity' (aka don't compress) to the end of the list of compressions supported while most other browsers replace the list with 'identity'. Also it should be noted that this is not a user configurable header so you can't actually try to override it. There is a second layer to the bug in that github pages should almost certainly not be sending back slices of compressed files even if gzip is listed before identity and some change to something in the github stack probably exposed the bug that was there in firefox all along. I literally just stumbled on this last week while doing a side project that involved browser range requests so this is fresh in my head. 1. it comes down to whether it was meant that 'Accept-Encoding:identity' should be appended to the list of header values possibly overwriting the one that already was there or if 'identity' should be appended to the list of values already in the 'Accept-Encoding' header. Firefox does the latter, everyone else does the former.
zX41ZdbW 11mo ago
It's old news, but you can even host ClickHouse databases on GitHub pages: https://github.com/ClickHouse/web-tables-demo https://github.com/ClickHouse/web-tables-demo
bomewish 11mo ago
Where does the 600mb SQLite file live though? GitHub won’t host that. Lfs server?
amighol 11mo ago
Is it just me or It does not run on Firefox. I ran on edge it works. In Firefox I get: [error: NetworkError: A network error occurred.]

86 points by WA9ACE 11mo ago

pspeter3 11mo ago

I wonder if the author would use DuckDB WASM now?

ngc6677 11mo ago
is it doing HTTP range request, to not download an entire db?
mrtimo 11mo ago
I'm using DuckDB WASM on github pages. This will take about 10 seconds to load [1] and shows business trends in my county (Spokane County). This site is built using data-explorer [2] which uses many other open-source projects including malloy and malloy-explorer. One cool thing... if you use the UI to make a query on the data - you can share the URL with someone and they will see the same result / query (it's all embedded in the URL). [1] - https://mrtimo.github.io/spokane-co-biz/#/model/businesses/explorer/businesses?query=run:%20businesses-%3Ebig_dashboard&run=true https://mrtimo.github.io/spokane-co-biz/#/model/businesses/e... [2] - https://github.com/aszenz/data-explorer https://github.com/aszenz/data-explorer
westurner 11mo ago
From "Show HN: TeaTime – distributed book library powered by SQLite, IPFS and GitHub" https://news.ycombinator.com/item?id=42264274 https://news.ycombinator.com/item?id=42264274 : >> phiresky/sql.js-httpvfs: https://github.com/phiresky/sql.js-httpvfs https://github.com/phiresky/sql.js-httpvfs >> mmomtchev/sqlite-wasm-http: https://github.com/mmomtchev/sqlite-wasm-http https://github.com/mmomtchev/sqlite-wasm-http >> This project is inspired from @phiresky/sql.js-httpvfs but uses the new official SQLite WASM distribution duckdb/duckdb-wasm: https://github.com/duckdb/duckdb-wasm https://github.com/duckdb/duckdb-wasm
- westurner 11mo ago
  "PSA: SQLite WAL checksums fail silently and may lose data" so that's probably not how to sync sqlite; https://news.ycombinator.com/item?id=44672902 https://news.ycombinator.com/item?id=44672902 electric-sql/electric: https://github.com/electric-sql/electric https://github.com/electric-sql/electric : > Specifically, Electric is a read-path sync engine for Postgres. It syncs data out of Postgres into ... anything you like. The core sync protocol is based on a low-level HTTP API. This integrates with CDNs for highly-scalable data delivery. electric-sql/pglite: https://github.com/electric-sql/pglite https://github.com/electric-sql/pglite : > Embeddable Postgres with real-time, reactive bindings. "Using Postgres for Everything" https://news.ycombinator.com/item?id=40420474 https://news.ycombinator.com/item?id=40420474

crtasm 11mo ago

(2021) >(or IPFS or any static file hoster)

deleted 11mo ago

[deleted]

misthop 11mo ago

It would be a it more compelling if every example, at least for me, didn't return `[error: NetworkError: A network error occurred.]`

cwmma 11mo ago
I'm assuming your using firefox, if so it's this bug[1] that basically prevents range requests from working. Basically firefox says it will accept gziped data even though it's a range request and github pages dudifly sends back an unreadable slice of a gziped file. 1. https://bugzilla.mozilla.org/show_bug.cgi?id=1874840 https://bugzilla.mozilla.org/show_bug.cgi?id=1874840
- smithcoin 11mo ago
  It doesn't work for me on safari either.
  - cwmma 11mo ago
    works fine on safari desktop for me
- Groxx 11mo ago
  huh. is this due to ambiguity in whether you want gzipped content vs gzipped transport (of arbitrary content), and/or which range the bytes are requesting? I can see both being useful, but I don't know what headers are available for these intentions...
  - cwmma 11mo ago
    my understanding is that technically only gzipped content is supported, not gzipped transport of arbitrary content. Due to ambiguity around the word 'append' [1] in the spec, firefox adds 'identity' (aka don't compress) to the end of the list of compressions supported while most other browsers replace the list with 'identity'. Also it should be noted that this is not a user configurable header so you can't actually try to override it. There is a second layer to the bug in that github pages should almost certainly not be sending back slices of compressed files even if gzip is listed before identity and some change to something in the github stack probably exposed the bug that was there in firefox all along. I literally just stumbled on this last week while doing a side project that involved browser range requests so this is fresh in my head. 1. it comes down to whether it was meant that 'Accept-Encoding:identity' should be appended to the list of header values possibly overwriting the one that already was there or if 'identity' should be appended to the list of values already in the 'Accept-Encoding' header. Firefox does the latter, everyone else does the former.

zX41ZdbW 11mo ago

It's old news, but you can even host ClickHouse databases on GitHub pages: https://github.com/ClickHouse/web-tables-demo https://github.com/ClickHouse/web-tables-demo

bomewish 11mo ago

Where does the 600mb SQLite file live though? GitHub won’t host that. Lfs server?

amighol 11mo ago

Is it just me or It does not run on Firefox. I ran on edge it works. In Firefox I get: [error: NetworkError: A network error occurred.]