8 ms·
One nice thing about CSV files being zipped and served via the web is they can be streamed directly into the database incredibly fast without having to persist
by wefarrell 3y ago
One nice thing about CSV files being zipped and served via the web is they can be streamed directly into the database incredibly fast without having to persist them anywhere (aside from the db).
You can load the zip file as a stream, read the CSV line by line, transform it, and then load it to the db using COPY FROM stdin (assuming Postgres).
- thibaut_barrere 3y agoDefinitely, it is much easier to stream CSV than say JSON or XML (even if JSONL/Sax parsers exist etc).
- eastbound 3y agoI you feel risky, try a Foreign Data Wrapper ;)
- heavenlyblue 3y agoThat doesn't sound like an amazingly safe idea
- dambi0 3y agoWhat specific risks do you foresee with this approach?
- diroussel 3y agoSeem totally fine to me. As long as you can rollback if the download is truncated or the crc checksum doesn’t match.
- chii 3y ago> or the crc checksum doesn’t match. which wouldn't exist if the api is simply just a single CSV file? at least with a zip, the CRC exists (an incomplete zip file is detectable, an incomplete, but syntactically correct CSV file is not)
- NL807 3y agoDROP DATABASE blah;
- aidos 3y agoThat’s not how COPY FROM works in postgres. You give it a csv and a table matching the structure and it hammers the data into the table faster than anything else can.
- berkes 3y agoIt isn't. But that's easily mitigated with temp tables, ephemeral database and COPY etc. Upstream can easily f-up and (accidentally) delete production data if you do this on a live db. Which is why PostgreSQL and nearly all other DBS have a miriad of tools to solve this by not doing it directly on a production database
- wefarrell 3y agoMaybe I'm missing something but I don't see how it's possible for a COPY statement alone to remove existing data.
- LgWoodenBadger 3y agoIf in the regular scenario you load 10000 rows of new data and delete the old then it’s fine. What if someone screws up the zip and instead of 10000 today, it’s only 10?
- aidos 3y agoI had this last week, but instead it was a 3rd party api and their service started returning null instead of true for the has_more property beyond the second page of results. In either the solution is probably to check rough counts and error if not reasonable.
- camgunz 3y agoI think generally don't replace the prod db until the new one passes tests.