6 ms·

Dropbox-as-a-Database

84 points by cedrics 14y ago

sehugg 14y ago
Reminds me of the GMail filesystem: http://www.viksoe.dk/code/gmail.htm http://www.viksoe.dk/code/gmail.htm
- bduerst 14y ago
  What a relic! website copyright 2001
- trevorcreech 14y ago
  Ha, I remember using this for backups before I had even heard of ftp.
MatthewPhillips 14y ago
I've used .json files in place of a database before on a read-only site. It worked very well. I do believe that we over-engineer some times and use databases when flat files will do just fine.
- codewright 14y ago
  Using sqlite as the "this is just an idea" phase positions you better for scale-out/build-up if it becomes necessary. Helps keep your head straight but doesn't slow you down. I say this as someone who worships Crockford for inventing JSON. I've seen too many projects get fucked hard because they used flatfiles. Edit: Seriously though, managing synchronization/atomic changes to flatfiles manually is a Bad Idea (TM). It's somewhere up there with, "I have a performance problem. I'll use pthreads!" Now you have 1,02-SEGMENTATION FAULT problems.
  - MatthewPhillips 14y ago
    Like I said, in my case there were no changes, I only read from them. So I let the server cache the hell out of them and the entire site was fast because of it.
- IsaacL 14y ago
  You're not alone. Guess which famous tech website uses flat files as a database? Hint: orange header, heirarchical comments...
  - habosa 14y ago
    Is this true? Is there some post about this?
gdp 14y ago
It seems rather generous to call this a database. It's files on a filesystem. The lack of any kind of transactional consistency guarantees (Dropbox is notoriously awful at this), and the lack of any sane mechanism for querying this "database" means that it's just "data", not a "database".
- yefrederic 14y ago
  I think he calls it "database" because it is strongly integrated with the database syntax of the Opa language. You can declare and use this "dropbox-database" just like a Mongo one, independently of the notion of safe-ness of the database.
  - gdp 14y ago
    Consistent with the complaint leveled against my previous comment, I should think that this is really consistent with the DBMS syntax of the Opa language</pedantry>
- Supreme 14y ago
  In the world of SaaS, social media, NoSQL and NodeJS, everything which has already been thought of and brought to maturity is suddenly new, shiny, incomplete, buggy and has a new name. It's a brave new world - enjoy your stay.
- mistercow 14y ago
  Transactions are not a requirement for databases, and querying is the domain of the DBMS, not the database itself.
  - gdp 14y ago
    You're quite right. I think my complaint was that it reduces the definition of "database" to "anything that is used to store data", rather than the database being the json file(s) themselves, at which point we have degenerated to about the same terminological point where it would be reasonable to call saving CSV or MS-Access files into a Dropbox folder "Dropbox-as-a-Database", which seems an unhelpful label, if nothing else.
tarr11 14y ago
It is a fun exercise to do these kinds of things. Obviously, they are very inefficient. But that is not the point! Perhaps there's some residual value related to some Dropbox feature (such as automatic replication) that makes this useful . But that is countered by getting your account rate-limited by Dropbox. I wonder, is there some way to create a key-value store out of any physical or virtual object that can store state? I know there is a lot of work done with quantum computing to store state in quarks. I can see that has value. But, what is the most inefficient data storage mechanism you can come up with? Here are a few I can think of: Instagram-as-a-database - You issue a request to save your data, we encode it using some visual encoding scheme, take a picture, and then upload it somewhere. You get returned a URL, and can then decode it or update it. MTurk-as-a-database - The possibilities are endless here. Someone could write down your request on a piece of paper and then you could send another request where they type it back. Twilio-as-a-database - Encode your data as an audio stream. Store it in random voicemail boxes. Retrieve it later by calling up, listening to your voicemail, decoding the messages.
- Jabbles 14y ago
  quantum computing to store state in quarks Atoms or electrons, but not quarks.
- Erwin 14y ago
  Humanity as a database: after infiltrating a religious order mandate arranged marriages encoding your input data in predicted genetic mutations of offspring. Ensure the religious order is hermetic so no outside data appears and that its teaching require all members to appear at a certain location at a certain time. Retrieve your data 1000 years later by appearing at the preordained spot for the Holy Query Ceremony, and examining the eye/hair colours of those gathered. Good error correction protocols are required. (This might well already be part of a Neal Stephenson novel).
- plaguuuuuu 14y ago
  A Minecraft game would make a hilarious database. The fact that it's probably pretty trivial to CRUD it programmatically outside the Minecraft client by manipulating save files is a great bonus. Start using the DB. If you get bored enough you can examine the data physically in-game.
mattmanser 14y ago
Your blog design is bad btw, popout at side overlays scroll bar in Windows/Chrome.
- yefrederic 14y ago
  It's one of the design proposed by Google, blame them :)
  - bduerst 14y ago
    Google isn't the one using it to demonstrate their work as a developer ;-)
- ehutch79 14y ago
  Plus, i can barely read the text.
formatjam 14y ago
Actually I have the exact same idea like a few weeks ago. The potential for this idea is big. It can be widely used by small web apps (Chrome App). Instead of hosting a server and a database, every thing can be done and stored in the client, isn't that neat?
- yefrederic 14y ago
  You should have a look at filepicker.io then
- ananddass 14y ago
  as yefrederic pointed out you could use filepicker.io to build static js applications. We handle all the server side logic for you.
taskstrike 14y ago
Dropbox as a database is a actually great idea. It takes away scaling problems from the developer and puts the data control back to the user. I'm actually working on NimbusBase, a product that allows you to write clientside javascript applications where there is no server code and data is stored inside of Dropbox. It also integrates with backbone. Shameless plug: sign up for our beta! http://nimbusbase.com http://nimbusbase.com
pencilcode 14y ago
is this part of the app open-sourced? it's a great idea.
- yefrederic 14y ago
  You can find the source code here : https://github.com/cedricss/server-monitor/ https://github.com/cedricss/server-monitor/
  - hbbio 14y ago
    The list of commits is interesting: https://github.com/cedricss/server-monitor/commits/master https://github.com/cedricss/server-monitor/commits/master And also shows its really a one-day app!
- cedrics 14y ago
  http://server-monitor.herokuapp.com/ http://server-monitor.herokuapp.com/ is open-source (MIT), and use the Opa framework (also open-source, MIT). The Dropbox-as-a-Database is integrated in Opa by default.
shykes 14y ago
"update: Heroku is down [...] the demo in now on dotCloud." :)

84 points by cedrics 14y ago

sehugg 14y ago

Reminds me of the GMail filesystem: http://www.viksoe.dk/code/gmail.htm http://www.viksoe.dk/code/gmail.htm

bduerst 14y ago
What a relic! website copyright 2001
trevorcreech 14y ago
Ha, I remember using this for backups before I had even heard of ftp.

MatthewPhillips 14y ago

I've used .json files in place of a database before on a read-only site. It worked very well. I do believe that we over-engineer some times and use databases when flat files will do just fine.

codewright 14y ago
Using sqlite as the "this is just an idea" phase positions you better for scale-out/build-up if it becomes necessary. Helps keep your head straight but doesn't slow you down. I say this as someone who worships Crockford for inventing JSON. I've seen too many projects get fucked hard because they used flatfiles. Edit: Seriously though, managing synchronization/atomic changes to flatfiles manually is a Bad Idea (TM). It's somewhere up there with, "I have a performance problem. I'll use pthreads!" Now you have 1,02-SEGMENTATION FAULT problems.
- MatthewPhillips 14y ago
  Like I said, in my case there were no changes, I only read from them. So I let the server cache the hell out of them and the entire site was fast because of it.
IsaacL 14y ago
You're not alone. Guess which famous tech website uses flat files as a database? Hint: orange header, heirarchical comments...
- habosa 14y ago
  Is this true? Is there some post about this?

gdp 14y ago

It seems rather generous to call this a database. It's files on a filesystem. The lack of any kind of transactional consistency guarantees (Dropbox is notoriously awful at this), and the lack of any sane mechanism for querying this "database" means that it's just "data", not a "database".

yefrederic 14y ago
I think he calls it "database" because it is strongly integrated with the database syntax of the Opa language. You can declare and use this "dropbox-database" just like a Mongo one, independently of the notion of safe-ness of the database.
- gdp 14y ago
  Consistent with the complaint leveled against my previous comment, I should think that this is really consistent with the DBMS syntax of the Opa language</pedantry>
Supreme 14y ago
In the world of SaaS, social media, NoSQL and NodeJS, everything which has already been thought of and brought to maturity is suddenly new, shiny, incomplete, buggy and has a new name. It's a brave new world - enjoy your stay.
mistercow 14y ago
Transactions are not a requirement for databases, and querying is the domain of the DBMS, not the database itself.
- gdp 14y ago
  You're quite right. I think my complaint was that it reduces the definition of "database" to "anything that is used to store data", rather than the database being the json file(s) themselves, at which point we have degenerated to about the same terminological point where it would be reasonable to call saving CSV or MS-Access files into a Dropbox folder "Dropbox-as-a-Database", which seems an unhelpful label, if nothing else.

tarr11 14y ago

It is a fun exercise to do these kinds of things. Obviously, they are very inefficient. But that is not the point! Perhaps there's some residual value related to some Dropbox feature (such as automatic replication) that makes this useful . But that is countered by getting your account rate-limited by Dropbox. I wonder, is there some way to create a key-value store out of any physical or virtual object that can store state? I know there is a lot of work done with quantum computing to store state in quarks. I can see that has value. But, what is the most inefficient data storage mechanism you can come up with? Here are a few I can think of: Instagram-as-a-database - You issue a request to save your data, we encode it using some visual encoding scheme, take a picture, and then upload it somewhere. You get returned a URL, and can then decode it or update it. MTurk-as-a-database - The possibilities are endless here. Someone could write down your request on a piece of paper and then you could send another request where they type it back. Twilio-as-a-database - Encode your data as an audio stream. Store it in random voicemail boxes. Retrieve it later by calling up, listening to your voicemail, decoding the messages.

Jabbles 14y ago
quantum computing to store state in quarks Atoms or electrons, but not quarks.
Erwin 14y ago
Humanity as a database: after infiltrating a religious order mandate arranged marriages encoding your input data in predicted genetic mutations of offspring. Ensure the religious order is hermetic so no outside data appears and that its teaching require all members to appear at a certain location at a certain time. Retrieve your data 1000 years later by appearing at the preordained spot for the Holy Query Ceremony, and examining the eye/hair colours of those gathered. Good error correction protocols are required. (This might well already be part of a Neal Stephenson novel).
plaguuuuuu 14y ago
A Minecraft game would make a hilarious database. The fact that it's probably pretty trivial to CRUD it programmatically outside the Minecraft client by manipulating save files is a great bonus. Start using the DB. If you get bored enough you can examine the data physically in-game.

mattmanser 14y ago

Your blog design is bad btw, popout at side overlays scroll bar in Windows/Chrome.

yefrederic 14y ago
It's one of the design proposed by Google, blame them :)
- bduerst 14y ago
  Google isn't the one using it to demonstrate their work as a developer ;-)
ehutch79 14y ago
Plus, i can barely read the text.

formatjam 14y ago

Actually I have the exact same idea like a few weeks ago. The potential for this idea is big. It can be widely used by small web apps (Chrome App). Instead of hosting a server and a database, every thing can be done and stored in the client, isn't that neat?

yefrederic 14y ago
You should have a look at filepicker.io then
ananddass 14y ago
as yefrederic pointed out you could use filepicker.io to build static js applications. We handle all the server side logic for you.

taskstrike 14y ago

Dropbox as a database is a actually great idea. It takes away scaling problems from the developer and puts the data control back to the user. I'm actually working on NimbusBase, a product that allows you to write clientside javascript applications where there is no server code and data is stored inside of Dropbox. It also integrates with backbone. Shameless plug: sign up for our beta! http://nimbusbase.com http://nimbusbase.com

pencilcode 14y ago

is this part of the app open-sourced? it's a great idea.

yefrederic 14y ago
You can find the source code here : https://github.com/cedricss/server-monitor/ https://github.com/cedricss/server-monitor/
- hbbio 14y ago
  The list of commits is interesting: https://github.com/cedricss/server-monitor/commits/master https://github.com/cedricss/server-monitor/commits/master And also shows its really a one-day app!
cedrics 14y ago
http://server-monitor.herokuapp.com/ http://server-monitor.herokuapp.com/ is open-source (MIT), and use the Opa framework (also open-source, MIT). The Dropbox-as-a-Database is integrated in Opa by default.

shykes 14y ago

"update: Heroku is down [...] the demo in now on dotCloud." :)