5 ms·

How to Write a New Git Protocol

162 points by alecrn 12y ago

ossreality 12y ago
Holy crap why?! Pre/post hooks to encrypt/decrypt seems like it'd be 100000x easier.
fortytw2 12y ago
Nice job! I had no idea anything like this was even possible.
cornstalks 12y ago
This would have been very nice to have a few months ago! At Aspera we wrote a protocol handler for Git[1] that let us send data over fasp (a reliable UDP protocol for fast bulk-data transfers). I learned a ton by reading Git's source code to figure out how to do it, but this blog post is much more pain-free. Very good write up! [1]: https://www.youtube.com/watch?v=ECq7dCwixaY https://www.youtube.com/watch?v=ECq7dCwixaY
- alecrn 12y ago
  That's very interesting! The alternatives to TCP that can be written in UDP which are optimized for a certain application are especially interesting to me. Working at the UDP level can present a lot of interesting performance options, kinda like working in C instead of Python. Something I've been curious about is if there are proxies that can convert protocols. So for instance, on your local machine, you could have a proxy that turns TCP connections on port 8000 into FASP connections to another machine. This would let you use an ordinary web browser over FASP. You could even pipe the proxies, e.g., TCP -> FASP -> MinimaLT [1]. That way any program could have really fast data transfer over an encrypted tunnel. [1] http://cr.yp.to/tcpip/minimalt-20130522.pdf http://cr.yp.to/tcpip/minimalt-20130522.pdf
michaelmior 12y ago
Thanks for sharing this. It's a good walkthrough and potentially quite useful for anyone wanting to implement a new protocol. It seems like the wrong tool for the job to me however. This seems like it would be more easily implemented with smudge/clean filters[0]. In fact, this is exactly what git-crypt[1] does. The idea is that you run a filter on every file as it's checked out to do the decryption and every time a file is staged to do encryption. This requires nothing extra on the remote repository and you can you git commands as normal. [0] http://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes http://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes [1] https://github.com/AGWA/git-crypt https://github.com/AGWA/git-crypt
- JoshTriplett 12y ago
  Depends on the behavior you want. With smudge/clean filters, the files in the repository history are encrypted. That means you don't get the benefit of delta encoding, and any tools you have that look directly at the repository (such as gitk or anything based on libgit2) cannot operate on the cleartext data. On the other hand, git-remote-gcrypt (https://github.com/joeyh/git-remote-gcrypt/ https://github.com/joeyh/git-remote-gcrypt/) encrypts when pushing and decrypts when pulling, which leaves the local repository unencrypted but keeps it encrypted on the untrusted remote server.
  - michaelmior 12y ago
    That's a fair point :) Although I do like the fact that the smudge/blur approach means that you can selectively encrypt files as well as still share the entire repository publicly via any protocol.
    - JoshTriplett 12y ago
      True. And it also has the advantage of failing closed (if you accidentally push to the wrong place) rather than failing open.
- frutiger 12y ago
  Smudge/clean filters will reveal the metadata of the repository (directory structure, file names, etc.)
fiatjaf 12y ago
For people wanting to store their passwords, personal data and large files in a git-like fashion, I recommend Git-Annex: http://git-annex.branchable.com/ http://git-annex.branchable.com/
callesgg 12y ago
git over ssh should be as safe as ssh, right? Does it somehow go outside the ssh tunnel?
- beagle3 12y ago
  He is worried about data-at-rest after the ssh. Specifically, he mentions wanting copies of his private keys backed up, but NOT having copies of his private keys in a github repository -- which would be the case with a standard push through ssh.
packetized 12y ago
This seems interesting to me from the perspective of being able to securely & incrementally back up a media collection to a cloud storage provider (S3, etc) without fear of it being scanned for copyrighted content, a la Dropbox. There are probably solutions out there to do this already; I'm just not aware of them. Very interesting article, and well-written walkthrough.
- alecrn 12y ago
  That's the reason I wrote this for myself, and why I made https://filegrave.com https://filegrave.com, as I thought it could be useful to others as well. Git is already a great tool with a workflow for handling incremental updates that I happen to be used to working with, so there was no need to reinvent the wheel.

162 points by alecrn 12y ago

ossreality 12y ago

Holy crap why?! Pre/post hooks to encrypt/decrypt seems like it'd be 100000x easier.

fortytw2 12y ago

Nice job! I had no idea anything like this was even possible.

cornstalks 12y ago

This would have been very nice to have a few months ago! At Aspera we wrote a protocol handler for Git[1] that let us send data over fasp (a reliable UDP protocol for fast bulk-data transfers). I learned a ton by reading Git's source code to figure out how to do it, but this blog post is much more pain-free. Very good write up! [1]: https://www.youtube.com/watch?v=ECq7dCwixaY https://www.youtube.com/watch?v=ECq7dCwixaY

alecrn 12y ago
That's very interesting! The alternatives to TCP that can be written in UDP which are optimized for a certain application are especially interesting to me. Working at the UDP level can present a lot of interesting performance options, kinda like working in C instead of Python. Something I've been curious about is if there are proxies that can convert protocols. So for instance, on your local machine, you could have a proxy that turns TCP connections on port 8000 into FASP connections to another machine. This would let you use an ordinary web browser over FASP. You could even pipe the proxies, e.g., TCP -> FASP -> MinimaLT [1]. That way any program could have really fast data transfer over an encrypted tunnel. [1] http://cr.yp.to/tcpip/minimalt-20130522.pdf http://cr.yp.to/tcpip/minimalt-20130522.pdf

michaelmior 12y ago

Thanks for sharing this. It's a good walkthrough and potentially quite useful for anyone wanting to implement a new protocol. It seems like the wrong tool for the job to me however. This seems like it would be more easily implemented with smudge/clean filters[0]. In fact, this is exactly what git-crypt[1] does. The idea is that you run a filter on every file as it's checked out to do the decryption and every time a file is staged to do encryption. This requires nothing extra on the remote repository and you can you git commands as normal. [0] http://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes http://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes [1] https://github.com/AGWA/git-crypt https://github.com/AGWA/git-crypt

JoshTriplett 12y ago
Depends on the behavior you want. With smudge/clean filters, the files in the repository history are encrypted. That means you don't get the benefit of delta encoding, and any tools you have that look directly at the repository (such as gitk or anything based on libgit2) cannot operate on the cleartext data. On the other hand, git-remote-gcrypt (https://github.com/joeyh/git-remote-gcrypt/ https://github.com/joeyh/git-remote-gcrypt/) encrypts when pushing and decrypts when pulling, which leaves the local repository unencrypted but keeps it encrypted on the untrusted remote server.
- michaelmior 12y ago
  That's a fair point :) Although I do like the fact that the smudge/blur approach means that you can selectively encrypt files as well as still share the entire repository publicly via any protocol.
  - JoshTriplett 12y ago
    True. And it also has the advantage of failing closed (if you accidentally push to the wrong place) rather than failing open.
frutiger 12y ago
Smudge/clean filters will reveal the metadata of the repository (directory structure, file names, etc.)

fiatjaf 12y ago

For people wanting to store their passwords, personal data and large files in a git-like fashion, I recommend Git-Annex: http://git-annex.branchable.com/ http://git-annex.branchable.com/

callesgg 12y ago

git over ssh should be as safe as ssh, right? Does it somehow go outside the ssh tunnel?

beagle3 12y ago
He is worried about data-at-rest after the ssh. Specifically, he mentions wanting copies of his private keys backed up, but NOT having copies of his private keys in a github repository -- which would be the case with a standard push through ssh.

packetized 12y ago

This seems interesting to me from the perspective of being able to securely & incrementally back up a media collection to a cloud storage provider (S3, etc) without fear of it being scanned for copyrighted content, a la Dropbox. There are probably solutions out there to do this already; I'm just not aware of them. Very interesting article, and well-written walkthrough.

alecrn 12y ago
That's the reason I wrote this for myself, and why I made https://filegrave.com https://filegrave.com, as I thought it could be useful to others as well. Git is already a great tool with a workflow for handling incremental updates that I happen to be used to working with, so there was no need to reinvent the wheel.