10 ms·

3K, 60fps, 130ms: achieving it with Rust

427 points by clpwn 6y ago

deleted 6y ago
[deleted]
jchw 6y ago
Nitpick: “audiophile-quality sound” it seems, is becoming the new “military-grade encryption.” I don’t have many other comments to make other than I am surprised rust-analyzer was only mentioned in passing.
- youeseh 6y ago
  Whose ears and which military? :D
- dijit 6y ago
  The issue with 'military-grade' is that anyone in the military will attest it translates to: Cheapest possible thing that gets the job done. Audiophile grade at least has roots in high fidelity.
  - kevin_thibedeau 6y ago
    3DES is still military grade.
    - user5994461 6y ago
      No it's not. It stopped being approved for usage by NIST a few years ago.
      - kevin_thibedeau 6y ago
        112-bit keys are still allowed precisely because of 3DES.
      - Twirrim 6y ago
        Really? 3DES still appears here, https://csrc.nist.gov/projects/block-cipher-techniques https://csrc.nist.gov/projects/block-cipher-techniques, with DES and Skipjack being called out as deprecated.
        user5994461 6y ago
        That page says that 3DES is prohibited from usage in new applications and is prohibited for encrypting more than 1 GB of data, since 2017. The attached documents have additional information on implementation and (non) usage, including deadline to migrate legacy military systems. It's sadly quite cumbersome to go through the tens of PDF to find the relevant information.
  - Joeboy 6y ago
    > Audiophile grade at least has roots in high fidelity. Does it though? Audiophiles generally seem to eschew fidelity in favour of something that sounds subjectively nice, including the psychoacoustic effects of spending a lot of money. Eg. they seem very fond of "warmth". If you asked me to make something sound "warm", I'd be applying some soft clipping and dampening the top end, not eliminating sources of distortion. Edit: If you actually wanted high fidelity, you'd use studio headphones / monitors, which are designed to be "unflattering", so you can be confident you'll hear any issues when mixing / mastering. People don't normally listen for pleasure with those, because they become fatiguing after a few hours. Choosing equipment because you like the sound is a very reasonable thing to do, but it's not the same as pursuing fidelity.
    - SomeoneFromCA 6y ago
      You've never listened to audiophile equipment have you? If apply "some soft clipping" it will sound bad, I guarantee you, no audiophile would like it.
      - Joeboy 6y ago
        > You've never listened to audiophile equipment have you? You're saying that I ought to judge the merits of audiophile equipment by the subjective measure of whether I like the sound of it. Which is the metric I said audiophiles would favour. > If apply "some soft clipping" it will sound bad Soft clipping often sounds nice, which is why it's very commonly applied to music. You're saying that eg. the sound of a classic Vox amp is bad, which I guess you're free to believe if that's what your ears tell you, but it's certainly not an objective truth.
        SomeoneFromCA 6y ago
        Because what you are describing is a simplistic picture, describing whole class of people as stupid simpletons who cannot tell low THD and low IMD audio from "soft clipping which sounds nice". If you are referring to vacuum tube amps, soft clipping is only partially the reason why they sound the way they do; in fact most of the time amps are not clipping and are outputting close to 1% of their their total power. Reasons why tube equipment sounds better/different from the solid state amps are a lot more complex than the "common wisdom" of soft clipping.
  - Macha 6y ago
    Similarly "medical grade" = "single use" in many actual medical contexts.
- simias 6y ago
  As far as I'm concerned "audiophile" has been synonymous with "overpriced placebo" basically forever. Beyond that I wish the article had explain a bit better why it chose these "better-than-std" crates. I'm actually using all the std variants in my projects, I'm curious to know if I'm missing out or if I just happen not to hit on their limitations.
  - cesarb 6y ago
    > Beyond that I wish the article had explain a bit better why it chose these "better-than-std" crates. At least for parking_lot, its README has a long list with its advantages over std: https://github.com/Amanieu/parking_lot/blob/master/README.md https://github.com/Amanieu/parking_lot/blob/master/README.md
    - simias 6y ago
      I saw that but I was interested to know if TFA had decided to go with it because it looks better on paper or if it's because they hit a roadblock using the std counterparts and migrated to using those. That being said since they're drop-in replacements for the most part I suppose I could just try to rebuild my project with this crate and see if I notice a difference performance-wise.
      - clpwn 6y ago
        In our case, it wasn't a matter of hitting a specific roadblock as much as it was past experience in performance-sensitive projects, and knowing that if we start with those crates we'll probably be fine, and if we don't, we'll probably switch to them eventually for some reason. With crossbeam for example, you can hit roadblocks with std since their channels are MPSC, whereas crossbeam supports MPMC channels (and is faster than std in every meaningful measurement last I checked).
        simias 6y ago
        That's great to know, thanks! Reading the description it almost seemed too good to be true but if it's indeed objectively better in basically every situation I should probably give it a try.
- simlevesque 6y ago
  yeah audiophile can be so may things. To me it means 24bits or more.
  - kevin_thibedeau 6y ago
    More than 21-bits is meaningless. It's all hype beyond 24-bits.
    - SomeoneFromCA 6y ago
      You might be right, however 16-bit sounds really harsh to my ears, and 24-bits is the only widely used standard, better than 16-bit.
      - nseggs 6y ago
        I think the bigger issue is likely to be a trash computer mic, a trash preamp/adc, trash dac, trash speakers, trash room. I don't care if at some point you're sampling and sending that signal at 1000-bit or whatever, it's still trash, just very accurately sampled trash.
        SomeoneFromCA 6y ago
        I disagree, I do not own trash equipment. Every time I install Linux, i switch Pulseaudio settings from 16-bit to 24-bit; the difference is immediate, although subtle. Everyone I know who tried to do this, noted that listening fatigue is a lot lower with the new settings.
        sneak 6y ago
        In my direct experience, everyone who claims this to me, so far, is unable to distinguish 16 bit and 24 bit recordings in an ABX. The audiophile world would do well to adopt the concept of double-blind study.
        6y ago
- qppo 6y ago
  especially because all VoIP codecs sound like shit. It's intelligible, but the bar isn't high for fidelity.
- clpwn 6y ago
  You're right, that sounds way too fluffy. To clarify, we're targeting "transparent" sounding audio, not "FLACs or bust" audio. Right now we send stereo 48kHz 96kb/s Opus (CELT, not SILK) that we found hit the voice transparency sweet-spot compared to the lossless audio source. We had used higher bitrates in the past, and could easily go back to them, but quality plateaued at around 96k in our experimentation. More than choosing sane transparent-sounding encoding parameters, the biggest difference in fidelity by far was choosing the correct microphones and speakers for accurate reproduction of voices.
  - sneak 6y ago
    Voice does not extend above 22.05khz, so using sampling rates above 44.1khz is entirely objectively wasteful and useless, unless your codec only works at 48khz input or something. Are you using 48khz for a specific reason?
    - namibj 6y ago
      Please read the official Opus FAQ to sampling rates: https://wiki.xiph.org/OpusFAQ#But_won.27t_the_resampler_hurt_the_quality.3F_Isn.27t_it_better_to_use_44.1_kHz_directly.3F https://wiki.xiph.org/OpusFAQ#But_won.27t_the_resampler_hurt...
    - eqvinox 6y ago
      44.1 kHz is essentially deprecated on the hardware level since it's annoying to deal with the extra clock. It's a few cents for an extra crystal, way too expensive ;). 44100 also makes for very poor multipliers/dividers to other clocks since it includes 3²×5²×7² as factors. 48000 is much nicer with 3×5³.
- masklinn 6y ago
  > Nitpick: “audiophile-quality sound” it seems, is becoming the new “military-grade encryption.” It's too bad they didn't explain it. I expected they meant allowance for "full bandwidth" audio (possibly including music you can listen to). Video conferencing systems generally use voice-only codecs compressed to shit, full of artifacts in the voice range and utterly dead outside of it.
- duskwuff 6y ago
  It can join the ranks of meaningless phrases like "aircraft-grade aluminum", "chef-grade cookware", and "contractor-grade tools".
- efreak 6y ago
  To me, "military grade encryption" means following industry standard. "Audiophile quality" means higher quality than you need, care about, or can even tell apart from lower quality.
  - Skunkleton 6y ago
    No, "military grade encryption" means nothing. If it referenced a standard, than that might mean something. I've worked on products for the military that still used single pass DES encryption. So that was military grade. It might as well have been ROT13.
ttul 6y ago
If this actually works, I am desperately keen to get my hands on it. If you have the capacity for high bandwidth, why not use it? Zoom’s model must work on whatever crappy broadband people have in their home office. If you have gigabit, it doesn’t seem to make use of that extra capacity to improve video quality. As for sound, I don’t think audiophile quality is necessary...
- onion2k 6y ago
  As for sound, I don’t think audiophile quality is necessary... Given you'll need about 10Mbps upstream for 60fps 3K video it seems a little unreasonable not to add on a 320Kpbs (or more) audio stream. It could make this useful for things like streaming music concerts.
  - wongarsu 6y ago
    Also, audio quality seems to be more important for the subjective experience than video quality, even in regular video content.
  - wenc 6y ago
    Semi-related note: there's work being done at Stanford to make it possible for remote musicians to play together in an ensemble at low latencies. JackTrip is the resulting software -- not end-user friendly, but apparently it works. https://ccrma.stanford.edu/groups/soundwire/software/jacktrip/ https://ccrma.stanford.edu/groups/soundwire/software/jacktri... (Some basic numbers: sounds takes 1 ms to travel a foot, every ms is a foot of separation between musicians, 30ms of latency = 30 ft separation = the max for jamming. So 130ms is not low enough.)
- nerdbaggy 6y ago
  If you only need a P2P video stream https://github.com/CESNET/UltraGrid/wiki https://github.com/CESNET/UltraGrid/wiki is amazing and lower latency
- dougmwne 6y ago
  Let's turn that statement around and instead of thinking about audio bitrates, focus on experience. A great "audiophile" setup can make the performers sound there in the room with you. No matter how much BS the hobby spews, when you hear a really great setup, that guitar truly sounds 6 feet away from you. Zoom calls do not sound there in the room with you. Microphones are terrible, there's compression artifacts, latency, packet loss, background noise, and tiny speakers. No one could possibly close their eyes and forget that the other person is not there in the room with them, on any POTS or VOIP technology that exists. But what if you could create an audio communications system with an actual illusion of auditory presence. Sounds amazing! And given that this company is trying to create wall-screen, life size ultra-HD video conferences, I'm pretty sure that "audiophile" exactly what they're going for. Personally as a remote worker, I would absolutely swoon for this.
ccostes 6y ago
Aside from the Rust aspect (which is cool!), I can't believe we've come this far and still don't have low-latency video conferencing. Maybe I'm overly sensitive, but people talking over each other and the lack of conversational flow drives me crazy with things like hangouts.
- GuiA 6y ago
  There are hard limits at play. No matter what you do, you can't go from New York to London in less than ~20ms; add video/audio encoding, packet switching, decoding, etc. and it's easy to see why any latency under the 100ms mark at that spatial scale in a scalable, mainstream product would be close to a miracle. The thing is that when we talk in a room, sound will take <10ms to reach my ears from your mouth. This is what "enables" all of the human turn taking cues in conversation (eye contact, picking up whether a sentence is about to end/whether it's a good time to chime in/etc) - I've been looking for work from people who've tried to see at what point things start feeling really bad (is it 10ms, or 50ms?), but haven't found much so far. No matter what it is though, it's likely that long distance digital communications just cannot match it. See also this interesting comment about the feeling of "closeness" from phone copper wires: https://news.ycombinator.com/item?id=22931809 https://news.ycombinator.com/item?id=22931809 Landlines were so fast and so "direct" in their latency (where distance correlates very directly with time, due to a lack of "hops") that local phone calls were faster than the speed of sound across a table, and for a bit after they came out--before people generally got used to seemingly random latency--local calls felt "intimate", like as if you were talking to someone in bed with their head right next to you; I also have heard stories of negotiators who had gotten really tuned to analyzing people's wait times while thinking that long distance calls were confusing and threw them off their game.
  - jokoon 6y ago
    > it's easy to see why any latency under the 100ms mark at that spatial scale in a scalable, mainstream product would be close to a miracle. It seems normal phones are able to do it, though. At least it seems normal phones suffer less latency problem. In a way, simplicity in technology often means better performance.
vertex-four 6y ago
Are they still using WebRTC, just their own implementation? Or have they switched to something else on the wire?
- markdog12 6y ago
  There's a section in the article about it: "In the beginning (or: why we're not WebRTC)"
  - vertex-four 6y ago
    I'm interested in what they are using if not WebRTC - there's several good options in this space (SRT would be my go-to choice), so it'd be really interesting to see if they rolled their own wire protocol or used something else.
    - beowulfey 6y ago
      They built it from scratch
      - vertex-four 6y ago
        Their blog post suggests they wrote something from scratch, but gives no clue as to what, whether they considered building on a more modern protocol specification than RTP (which is a couple decades old at this point), what they've taken from other more modern protocol specs if they didn't use one directly, or anything aside from that they wrote some code really.
nerdbaggy 6y ago
I wonder how much bandwidth this uses. The less bandwidth it uses the higher the latency because of compression. Its much easier to get low latency video when you have large (Gbit+) links
jonnypotty 6y ago
I wish I read more things like this on hn. "We wanted to know and understand every line of code being run on our hardware, and it should be designed for the exact hardware we wanted"
- wmf 6y ago
  As long as you're also comfortable reading the "over-engineering made our product inflexible, late to market, and too expensive" blog posts later.
  - noir_lord 6y ago
    I mean you could also have the "we used commodity everything, where first to market but the next folks did it better and cheaper because they could" posts - hindsight is 20/20. I bet IBM didn't expect using off the shelf components would mean that the IBM PC was the standard for the next 30 years and it wouldn't be theirs. ----- As someone who works at a company that relied heavily on video conferencing (half the devs off shore) - every single major solution absolutely sucks, they are flaky, unreliable, sound quality is poor, video rate is poor (and this is with fat pipes at both ends) and worst of all latency, latency when trying to have a round table conversation with people remotely is horrific, it is good to see someone pushing the limits, Skype et al haven't gotten much better in the last decade yet my internet connection at home/work is x50 times faster and even mid range business laptops have much improved graphics grunt.
- freeopinion 6y ago
  "porting [webrtc-audio-processing] to Rust in the near-term is not likely (it's around 80k lines of C and C++ code)." That's just one of their dependencies. It's possible to know every line without rewriting. And it's possible to rewrite and still not know every line. They seem to strike a reasonable balance.
- kingosticks 6y ago
  But that statement seems at odds with a dependency on the enormous WebRTC AudioProcessing C++ module. But then they also say they don't use WebRTC so maybe I misunderstand what's going on.
GuiA 6y ago
Very cool from a tech standpoint. From a product point of view, I find it interesting that the illustrations/concept videos for these things always show people interacting very closely to the wall - e.g. playing chess, sitting around a table, etc. https://tonari.no/static/media/family.48218197.svg https://tonari.no/static/media/family.48218197.svg But in practice, people tend to keep their distance from it. E.g. the pictures of this setup tend to show people clustered in their own group on each side of the wall, with a solid 2-3 meters from the wall. https://blog.tonari.no/images/ea56c74d-a55d-4183-9a7b-d697954c5159-tonari-frontier-2.png.optimized.jpg https://blog.tonari.no/images/ea56c74d-a55d-4183-9a7b-d69795... It makes sense, it's awkward to be close to a large solid (emissive) surface, and humans instinctively get closer to their in group when faced with an out group. I wonder how the system could be designed to encourage participants being closer, if there is an advantage to that.
- STRML 6y ago
  A practical problem to solve there: where do you put the cameras? I would actually prefer putting them behind the screen if possible - a few small pinholes wouldn't be that noticeable. If you could put multiple wide-angle cameras in multiple places, you could stitch them together in software and create a real feeling of closeness.
- renewiltord 6y ago
  I'd sit closer but the picture then distorts and I am distorted for my conversation partner.
nerdbaggy 6y ago
If anybody is looking for a low latency high bandwidth P2P video streaming solution there is https://github.com/CESNET/UltraGrid/wiki https://github.com/CESNET/UltraGrid/wiki It can do less than 80ms of latency
- ClumsyPilot 6y ago
  This is cool, thanks for the link. Is this Nvidia GPU only? Mighr give it a try at some point
  - nerdbaggy 6y ago
    You don't need the GPU, just depends on the type of compression you want. It supports intel VA-API as well as NVIDIA VDPAU
- _visgean 6y ago
  How do you use this over network? I have installed it but its very unclear to me what I need to do in order to call my colleague in another city. It seems that it can only connect to publicly visible hosts? Overall it looks like somebody should develop an application on top of this.
systemvoltage 6y ago
@dang - Suggest altering the title to say what it is "Achieving 3K, 60fps, 130ms Video Conferencing with Rust".
codefined 6y ago
Any suggestions on a group video conferencing tool for use on a local network (Ethernet) that's effective? Either self-hosted or online, just for personal usage to talk with others?
lc5G 6y ago
for crate in $(ls */Cargo.toml | xargs dirname); do cargo build Why do this instead of cargo --workspace build Is it so you can time the individual crates?
- clpwn 6y ago
  Yep!
- ninkendo 6y ago
  Yeah it looks like they wanted to know how long each crate took to build individually. But as long as we're nitpicking, nobody should just pipe `ls` into `xargs` like this, since it fails if anything has spaces in it. Instead, do: for cargo_toml in */Cargo.toml; do crate="$(dirname "${cargo_toml}")" pushd $crate # ... done Don't be that person who writes a script which won't tolerate spaces in filenames!
  - OJFord 6y ago
    Alternatively: Don't be that person who clones the repo at a path with spaces in!
    - ninkendo 6y ago
      Not having spaces in your directory names is certainly a good idea, but I'll be damned if I let any of my code have issues with them. Just because something's a good idea doesn't mean it should be a requirement :) (The main reason for the advice of "Don't put spaces in paths" is really only because it breaks lots of poorly-written software... but that's not an excuse for your software to be poorly-written!)
pier25 6y ago
Woah this portal thing into another place seems super exciting if they can really pull it off and maintain low latency in the real world.
remmargorp64 6y ago
But does it have middle out compression?
zelly 6y ago
The bottleneck is not on the CPU. I'm afraid this company may have wasted their time trying to reinvent WebRTC. If you really want to get realtime video, I think the best approach is a custom codec on CUDA or better yet custom hardware (FPGA). You can only go so far on general purpose hardware before you hit a wall and get Zoom/WebEx quality.
- busrf 6y ago
  Can you recommend some resources for the current state of the art for low latency video? Somebody else in the comments posted https://github.com/CESNET/UltraGrid/wiki https://github.com/CESNET/UltraGrid/wiki, but I’m curious to learn more.
- ronyfadel 6y ago
  Is or is not? I’m confused: if the bottleneck is not on the CPU what does CUDA solve?
  - zelly 6y ago
    The bottleneck is the video encoding/decoding/rendering, which is done on GPUs to begin with. Of course if it were done on CPU instead, then it would be significantly worse, but that's not where we're starting from. Improving stuff on the host side by, say, rewriting WebRTC in Rust won't improve the latency of your video by much or at all.
Scaevolus 6y ago
What's the codec stack for this? x264 --tune zerolatency + opus with opus_delay=20ms?
- namibj 6y ago
  20ms is wasteful. Use minimum latency where SILK still works, afaik that's 7.5ms.
  - codys 6y ago
    This assumes that video encoding latency is lower than the audio latency.
    - namibj 6y ago
      Shouldn't be more than a single frame, which is 16 2/3 ms for 60 fps. And for e.g. JPEG it can be even shorter, especially with a rolling shutter.
usefulcat 6y ago
"We wanted to know and understand every line of code being run on our hardware, and it should be designed for the exact hardware we wanted." This rings very true for every high-performance thing I've ever worked on, from games to trading systems.
swsieber 6y ago
This is welcome news. I have been itching to convert a small headshot videostream (thing under 100x100px) to audio, stream it over mumble and then convert it back to video, just to see what the latency is like. It would obviously be a big undertaking, but not as big as this methinks.
LockAndLol 6y ago
After reading it, I'm still not entirely sure what's being done. Is it live streaming or is it the transport? Are they doing video encoding (the audio encoding seems to be done by that webrtc-audio thing)? Have they chosen a progressive encoding format that compresses frames and pumps them out to the wire as soon as they're done? Is TCP or UDP involved or a new Layer 3 protocol entirely? Have I just missed all of those parts or were they really missing amid all the Rust celebration?
- clpwn 6y ago
  > After reading it, I'm still not entirely sure what's being done. > Is it live streaming or is it the transport? tonari is the entire stack, similar in "feature scope" to WebRTC but with different goals and target environments. > Are they doing video encoding (the audio encoding seems to be done by that webrtc-audio thing)? Yep, this includes video encoding and transport. We don't use the WebRTC audio library for encoding or transport, just for echo cancellation and other helpful acoustic processing. > Have they chosen a progressive encoding format that compresses frames and pumps them out to the wire as soon as they're done? Yep, basically, if by that you mean we don't use B-frames or other codec features that would require buffering multiple video frames before receiving a compressed stream, so we're able to send out encoded frames as they arrive. > Is TCP or UDP involved or a new Layer 3 protocol entirely? We encapsulate our protocol in UDP since we operate on normal internet - a new protocol is out of the question without a huge lobbying force and 15 years of patience on your side. > Have I just missed all of those parts or were they really missing amid all the Rust celebration? We intentionally didn't get into the protocol details because we are saving that for a dedicated post (and code to back it up).
  - LockAndLol 6y ago
    Thank you very much for the answers. Glad I wasn't too far off. Looking forward to the technical post. If you're planning on releasing all of this royalty-free and opensource, you'd be quite a boon to the free and open internet. Getting this picked up by the likes of Mozilla and getting it into a browser would be amazing.
MR4D 6y ago
Gotta love a writeup with this line in it: like Brian's 1970s-era MacBook Pro That's a writer(s) who knows what it's like to read long (aka thorough) technical articles and not bore the readers to death. Great article!
alpineidyll3 6y ago
Awesome post
eadan 6y ago
This is amazing! The first thing that popped to my mind seeing the life sized "portal" was the farcaster portals from the sci-fi novel Hyperion https://hyperioncantos.fandom.com/wiki/Farcaster https://hyperioncantos.fandom.com/wiki/Farcaster
imtringued 6y ago
My WebRTC projects haven't suffered that much from latency. The biggest source of delays is usually caused by encoding video for me. I've had to limit streams to 720p and 25fps to reduce the time spent on CPU encoding a vp8 stream. There are also bandwidth considerations (real time encoding = significantly less compression) but the end result is slightly less than 200ms one way latency (including input lag from mouse, 15ms network latency and display lag) without any special settings. All I'm doing is feeding a ffmpeg stream to kurento and letting it broadcast it via WebRTC. This is not a web conferencing application and it is also not using WebRTC via p2p. It's closer to conventional live streaming with a sane amount of latency (compared to up to 30s of latency you commonly see on twitch). Of course I personally would prefer it if the latency can be brought down even further. 100ms or lower is like the holy grail for me and only appears to be doable with codecs that aren't supported by WebRTC. However, people don't want to install apps just for my little service and I certainly won't encode every stream via several codecs just for the tiny minority of the user base that actually ends up using the app.
chubs 6y ago
Sounds impressive, but i'm dying to know: what video codec are they using?
novok 6y ago
I wonder how it compares with apple facetime on two new macbooks with ethernet connections on both sides. They actually work on reducing latency and pushing high res video if your connection supports it.
- bschwindHN 6y ago
  That's a great idea, I've always preferred facetime at least for the video quality. We'll do a latency test sometime, I suspect it'll be quite good!
jupp0r 6y ago
I love Rust, but them deciding to redesign/reimplement webrtc after being frustrated after a week seems like a prime candidate for not invented here syndrome with Rust being the justification. There is a reason webrtc is as big as it is, it’s a complex problem to solve. Regarding the premise of high latency in webrtc: Google Stadia has ~160ms round trip latency at 4k from my Macbook to a data center, so it’s not like that’s unachievable.
- Apofis 6y ago
  Google is colocating in your basement.
jonny383 6y ago
"we truly don't believe we could have achieved these numbers with this level of stability without Rust" Oh please. This is just rust sensationalism. People don't truly believe rust is faster than C do they?
- gpm 6y ago
  Developing stable complex software in C takes a hell of a lot more effort and skill than it does in Rust IMO.
- bufferoverflow 6y ago
  In some problems Rust is the fastest: https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/spectralnorm.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/revcomp.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/binarytrees.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
  - jonny383 6y ago
    Every single one of those rust implementations is using unsafe {}, thus defeating the purpose of using rust in the first place. Run the same benchmark without unsafe {}.
    - turndown 6y ago
      >defeating the purpose of using rust in the first place I don't think this is true; the whole point of Rust is that unsafe operations are explicit, not that you never do so. Also, I looked at the first one, and it's only using unsafe on what are basically op code calls; I don't think it is realistic to complain about that.
    - igouy 6y ago
      Not true. Of those 4 tasks, the rust programs for task 3 and task 4 do not use the keyword "unsafe". For task 2, the spectral-norm Rust #6 program does use "unsafe" but #5 does not and it's almost as fast.
Exuma 6y ago
I love their homepage https://tonari.no/ https://tonari.no/
- deleted 6y ago
  [deleted]
- aiotokyo 6y ago
  Thank you so much! Keep an eye for further updates.
zamalek 6y ago
> We just enforce rustfmt. After interaction with both rustfmt and go fmt, I have concluded that .editorconfig is solving a problem that really shouldn't be solved. We went through the ordeal of defining our C# coding standards where I work and, let me tell you, people (myself included) care very deeply about their way of structuring code. And it's a bloody waste of their time. Having the language designers say, "here is how our language should be structured" is a breath of fresh air.
sephamorr 6y ago
Why exactly do existing video streaming solutions use such small amounts of bandwidth and have terrible quality as a result? Does anyone have a deep dive into why this is the case? It seems that it would be a killer feature to make better utilization of bandwidth. Even over wifi, speedtest shows 4ms/100mb/100mb on my internet connection, but Zoom, FaceTime, and others never use more than about 0.8Mbit/s for a video stream, and the resulting quality of audio and video is...understandably poor. Latency too totally feels like a software problem, perhaps with too many layers of abstraction. (60fps->16ms for the camera, ~10ms for encoding with NVENC/equivalents, 35ms measured one-way latency from my laptop to my parents 4000km away, ~10ms decode, 16ms frame delay = 87ms one way). Maybe I'm asking for too much from non-realtime systems (I'm used to RTOS, extensive use of DMA, zero-copy network drivers, etc), but it seems that there is a lot of room to improve.
- rasz 6y ago
  OnLive "solved" encoder latency 15 years ago. You dont wait 16ms for the next frame. Instead you progressively start encoding after receiving first tens of lines. This way your encoded video stream lags just couple of milliseconds behind, same for decoding. You could crudely emulate this by dividing screen into 4 rows and sending 4 concurrent video stream, instant 1/4 latency drop.
  - sephamorr 6y ago
    Sure, many of the operations in the list can be pipelined as you mention. Something like G-Sync would also allow you to sync the destination display to the arrival of the (start of) frame.
- clpwn 6y ago
  It's worth mentioning that in our case, a significant chunk of the latency in our 130ms measurement is just the input lag of our display that we currently use. We were surprised by how slow they can be.
realchucknorris 6y ago
would loved to see a demo
dbrgn 6y ago
"A week of struggling with WebRTC’s nearly 750,000 LoC behemoth of a codebase revealed just how painful a single small change could be — how hard it was to test, and feel truly safe, with the code you were dealing with." I totally feel you. It's impressive what the WebRTC implementation has achieved, but it's just not pleasant at all to work with it.
snvzz 6y ago
130ms is a world better than 500ms and a much welcome improvement, but it is still terrible. Latency happens throughout the whole stack; Unfortunately much would need to be fixed outside this project to achieve any further significant improvement. Operating System, firmware, blackbox hardware are some other non-negligible sources of latency. Everything adds up.

427 points by clpwn 6y ago

deleted 6y ago

[deleted]

jchw 6y ago

Nitpick: “audiophile-quality sound” it seems, is becoming the new “military-grade encryption.” I don’t have many other comments to make other than I am surprised rust-analyzer was only mentioned in passing.

youeseh 6y ago
Whose ears and which military? :D
dijit 6y ago
The issue with 'military-grade' is that anyone in the military will attest it translates to: Cheapest possible thing that gets the job done. Audiophile grade at least has roots in high fidelity.
- kevin_thibedeau 6y ago
  3DES is still military grade.
  - user5994461 6y ago
    No it's not. It stopped being approved for usage by NIST a few years ago.
    - kevin_thibedeau 6y ago
      112-bit keys are still allowed precisely because of 3DES.
    - Twirrim 6y ago
      Really? 3DES still appears here, https://csrc.nist.gov/projects/block-cipher-techniques https://csrc.nist.gov/projects/block-cipher-techniques, with DES and Skipjack being called out as deprecated.
      - user5994461 6y ago
        That page says that 3DES is prohibited from usage in new applications and is prohibited for encrypting more than 1 GB of data, since 2017. The attached documents have additional information on implementation and (non) usage, including deadline to migrate legacy military systems. It's sadly quite cumbersome to go through the tens of PDF to find the relevant information.
- Joeboy 6y ago
  > Audiophile grade at least has roots in high fidelity. Does it though? Audiophiles generally seem to eschew fidelity in favour of something that sounds subjectively nice, including the psychoacoustic effects of spending a lot of money. Eg. they seem very fond of "warmth". If you asked me to make something sound "warm", I'd be applying some soft clipping and dampening the top end, not eliminating sources of distortion. Edit: If you actually wanted high fidelity, you'd use studio headphones / monitors, which are designed to be "unflattering", so you can be confident you'll hear any issues when mixing / mastering. People don't normally listen for pleasure with those, because they become fatiguing after a few hours. Choosing equipment because you like the sound is a very reasonable thing to do, but it's not the same as pursuing fidelity.
  - SomeoneFromCA 6y ago
    You've never listened to audiophile equipment have you? If apply "some soft clipping" it will sound bad, I guarantee you, no audiophile would like it.
    - Joeboy 6y ago
      > You've never listened to audiophile equipment have you? You're saying that I ought to judge the merits of audiophile equipment by the subjective measure of whether I like the sound of it. Which is the metric I said audiophiles would favour. > If apply "some soft clipping" it will sound bad Soft clipping often sounds nice, which is why it's very commonly applied to music. You're saying that eg. the sound of a classic Vox amp is bad, which I guess you're free to believe if that's what your ears tell you, but it's certainly not an objective truth.
      - SomeoneFromCA 6y ago
        Because what you are describing is a simplistic picture, describing whole class of people as stupid simpletons who cannot tell low THD and low IMD audio from "soft clipping which sounds nice". If you are referring to vacuum tube amps, soft clipping is only partially the reason why they sound the way they do; in fact most of the time amps are not clipping and are outputting close to 1% of their their total power. Reasons why tube equipment sounds better/different from the solid state amps are a lot more complex than the "common wisdom" of soft clipping.
- Macha 6y ago
  Similarly "medical grade" = "single use" in many actual medical contexts.
simias 6y ago
As far as I'm concerned "audiophile" has been synonymous with "overpriced placebo" basically forever. Beyond that I wish the article had explain a bit better why it chose these "better-than-std" crates. I'm actually using all the std variants in my projects, I'm curious to know if I'm missing out or if I just happen not to hit on their limitations.
- cesarb 6y ago
  > Beyond that I wish the article had explain a bit better why it chose these "better-than-std" crates. At least for parking_lot, its README has a long list with its advantages over std: https://github.com/Amanieu/parking_lot/blob/master/README.md https://github.com/Amanieu/parking_lot/blob/master/README.md
  - simias 6y ago
    I saw that but I was interested to know if TFA had decided to go with it because it looks better on paper or if it's because they hit a roadblock using the std counterparts and migrated to using those. That being said since they're drop-in replacements for the most part I suppose I could just try to rebuild my project with this crate and see if I notice a difference performance-wise.
    - clpwn 6y ago
      In our case, it wasn't a matter of hitting a specific roadblock as much as it was past experience in performance-sensitive projects, and knowing that if we start with those crates we'll probably be fine, and if we don't, we'll probably switch to them eventually for some reason. With crossbeam for example, you can hit roadblocks with std since their channels are MPSC, whereas crossbeam supports MPMC channels (and is faster than std in every meaningful measurement last I checked).
      - simias 6y ago
        That's great to know, thanks! Reading the description it almost seemed too good to be true but if it's indeed objectively better in basically every situation I should probably give it a try.
simlevesque 6y ago
yeah audiophile can be so may things. To me it means 24bits or more.
- kevin_thibedeau 6y ago
  More than 21-bits is meaningless. It's all hype beyond 24-bits.
  - SomeoneFromCA 6y ago
    You might be right, however 16-bit sounds really harsh to my ears, and 24-bits is the only widely used standard, better than 16-bit.
    - nseggs 6y ago
      I think the bigger issue is likely to be a trash computer mic, a trash preamp/adc, trash dac, trash speakers, trash room. I don't care if at some point you're sampling and sending that signal at 1000-bit or whatever, it's still trash, just very accurately sampled trash.
      - SomeoneFromCA 6y ago
        I disagree, I do not own trash equipment. Every time I install Linux, i switch Pulseaudio settings from 16-bit to 24-bit; the difference is immediate, although subtle. Everyone I know who tried to do this, noted that listening fatigue is a lot lower with the new settings.
        sneak 6y ago
        In my direct experience, everyone who claims this to me, so far, is unable to distinguish 16 bit and 24 bit recordings in an ABX. The audiophile world would do well to adopt the concept of double-blind study.
        6y ago
qppo 6y ago
especially because all VoIP codecs sound like shit. It's intelligible, but the bar isn't high for fidelity.
clpwn 6y ago
You're right, that sounds way too fluffy. To clarify, we're targeting "transparent" sounding audio, not "FLACs or bust" audio. Right now we send stereo 48kHz 96kb/s Opus (CELT, not SILK) that we found hit the voice transparency sweet-spot compared to the lossless audio source. We had used higher bitrates in the past, and could easily go back to them, but quality plateaued at around 96k in our experimentation. More than choosing sane transparent-sounding encoding parameters, the biggest difference in fidelity by far was choosing the correct microphones and speakers for accurate reproduction of voices.
- sneak 6y ago
  Voice does not extend above 22.05khz, so using sampling rates above 44.1khz is entirely objectively wasteful and useless, unless your codec only works at 48khz input or something. Are you using 48khz for a specific reason?
  - namibj 6y ago
    Please read the official Opus FAQ to sampling rates: https://wiki.xiph.org/OpusFAQ#But_won.27t_the_resampler_hurt_the_quality.3F_Isn.27t_it_better_to_use_44.1_kHz_directly.3F https://wiki.xiph.org/OpusFAQ#But_won.27t_the_resampler_hurt...
  - eqvinox 6y ago
    44.1 kHz is essentially deprecated on the hardware level since it's annoying to deal with the extra clock. It's a few cents for an extra crystal, way too expensive ;). 44100 also makes for very poor multipliers/dividers to other clocks since it includes 3²×5²×7² as factors. 48000 is much nicer with 3×5³.
masklinn 6y ago
> Nitpick: “audiophile-quality sound” it seems, is becoming the new “military-grade encryption.” It's too bad they didn't explain it. I expected they meant allowance for "full bandwidth" audio (possibly including music you can listen to). Video conferencing systems generally use voice-only codecs compressed to shit, full of artifacts in the voice range and utterly dead outside of it.
duskwuff 6y ago
It can join the ranks of meaningless phrases like "aircraft-grade aluminum", "chef-grade cookware", and "contractor-grade tools".
efreak 6y ago
To me, "military grade encryption" means following industry standard. "Audiophile quality" means higher quality than you need, care about, or can even tell apart from lower quality.
- Skunkleton 6y ago
  No, "military grade encryption" means nothing. If it referenced a standard, than that might mean something. I've worked on products for the military that still used single pass DES encryption. So that was military grade. It might as well have been ROT13.

ttul 6y ago

If this actually works, I am desperately keen to get my hands on it. If you have the capacity for high bandwidth, why not use it? Zoom’s model must work on whatever crappy broadband people have in their home office. If you have gigabit, it doesn’t seem to make use of that extra capacity to improve video quality. As for sound, I don’t think audiophile quality is necessary...

onion2k 6y ago
As for sound, I don’t think audiophile quality is necessary... Given you'll need about 10Mbps upstream for 60fps 3K video it seems a little unreasonable not to add on a 320Kpbs (or more) audio stream. It could make this useful for things like streaming music concerts.
- wongarsu 6y ago
  Also, audio quality seems to be more important for the subjective experience than video quality, even in regular video content.
- wenc 6y ago
  Semi-related note: there's work being done at Stanford to make it possible for remote musicians to play together in an ensemble at low latencies. JackTrip is the resulting software -- not end-user friendly, but apparently it works. https://ccrma.stanford.edu/groups/soundwire/software/jacktrip/ https://ccrma.stanford.edu/groups/soundwire/software/jacktri... (Some basic numbers: sounds takes 1 ms to travel a foot, every ms is a foot of separation between musicians, 30ms of latency = 30 ft separation = the max for jamming. So 130ms is not low enough.)
nerdbaggy 6y ago
If you only need a P2P video stream https://github.com/CESNET/UltraGrid/wiki https://github.com/CESNET/UltraGrid/wiki is amazing and lower latency
dougmwne 6y ago
Let's turn that statement around and instead of thinking about audio bitrates, focus on experience. A great "audiophile" setup can make the performers sound there in the room with you. No matter how much BS the hobby spews, when you hear a really great setup, that guitar truly sounds 6 feet away from you. Zoom calls do not sound there in the room with you. Microphones are terrible, there's compression artifacts, latency, packet loss, background noise, and tiny speakers. No one could possibly close their eyes and forget that the other person is not there in the room with them, on any POTS or VOIP technology that exists. But what if you could create an audio communications system with an actual illusion of auditory presence. Sounds amazing! And given that this company is trying to create wall-screen, life size ultra-HD video conferences, I'm pretty sure that "audiophile" exactly what they're going for. Personally as a remote worker, I would absolutely swoon for this.

ccostes 6y ago

Aside from the Rust aspect (which is cool!), I can't believe we've come this far and still don't have low-latency video conferencing. Maybe I'm overly sensitive, but people talking over each other and the lack of conversational flow drives me crazy with things like hangouts.

GuiA 6y ago
There are hard limits at play. No matter what you do, you can't go from New York to London in less than ~20ms; add video/audio encoding, packet switching, decoding, etc. and it's easy to see why any latency under the 100ms mark at that spatial scale in a scalable, mainstream product would be close to a miracle. The thing is that when we talk in a room, sound will take <10ms to reach my ears from your mouth. This is what "enables" all of the human turn taking cues in conversation (eye contact, picking up whether a sentence is about to end/whether it's a good time to chime in/etc) - I've been looking for work from people who've tried to see at what point things start feeling really bad (is it 10ms, or 50ms?), but haven't found much so far. No matter what it is though, it's likely that long distance digital communications just cannot match it. See also this interesting comment about the feeling of "closeness" from phone copper wires: https://news.ycombinator.com/item?id=22931809 https://news.ycombinator.com/item?id=22931809 Landlines were so fast and so "direct" in their latency (where distance correlates very directly with time, due to a lack of "hops") that local phone calls were faster than the speed of sound across a table, and for a bit after they came out--before people generally got used to seemingly random latency--local calls felt "intimate", like as if you were talking to someone in bed with their head right next to you; I also have heard stories of negotiators who had gotten really tuned to analyzing people's wait times while thinking that long distance calls were confusing and threw them off their game.
- jokoon 6y ago
  > it's easy to see why any latency under the 100ms mark at that spatial scale in a scalable, mainstream product would be close to a miracle. It seems normal phones are able to do it, though. At least it seems normal phones suffer less latency problem. In a way, simplicity in technology often means better performance.

vertex-four 6y ago

Are they still using WebRTC, just their own implementation? Or have they switched to something else on the wire?

markdog12 6y ago
There's a section in the article about it: "In the beginning (or: why we're not WebRTC)"
- vertex-four 6y ago
  I'm interested in what they are using if not WebRTC - there's several good options in this space (SRT would be my go-to choice), so it'd be really interesting to see if they rolled their own wire protocol or used something else.
  - beowulfey 6y ago
    They built it from scratch
    - vertex-four 6y ago
      Their blog post suggests they wrote something from scratch, but gives no clue as to what, whether they considered building on a more modern protocol specification than RTP (which is a couple decades old at this point), what they've taken from other more modern protocol specs if they didn't use one directly, or anything aside from that they wrote some code really.

nerdbaggy 6y ago

I wonder how much bandwidth this uses. The less bandwidth it uses the higher the latency because of compression. Its much easier to get low latency video when you have large (Gbit+) links

jonnypotty 6y ago

I wish I read more things like this on hn. "We wanted to know and understand every line of code being run on our hardware, and it should be designed for the exact hardware we wanted"

wmf 6y ago
As long as you're also comfortable reading the "over-engineering made our product inflexible, late to market, and too expensive" blog posts later.
- noir_lord 6y ago
  I mean you could also have the "we used commodity everything, where first to market but the next folks did it better and cheaper because they could" posts - hindsight is 20/20. I bet IBM didn't expect using off the shelf components would mean that the IBM PC was the standard for the next 30 years and it wouldn't be theirs. ----- As someone who works at a company that relied heavily on video conferencing (half the devs off shore) - every single major solution absolutely sucks, they are flaky, unreliable, sound quality is poor, video rate is poor (and this is with fat pipes at both ends) and worst of all latency, latency when trying to have a round table conversation with people remotely is horrific, it is good to see someone pushing the limits, Skype et al haven't gotten much better in the last decade yet my internet connection at home/work is x50 times faster and even mid range business laptops have much improved graphics grunt.
freeopinion 6y ago
"porting [webrtc-audio-processing] to Rust in the near-term is not likely (it's around 80k lines of C and C++ code)." That's just one of their dependencies. It's possible to know every line without rewriting. And it's possible to rewrite and still not know every line. They seem to strike a reasonable balance.
kingosticks 6y ago
But that statement seems at odds with a dependency on the enormous WebRTC AudioProcessing C++ module. But then they also say they don't use WebRTC so maybe I misunderstand what's going on.

GuiA 6y ago

Very cool from a tech standpoint. From a product point of view, I find it interesting that the illustrations/concept videos for these things always show people interacting very closely to the wall - e.g. playing chess, sitting around a table, etc. https://tonari.no/static/media/family.48218197.svg https://tonari.no/static/media/family.48218197.svg But in practice, people tend to keep their distance from it. E.g. the pictures of this setup tend to show people clustered in their own group on each side of the wall, with a solid 2-3 meters from the wall. https://blog.tonari.no/images/ea56c74d-a55d-4183-9a7b-d697954c5159-tonari-frontier-2.png.optimized.jpg https://blog.tonari.no/images/ea56c74d-a55d-4183-9a7b-d69795... It makes sense, it's awkward to be close to a large solid (emissive) surface, and humans instinctively get closer to their in group when faced with an out group. I wonder how the system could be designed to encourage participants being closer, if there is an advantage to that.

STRML 6y ago
A practical problem to solve there: where do you put the cameras? I would actually prefer putting them behind the screen if possible - a few small pinholes wouldn't be that noticeable. If you could put multiple wide-angle cameras in multiple places, you could stitch them together in software and create a real feeling of closeness.
renewiltord 6y ago
I'd sit closer but the picture then distorts and I am distorted for my conversation partner.

nerdbaggy 6y ago

If anybody is looking for a low latency high bandwidth P2P video streaming solution there is https://github.com/CESNET/UltraGrid/wiki https://github.com/CESNET/UltraGrid/wiki It can do less than 80ms of latency

ClumsyPilot 6y ago
This is cool, thanks for the link. Is this Nvidia GPU only? Mighr give it a try at some point
- nerdbaggy 6y ago
  You don't need the GPU, just depends on the type of compression you want. It supports intel VA-API as well as NVIDIA VDPAU
_visgean 6y ago
How do you use this over network? I have installed it but its very unclear to me what I need to do in order to call my colleague in another city. It seems that it can only connect to publicly visible hosts? Overall it looks like somebody should develop an application on top of this.

systemvoltage 6y ago

@dang - Suggest altering the title to say what it is "Achieving 3K, 60fps, 130ms Video Conferencing with Rust".

codefined 6y ago

Any suggestions on a group video conferencing tool for use on a local network (Ethernet) that's effective? Either self-hosted or online, just for personal usage to talk with others?

lc5G 6y ago

for crate in $(ls */Cargo.toml | xargs dirname); do cargo build Why do this instead of cargo --workspace build Is it so you can time the individual crates?

clpwn 6y ago
Yep!
ninkendo 6y ago
Yeah it looks like they wanted to know how long each crate took to build individually. But as long as we're nitpicking, nobody should just pipe `ls` into `xargs` like this, since it fails if anything has spaces in it. Instead, do: for cargo_toml in */Cargo.toml; do crate="$(dirname "${cargo_toml}")" pushd $crate # ... done Don't be that person who writes a script which won't tolerate spaces in filenames!
- OJFord 6y ago
  Alternatively: Don't be that person who clones the repo at a path with spaces in!
  - ninkendo 6y ago
    Not having spaces in your directory names is certainly a good idea, but I'll be damned if I let any of my code have issues with them. Just because something's a good idea doesn't mean it should be a requirement :) (The main reason for the advice of "Don't put spaces in paths" is really only because it breaks lots of poorly-written software... but that's not an excuse for your software to be poorly-written!)

pier25 6y ago

Woah this portal thing into another place seems super exciting if they can really pull it off and maintain low latency in the real world.

remmargorp64 6y ago

But does it have middle out compression?

zelly 6y ago

The bottleneck is not on the CPU. I'm afraid this company may have wasted their time trying to reinvent WebRTC. If you really want to get realtime video, I think the best approach is a custom codec on CUDA or better yet custom hardware (FPGA). You can only go so far on general purpose hardware before you hit a wall and get Zoom/WebEx quality.

busrf 6y ago
Can you recommend some resources for the current state of the art for low latency video? Somebody else in the comments posted https://github.com/CESNET/UltraGrid/wiki https://github.com/CESNET/UltraGrid/wiki, but I’m curious to learn more.
ronyfadel 6y ago
Is or is not? I’m confused: if the bottleneck is not on the CPU what does CUDA solve?
- zelly 6y ago
  The bottleneck is the video encoding/decoding/rendering, which is done on GPUs to begin with. Of course if it were done on CPU instead, then it would be significantly worse, but that's not where we're starting from. Improving stuff on the host side by, say, rewriting WebRTC in Rust won't improve the latency of your video by much or at all.

Scaevolus 6y ago

What's the codec stack for this? x264 --tune zerolatency + opus with opus_delay=20ms?

namibj 6y ago
20ms is wasteful. Use minimum latency where SILK still works, afaik that's 7.5ms.
- codys 6y ago
  This assumes that video encoding latency is lower than the audio latency.
  - namibj 6y ago
    Shouldn't be more than a single frame, which is 16 2/3 ms for 60 fps. And for e.g. JPEG it can be even shorter, especially with a rolling shutter.

usefulcat 6y ago

"We wanted to know and understand every line of code being run on our hardware, and it should be designed for the exact hardware we wanted." This rings very true for every high-performance thing I've ever worked on, from games to trading systems.

swsieber 6y ago

This is welcome news. I have been itching to convert a small headshot videostream (thing under 100x100px) to audio, stream it over mumble and then convert it back to video, just to see what the latency is like. It would obviously be a big undertaking, but not as big as this methinks.

LockAndLol 6y ago

After reading it, I'm still not entirely sure what's being done. Is it live streaming or is it the transport? Are they doing video encoding (the audio encoding seems to be done by that webrtc-audio thing)? Have they chosen a progressive encoding format that compresses frames and pumps them out to the wire as soon as they're done? Is TCP or UDP involved or a new Layer 3 protocol entirely? Have I just missed all of those parts or were they really missing amid all the Rust celebration?

clpwn 6y ago
> After reading it, I'm still not entirely sure what's being done. > Is it live streaming or is it the transport? tonari is the entire stack, similar in "feature scope" to WebRTC but with different goals and target environments. > Are they doing video encoding (the audio encoding seems to be done by that webrtc-audio thing)? Yep, this includes video encoding and transport. We don't use the WebRTC audio library for encoding or transport, just for echo cancellation and other helpful acoustic processing. > Have they chosen a progressive encoding format that compresses frames and pumps them out to the wire as soon as they're done? Yep, basically, if by that you mean we don't use B-frames or other codec features that would require buffering multiple video frames before receiving a compressed stream, so we're able to send out encoded frames as they arrive. > Is TCP or UDP involved or a new Layer 3 protocol entirely? We encapsulate our protocol in UDP since we operate on normal internet - a new protocol is out of the question without a huge lobbying force and 15 years of patience on your side. > Have I just missed all of those parts or were they really missing amid all the Rust celebration? We intentionally didn't get into the protocol details because we are saving that for a dedicated post (and code to back it up).
- LockAndLol 6y ago
  Thank you very much for the answers. Glad I wasn't too far off. Looking forward to the technical post. If you're planning on releasing all of this royalty-free and opensource, you'd be quite a boon to the free and open internet. Getting this picked up by the likes of Mozilla and getting it into a browser would be amazing.

MR4D 6y ago

Gotta love a writeup with this line in it: like Brian's 1970s-era MacBook Pro That's a writer(s) who knows what it's like to read long (aka thorough) technical articles and not bore the readers to death. Great article!

alpineidyll3 6y ago

Awesome post

eadan 6y ago

This is amazing! The first thing that popped to my mind seeing the life sized "portal" was the farcaster portals from the sci-fi novel Hyperion https://hyperioncantos.fandom.com/wiki/Farcaster https://hyperioncantos.fandom.com/wiki/Farcaster

imtringued 6y ago

My WebRTC projects haven't suffered that much from latency. The biggest source of delays is usually caused by encoding video for me. I've had to limit streams to 720p and 25fps to reduce the time spent on CPU encoding a vp8 stream. There are also bandwidth considerations (real time encoding = significantly less compression) but the end result is slightly less than 200ms one way latency (including input lag from mouse, 15ms network latency and display lag) without any special settings. All I'm doing is feeding a ffmpeg stream to kurento and letting it broadcast it via WebRTC. This is not a web conferencing application and it is also not using WebRTC via p2p. It's closer to conventional live streaming with a sane amount of latency (compared to up to 30s of latency you commonly see on twitch). Of course I personally would prefer it if the latency can be brought down even further. 100ms or lower is like the holy grail for me and only appears to be doable with codecs that aren't supported by WebRTC. However, people don't want to install apps just for my little service and I certainly won't encode every stream via several codecs just for the tiny minority of the user base that actually ends up using the app.

chubs 6y ago

Sounds impressive, but i'm dying to know: what video codec are they using?

novok 6y ago

I wonder how it compares with apple facetime on two new macbooks with ethernet connections on both sides. They actually work on reducing latency and pushing high res video if your connection supports it.

bschwindHN 6y ago
That's a great idea, I've always preferred facetime at least for the video quality. We'll do a latency test sometime, I suspect it'll be quite good!

jupp0r 6y ago

I love Rust, but them deciding to redesign/reimplement webrtc after being frustrated after a week seems like a prime candidate for not invented here syndrome with Rust being the justification. There is a reason webrtc is as big as it is, it’s a complex problem to solve. Regarding the premise of high latency in webrtc: Google Stadia has ~160ms round trip latency at 4k from my Macbook to a data center, so it’s not like that’s unachievable.

Apofis 6y ago
Google is colocating in your basement.

jonny383 6y ago

"we truly don't believe we could have achieved these numbers with this level of stability without Rust" Oh please. This is just rust sensationalism. People don't truly believe rust is faster than C do they?

gpm 6y ago
Developing stable complex software in C takes a hell of a lot more effort and skill than it does in Rust IMO.
bufferoverflow 6y ago
In some problems Rust is the fastest: https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/spectralnorm.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/revcomp.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/binarytrees.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- jonny383 6y ago
  Every single one of those rust implementations is using unsafe {}, thus defeating the purpose of using rust in the first place. Run the same benchmark without unsafe {}.
  - turndown 6y ago
    >defeating the purpose of using rust in the first place I don't think this is true; the whole point of Rust is that unsafe operations are explicit, not that you never do so. Also, I looked at the first one, and it's only using unsafe on what are basically op code calls; I don't think it is realistic to complain about that.
  - igouy 6y ago
    Not true. Of those 4 tasks, the rust programs for task 3 and task 4 do not use the keyword "unsafe". For task 2, the spectral-norm Rust #6 program does use "unsafe" but #5 does not and it's almost as fast.

Exuma 6y ago

I love their homepage https://tonari.no/ https://tonari.no/

deleted 6y ago
[deleted]
aiotokyo 6y ago
Thank you so much! Keep an eye for further updates.

zamalek 6y ago

> We just enforce rustfmt. After interaction with both rustfmt and go fmt, I have concluded that .editorconfig is solving a problem that really shouldn't be solved. We went through the ordeal of defining our C# coding standards where I work and, let me tell you, people (myself included) care very deeply about their way of structuring code. And it's a bloody waste of their time. Having the language designers say, "here is how our language should be structured" is a breath of fresh air.

sephamorr 6y ago

Why exactly do existing video streaming solutions use such small amounts of bandwidth and have terrible quality as a result? Does anyone have a deep dive into why this is the case? It seems that it would be a killer feature to make better utilization of bandwidth. Even over wifi, speedtest shows 4ms/100mb/100mb on my internet connection, but Zoom, FaceTime, and others never use more than about 0.8Mbit/s for a video stream, and the resulting quality of audio and video is...understandably poor. Latency too totally feels like a software problem, perhaps with too many layers of abstraction. (60fps->16ms for the camera, ~10ms for encoding with NVENC/equivalents, 35ms measured one-way latency from my laptop to my parents 4000km away, ~10ms decode, 16ms frame delay = 87ms one way). Maybe I'm asking for too much from non-realtime systems (I'm used to RTOS, extensive use of DMA, zero-copy network drivers, etc), but it seems that there is a lot of room to improve.

rasz 6y ago
OnLive "solved" encoder latency 15 years ago. You dont wait 16ms for the next frame. Instead you progressively start encoding after receiving first tens of lines. This way your encoded video stream lags just couple of milliseconds behind, same for decoding. You could crudely emulate this by dividing screen into 4 rows and sending 4 concurrent video stream, instant 1/4 latency drop.
- sephamorr 6y ago
  Sure, many of the operations in the list can be pipelined as you mention. Something like G-Sync would also allow you to sync the destination display to the arrival of the (start of) frame.
clpwn 6y ago
It's worth mentioning that in our case, a significant chunk of the latency in our 130ms measurement is just the input lag of our display that we currently use. We were surprised by how slow they can be.

realchucknorris 6y ago

would loved to see a demo

dbrgn 6y ago

"A week of struggling with WebRTC’s nearly 750,000 LoC behemoth of a codebase revealed just how painful a single small change could be — how hard it was to test, and feel truly safe, with the code you were dealing with." I totally feel you. It's impressive what the WebRTC implementation has achieved, but it's just not pleasant at all to work with it.

snvzz 6y ago

130ms is a world better than 500ms and a much welcome improvement, but it is still terrible. Latency happens throughout the whole stack; Unfortunately much would need to be fixed outside this project to achieve any further significant improvement. Operating System, firmware, blackbox hardware are some other non-negligible sources of latency. Everything adds up.