5 ms·

How to Animate Multiplayer Cursors

235 points by stevenfabre 4y ago

agitator 4y ago
I was hoping that this article would go into deeper techniques than just interpolation and splines. I would be cool to see an example of how a Kalman filter approach would compare in terms of precision and latency. My expectation is that it would be the best of both worlds.
- mcphage 4y ago
  Or even a physics simulation—if the mouse is moving with a given velocity, that velocity won't change super fast. So even if your data is a bit behind, you can use physics to estimate where they probably are now. And if you get new data, and your estimation is too far off, then move the mouse to the right spot. If it's mostly correct, just base your future simulation off of the new information, so that it moves smoothly towards the correct value.
EllenChisa 4y ago
Loved the deep dive into the difference between timing functions and spring/spline animations (ctrl-f comparison to jump to the animation).
sghosh2 4y ago
The animations on the blog itself are incredible. Would love to read something in the future about how you guys put this together (i.e. what frameworks, components, tooling is used to generate these blogs).
- stevenfabre 4y ago
  That would be pretty meta! "how to create how to interactive articles" :)
  - cableshaft 4y ago
    I'm super interested in this. I have a few interactive articles I'd like to make, but I'm not sure what tech to use for it to make it relatively quickly and without taxing the page too much. Was thinking maybe the Phaser game framework, but that might be too heavy, especially if there needs to be 7 or 8 of them on the same page.
    - stevenfabre 4y ago
      We build our interactive visuals with React and embed those into the mdx file for the blog post. I made note for us to write a blog post about it. Stay tuned :)
      - cableshaft 4y ago
        Already learned something. Wasn't aware of mdx and it looks right up my alley, thanks for mentioning it. I look forward to your blog post!
- JayStavis 4y ago
  This may interest you https://www.joshwcomeau.com/blog/how-i-built-my-blog/ https://www.joshwcomeau.com/blog/how-i-built-my-blog/
steventey 4y ago
The interactive animations in this post are mind-blowingly good!!!
- JamesSwift 4y ago
  I opened the article just to check and you're right. This is A+, top-tier stuff. Sets a standard for many other technical articles.
jewel 4y ago
There's another approach that might work better. Instead of sampling the mouse position every 100ms, you'd save off all the mouse positions, and then send the latest batch every 100ms. The other side would then replay the exact positions, just delayed by 100ms. It'll end up with the same latency as these motion smoothing approaches, while only using slightly more bandwidth.
- cousin_it 4y ago
  Yeah, wanted to say the same thing. If latency is the only problem, just batch up the updates and replay on the other end. If bandwidth also becomes a problem, only then start compressing the data. But to be honest, if we can stream video over the internet, we surely have enough bandwidth to stream cursor positions.
  - stevenfabre 4y ago
    Good point, it's also important to consider this in the context of the application that will use those multiplayer cursors. It's important that the state of the document matches the cursors, so it makes sense to have both presence (cursors, selection) and storage (document data) be perfectly in sync even if that means having a slight 80-200ms delay.
  - Aeolun 4y ago
    I think this is one of those situations where “just because we can doesn’t mean we should” applies. The marginal benefit of exact cursor positions is so low that sending all that data still feels like a waste.
  - zaptrem 4y ago
    Could also have a positive impact on battery life as it would let the WiFi/cellular radio sleep longer.
- acomms 4y ago
  I like this solution but would latency create issues? You'd only be sending 100ms worth of motion every ~120ms. Would you just drift 200ms behind with every passing second (20ms behind after each 100ms batch)? I think I may be missing something though.
vincentriemer 4y ago
Another potential approach could be using the Web Animations API with an additive/relative animation approach via the `composite: 'add'` option. I say potential as it really only fully works in Chrome atm — Firefox has it but has weird rendering bugs and Safari says it has experimental support but I've never managed to get it working. I like the potential of this approach as it lets you get smoother results than just CSS transitions yet doesn't require you to use a RAF loop-based animation library.
Stampo00 4y ago
The spline stuff is super useful to me for another non-cursor idea I've had simmering. But I'm torn by this. The spline approach seems to look the most accurate. But when all three approaches are shown at the same time at the end, I think the spring animation might look more visually pleasing. But then, if the spring approach is only degrees better than CSS transitions, is it really worth all the extra code?
- Etherlord87 4y ago
  You're comparing the end-path, but the CSS with easing just isn't smooth, because the easing abruptly stops when the target changes. Without easing you still have abrupt changes in cursor speed. In the end, as calculated above, you only need 90 bytes per second for 30 FPS cursor movement, so why bother... Maybe in your case there's more objects and then it makes sense to approximate, but then if you can deal with some CPU usage for that, you can build a detailed bezier curve, and then unsubdivide it where it wouldn't change the overall shape much (probably detectable with segment length). This way the approximation is done on the sender that has the entire data, rather than on the receiver.
AnhTho_FR 4y ago
Awesome deep dive!
schneegansmarie 4y ago
I always felt like using a CSS ease transition between points would be enough. Love the spline approach with perfect-cursors, cursors really feel like they're being moved by real people.
fullstackwife 4y ago
It depends on the app, but in some cases you should also include content identifier to the usual X,Y coordinates, as from the user perspective it is more important to see the cursor hovering on top of something instead of being "very close". The meaning is different in such case.
- stevenfabre 4y ago
  Exactly! With Liveblocks, you can also store anything presence related. You can for instance store a selectedId and use that on the other side to highlight the selected element.
dwallin 4y ago
I think it's a mistake to just drop most of the data and then try to rebuild the lost data afterwards. Nothing requires you send just a set of points across. A better approach is to transform and compress the data BEFORE sending it across the wire. This way you get the benefit of using all the available data to create a more accurate simplification. For example, each update take your cursor point set and construct a bezier curve that best fits the data.
- philsnow 4y ago
  Nobody will ever see this response since the article was posted 3 days ago but: Another advantage of doing it this way is that the sender can downsample on large/gross movements that would render faithfully as splines with fewer points, and send more samples on tight movements. If you're not sending uniformly-spaced (in time) samples, though, you'll want some kind of timing information encoded as well.
somehnacct3757 4y ago
This is analogous to player position in any multiplayer game. One additional parameter to keep in mind is how real-time does your simulation of remote players need to be? If you don't need real-time positioning there's a whole other dimension of shortcuts you can take by introducing what amounts to a 'streaming delay'. In a lot of these apps the cursors can't interact with each other so you have no need for real-time positioning and its accompanying smoothing techniques. Cheat cheat cheat! That's how games get their performance, way more often than being smart they're clever opportunistic cheaters!
- capableweb 4y ago
  > This is analogous to player position in any multiplayer game. Not sure. Multiplayer games are easier to predict player movement. Once someone starts moving forward, you can predict that they'll move forward for a little while, start turning, continue forward and so on. Add in physics (like the motions/movements of a car, or the running of a human) and there will be constraints the player can't break (when you stop moving forward, you'll move forward slower and slower until you stop, maybe over 100ms or so) But mouse movement is highly erratic. It'll be short of impossible to add any sort of prediction, as it'll be incorrect most of the time, instead of being mostly correct but sometimes not.
  - tomovo 4y ago
    The real fun starts when online players start shooting at each other. How to decide if a hit was fair AND make it look believable for both parties to minimize outrage?
    - Aeolun 4y ago
      > AND make it look believable for both parties to minimize outrage? Call of Duty MW2 used to replay what your opponent saw, after your death. That always made an inexplicable death much easier to swallow. Of course, they probably had to add that feature since they did away with dedicated servers, but it was still nice.
  - 4y ago
DonHopkins 4y ago
I love this article! Great analysis and solutions. It's nice when user interface designers care enough to think through, implement, try out, and refine such important details. I developed a multi player version of SimCity for X11 that I released in 1993 and demonstrated at the InterCHI '93 Interactive Experience, which showed you other player's cursors moving around and editing the map, but of course it required a fast network to run on and updated all the clients synchronously, due to the limitations of X-Windows, so there were no interpolation issues. (X-Windows clients aren't even capable of performing local computation and feedback the way NeWS clients could and web browser clients now can, so it was a moot point.) https://www.donhopkins.com/home/catalog/simcity/simcity-announcement.html https://www.donhopkins.com/home/catalog/simcity/simcity-anno... https://www.donhopkins.com/home/catalog/simcity/simcitynet.html https://www.donhopkins.com/home/catalog/simcity/simcitynet.h... The multi player demo showing different kinds of voting dialogs, multiple cursors, the tool palette and pie menus, and the voting "yes" shortcut of building the same thing in the same place, starts at 5m45s: https://www.youtube.com/watch?v=_fVl4dGwUrA&t=5m45s https://www.youtube.com/watch?v=_fVl4dGwUrA&t=5m45s One interesting thing about the SimCity cursor was that it was color and shape coded to show which tool was selected. The tool palette (and also the pie menus which had the same icons and layout as the tool palette) served as a legend for the cursor by showing the same color coded outline around the icons as the cursor used. So you could tell which tool other users had selected. You could hide the tool palette to make the map bigger, and use the pie menus instead, which were much more efficient. Multi Player SimCityNet for X11 on Linux: Demo of the latest optimized Linux version of Multi Player SimCity for X11. Ported to Unix, optimized for Linux and demonstrated by Don Hopkins: https://www.youtube.com/watch?v=_fVl4dGwUrA https://www.youtube.com/watch?v=_fVl4dGwUrA Micropolis Online (SimCity) Web Demo: A demo of the open source Micropolis Online game (based on the original SimCity Classic source code from Maxis), running on a web server, written in C++ and Python, and displaying in a web browser, written in OpenLaszlo and JavaScript, running in the Flash player. Developed by Don Hopkins: https://www.youtube.com/watch?v=8snnqQSI0GE https://www.youtube.com/watch?v=8snnqQSI0GE Source Code: https://github.com/SimHacker/micropolis https://github.com/SimHacker/micropolis HAR 2009 talk: Constructionist Educational Open Source SimCity: https://donhopkins.medium.com/har-2009-lightning-talk-transcript-constructionist-educational-open-source-simcity-by-don-3a9e010bf305 https://donhopkins.medium.com/har-2009-lightning-talk-transc...
paulryanrogers 4y ago
These are pointers and not cursors, unless I misunderstand. Cursor made me think of sharing tmux or IDE text sessions.
- Sohcahtoa82 4y ago
  Not only is this is needless pedantry, it's also incorrect, making it a very typical HN comment. It is not uncommon to refer to pointers as "mouse cursors" or shorten it to just "cursor". In fact, Wikipedia considers pointers to be a subset of cursors. https://en.wikipedia.org/wiki/Cursor_(user_interface)#Pointer https://en.wikipedia.org/wiki/Cursor_(user_interface)#Pointe... "In computer user interfaces, a cursor is an indicator used to show the current position for user interaction on a computer monitor or other display device that will respond to input from a text input or pointing device."
  - franklampard 4y ago
    Exactly. In css this is exactly called cursor:pointer/crosshair
  - paulryanrogers 4y ago
    Is it so needless? Perhaps I'm misremembering but the distinction was at least useful in the past as GUIs evolved out of TUIs. If they had said pointers there would be no confusion.
    - Sohcahtoa82 4y ago
      Since its wrong, I'd think it's basically needless by definition.
bluelightning2k 4y ago
Can we just take a moment to appreciate how well done this article is. It's like the AAA studio equivalent of a tech article.
deleted 4y ago
[deleted]
blt 4y ago
Would be interested to hear a signal processing perspective on this problem. I feel like splines might be susceptible to creating false information (e.g. ringing/overshoot), but I don't know enough about it.
ynx 4y ago
Not quite the same thing, but the author (and others) might be interested in some modern techniques used in the game industry to filter inputs: https://twitter.com/evil_arev/status/1128062338156900353 https://twitter.com/evil_arev/status/1128062338156900353 https://www.shadertoy.com/view/NlcXWM https://www.shadertoy.com/view/NlcXWM
franklampard 4y ago
Good article!
deleted 4y ago
[deleted]

235 points by stevenfabre 4y ago

I was hoping that this article would go into deeper techniques than just interpolation and splines. I would be cool to see an example of how a Kalman filter approach would compare in terms of precision and latency. My expectation is that it would be the best of both worlds.

mcphage 4y ago
Or even a physics simulation—if the mouse is moving with a given velocity, that velocity won't change super fast. So even if your data is a bit behind, you can use physics to estimate where they probably are now. And if you get new data, and your estimation is too far off, then move the mouse to the right spot. If it's mostly correct, just base your future simulation off of the new information, so that it moves smoothly towards the correct value.

EllenChisa 4y ago

Loved the deep dive into the difference between timing functions and spring/spline animations (ctrl-f comparison to jump to the animation).

sghosh2 4y ago

The animations on the blog itself are incredible. Would love to read something in the future about how you guys put this together (i.e. what frameworks, components, tooling is used to generate these blogs).

stevenfabre 4y ago
That would be pretty meta! "how to create how to interactive articles" :)
- cableshaft 4y ago
  I'm super interested in this. I have a few interactive articles I'd like to make, but I'm not sure what tech to use for it to make it relatively quickly and without taxing the page too much. Was thinking maybe the Phaser game framework, but that might be too heavy, especially if there needs to be 7 or 8 of them on the same page.
  - stevenfabre 4y ago
    We build our interactive visuals with React and embed those into the mdx file for the blog post. I made note for us to write a blog post about it. Stay tuned :)
    - cableshaft 4y ago
      Already learned something. Wasn't aware of mdx and it looks right up my alley, thanks for mentioning it. I look forward to your blog post!
JayStavis 4y ago
This may interest you https://www.joshwcomeau.com/blog/how-i-built-my-blog/ https://www.joshwcomeau.com/blog/how-i-built-my-blog/

steventey 4y ago

The interactive animations in this post are mind-blowingly good!!!

JamesSwift 4y ago
I opened the article just to check and you're right. This is A+, top-tier stuff. Sets a standard for many other technical articles.

jewel 4y ago

There's another approach that might work better. Instead of sampling the mouse position every 100ms, you'd save off all the mouse positions, and then send the latest batch every 100ms. The other side would then replay the exact positions, just delayed by 100ms. It'll end up with the same latency as these motion smoothing approaches, while only using slightly more bandwidth.

cousin_it 4y ago
Yeah, wanted to say the same thing. If latency is the only problem, just batch up the updates and replay on the other end. If bandwidth also becomes a problem, only then start compressing the data. But to be honest, if we can stream video over the internet, we surely have enough bandwidth to stream cursor positions.
- stevenfabre 4y ago
  Good point, it's also important to consider this in the context of the application that will use those multiplayer cursors. It's important that the state of the document matches the cursors, so it makes sense to have both presence (cursors, selection) and storage (document data) be perfectly in sync even if that means having a slight 80-200ms delay.
- Aeolun 4y ago
  I think this is one of those situations where “just because we can doesn’t mean we should” applies. The marginal benefit of exact cursor positions is so low that sending all that data still feels like a waste.
- zaptrem 4y ago
  Could also have a positive impact on battery life as it would let the WiFi/cellular radio sleep longer.
acomms 4y ago
I like this solution but would latency create issues? You'd only be sending 100ms worth of motion every ~120ms. Would you just drift 200ms behind with every passing second (20ms behind after each 100ms batch)? I think I may be missing something though.

vincentriemer 4y ago

Another potential approach could be using the Web Animations API with an additive/relative animation approach via the `composite: 'add'` option. I say potential as it really only fully works in Chrome atm — Firefox has it but has weird rendering bugs and Safari says it has experimental support but I've never managed to get it working. I like the potential of this approach as it lets you get smoother results than just CSS transitions yet doesn't require you to use a RAF loop-based animation library.

Stampo00 4y ago

The spline stuff is super useful to me for another non-cursor idea I've had simmering. But I'm torn by this. The spline approach seems to look the most accurate. But when all three approaches are shown at the same time at the end, I think the spring animation might look more visually pleasing. But then, if the spring approach is only degrees better than CSS transitions, is it really worth all the extra code?

Etherlord87 4y ago
You're comparing the end-path, but the CSS with easing just isn't smooth, because the easing abruptly stops when the target changes. Without easing you still have abrupt changes in cursor speed. In the end, as calculated above, you only need 90 bytes per second for 30 FPS cursor movement, so why bother... Maybe in your case there's more objects and then it makes sense to approximate, but then if you can deal with some CPU usage for that, you can build a detailed bezier curve, and then unsubdivide it where it wouldn't change the overall shape much (probably detectable with segment length). This way the approximation is done on the sender that has the entire data, rather than on the receiver.

AnhTho_FR 4y ago

Awesome deep dive!

schneegansmarie 4y ago

I always felt like using a CSS ease transition between points would be enough. Love the spline approach with perfect-cursors, cursors really feel like they're being moved by real people.

fullstackwife 4y ago

It depends on the app, but in some cases you should also include content identifier to the usual X,Y coordinates, as from the user perspective it is more important to see the cursor hovering on top of something instead of being "very close". The meaning is different in such case.

stevenfabre 4y ago
Exactly! With Liveblocks, you can also store anything presence related. You can for instance store a selectedId and use that on the other side to highlight the selected element.

dwallin 4y ago

I think it's a mistake to just drop most of the data and then try to rebuild the lost data afterwards. Nothing requires you send just a set of points across. A better approach is to transform and compress the data BEFORE sending it across the wire. This way you get the benefit of using all the available data to create a more accurate simplification. For example, each update take your cursor point set and construct a bezier curve that best fits the data.

philsnow 4y ago
Nobody will ever see this response since the article was posted 3 days ago but: Another advantage of doing it this way is that the sender can downsample on large/gross movements that would render faithfully as splines with fewer points, and send more samples on tight movements. If you're not sending uniformly-spaced (in time) samples, though, you'll want some kind of timing information encoded as well.

somehnacct3757 4y ago

This is analogous to player position in any multiplayer game. One additional parameter to keep in mind is how real-time does your simulation of remote players need to be? If you don't need real-time positioning there's a whole other dimension of shortcuts you can take by introducing what amounts to a 'streaming delay'. In a lot of these apps the cursors can't interact with each other so you have no need for real-time positioning and its accompanying smoothing techniques. Cheat cheat cheat! That's how games get their performance, way more often than being smart they're clever opportunistic cheaters!

capableweb 4y ago
> This is analogous to player position in any multiplayer game. Not sure. Multiplayer games are easier to predict player movement. Once someone starts moving forward, you can predict that they'll move forward for a little while, start turning, continue forward and so on. Add in physics (like the motions/movements of a car, or the running of a human) and there will be constraints the player can't break (when you stop moving forward, you'll move forward slower and slower until you stop, maybe over 100ms or so) But mouse movement is highly erratic. It'll be short of impossible to add any sort of prediction, as it'll be incorrect most of the time, instead of being mostly correct but sometimes not.
- tomovo 4y ago
  The real fun starts when online players start shooting at each other. How to decide if a hit was fair AND make it look believable for both parties to minimize outrage?
  - Aeolun 4y ago
    > AND make it look believable for both parties to minimize outrage? Call of Duty MW2 used to replay what your opponent saw, after your death. That always made an inexplicable death much easier to swallow. Of course, they probably had to add that feature since they did away with dedicated servers, but it was still nice.
- 4y ago

DonHopkins 4y ago

I love this article! Great analysis and solutions. It's nice when user interface designers care enough to think through, implement, try out, and refine such important details. I developed a multi player version of SimCity for X11 that I released in 1993 and demonstrated at the InterCHI '93 Interactive Experience, which showed you other player's cursors moving around and editing the map, but of course it required a fast network to run on and updated all the clients synchronously, due to the limitations of X-Windows, so there were no interpolation issues. (X-Windows clients aren't even capable of performing local computation and feedback the way NeWS clients could and web browser clients now can, so it was a moot point.) https://www.donhopkins.com/home/catalog/simcity/simcity-announcement.html https://www.donhopkins.com/home/catalog/simcity/simcity-anno... https://www.donhopkins.com/home/catalog/simcity/simcitynet.html https://www.donhopkins.com/home/catalog/simcity/simcitynet.h... The multi player demo showing different kinds of voting dialogs, multiple cursors, the tool palette and pie menus, and the voting "yes" shortcut of building the same thing in the same place, starts at 5m45s: https://www.youtube.com/watch?v=_fVl4dGwUrA&t=5m45s https://www.youtube.com/watch?v=_fVl4dGwUrA&t=5m45s One interesting thing about the SimCity cursor was that it was color and shape coded to show which tool was selected. The tool palette (and also the pie menus which had the same icons and layout as the tool palette) served as a legend for the cursor by showing the same color coded outline around the icons as the cursor used. So you could tell which tool other users had selected. You could hide the tool palette to make the map bigger, and use the pie menus instead, which were much more efficient. Multi Player SimCityNet for X11 on Linux: Demo of the latest optimized Linux version of Multi Player SimCity for X11. Ported to Unix, optimized for Linux and demonstrated by Don Hopkins: https://www.youtube.com/watch?v=_fVl4dGwUrA https://www.youtube.com/watch?v=_fVl4dGwUrA Micropolis Online (SimCity) Web Demo: A demo of the open source Micropolis Online game (based on the original SimCity Classic source code from Maxis), running on a web server, written in C++ and Python, and displaying in a web browser, written in OpenLaszlo and JavaScript, running in the Flash player. Developed by Don Hopkins: https://www.youtube.com/watch?v=8snnqQSI0GE https://www.youtube.com/watch?v=8snnqQSI0GE Source Code: https://github.com/SimHacker/micropolis https://github.com/SimHacker/micropolis HAR 2009 talk: Constructionist Educational Open Source SimCity: https://donhopkins.medium.com/har-2009-lightning-talk-transcript-constructionist-educational-open-source-simcity-by-don-3a9e010bf305 https://donhopkins.medium.com/har-2009-lightning-talk-transc...

paulryanrogers 4y ago

These are pointers and not cursors, unless I misunderstand. Cursor made me think of sharing tmux or IDE text sessions.

Sohcahtoa82 4y ago
Not only is this is needless pedantry, it's also incorrect, making it a very typical HN comment. It is not uncommon to refer to pointers as "mouse cursors" or shorten it to just "cursor". In fact, Wikipedia considers pointers to be a subset of cursors. https://en.wikipedia.org/wiki/Cursor_(user_interface)#Pointer https://en.wikipedia.org/wiki/Cursor_(user_interface)#Pointe... "In computer user interfaces, a cursor is an indicator used to show the current position for user interaction on a computer monitor or other display device that will respond to input from a text input or pointing device."
- franklampard 4y ago
  Exactly. In css this is exactly called cursor:pointer/crosshair
- paulryanrogers 4y ago
  Is it so needless? Perhaps I'm misremembering but the distinction was at least useful in the past as GUIs evolved out of TUIs. If they had said pointers there would be no confusion.
  - Sohcahtoa82 4y ago
    Since its wrong, I'd think it's basically needless by definition.

bluelightning2k 4y ago

Can we just take a moment to appreciate how well done this article is. It's like the AAA studio equivalent of a tech article.

deleted 4y ago

[deleted]

blt 4y ago

Would be interested to hear a signal processing perspective on this problem. I feel like splines might be susceptible to creating false information (e.g. ringing/overshoot), but I don't know enough about it.

ynx 4y ago

Not quite the same thing, but the author (and others) might be interested in some modern techniques used in the game industry to filter inputs: https://twitter.com/evil_arev/status/1128062338156900353 https://twitter.com/evil_arev/status/1128062338156900353 https://www.shadertoy.com/view/NlcXWM https://www.shadertoy.com/view/NlcXWM

franklampard 4y ago

Good article!

deleted 4y ago

[deleted]