To me, the the most interesting part of Bret Victors ideas lie in the re-embodiment of disembodied interactions. The computers choreograph us anyhow, but often

To me, the the most interesting part of Bret Victors ideas lie in the re-embodiment of disembodied interactions. The computers choreograph us anyhow, but often in a very poor and limited way – only our fingertips and our eyes move a bit. Why not make the choreography of interaction richer? Why not create a computer-aided full-body choreography of interaction? In a way, the interactions can be broken up in subject, verb and object. I edit text. I crop a photograph. The subject is the user, and all the representations that extend the user. The mouse pointer in that sense is part of the user. The verb is the action. To select text on screen, I put pressure on a touchpad, move my fingers and release. This is a learned interaction, there are no inherent affordances to a medium (only, at best, inherent affordances to a tool that extends you). The object is a representation. With computer interfaces, you hardly ever interact with something in an immediate way. I want a comment to appear on this site but instead I am writing this text in a white box and not where the comment would appear. All the computer interactions are mediated by these in-between steps. (An example for unmediated interaction would be cooking. What you chop is what you get.) To get a rich feedback loop in such a mediated environment you need to try and make it as unmediated as possible. To make the subject less mediated, the whole body of the user, the quality of the movements could be integrated into integrations. Here I have hopes for AI supported movement recognition. In addition, the representations of the user (e.g. the mouse pointer in existing systems) could become less binary in their state (I don't have a good example for it, but a hammer can be used with a whole range of intensity while a mouse pointer either performs a click or not). To make the object less mediated, its representation should ideally be as transparent as possible. AR in this regard is much more promising than VR, since in VR the representation takes place within another representation. To make the action less mediated, the action should be able to be embodied by the (technologically extended) user, as well as being inherent to the medium which represents the subject. Here we are building a bridge between the human body and a way to represent things that physically do not exist. It's never going to be ideal, but it could be better than what we have now. So for such a task you'd need to be a UX person, a choreographer, a programmer, a AR/AI person, and ideally have some insights into media theory. It's just not an easy task.