6 ms·

I've built an app that has the same goals (not operate a mouse) but approach it completely different. Rather than try to simulate the moving the mouse itself,

I've built an app that has the same goals (not operate a mouse) but approach it completely different. Rather than try to simulate the moving the mouse itself, Shortcat [https://shortcat.app/ https://shortcat.app/] indexes the user interface (buttons, text fields, links, menus, etc) and enables fast fuzzy search of the interface. Type a word, abbreviations, or hints and hit Enter to click or action the element. Works almost everywhere on macOS, including browsers, Electron apps, and even iOS apps! The goal is to minimise cognitive overhead to achieve a particular intent, so being able to type a word to hit a button, or active a deep menu item when you don't know the shortcut is quick and easy. I'm currently working on a modal option which enables staying within Shortcat to navigate an interface, as well as chords for simulating scrolling and arrow keys. Shortcat relies on using the Accessibility API to index UI elements however, and is dependent on how well an app or website has implemented it. One of the goals is to help improve accessibility implementations by exposing more people to its implementations and pushing for developers to fix broken or incorrectly implemented accessibility tagging. Shortcat is macOS only for now as I haven't been able to investigate how viable doing this on Windows or Linux would be, especially on Linux considering all the different toolkits that exist.

LASR 4y ago
This is excellent. I will be trying this out.
kache_ 4y ago
Have you considered using ML/OCR to figure out the position of the text relative to the screen? Seems much simpler than relying on accessibility APIs Thank you for your hard work!
- _chendo_ 4y ago
  I have plans to use ML/OCR to augment results down the road but the AX APIs and ecosystem on most apps (that I encounter, at least) are generally decent. Also, OCR means it won’t understand buttons with just icons, whereas AX APIs can grab em just fine. Thanks! It’s easily my longest running project at a decade
hawski 4y ago
Very interesting thing. I wonder if the gap of apps not supporting a11y could be reduced by using Tesseract to OCR the text.
- ancienthope 4y ago
  That's what https://superkey.app/ https://superkey.app/ does.
discodachshund 4y ago
Excited to try this out! Is it planned to open source? I would love to try integrating this into Raycast
- punnerud 4y ago
  Is Raycast open source? Could only find that the plugins are on GitHub.
- _chendo_ 4y ago
  It won't be open source, but I will be adding an API so it can be integrated with other apps and scripted
Luc 4y ago
This app is quality. I can tell you have been working on it for years. Why not charge for it?
- _chendo_ 4y ago
  Haha, thanks :) I did charge for it a couple of years ago, however I rebuilt the whole thing from scratch after a long hiatus and hadn't had bothered to reimplement licensing because the existing options all kinda suck, and figured I'd focus the time on features and usability first. I think with the modal mode in the next release will bring it much closer to a 1.0 release.
  - justusw 4y ago
    If you bundle it and release a paid for application on the App Store, I would totally buy it and even roll it out to my staff. The magic of the App Store allows you to do company wide roll-outs quite easily.
    - _chendo_ 4y ago
      I'm not sure if an app like Shortcat can be released on the App Store given it uses the Accessibility APIs (sandboxing etc), also the 15-30% cut they take is a bit ooooof, but I do have plans to support company/teams licensing!
  - tfsh 4y ago
    +1 this is awesome! I'd like to donate if I can :) edit: nvm me, found the option in settings (on activation show shortcuts immediately). Quick question, I've been playing around with Shortcat for a while. When I press the activation hot-key it takes about 4 seconds for the yellow two-letter denoated highlights to show up, despite the app's text stating "found n elements in ~0.20s". Is there a config option to instantly show the yellow highlights?
    - _chendo_ 4y ago
      Thanks! I don't have a way to take tips yet, but you can support by pushing for developers to improve their accessibility implementations when you run into issues! I see you found the setting for that. It was a deliberate default initially as the intended way to use Shortcat is to activate Shortcat and type what you want without waiting to see hints, as this is generally faster and less mental overhead IMO, especially for fast typists and well-structured interfaces. However, some people prefer minimal keystrokes and I get that. I'm trying to figure out the right set of defaults to make it friendly to new users while nudging people to how Shortcat is designed to be used and will be tweaking it as I go.
yewenjie 4y ago
I have been meaning to build something like this for myself, albeit for Linux. Does anybody know if there is any already existing efforts there? Given that Linux doesn't have anything like an accessibility API, I think the only option is training ML models.
pabs3 4y ago
Should be doable on Linux for most mainstream apps due to the toolkits having a11y support, but obviously not all apps use mainstream toolkits.
zvmaz 4y ago
Excellent! I'd love to have that on Linux.
smcleod 4y ago
I've been playing around with ShortCat recently - really cool app! Keep up the good work.
- _chendo_ 4y ago
  Thanks :D
m1r5h1 4y ago
I never comment on HN however I just want to say I've downloaded your app and it's very impressive - I'm going to try and incorporate this into my workflow the best I can. Thanks!
Tepix 4y ago
Looks very cool, have you considered adding voice input or is it already possible?
- least 4y ago
  There’s already a very sophisticated system on MacOS for voice input so I feel like it’d probably be superfluous.
Daynil 4y ago
I love this new wave of tools coming out for mouseless computer use. Chronic mouse use has destroyed my wrist so I have to avoid using it as much as possible. I love Shortcat's approach in general, indexing the UI. However, the reliance on the Accessibility API is actually a significant downside in the real world in my experience since so many apps don't properly implement it. I feel like Warpd is a good complement to this, you could use Hint or Grid mode as a fallback when the indexing approach fails. I wish I could use shortcat or Warpd, but unfortunately I'm on windows. Curious if anyone has any good tool recommendations for windows? Currently, I'm using: 1. Vimium for Chrome (so good, wish I could just use it across the OS). 2. Hunt and Peck: https://github.com/zsims/hunt-and-peck https://github.com/zsims/hunt-and-peck has been my favorite for OS-level use, a simple version of shortcat for windows. But, it's not maintained and not as slick as some of these newer tools.
- forgotmypw17 4y ago
  If you're already using Vimium, I suggest trying qutebrowser, which takes keyboard accessibility to a whole new level, by making it a first-class feature for the entire browser. It does basically cut out the mouse, and had a several-days learning curve for me, but after that it's pretty great. Here are some cool features, off the top of my head: * Python-scriptable, though I haven't figured out how to use this yet. * Bind javascript bookmarklets to a keyboard shortcut (use :bind with the jseval command) * Toggle not only javascript, but image loading and a whole slew of other features, with a keyboard macro. * Vertical tabs. * All config is adjustable via commands. * Keyboard macros like "pop tab into a new window", "clone tab", "close all other tabs", etc. * Text selection using the keyboard. * Quite similar keyboard dynamics to vim. It has a built-in ad blocker, and you should run :adblock-update when you first use it. Another browser which is similar, but which I haven't gotten into as much, is Luakit.
- EsportToys 4y ago
  A bit late to the party, but I’ve just released v1.0 of TPMouse, for Windows: https://github.com/EsportToys/TPMouse https://github.com/EsportToys/TPMouse
  - 4y ago
joshspankit 4y ago
This is what CMD-Shift-? should be. I think this paradigm along with more app developers putting all the important functions in menus is a strong contender for Maximum Intuitive Productivity
madacol 4y ago
Looks really cool. Is it able to select text for copying?
- _chendo_ 4y ago
  I'm working on a version that allows sending arrow keys with modifiers to the targeted application, so soon!
vips7L 4y ago
Looks awesome! Wish I had this for windows!
- nthypes 4y ago
  https://github.com/GavinPen/AhkCoordGrid https://github.com/GavinPen/AhkCoordGrid
bloopernova 4y ago
Shortcat is utterly amazing. I really hope I can work this into my entire MacOS usage. You should be really proud of what you've made because this is fantastic!!
abalaji 4y ago
Any plans to add scrolling functionality to shortcat? I'd be able to move over completely from vimac if that gets added.
- _chendo_ 4y ago
  Working on that right now :)
fcoury 4y ago
Oh my! I came to the comment section to ask about a Mac app that I've seen a long time ago that did this. Lo and behold, you, the author, have written the first comment. :-) Thank you for Shortcat, I used it a long time ago and loved it. Excited to giv it another go!
- _chendo_ 4y ago
  No worries! Glad you love Shortcat :D
ta988 4y ago
I just tried that, this is excellent.

LASR 4y ago

This is excellent. I will be trying this out.

kache_ 4y ago

Have you considered using ML/OCR to figure out the position of the text relative to the screen? Seems much simpler than relying on accessibility APIs Thank you for your hard work!

_chendo_ 4y ago
I have plans to use ML/OCR to augment results down the road but the AX APIs and ecosystem on most apps (that I encounter, at least) are generally decent. Also, OCR means it won’t understand buttons with just icons, whereas AX APIs can grab em just fine. Thanks! It’s easily my longest running project at a decade

hawski 4y ago

Very interesting thing. I wonder if the gap of apps not supporting a11y could be reduced by using Tesseract to OCR the text.

ancienthope 4y ago
That's what https://superkey.app/ https://superkey.app/ does.

discodachshund 4y ago

Excited to try this out! Is it planned to open source? I would love to try integrating this into Raycast

punnerud 4y ago
Is Raycast open source? Could only find that the plugins are on GitHub.
_chendo_ 4y ago
It won't be open source, but I will be adding an API so it can be integrated with other apps and scripted

Luc 4y ago

This app is quality. I can tell you have been working on it for years. Why not charge for it?

_chendo_ 4y ago
Haha, thanks :) I did charge for it a couple of years ago, however I rebuilt the whole thing from scratch after a long hiatus and hadn't had bothered to reimplement licensing because the existing options all kinda suck, and figured I'd focus the time on features and usability first. I think with the modal mode in the next release will bring it much closer to a 1.0 release.
- justusw 4y ago
  If you bundle it and release a paid for application on the App Store, I would totally buy it and even roll it out to my staff. The magic of the App Store allows you to do company wide roll-outs quite easily.
  - _chendo_ 4y ago
    I'm not sure if an app like Shortcat can be released on the App Store given it uses the Accessibility APIs (sandboxing etc), also the 15-30% cut they take is a bit ooooof, but I do have plans to support company/teams licensing!
- tfsh 4y ago
  +1 this is awesome! I'd like to donate if I can :) edit: nvm me, found the option in settings (on activation show shortcuts immediately). Quick question, I've been playing around with Shortcat for a while. When I press the activation hot-key it takes about 4 seconds for the yellow two-letter denoated highlights to show up, despite the app's text stating "found n elements in ~0.20s". Is there a config option to instantly show the yellow highlights?
  - _chendo_ 4y ago
    Thanks! I don't have a way to take tips yet, but you can support by pushing for developers to improve their accessibility implementations when you run into issues! I see you found the setting for that. It was a deliberate default initially as the intended way to use Shortcat is to activate Shortcat and type what you want without waiting to see hints, as this is generally faster and less mental overhead IMO, especially for fast typists and well-structured interfaces. However, some people prefer minimal keystrokes and I get that. I'm trying to figure out the right set of defaults to make it friendly to new users while nudging people to how Shortcat is designed to be used and will be tweaking it as I go.

yewenjie 4y ago

I have been meaning to build something like this for myself, albeit for Linux. Does anybody know if there is any already existing efforts there? Given that Linux doesn't have anything like an accessibility API, I think the only option is training ML models.

pabs3 4y ago

Should be doable on Linux for most mainstream apps due to the toolkits having a11y support, but obviously not all apps use mainstream toolkits.

zvmaz 4y ago

Excellent! I'd love to have that on Linux.

smcleod 4y ago

I've been playing around with ShortCat recently - really cool app! Keep up the good work.

_chendo_ 4y ago
Thanks :D

m1r5h1 4y ago

I never comment on HN however I just want to say I've downloaded your app and it's very impressive - I'm going to try and incorporate this into my workflow the best I can. Thanks!

Tepix 4y ago

Looks very cool, have you considered adding voice input or is it already possible?

least 4y ago
There’s already a very sophisticated system on MacOS for voice input so I feel like it’d probably be superfluous.

Daynil 4y ago

I love this new wave of tools coming out for mouseless computer use. Chronic mouse use has destroyed my wrist so I have to avoid using it as much as possible. I love Shortcat's approach in general, indexing the UI. However, the reliance on the Accessibility API is actually a significant downside in the real world in my experience since so many apps don't properly implement it. I feel like Warpd is a good complement to this, you could use Hint or Grid mode as a fallback when the indexing approach fails. I wish I could use shortcat or Warpd, but unfortunately I'm on windows. Curious if anyone has any good tool recommendations for windows? Currently, I'm using: 1. Vimium for Chrome (so good, wish I could just use it across the OS). 2. Hunt and Peck: https://github.com/zsims/hunt-and-peck https://github.com/zsims/hunt-and-peck has been my favorite for OS-level use, a simple version of shortcat for windows. But, it's not maintained and not as slick as some of these newer tools.

forgotmypw17 4y ago
If you're already using Vimium, I suggest trying qutebrowser, which takes keyboard accessibility to a whole new level, by making it a first-class feature for the entire browser. It does basically cut out the mouse, and had a several-days learning curve for me, but after that it's pretty great. Here are some cool features, off the top of my head: * Python-scriptable, though I haven't figured out how to use this yet. * Bind javascript bookmarklets to a keyboard shortcut (use :bind with the jseval command) * Toggle not only javascript, but image loading and a whole slew of other features, with a keyboard macro. * Vertical tabs. * All config is adjustable via commands. * Keyboard macros like "pop tab into a new window", "clone tab", "close all other tabs", etc. * Text selection using the keyboard. * Quite similar keyboard dynamics to vim. It has a built-in ad blocker, and you should run :adblock-update when you first use it. Another browser which is similar, but which I haven't gotten into as much, is Luakit.
EsportToys 4y ago
A bit late to the party, but I’ve just released v1.0 of TPMouse, for Windows: https://github.com/EsportToys/TPMouse https://github.com/EsportToys/TPMouse
- 4y ago

joshspankit 4y ago

This is what CMD-Shift-? should be. I think this paradigm along with more app developers putting all the important functions in menus is a strong contender for Maximum Intuitive Productivity

madacol 4y ago

Looks really cool. Is it able to select text for copying?

_chendo_ 4y ago
I'm working on a version that allows sending arrow keys with modifiers to the targeted application, so soon!

vips7L 4y ago

Looks awesome! Wish I had this for windows!

nthypes 4y ago
https://github.com/GavinPen/AhkCoordGrid https://github.com/GavinPen/AhkCoordGrid

bloopernova 4y ago

Shortcat is utterly amazing. I really hope I can work this into my entire MacOS usage. You should be really proud of what you've made because this is fantastic!!

abalaji 4y ago

Any plans to add scrolling functionality to shortcat? I'd be able to move over completely from vimac if that gets added.

_chendo_ 4y ago
Working on that right now :)

fcoury 4y ago

Oh my! I came to the comment section to ask about a Mac app that I've seen a long time ago that did this. Lo and behold, you, the author, have written the first comment. :-) Thank you for Shortcat, I used it a long time ago and loved it. Excited to giv it another go!

_chendo_ 4y ago
No worries! Glad you love Shortcat :D

ta988 4y ago

I just tried that, this is excellent.