Tech details discussion about edge cases of the macOS accessibility API. #734

blindFS · 2026-04-30T09:48:59Z

blindFS
Apr 30, 2026

First of all, I'd like to show my sincere gratitude to the creator of this project. The macOS hint mode directly inspired me to create this PopClip alternative.

So far I'm pretty happy playing with it, but there're some nasty edge cases, like here in the Discord app:

Basically there're non-clickable elements found for the channel icons. Do you have any clue of how to work around such cases? Subrole or DomClass maybe?

y3owk1n · 2026-04-30T10:49:28Z

y3owk1n
Apr 30, 2026
Maintainer

Hey there, do you already try to inspect those element and check out what are available to target in term of accessibility? It must be something for you to target for that element, the only thing is that it might be a very generic role say AXText without much identifier, and you probably can't distinguish it from other AXText. Note that I haven't inspect anything at this moment. Just guessing.

Unfortunately, macOS accessibility is hard, and also not all electron apps are implementing proper accessibility too. Thats also why we started to support grid based navigations.

37 replies

y3owk1n May 2, 2026
Maintainer

@y3owk1n I start to see the dark side of electron apps. My another finding of the day is that when you do a show_menu action, or right click on something, the popup menu in some electron apps is not a descendant of the "focused AXWindow", but is a descendant of current AXApplication, probably belongs to a secondary window or something. And that's not the case for native apps.

Yeap, those are the quirks, what's worst is that, all electron apps doesn't behave the same, in order to provide a good experience, it's probably necessary to maintain a list of targeted heuristics.

I never really trust any AXAction at all, that's why I started to move into hardware clicks as well.

blindFS May 2, 2026
Author

I never really trust any AXAction at all, that's why I started to move into hardware clicks as well.

I simulate mouse clicking for "electronish" apps, also when AXAction fails with specific error code.

There're still edge cases, for example navigation buttons of App Store execute AXAction without error, but nothing happens. I might totally switch to the mouse event side someday, I'm not sure.

y3owk1n May 2, 2026
Maintainer

@y3owk1n Purely judging from your screen recording above, I guess you start traversing from the root of AXApplication? Does that cause trouble if there're let's say multiple safari windows stacked together?

I didn't check yet, but as far as I remember, I think we are doing AXWindow (frontmost window) instead of AXApplication, so there's no issue when multiple windows stacked together in my memory. I uses yabai all the time, windows never get stacked together, but i think i made sure this works properly back then.

However, other than AXWindow, we also includes supplementary elements to be targeted in the config (menubar, dock, notification center, stage manager, and mission control). Enabling all of them will makes almost everything clickable on macOS, downside, more traversing.

I never really trust any AXAction at all, that's why I started to move into hardware clicks as well.

I simulate mouse clicking for "electronish" apps, also when AXAction fails with specific error code.

There're still edge cases, for example navigation buttons of App Store execute AXAction without error, but nothing happens. I might totally switch to the mouse event side someday, I'm not sure.

I remember i hated AXAction back then because on chromium (Brave i mean), the AXPress does nothing (no success but no error either)... not sure if I did something wrong back then or what. And i gave up and move on with hardware clicks.

blindFS May 2, 2026
Author

However, other than AXWindow, we also includes supplementary elements to be targeted in the config (menubar, dock, notification center, stage manager, and mission control). Enabling all of them will makes almost everything clickable on macOS, downside, more traversing.

Cool, that should be good for the multi-window stacking problem. But you might want to check the electron popup menu issue I mentioned above, as it's not accessible from the focused AXWindow.

I remember i hated AXAction back then because on chromium (Brave i mean), the AXPress does nothing (no success but no error either)... not sure if I did something wrong back then or what. And i gave up and move on with hardware clicks.

Same, AXPress does nothing on all "electronish" apps. By "electronish", I mean there're certain frameworks included in the app content bundle. That's why I have to implement simulation of mouse events, even though I don't like to.

y3owk1n May 3, 2026
Maintainer

Cool, that should be good for the multi-window stacking problem. But you might want to check the electron popup menu issue I mentioned above, as it's not accessible from the focused AXWindow.

I just briefly look into it and did some research. Apparently, there are few ways to create a popup in electron, doing it differently will causes different outcome in term of accessibility tree, face palm. So whether or not the popup is a properly child, it all depends on how the developer decide to go with the implementation...

Same, AXPress does nothing on all "electronish" apps. By "electronish", I mean there're certain frameworks included in the app content bundle. That's why I have to implement simulation of mouse events, even though I don't like to.

Thats what it is, but i think if you started to implement some mouse event simulation, might as well just use that and maintain a single truth for all the actions, rather than trying to do complex error detection and fallback and stuffs.

blindFS · 2026-05-03T10:51:00Z

blindFS
May 3, 2026
Author

Hi @y3owk1n, I'd like to share this new feature that I introduced in glyphlow 0.0.10 with you. I think it has endless possibilities, but I can't think of many useful scenarios except for those nested menu items.

I'm not sure whether I should introduce:

app specific workflow, or app blacklist/whitelist for each workflow
workflow for images or other elements

Maybe you have some thoughts on that?

18 replies

blindFS May 5, 2026
Author

Yeap, got it, i guess this has no easy way to eliminate tho. At accessibility point of view, they are not hidden, just covered by other element. It's possible to do some targeted calculation with position to determine what should show and what should not, but in my opinion, it's just not worth it, and very fragile.

100%, in general any popup menu on a webpage can cover some underlying clickable items, it's pretty hard and probably not cheap to analyze the z stack order.

IMO, false positives are more benign than false negatives, as long as they don't mess around with the true elements of intention, that's why I introduced this colored frames in the first place, so it's easier to distinguish hint labels placed closely.

You can see label collision is a problem for text mode

I don't know if you've encountered such clustered clickable items in Neru, feel free to drop your two cents on how to make it more visually pleasing. One thing I can think of is a navigation mode after selecting any element, for example, in the screenshot above, say you accidentally selected BD, but what you actually want is DD, it might be nice to be able to:

press a key to enter navigation mode, only a few elements closest to current false selection are shown, or other hints are dimmed to reduce distraction.
press hjkl to pick an adjacent one to replace, so on and so forth

blindFS May 5, 2026
Author

@y3owk1n another thing I noticed is that in Neru, hints with common prefix tend to be placed nearby, while in glyphlow, I deliberately reverse them so when you type a hint character, there're fewer candidates in the area of attention. Does that make sense to you, maybe you would consider adding that as an option?

y3owk1n May 5, 2026
Maintainer

say you accidentally selected BD, but what you actually want is DD

i think there's a lot of ways to achieve this depends on the UX. For neru wise, i deliberately not to sort anything at all, to ensure we have a set of deterministic label (as long as nothing has changed, the label never change), and the default don't click on select, it just moves the mouse to the position. So it's easy to reselect a correct label if you made a typo. A quick example:

Screen.Recording.2026-05-05.at.11.38.48.PM.mov

another thing I noticed is that in Neru, hints with common prefix tend to be placed nearby, while in glyphlow, I deliberately reverse them so when you type a hint character, there're fewer candidates in the area of attention. Does that make sense to you, maybe you would consider adding that as an option?

this actually make a lot of sense too, considering the potential of label collision, worth to consider as an option. tho, i have to be honest, now a days I don't really use the hint mode anymore, but I know a lot of people like it, so it's there. If you never open up this discussion, i might never touch any of the code anymore unless there's external PR from contributors. hahaha.

y3owk1n May 5, 2026
Maintainer

Also just to share a bit, i think i just added a powerful feature that might inspire you (I am not sure about it) a bit in term of automation. Check this out ~ #759. At least for neru, we can automate more repetitive works easily.

blindFS May 5, 2026
Author

Also just to share a bit, i think i just added a powerful feature that might inspire you a bit in term of automation (I am not sure about it). Check this out ~ #759. I am feeling happy about this thing. hahaha

Cool, I've also made some progress on the workflow feature, users can achieve similar things with that, with a more verbose toml config though. Glad to know that we share the taste of good interaction logic.

CLI interface is on-hold for glyphlow, the Clap crate dependency will increase the final binary size quite a bit, I'm not sure that's what I wanted.

blindFS · 2026-05-05T16:25:19Z

blindFS
May 5, 2026
Author

@y3owk1n Maybe we can share some interesting use cases of workflow/macro/action feed?

Here're the default ones that I enabled in glyphlow:

[[workflows]]
display = " ProofRead"
key = "R"
starting_role = "TextField"
actions = [
    "Focus",
    "SelectAll",
    "ShowMenu",
    { Sleep = 150 },
    { SearchFor = { role = "MenuItem", title = "Proofread" } },
    "Press",
]

[[workflows]]
display = "⮺ Copy"
key = "C"
starting_role = "Image"
actions = [
    "ShowMenu",
    { Sleep = 150 },
    { SearchFor = { role = "MenuItem", title = "Copy Image" } },
    "Press",
]

[[workflows]]
display = " Copy Link"
key = "L"
starting_role = "Image"
actions = [
    "ShowMenu",
    { Sleep = 150 },
    { SearchFor = { role = "MenuItem", title = "Copy Image Address" } },
    "Press",
]

[[workflows]]
display = "󰳽 Press [Left Click]"
key = "["
starting_role = "Generic"
actions = ["Press"]

[[workflows]]
display = " Menu [Right Click]"
key = "]"
starting_role = "Generic"
actions = [
    "ShowMenu",
    { Sleep = 150 },
    { SearchFor = { role = "MenuItem" } },
    "Press",
]

Basically some copy actions for images, click/right click for all kinds of elements, apple intelligence proofread for text field input.

1 reply

y3owk1n May 6, 2026
Maintainer

Out of the blue, i cant think use cases yet, but this specific thing inspired me to implement this feature:

On brave browser, it has a new feature like firefox container (but it's very very new, and they dont have an easy way to open with new container yet)

but i notice that i need to do this a lot of the times, so i added this feature to be able to do:

[recursive_grid.hotkeys]
"1" = ["action right_click", "action sleep 0.1", "action feed down down down down down enter enter"]
"2" = ["action right_click", "action sleep 0.1", "action feed down down down down down enter down enter"]
"3" = ["action right_click", "action sleep 0.1", "action feed down down down down down enter down down enter"]
"4" = ["action right_click", "action sleep 0.1", "action feed down down down down down enter down down down enter"]
"5" = ["action right_click", "action sleep 0.1", "action feed down down down down down enter down down down down enter"]

# instead of down down down, i figured that i can also use some keyboard prefixes
"1" = ["action right_click", "action sleep 0.1", "action feed o p e n enter m d a enter"]
"2" = ["action right_click", "action sleep 0.1", "action feed o p e n enter s k b enter"]
"3" = ["action right_click", "action sleep 0.1", "action feed o p e n enter t r a enter"]
"4" = ["action right_click", "action sleep 0.1", "action feed o p e n enter m a d enter"]
"5" = ["action right_click", "action sleep 0.1", "action feed o p e n enter w a k enter"]

it's ugly for now, but it does the work fine. hahahaha. I foresee that this can potential do a lot of things, but we'll see.

I guess the biggest differences is that Neru doesn't and do not want to incorporate accessibility queries, but just purely feed the key back to macOS bypassing Neru overlays. So user can just do the thing that they used to, without exiting the overlay.

Also if that's a thing, it can potentially be something like copy a text on my cursor, and then go to a website say google and paste it:

[recursive_grid.hotkeys]
# triple click to select text -> copy -> new tab -> visit google.com -> paste -> enter
"9" = [
    "action left_click",
    "action left_click",
    "action left_click",
    "action sleep 0.2",
    "action feed cmd+c",
    "action sleep 0.2",
    "action feed cmd+t",
    "action sleep 0.3",
    "action feed g o o g l e . c o m enter",
    "action sleep 1",
    "action feed cmd+v",
    "action sleep 0.5",
    "action feed enter"
]

And maybe text insertion like email signature:

[recursive_grid.hotkeys]
"0" = "action feed B e s t space r e g a r d s , enter K y l e"

For neru feed action, I think as long as you can do it with your keyboard and mouse, it will be able to automate. Unfortunately, since it's not accessibility based, it doesn't understand anything on the screen, but just do the thing. Configured flows might not survive app specific or OS update, as thing might change. But on the another side, it literally works everywhere.

blindFS · 2026-05-06T09:00:29Z

blindFS
May 6, 2026
Author

Hi @y3owk1n, TIL .focused_window() will miss the real frontmost AXPopover sometimes (e.g. apple music search popover), definitely sounds like a bug of accessibility to me. I don't know if you've noticed that already, just a reminder.

28 replies

blindFS May 20, 2026
Author

@y3owk1n Thanks for the intuitive way to do the comparison, that's pretty clever. I might test it against vimari sometime after the streamlined drawing refactoring.

On most situations that's just not necessary, but no harm either.
On those tail cases, it simply improves UX, and it might be handy for debugging because now you can see the slowest part on screen, there might be some smart hacks helpful to prune that without harming the rest.

y3owk1n May 21, 2026
Maintainer

@y3owk1n Thanks for the intuitive way to do the comparison, that's pretty clever. I might test it against vimari sometime after the streamlined drawing refactoring.

No problem ~ I almost forgot about existence of vimari. lol.

On most situations that's just not necessary, but no harm either.
On those tail cases, it simply improves UX, and it might be handy for debugging because now you can see the slowest part on screen, there might be some smart hacks helpful to prune that without harming the rest.

Agree, it's also in my opinion visually appealing. Hahaha. I like those element streaming in. This is what I wanted achieve when I first started, but I have no idea how to do it. Lmao.

blindFS May 21, 2026
Author

@y3owk1n FYI, I implemented it using a message passing style

Traverse UI elements in a background thread which sends the info of each element candidate upon finding
On the main thread, it listens to the receiver, whenever a message of a new candidate come, simply draw its layers
Upon finishing the traversal, do some incremental adjustments, e.g.

repositioning, text update (e.g. "A" to "AAA")
maybe hide some that couldn't pass the hit-test in your case

y3owk1n May 21, 2026
Maintainer

Amazing, thanks for the info. I'll try to make a prototype and see how it feels on my side.

y3owk1n May 21, 2026
Maintainer

Quick prototype, needed quite a bit of refactor on how i traverse the tree, but seems cool and pretty! I need to think through the whole UX here to see if this is a good idea, but it works ~

Screen.Recording.2026-05-21.at.2.55.55.PM.mp4

y3owk1n · 2026-06-01T04:19:33Z

y3owk1n
Jun 1, 2026
Maintainer

Just to share, i recently created a small utility https://github.com/y3owk1n/mimi that you might be interested. It's very early stage tho. TLDR, it is a macOS event daemon that runs your shell commands when things happen. The events are still very limited, but the list will be more comprehensive along the way i believe.

Whether or not this project will live or died quietly, I don't know yet, but I think there's not a lot of project like this in the scene now.

0 replies

Uh oh!

Tech details discussion about edge cases of the macOS accessibility API. #734

Uh oh!

blindFS Apr 30, 2026

Replies: 5 comments · 84 replies

Uh oh!

Uh oh!

y3owk1n Apr 30, 2026 Maintainer

Uh oh!

y3owk1n May 2, 2026 Maintainer

Uh oh!

blindFS May 2, 2026 Author

Uh oh!

Uh oh!

y3owk1n May 2, 2026 Maintainer

Uh oh!

blindFS May 2, 2026 Author

Uh oh!

y3owk1n May 3, 2026 Maintainer

Uh oh!

blindFS May 3, 2026 Author

Uh oh!

Uh oh!

blindFS May 5, 2026 Author

Uh oh!

blindFS May 5, 2026 Author

Uh oh!

y3owk1n May 5, 2026 Maintainer

Uh oh!

y3owk1n May 5, 2026 Maintainer

Uh oh!

blindFS May 5, 2026 Author

Uh oh!

blindFS May 5, 2026 Author

Uh oh!

Uh oh!

y3owk1n May 6, 2026 Maintainer

Uh oh!

blindFS May 6, 2026 Author

Uh oh!

blindFS May 20, 2026 Author

Uh oh!

y3owk1n May 21, 2026 Maintainer

Uh oh!

blindFS May 21, 2026 Author

Uh oh!

y3owk1n May 21, 2026 Maintainer

Uh oh!

y3owk1n May 21, 2026 Maintainer

Uh oh!

y3owk1n Jun 1, 2026 Maintainer

blindFS
Apr 30, 2026

Replies: 5 comments 84 replies

y3owk1n
Apr 30, 2026
Maintainer

y3owk1n May 2, 2026
Maintainer

blindFS May 2, 2026
Author

y3owk1n May 2, 2026
Maintainer

blindFS May 2, 2026
Author

y3owk1n May 3, 2026
Maintainer

blindFS
May 3, 2026
Author

blindFS May 5, 2026
Author

blindFS May 5, 2026
Author

y3owk1n May 5, 2026
Maintainer

y3owk1n May 5, 2026
Maintainer

blindFS May 5, 2026
Author

blindFS
May 5, 2026
Author

y3owk1n May 6, 2026
Maintainer

blindFS
May 6, 2026
Author

blindFS May 20, 2026
Author

y3owk1n May 21, 2026
Maintainer

blindFS May 21, 2026
Author

y3owk1n May 21, 2026
Maintainer

y3owk1n May 21, 2026
Maintainer

y3owk1n
Jun 1, 2026
Maintainer