Blink: Remapping Caps Lock to Ctrl does not work on iPadOS 13

[nanzhong]

I've got blink (latest from the raw branch) installed on my iPad running the developer preview of iPadOS 13 and the caps lock remapping no longer seems to work.

I unfortunately am not familiar with the event handling of these key commands so I've not been able to figure out why things broke. keyCommands seems to be returning the correct set of UIKeyCommands (eg. alpha shift modifier + "a"), it just never seems to trigger ctrlSeq: or ctrSeqWithInput:.

[nanzhong]

I've had a chance to play around with this some more and it seem like the key event for caps lock is correctly being sent and can be intercepted via _handleKeyUIEvent.

I've been able to intercept caps lock there and implement a rudimentary alternative that seems to work. Is there a reason why the logic for handling all the key remapping can't happen in that function? Is it considered an internal API that will fail app store review?

[yury]

Hi @nanzhong,

Thank you for investigation. I’m flying home today. So I will be able to look tomorrow. Any PRs are welcome!

[nanzhong]

I'll work on this a bit more and see where I get. I hesitate with my current approach because it doesn't work in all cases (I've only mapped the commonly used keys I need for work, caps lock + {a, e, n, p, x} so I don't have to restore to iOS 12), and also breaks some other things.

The approach I've taken is to keep track of modifier key presses in _handleKeyUIEvent and then rely on keyCommands only for the inputs.

The real underlying reason as to why this broke escapes me. At first I thought this could be an ordering issue with keyCommands, but I've made sure that even with the Caps Lock modifier UIKeyCommand first, autoRepeatSeq is still called instead of ctrlSeq getting called. I wonder if iPadOS 13 is just either losing the UIKeyModifierAlphaShift somewhere or is just behaving differently when it comes to key command handling...

[glberen]

I can’t get blink to launch in ios 13 at all. Just get a black screen

[nanzhong]

@glberen I've been building it myself on Xcode 11 beta, I wonder if that is why I'm able to at least have it running?

[glberen]

Most likely. I’m using the App Store version.

[nanzhong]

I think I've narrowed it down to the underlying reason why this is no longer working.

The following UIKeyCommand

[UIKeyCommand keyCommandWithInput:@"" modifierFlags:UIKeyModifierAlphaShift action:@selector(capsPressed:)]

no longer works. _handleKeyUIEvent correctly sees the event with the correct modifier flags, keyCommands is called as expected, but the selector defined by that command never gets called.

[nanzhong]

To me this either feels like a regression in the new OS version, or they intentionally removed support for this.

In any case a workaround for the time being that I am using can be found in https://github.com/blinksh/blink/compare/raw...nanzhong:capslock-handling?expand=1.

I remap caps lock to control, so my branch hard codes that.

I feel like if we relied on handleKeyUIEvent completely a lot of TermInput could be simplified... but I have a feeling it's intentionally done this way because that's an internal API that shouldn't be used? @yury is that correct?

[carloscabanero]

We do it in that convoluted way because it was the only way to support it within the AppStore constraints.

But given that I'm at WWDC, I will make them this question on Thursday! 👍🏼

[nanzhong]

Great, looking forward to see what Apple says. :D

[carloscabanero]

Thanks a lot for debugging this! I couldn't even get the new iPadOS to install on my iPad, may need support for that too :D

[nanzhong]

No problem. Thank you and all the contributors for building this! I work full time using an iPad only setup so I have a strong vested interest in Blink. 😄

I had trouble getting iPadOS to install until I installed Xcode 11 beta on my macbook; it was previously complaining about the update not existing on the update server.

[carloscabanero]

Great news! Just talked to Apple about this. It isn’t intentional and they will fix it. They have the link of this issue to keep track of it (smile! ☺️) plus we will open another couple radars on related stuff with the keyboard.

Thanks again!!

PS: And we got a feature request to add vim! 😅

[nanzhong]

Amazing, this is great news! I'm an emacser so the vim part I'll pretend I didn't see.

[kickingvegas]

@carloscabanero thanks so much for this feature and for getting in touch with Apple on this. Any chance they would provide full support for swapping keys in Settings like in macOS System Preferences?

[carloscabanero]

I don’t think they will go that far, but at least give access to the same Key events as are available on the Mac, we now know there is a real chance that may happen sooner rather than later.

And I think we may even have a case to get that caps lock key disabled🤞 .

[yury]

Installing 13b2 🤞

[yury]

Nope :( still an issue with beta2

[nanzhong]

Yeah... just verified this as well. Beta 2 actually breaks things even more... I use the dvorak keyboard layout and blink no longer correctly respects it. All other iOS apps seem to work still, so this is likely isolated to the way UIKeyCommands are handled...

To fix this, I had to change all instances in TermInput.m that was accessing the input property of the UIKeyCommand with

NSString *input = [[cmd valueForKey:@"_layoutAwareInput"] stringValue];

[nanzhong]

Mhmm this is really confusing... It seems like the keyboard layout is inconsistently getting stored. Sometimes it recognizes dvorak, but sometime is reverts to qwerty... So my "fix" only works sometimes, and actually makes things worse other times... 🤦‍♂

I wish I could downgrade to Beta 1...

[nanzhong]

I've spent a few hours tonight trying to understand this, but unfortunately since I use my iPad for day to day work, I've had to restore back to iPadOS 13 beta 1 before figuring out exactly what went wrong.

What's really interesting and probably related to this regression is the way keyboard layout and the _layoutAwareInput interacts, might be something worth digging into, but I have a hunch dvorak users like myself are far and few in between. I'll try again when I have some more time or when beta 3 comes out.

[carloscabanero]

Hi @nanzhong, hope your weekend is going great! Thanks a lot again for all your effort testing this. I have already reported the other issues and suggestions, but I am struggling with how to phrase this last one.

Is the problem that with Dvorak layouts the keys still passed on UIKeyCommand sequences are from a different layout?

[nanzhong]

@carloscabanero exactly. It seems like the input property on UIKeyCommand isn't getting set properly (not taking into account the keyboard layout). Inspecting the object in the debugger you can tell that there's a hidden property called _layoutAwareInput that actually holds the correct value.

[nanzhong]

I had a chance to play around with iPadOS 13 Beta 3 today, and the broken behaviour I saw on Beta 2 has been fixed. The mapping issues introduced in Beta 1 are still present, and my temporary fix still seems to work.

[nanzhong]

I spoke too soon :(. It still happens but less frequently. I don't have time to debug this further but I might have some more time later tonight or tomorrow.

[nanzhong]

If anyone else also uses non standard keyboard layouts, I've been able to fix this when it happens by:

1. change the hardware keyboard layout in settings to English
2. switch back to Blink
3. manually disconnect the smart keyboard cover
4. force close the app
5. change the hardware keyboard layout in settings back to the desired layout (dvorak in my case)
6. switch back to Blink

[aspickard]

Thanks for investigating this @nanzhong. Would I still need to perform a manual build with your patch in combination with the above steps in order to introduce the fix? I’ve tried them (for colemak) without the patch on Beta 3 with no success unfortunately.

[nanzhong]

@aspickard sorry I have been on vacation. I just tried some more in beta 4 and the keyboard layout issues seems to be much better now. The caps lock mapping issue however still remains.

[yury]

Fixed in beta 5

[nanzhong]

😮 🍾 Installing now!

[normanjoyner]

Confirmed working for me 👍

[mikeandmore]

I’m on iOS 13.1 b3 and blink 13.0 (132) test flight. I think this bug is coming back again.

Once I set “Capslock as Ctrl Modifiers” and check “Capslock as Ctrl” option, Capslock + letter would reflect as the letter key. Also, Capslock seems to become a lock of control for just [ and ] keys.

[mikeandmore]

Capslock to control still not working with iPadOS 13.1 release.

I wonder if Apple had made this impossible?

[shohey1226]

This might be the issue on which Xcode you built on. When it's build on Xcode10, it looks good on iPadOS/iOS13 but not Xcode11 beta. And I saw Xcode11 comes on App Store app now.. Does anyone install Xcode11 and built on it? Capslock/Ctrl is okay on it?

[nanzhong]

I can confirm that it's broken again on 1.13.2 (latest testflight/HEAD of the raw branch). I've been digging into this for a few days now and it seems like apple has made a few really significant changes:

1. UIKeyModifierAlphaShift (modifier that represents caps lock) no longer works like the other modifiers. It does not indicate that the modifier key is depressed, but rather that caps lock in on.
2. There is an artificial delay in when caps lock activates which delays the key down event.

This really breaks the way blink handles caps lock remapping unfortunately and I haven't been able to come up with a good idea of how it can be fixed. 1. is the blocker, and 2. is a really unfortunate setback to responsiveness.

I suspect if blink was rebuilt using the ios 12 sdk things would work again (as the app store version still seems to work).

[yury]

Yep. Bad thing, that is system broken behavior.
Safari also reports has problems with handling it right.
https://javascript.info/keyboard-events (check keydown and keyup).

[nanzhong]

Has anyone tried the iOS 13.2 beta yet to see if it's the same on that?

[yury]

I’m on 13.2 beta. And it is still broken

[awper361]

I am using IPadOS 13.2 and blink 13.0. Can't swap caps to ctrl but swap caps to esc is working well.

[sideshowcoder]

This seems to have reappeared with the latest app store release. iPadOS 13.1.3 latest blink from app store as of today.

[tydavis]

Confirmed on iPadOS 13.2, "Blink: v13.0.155. Oct 29 2019"

Caps-as-control does not function, but smart keyboard ctrl key remains usable. If changed to caps-as-escape, still does not work and control key is also unusable.

[nanzhong]

From the digging that I've done, there isn't much that can don on blink's side for this. iPadOS 13.1 changed the behaviour of caps lock at an os level.

My current solution has been to run with a custom patch set that hooks into internal APIs, however, this would not be releasable on the app store and is also the reason @yury and @carloscabanero have not used the approach.

[sideshowcoder]

It seems to have been the change of so as previously it worked, and after updating blink it failed to remap? (App Store version) I also checked other with Terminus and it remaps capslock to ctrl fine. I wonder if it has todo with xcode version used to built as @shohey1226 suggested?

[chrismetcalf]

I can also confirm that Terminus works correctly. Please don’t make me use Terminus. :)

So if it’s compiled with XCode 10, it works? I may try manually compiling myself to see if I can get a fix for myself at least.

[colinc]

I can confirm that building with Xcode 10.3 does not reproduce the issue on iPad OS 13.2.

As nanzhong pointed out, all the way back in June, autoRepeatSeq is called instead of ctrlSeq when built with Xcode 11.

[sideshowcoder]

That would explain why others are able to make this work then. Can a new version built with Xcode 10 be released still? Or do I need to built myself? Also should this Issue be reopened in this case, as there are others being opened at the moment.

[colinc]

For clarity, I built the stable branch in Xcode 11.2 (which reproduces the issue in iPad OS 13.2) and deployed on an iPad mini 3 running 12.4 (the latest/last version of iOS that device will run). The problem is not reproduced on that device and ctrlSeq is properly called.

@sideshowcoder All app updates submitted after March 2020 will have to be built with Xcode 11 so it would only be a stopgap fix _if_ Apple is indeed going to fix the underlying issue. Otherwise, some other workaround, like what nanzhong has explored, would be necessary.

[yury]

We think we found solution. Working on that... Fix should be out this week

[chrismetcalf]

@yury If you need beta testers, I'm glad to give it a shot.

[cctile]

FWIW, Xcode 11.2.1 was released on 2019-11-05.

[rexs51]

FYI, I disabled
Settings->Keyboard->Special Keys: Auto Repeat
and now capslock is correctly acting as control with
Modifier Mappings: Capslock -> Cntl

• ipados 13.2, iPad Pro 12.9 + Smart Keyboard

[yury]

@rexs51 sad, but no.

You will get ctrl-A if you press a and caps lock is in ON mode.

[colinc]

In case anyone is wondering, the combination of Xcode 11.2.1 & iOS/iPadOS 13.3 unfortunately does not resolve the issue.

[kickingvegas]

Wonder out loud if this merits a mass radar to Apple. Argument would be that iOS/iPadOS should support key remapping for an external keyboard so that 3rd party devs don't have to resort to an un-natural solution.

[chrismetcalf]

1000 times what @kickingvegas said. But that's going to probably take a lot longer than fixing this bug. :)

[ElvishJerricco]

To be clear, this is a regression in Blink, not iPadOS, right? As I understand it, the issue popped up after a Blink update, not after an iPadOS update.

[yury]

@ElvishJerricco
It is iPadOS issue. When you compile Blink with latest XCode, Blink 12 and Blink 13 are broken.
In general Apple fixes UIKeyCommand detection, but it breaks Blink. We are working on solution (it is not easy)

[chrismetcalf]

Any updates, @yury?

[yury]

@chrismetcalf not yet. Reimplementing all input system with WKWebView. Takes a little bit longer than I expected.

[nanzhong]

@yury is the approach you are taking to have the input handling happen in js and getting rid of TermInput entirely? I tried that approach in the a past but had trouble with having the webview be the first responder in all cases and gave up, but that's probably because I don't fully grok the way the first responder is currently being juggled around.

Does caps lock key up/down get correctly handled in js now? Last time I checked they key up/down were still tied to caps lock being on and off rather than the key press.

[yury]

@nanzhong

I want to address all KB issues we have now. Like custom mappings (custom esc, shift, ctrl, alt), left/right-control/alt/shift/cmd, combo strokes (cmd+s cmd+1), and we also broke our IME support. WKWebView gives us up/down events (except CapsLock, for that we have to use UIKeyCommand), also WKWebView reports repeats (in all languages) and composing events (for IME).

For CapsLock, we want to make a trick with UIKeyCommand. It is kinda hack API, but we don't call it or obfuscate anything :)

Just applying this hack to current implementation still breaks things and very unreliable. So we have to reimplement our TermInput.

If you have non US kb, please let me know. I want you to run some experiments with logger.

[nanzhong]

@yury that sounds awesome. Let me know if there's anything I can help with. I've spent enough time mucking around with TermInput and maintaining my own build of blink were I do hook on internal APIs to have a reasonable understanding of how the current input mechanism works.

As for kb layout, use dvorak, but that's handled via the ios system layout settings and should behave the same as regular us qwerty. I do also use Chinese though, which will trigger the IME related things. Let me know what you would like me to test.

[chrismetcalf]

@yury It might be controversial, but I'd gladly pay for "advanced" keymapping support (above and beyond what was already in the app) as an in-app purchase. I love Blink and I use it all the time so I'd be glad to kick in a little more cash.

[yury]

@chrismetcalf we are thinking about patreon or GitHub sponsors, see #448

@nanzhong can we talk at our discord? I'm interested in dvorak layout key codes in js :)

[cctile]

FWIW iOS 13.2.3 is a thing.

[tydavis]

Blink is what allows me to work from my iPad and iPhone. If there is a way to chip in for a fix/feature, I support that.

[yury]

@tydavis workaround is under heavy development and it is highest priority now.

[dayuoba]

ipados 13.2.3, blink 13.0 . blink can’t start anymore. is there a way to use previous versions on apple store?

[mikeandmore]

ipados 13.2.3, blink 13.0 . blink can’t start anymore. is there a way to use previous versions on apple store?

This is not relevant to this bug. Consider submitting a new bug report.

[lookskywalker]

Hello, I'm using Blink shell extensively and the feature I really love is caps_lock to work as ctrl when it is pressed with other key, escape if pressed alone. But as you may already knows, Blink shell send esc key when it is caps_lock is pressed, even though it is pressed with another key. I guess that Blink has limitation due to IOS api or implementation(I'm not sure, as I'm not IOS developer).

What I want to ask is that is it possible to fix that problem at this time?(when caps_lock is pressed with other key, do not send esc key).

[dmitryzykov]

I really love is caps_lock to work as ctrl when it is pressed with other key, escape if pressed alone

I wish to have this behavior too. Having that on MacOS and really missing that in Blink. Very useful for vim.

[codebarber]

Just purchased on App Store and sadly ran into this issue. Terminus mapping works great, but I’d really like to use Blink as it seems faster and more refined as ssh client. I use tmux a lot and the CAPSLOCK mapping is vital to my hand health.

[carloscabanero]

We just released a TF version with changes to our keyboard functions. Not just we brought Caps back but we bundled a few more treats like complex combos. You can now do C-M-key or Shift-Ctrl-key, etc...

If anyone would like to try it and help us iron out issues, which would be greatly appreciated, please reach out to us on Twitter over DM and we will add you to TF.

[nanzhong]

I’ve had a chance to try it on iOS 13.2 and things are working well besides the shortcut to bring up the settings window.

Installing 13.3 now and will give it a try on that as well.

Great work!

[chrismetcalf]

Using the latest TF version I’m seeing the same as @nanzhong. My key remappings work again, but the new shortcut (Command + ,) to bring up the config doesn’t work.

I’ll continue to test more as I try out the additional key mapping functionality. The ability to remap keys both as modifiers and as keys (CapsLock as both Ctrl and Esc ) is very exciting!

[yury]

Command+, is fixed. Thank you. Pushing build 176.

[nanzhong]

Everything works as expected on 13.3 as well. Feels like this issue should be good to close now.

[awper361]

Everything works as expected on 13.3 as well. Feels like this issue should be good to close now.

Updated to 13.3 and latest appstore blink. The capslock still not working as ctrl. Could you share your blink version, and settings?

[nanzhong]

@awper361 The appstore version is not fixed yet. TestFlight builds or building from source yourself are the two ways to get the fixes right now.

[awper361]

@awper361 The appstore version is not fixed yet. TestFlight builds or building from source yourself are the two ways to get the fixes right now.

Thannk you!

[nanzhong]

@yury I'm noticing that regular Ctrl sequences (not using the remapped caps lock) seems to not be working as expected. Using the caps locked remapped to ctrl works however.

This is reproducible build from HEAD of the raw branch with sequences like Ctrl + c, Ctrl + .

[yury]

@nanzhong I believe it is dvorak thing.

iOS send broken codes for some keys if you hold Control key. I have patch function:

  if (e.ctrlKey) {
    if (
      (e.keyCode == 9 && e.code == 'KeyI') ||
      (e.keyCode == 8 && e.code == 'KeyH') ||
      (e.keyCode == 13 && e.code == 'KeyC') ||
      (e.keyCode == 13 && e.code == 'KeyM') ||
      (e.keyCode == 27 && e.code == 'BracketLeft')
    ) {
      keyDown.keyCode = keyMap.keyCode(e.code) || keyDown.keyCode;
    }
  }

In case of dvorak it works incorrectly.

Can you log all letters (row by row) on dorvak kb with Control pressed? Pay attention to letters that produce 8, 9, 13 and 27 codes instead of their codes :)

Thank you!

[yury]

@nanzhong pushed build 179. Please try.

[nanzhong]

Sorry for the delay @yury seem to work correctly now most of the time.

I've noticed that sometimes certain characters seem to get lost (c, t, and a few others), or cmd seems to be in an activated state even when it's not pressed down, but I'm assuming these are some different bugs related to edge cases of the new keyboard logic or some more issues with dvorak?

[yury]

@nanzhong what build you are trying?

[nanzhong]

I was on an earlier 18x build. I just installed the latest and will let you know how things are.

[yury]

@nanzhong it is related to cmd-tab. See #887 for details.

[idvorkin]

As an FYI - this is now a system option in IOS 13.4

[karlmcguire]

This is still an issue on iPhone IOS 13.4.1. I know I'm probably a minority using Blink on iPhone but there's no system option for remapping external keyboard keys, so it's up to Blink...

[yury]

@karlmcguire as far as I know there is an option to remap hardware kb keys on iPhone. You just need to attach kb first. Then that options will appear in settings.app