P5.js: iPad external keyboard issues with keyboard.js

[chrisorban]

Nature of issue?

• [x] Found a bug
• [ ] Existing feature enhancement
• [ ] New feature request

Most appropriate sub-area of p5.js?

• [ ] Color
• [ ] Core/Environment/Rendering
• [ ] Data
• [x] Events
• [ ] Image
• [ ] IO
• [ ] Math
• [ ] Typography
• [ ] Utilities
• [ ] WebGL
• [ ] Other (specify if possible)

Which platform were you using when you encountered this?

• [x] Mobile/Tablet (touch devices)
• [ ] Desktop/Laptop
• [ ] Others (specify if possible)

Details about the bug:

• p5.js version: 0.5.4 and 0.6.1 show the problem
• Web browser and version: safari, chrome and firefox exhibit the problem on iPad
• Operating System: recent iOs version
• Steps to reproduce this:
  keyPressed() doesn't work like it should on an iPad external keyboard. This example is all you need to see it:
  https://p5js.org/reference/#/p5/keyPressed
  It will recognized when a particular key is pressed, but it won't recognize when you stop pressing that key. This is different from the behavior on a regular computer or laptop. The keyup function in keyboard.js seems not to be working correctly on iPads.

Feature enhancement details:

A possible workaround is described at the link below. They suggest creating an event listener inside of keydown().
https://stackoverflow.com/questions/45924702/ipad-bluetooth-keyboard-returns-keycode-of-0-for-any-key-with-onkeyup

This bug has also been discussed as the p5.js web editor issue (see link below), but it turns out to be an issue with keyboard.js in p5.js rather than an issue with the web editor.
https://github.com/processing/p5.js-web-editor/issues/594

[lmccart]

@LadySith @lee2sman not sure where you are at with the mobile issue phase, is it finished already?
if not, this one is rather high priority and it'd be great to take a look at

[chrisorban]

@lee2sman Do you think you can spend some time on this? Beer is on me at the next processing community day.

[lee2sman]

hi @chrisorban ! @LadySith is working this summer as a GSOC Processing Fellow this summer and will be looking at this (and other moblie issues).

[LadySith]

Hi @chrisorban and @lmccart ! I'll be looking at this issue over the weekend.

[lmccart]

thanks @LadySith!

[LadySith]

Hi @chrisorban . I tried replicating the issue earlier using a friends apple magic bluetooth keyboard on an iPad but it seemed to work alright using that when we looked at the keyPressed() reference. I wanted to find out if this behavior is consistent with other external keyboards?

I also wanted to find out if for your usage, you will still be mapping the WASD keys to the arrows keys as an alternative for iOS usage?

[chrisorban]

Hi @LadySith Here is a video that documents the problem.
https://drive.google.com/file/d/17QOVo9kcEExHuaYkdTv8x-zxfjGXQs40/view?usp=sharing
This is with the apple brand keyboard that has a direct connection to an iPad pro (not bluetooth). Let me know if this helps. I suppose it is good to know that the issue doesn't affect the apple magic bluetooth keyboard.

If this can be fixed, my plan is to detect if the user is using iOS and then define UP_ARROW, DOWN_ARROW, LEFT_ARROW, and RIGHT_ARROW to refer to the WASD keys. Here is the code I was going to put in the setup() function:

    var iOS = /iPad/.test(navigator.userAgent) && !window.MSStream;

    // testing2345

  if (iOS) {
   print("You seem to be using an iPad. Control the ship with WASD keys instead of the arrow keys."); 
   UP_ARROW = 87;      // keycode for W
   LEFT_ARROW = 65;  // keycode for A
   DOWN_ARROW = 83;  // keycode for S
   RIGHT_ARROW = 68; // keycode for D
  }

The advantage of doing it this way is that I don't have to change anything in sketch.js to make things compatible with the iPad. I usually don't show my users the setup() function. I hide it in a functions.js file.

[chrisorban]

@kjhollen Thanks for taking this on. Any progress?

[kjhollen]

@chrisorban I'm getting the hardware to test with from a friend this weekend, and can work on it next week!

[chrisorban]

@kjhollen Just checking in. Are you able to reproduce the problem?

[kjhollen]

hi @chrisorban, I wasn't able to coordinate getting the hardware and testing with my schedule last week (I am getting ready to move across the country). my friend who has the keyboard + iPad is out of town until I move.

@outofambit is who I was planning to borrow the hardware from. I think she had tried to reproduce the issue and couldn't—tagging her here just in case!

[lmccart]

hi @chrisorban if you have a chance, can you try downloading this p5-test.js file and linking to that in your sketch to see if it fixes things?
http://files.lauren-mccarthy.com/share/p5-test.js

@kjhollen hope you don't mind my jumping in here. I just got curious about this one..

[chrisorban]

Hi @lmccart I tried that but it gives the same behavior as before. Here is the link:
http://alpha.editor.p5js.org/ChrisOrban/sketches/B1bkkIKEX

It works normally on my laptop, but on the iPad it only changes color the first time you press one of the keyboard keys. If you keep pressing that key it does nothing. Same problem as before. Sorry to be the bearer of bad news.

[kjhollen]

yes, that's totally fine @lmccart—go for it!

[lmccart]

Ok I looked into this more. I was able to reproduce the error. The problem is with bluetooth keyboard + iOS safari, and is described clearly in the link @chrisorban posted above in the post's question and first answer. This isn't an issue with the p5.js library itself.

The reason it is causing the p5.js keyPressed() function to fail is that keyPressed() checks whether a key is already down, and won't fire another keyPressed() function call until it is released again first. The relevant code is here. This is to give us the desired behavior that holding down a key fires only one keyPressed() call, and not multiple ones over time. This is different than the normal keyboard-browser behavior (for example, try holding down any key and you will see it start to repeat the letter you're typing).

We could do away with this behavior, but I think it may be important to keep, because it adds some clarity between keyIsPressed vs the keyPressed() function. That is, the variable can be tested in every draw loop for repeated action, whereas the function requires a completely new key press action.

@chrisorban is it possible that you could use the keyTyped() function instead? If you replace that function in the code example, it should work as desired.

var value = 0;

function draw() {
  fill(value);
  rect(25, 25, 50, 50);
}

function keyTyped() {
  if (value === 0) {
    value = 255;
  } else {
    value = 0;
  }
}

[chrisorban]

I agree it's a pretty deep issue that would require some pretty substantial changes to keyboard.js.

Unfortunately, keyTyped() is not going to work because I need to be able to figure out if two keys are being pressed so I can move a little circle in a diagonal direction. My original concern was getting keyIsDown() to work correctly on on the iPad keyboard, but it turns out it doesn't because keyPressed() doesn't work right on iPads.

What if there was a keyboard.js file that was only used for iPads? Or a p5-iPad.js library that could be used in place of the p5.js library? That way the current keyboard.js would remain the same, but there would be another version of keyboard.js that could be modified as needed to get it to work? Is that something worth considering?

[lmccart]

There's actually just not any good way to tell if a key is down while using a bluetooth keyboard because of the bug you found described in that post. Basically, the keyup event returns

{ 
  key: "dead", 
  which: 0
}

Where the "which" property normally displays the keycode of the key that was raised. So even if we had an alternative version of the keyboard.js file, there's not any way to detect which key was released since the release event from a bluetooth keyboard to iOS safari doesn't report it. The modification I suggested in the above comment wouldn't actually help the keyIsDown() reporting.

It sounds like a wired keyboard with ipad may work? I don't have one to test unfortunately.

[chrisorban]

Oh well. I wonder if Processing for the iPad has this same problem or not. That's the only other workaround I can think of at this point. Ohio State has committed to purchasing the apple brand keyboard for all 10,000 incoming freshman. There's no stopping that.

[chrisorban]

Maybe you're right. A wired keyboard might solve the problem.

[chrisorban]

I stumbled across this post which mentions that the ACE editor managed to fix a similar issue:

https://github.com/xtermjs/xterm.js/issues/1101

They provide a link to the source for ACE where someone figured out a workaround.

Let me know if this helps.

[lmccart]

thanks @chrisorban. this looks like a fix that would need to happen in the p5.js web editor, it can't be done by the p5.js library alone. you might suggest it over there.
sadly, I'm going to close this one for now as I think we've exhausted every reasonable possibility for addressing this with the library itself. I'm sorry we're not able to help with this one, but it's just a really frustrating issue with iOS and bluetooth that is beyond our control.