[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openjdk-2d-dev
Subject:    RFR: JDK-8302618: Problem typing uppercase letters with java.awt.Robot on Mac when moving mouse
From:       Harshitha Onkar <honkar () openjdk ! org>
Date:       2023-06-30 22:39:05
Message-ID: ZqhIYTFuyW36WTnzIX3oIqcYqZQ3F5OwsRvGstrIOdk=.d3ddc124-079a-483e-9f7a-17cf49443096 () github ! com
[Download RAW message or body]

**Problem:**

Robot erroneously produces lowercase letter when mouse is moved (manually) in unison \
with Robot's keyEvents on MacOS. This issue was originally logged by a developer of \
an on-screen accessibility keyboard  - TouchBoard. Originally reported at \
https://github.com/adoptium/adoptium-support/issues/710

The issue is reproducible on JDK versions 22 to 11, but works fine on JDK-8. 

This issue is not restricted for the Shift modifier key and causes problems with \
other modifier keys too and sometimes without any external mouse movement.

- This works correctly on JDK-8 up to JDK-9+129 when Accessibility APIs \
(AXUIElementCreateSystemWide/ AXUIElementPostKeyboardEvent) were used. Later on it \
was changed to CGEvents. 

- CGEvent APIs were introduced in \
[jdk-9+130](https://github.com/openjdk/jdk/commit/0d7becfcb7568417afb9c19f2db24e7797afc1d6#diff-ad38f25bd9f72b45c5cca2ba7f7fc6743c4820e7aeb1bbf5d7802c9d4e0e1479).


- With the present code, the issue occurs at \
[CRobot.m#L295](https://github.com/openjdk/jdk/blob/ac6af6a64099c182e982a0a718bc1b780cef616e/src/java.desktop/macosx/native/libawt_lwawt/awt/CRobot.m#L295.) \
The flags gets reset or cleared when mouse is moved physically in unison with Robot's \
key events.

- The physical mouse movement causes the event flags to be reset. 

**Impact:**

Modifier keys don't work as expected when using Robot with any simultaneous physical \
mouse movement and in case of TouchBoard, this behavior breaks the usability of the \
on-screen a11y keyboard. There is no known workaround for this particular use case \
except for reverting to JDK-8. More details on this use case \
[here.](https://github.com/adoptium/adoptium-support/issues/710#issuecomment-1594103280)


**Solution**

- In order to avoid the resetting of the CGEventFlags here \
[CRobot.m#L295](https://github.com/openjdk/jdk/blob/ac6af6a64099c182e982a0a718bc1b780cef616e/src/java.desktop/macosx/native/libawt_lwawt/awt/CRobot.m#L295.), \
the CGEvent flag state is obtained in `initRobot` (stored in initFlags) which is \
later used within `CRobot_keyEvent`.

- The incoming keyCode is used to determine whether it is a modifier key and the \
corresponding modifierFlagMask is either added or cleared from the initFlags based on \
whether the modifier key was pressed or released.

- Finally, only the required and known flag bits from initFlag are copied over to \
local flag which is used in `CGEventSetFlags()`.

**Testing**

The newly added test - RobotModifierMaskTest.java tests for Shift, Caps, Control, \
Option and Command keys. It should be tested for two cases:

CASE 1 : Run the test as an automated test WITHOUT any mouse movement and let the \
Robot go through all the test cases.

CASE 2 : Run the test in semi-automated mode. While the Robot in typing, manually \
move the mouse (without clicking/dragging). check if the test Passes or Fails.

CI Testing works as expected with the fix and does not break any existing Robot \
related tests.

-------------

Commit messages:
 - whitespace fix
 - test update
 - removed debug logs
 - test changes & allModifiersMask chnaged to static var
 - copyright header for test
 - localFlag fix, allModifiersMask added for correct flag bit calculation
 - initial test case added with debug logs
 - spacing
 - Dictionary implementation changes with NSLog
 - indent changes
 - ... and 1 more: https://git.openjdk.org/jdk/compare/ec45bd64...20f8fd07

Changes: https://git.openjdk.org/jdk/pull/14744/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=14744&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8302618
  Stats: 288 lines in 4 files changed: 284 ins; 0 del; 4 mod
  Patch: https://git.openjdk.org/jdk/pull/14744.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/14744/head:pull/14744

PR: https://git.openjdk.org/jdk/pull/14744


[prev in list] [next in list] [prev in thread] [next in thread]