flutter_flutter/shell/platform/windows/key_event_handler.cc
Chris Bracken 9365230ac2
Add support for IME-based text input on Windows (#23853)
This updates the Win32 desktop embedder to support input method (abbreviated IM
or IME) composing regions.

In contrast to languages such as English, where keyboard input is
managed keystroke-by-keystroke, languages such as Japanese require a
multi-step input process wherein the user begins a composing sequence,
during which point their keystrokes are captured by a system input
method and converted into a text sequence. During composing, the user is
able to edit the composing range and manage the conversion from keyboard
input to text before eventually committing the text to the underlying
text input field.

To illustrate this, in Japanese, this sequence might look something like
the following:

1. User types 'k'. The character 'k' is added to the composing region.
   Typically, the text 'k' will be inserted inline into the underlying
   text field but the composing range will be highlighted in some manner,
   frequently with a highlight or underline.
2. User types 'a'. The composing range is replaced with the phonetic
   kana character 'か' (ka). The composing range continues to be
   highlighted.
3. User types 'k'. The character 'k' is appended to the composing
   range such that the highlighted text is now 'かk'
4. User types 'u'. The trailing 'k' is replaced with the phonetic kana
   character 'く' (ku) such that the composing range now reads 'かく'
   The composing range continues to be highlighted.
5. The user presses the space bar to convert the kana characters to
   kanji. The composing range is replaced with '書く' (kaku: to write).
6. The user presses the space bar again to show other conversions. The
   user's configured input method (for example, ibus) pops up a
   completions menu populated with alternatives such as 各 (kaku:
   every), 描く (kaku: to draw), 核 (kaku: pit of a fruit, nucleus), 角
   (kaku: angle), etc.
7. The user uses the arrow keys to navigate the completions menu and
   select the alternative to input. As they do, the inline composing
   region in the text field is updated. It continues to be highlighted
   or underlined.
8. The user hits enter to commit the composing region. The text is
   committed to the underlying text field and the visual highlighting is
   removed.
9. If the user presses another key, a new composing sequence begins.

If a selection is present when composing begins, it is preserved until
the first keypress of input is received, at which point the selection is
deleted. If a composing sequence is aborted before the first keypress,
the selection is preserved. Creating a new selection (with the mouse,
for example) aborts composing and the composing region is automatically
committed. A composing range and selection, both with an extent, are
not permitted to co-exist.

During composing, keyboard navigation via the arrow keys, or home and
end (or equivalent shortcuts) is restricted to the composing range, as
are deletions via backspace and the delete key. This patch adds two new
private convenience methods, `editing_range` and `text_range`. The
former returns the range for which editing is currently active -- the
composing range, if composing, otherwise the full range of the text. The
latter, returns a range from position 0 (inclusive) to `text_.length()`
exclusive.

Windows IME support revolves around two main UI windows: the composition window
and the candidate window. The composition window is a system window overlaid
within the current window bounds which renders the composing string. Flutter
already renders this string itself, so we request that this window be hidden.
The candidate window is a system-rendered dropdown that displays all possible
conversions for the text in the composing region.  Since the contents of this
window are specific to the particular IME in use, and because the user may have
installed one or more third-party IMEs, Flutter does not attempt to render this
as a widget itself, but rather delegates to the system-rendered window.

The lifecycle of IME composing begins follows the following event order:
1. WM_IME_SETCONTEXT: on window creation this event is received. We strip the
   ISC_SHOWUICOMPOSITIONWINDOW bit from the event lparam before passing it to
   DefWindowProc() in order to hide the composition window, which Flutter
   already renders itself.
2. WM_IME_STARTCOMPOSITION: triggered whenever the user begins inputting new
   text. We use this event to set Flutter's TextInputModel into composing mode.
3. WM_IME_COMPOSITION: triggered on each keypress as the user adds, replaces,
   or deletes text in the composing region, navigates with their cursor within
   the composing region, or selects a new conversion candidate from the
   candidates list.
4. WM_IME_ENDCOMPOSITION: triggered when the user has finished editing the text
   in the composing region and decides to commit or abort the composition.

Additionally, the following IME-related events are emitted but not yet handled:
* WM_INPUTLANGCHANGE: triggered whenever the user selects a new language using
  the system language selection menu. Since there some language-specific
  behaviours to IMEs, we may want to make use of this in the future.
* WM_IME_NOTIFY: triggered to notify of various status events such as opening
  or closing the candidate window, setting the conversion mode, etc. None of
  these are relevant to Flutter at the moment.
* WM_IME_REQUEST: triggered to notify of various commands/requests such as
  triggering reconversion of text, which should begin composition mode, insert
  the selected text into the composing region, and allow the user to select new
  alternative candidates for the text in question before re-committing their
  new selection. This patch doesn't support this feature, but it's an important
  feature that we should support in future.
2021-01-24 12:56:08 -08:00

260 lines
9.2 KiB
C++

// Copyright 2013 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "flutter/shell/platform/windows/key_event_handler.h"
#include <windows.h>
#include <iostream>
#include "flutter/shell/platform/common/cpp/json_message_codec.h"
namespace flutter {
namespace {
static constexpr char kChannelName[] = "flutter/keyevent";
static constexpr char kKeyCodeKey[] = "keyCode";
static constexpr char kScanCodeKey[] = "scanCode";
static constexpr char kCharacterCodePointKey[] = "characterCodePoint";
static constexpr char kModifiersKey[] = "modifiers";
static constexpr char kKeyMapKey[] = "keymap";
static constexpr char kTypeKey[] = "type";
static constexpr char kHandledKey[] = "handled";
static constexpr char kWindowsKeyMap[] = "windows";
static constexpr char kKeyUp[] = "keyup";
static constexpr char kKeyDown[] = "keydown";
// The maximum number of pending events to keep before
// emitting a warning on the console about unhandled events.
static constexpr int kMaxPendingEvents = 1000;
// Re-definition of the modifiers for compatibility with the Flutter framework.
// These have to be in sync with the framework's RawKeyEventDataWindows
// modifiers definition.
// https://github.com/flutter/flutter/blob/19ff596979e407c484a32f4071420fca4f4c885f/packages/flutter/lib/src/services/raw_keyboard_windows.dart#L203
static constexpr int kShift = 1 << 0;
static constexpr int kShiftLeft = 1 << 1;
static constexpr int kShiftRight = 1 << 2;
static constexpr int kControl = 1 << 3;
static constexpr int kControlLeft = 1 << 4;
static constexpr int kControlRight = 1 << 5;
static constexpr int kAlt = 1 << 6;
static constexpr int kAltLeft = 1 << 7;
static constexpr int kAltRight = 1 << 8;
static constexpr int kWinLeft = 1 << 9;
static constexpr int kWinRight = 1 << 10;
static constexpr int kCapsLock = 1 << 11;
static constexpr int kNumLock = 1 << 12;
static constexpr int kScrollLock = 1 << 13;
/// Calls GetKeyState() an all modifier keys and packs the result in an int,
/// with the re-defined values declared above for compatibility with the Flutter
/// framework.
int GetModsForKeyState() {
int mods = 0;
if (GetKeyState(VK_SHIFT) < 0)
mods |= kShift;
if (GetKeyState(VK_LSHIFT) < 0)
mods |= kShiftLeft;
if (GetKeyState(VK_RSHIFT) < 0)
mods |= kShiftRight;
if (GetKeyState(VK_CONTROL) < 0)
mods |= kControl;
if (GetKeyState(VK_LCONTROL) < 0)
mods |= kControlLeft;
if (GetKeyState(VK_RCONTROL) < 0)
mods |= kControlRight;
if (GetKeyState(VK_MENU) < 0)
mods |= kAlt;
if (GetKeyState(VK_LMENU) < 0)
mods |= kAltLeft;
if (GetKeyState(VK_RMENU) < 0)
mods |= kAltRight;
if (GetKeyState(VK_LWIN) < 0)
mods |= kWinLeft;
if (GetKeyState(VK_RWIN) < 0)
mods |= kWinRight;
if (GetKeyState(VK_CAPITAL) < 0)
mods |= kCapsLock;
if (GetKeyState(VK_NUMLOCK) < 0)
mods |= kNumLock;
if (GetKeyState(VK_SCROLL) < 0)
mods |= kScrollLock;
return mods;
}
// This uses event data instead of generating a serial number because
// information can't be attached to the redispatched events, so it has to be
// possible to compute an ID from the identifying data in the event when it is
// received again in order to differentiate between events that are new, and
// events that have been redispatched.
//
// Another alternative would be to compute a checksum from all the data in the
// event (just compute it over the bytes in the struct, probably skipping
// timestamps), but the fields used below are enough to differentiate them, and
// since Windows does some processing on the events (coming up with virtual key
// codes, setting timestamps, etc.), it's not clear that the redispatched
// events would have the same checksums.
uint64_t CalculateEventId(int scancode, int action, bool extended) {
// Calculate a key event ID based on the scan code of the key pressed,
// and the flags we care about.
return scancode | (((action == WM_KEYUP ? KEYEVENTF_KEYUP : 0x0) |
(extended ? KEYEVENTF_EXTENDEDKEY : 0x0))
<< 16);
}
} // namespace
KeyEventHandler::KeyEventHandler(flutter::BinaryMessenger* messenger,
KeyEventHandler::SendInputDelegate send_input)
: channel_(
std::make_unique<flutter::BasicMessageChannel<rapidjson::Document>>(
messenger,
kChannelName,
&flutter::JsonMessageCodec::GetInstance())),
send_input_(send_input) {
assert(send_input != nullptr);
}
KeyEventHandler::~KeyEventHandler() = default;
void KeyEventHandler::TextHook(FlutterWindowsView* view,
const std::u16string& code_point) {}
KEYBDINPUT* KeyEventHandler::FindPendingEvent(uint64_t id) {
if (pending_events_.empty()) {
return nullptr;
}
for (auto iter = pending_events_.begin(); iter != pending_events_.end();
++iter) {
if (iter->first == id) {
return &iter->second;
}
}
return nullptr;
}
void KeyEventHandler::RemovePendingEvent(uint64_t id) {
for (auto iter = pending_events_.begin(); iter != pending_events_.end();
++iter) {
if (iter->first == id) {
pending_events_.erase(iter);
return;
}
}
std::cerr << "Tried to remove pending event with id " << id
<< ", but the event was not found." << std::endl;
}
void KeyEventHandler::AddPendingEvent(uint64_t id,
int scancode,
int action,
bool extended) {
if (pending_events_.size() > kMaxPendingEvents) {
std::cerr
<< "There are " << pending_events_.size()
<< " keyboard events that have not yet received a response from the "
<< "framework. Are responses being sent?" << std::endl;
}
KEYBDINPUT key_event = KEYBDINPUT{0};
key_event.wScan = scancode;
key_event.dwFlags = KEYEVENTF_SCANCODE |
(extended ? KEYEVENTF_EXTENDEDKEY : 0x0) |
(action == WM_KEYUP ? KEYEVENTF_KEYUP : 0x0);
pending_events_.push_back(std::make_pair(id, key_event));
}
void KeyEventHandler::HandleResponse(bool handled,
uint64_t id,
int action,
bool extended,
int scancode,
int character) {
if (handled) {
this->RemovePendingEvent(id);
} else {
// Since the framework didn't handle the event, we inject a newly
// synthesized one. We let Windows figure out the virtual key and
// character for the given scancode, as well as a new timestamp.
const KEYBDINPUT* key_event = this->FindPendingEvent(id);
if (key_event == nullptr) {
std::cerr << "Unable to find event " << id << " in pending events queue.";
return;
}
INPUT input_event;
input_event.type = INPUT_KEYBOARD;
input_event.ki = *key_event;
UINT accepted = send_input_(1, &input_event, sizeof(input_event));
if (accepted != 1) {
std::cerr << "Unable to synthesize event for unhandled keyboard event "
"with scancode "
<< scancode << " (character " << character << ")" << std::endl;
}
}
}
bool KeyEventHandler::KeyboardHook(FlutterWindowsView* view,
int key,
int scancode,
int action,
char32_t character,
bool extended) {
const uint64_t id = CalculateEventId(scancode, action, extended);
if (FindPendingEvent(id) != nullptr) {
// Don't pass messages that we synthesized to the framework again.
RemovePendingEvent(id);
return false;
}
// TODO: Translate to a cross-platform key code system rather than passing
// the native key code.
rapidjson::Document event(rapidjson::kObjectType);
auto& allocator = event.GetAllocator();
event.AddMember(kKeyCodeKey, key, allocator);
event.AddMember(kScanCodeKey, scancode, allocator);
event.AddMember(kCharacterCodePointKey, character, allocator);
event.AddMember(kKeyMapKey, kWindowsKeyMap, allocator);
event.AddMember(kModifiersKey, GetModsForKeyState(), allocator);
switch (action) {
case WM_KEYDOWN:
event.AddMember(kTypeKey, kKeyDown, allocator);
break;
case WM_KEYUP:
event.AddMember(kTypeKey, kKeyUp, allocator);
break;
default:
std::cerr << "Unknown key event action: " << action << std::endl;
return false;
}
AddPendingEvent(id, scancode, action, extended);
channel_->Send(event, [this, id, action, extended, scancode, character](
const uint8_t* reply, size_t reply_size) {
auto decoded = flutter::JsonMessageCodec::GetInstance().DecodeMessage(
reply, reply_size);
bool handled = (*decoded)[kHandledKey].GetBool();
this->HandleResponse(handled, id, action, extended, scancode, character);
});
return true;
}
void KeyEventHandler::ComposeBeginHook() {
// Ignore.
}
void KeyEventHandler::ComposeEndHook() {
// Ignore.
}
void KeyEventHandler::ComposeChangeHook(const std::u16string& text,
int cursor_pos) {
// Ignore.
}
} // namespace flutter