flutter_flutter/shell/platform/windows/text_input_plugin.cc
Chris Bracken 9365230ac2
Add support for IME-based text input on Windows (#23853)
This updates the Win32 desktop embedder to support input method (abbreviated IM
or IME) composing regions.

In contrast to languages such as English, where keyboard input is
managed keystroke-by-keystroke, languages such as Japanese require a
multi-step input process wherein the user begins a composing sequence,
during which point their keystrokes are captured by a system input
method and converted into a text sequence. During composing, the user is
able to edit the composing range and manage the conversion from keyboard
input to text before eventually committing the text to the underlying
text input field.

To illustrate this, in Japanese, this sequence might look something like
the following:

1. User types 'k'. The character 'k' is added to the composing region.
   Typically, the text 'k' will be inserted inline into the underlying
   text field but the composing range will be highlighted in some manner,
   frequently with a highlight or underline.
2. User types 'a'. The composing range is replaced with the phonetic
   kana character 'か' (ka). The composing range continues to be
   highlighted.
3. User types 'k'. The character 'k' is appended to the composing
   range such that the highlighted text is now 'かk'
4. User types 'u'. The trailing 'k' is replaced with the phonetic kana
   character 'く' (ku) such that the composing range now reads 'かく'
   The composing range continues to be highlighted.
5. The user presses the space bar to convert the kana characters to
   kanji. The composing range is replaced with '書く' (kaku: to write).
6. The user presses the space bar again to show other conversions. The
   user's configured input method (for example, ibus) pops up a
   completions menu populated with alternatives such as 各 (kaku:
   every), 描く (kaku: to draw), 核 (kaku: pit of a fruit, nucleus), 角
   (kaku: angle), etc.
7. The user uses the arrow keys to navigate the completions menu and
   select the alternative to input. As they do, the inline composing
   region in the text field is updated. It continues to be highlighted
   or underlined.
8. The user hits enter to commit the composing region. The text is
   committed to the underlying text field and the visual highlighting is
   removed.
9. If the user presses another key, a new composing sequence begins.

If a selection is present when composing begins, it is preserved until
the first keypress of input is received, at which point the selection is
deleted. If a composing sequence is aborted before the first keypress,
the selection is preserved. Creating a new selection (with the mouse,
for example) aborts composing and the composing region is automatically
committed. A composing range and selection, both with an extent, are
not permitted to co-exist.

During composing, keyboard navigation via the arrow keys, or home and
end (or equivalent shortcuts) is restricted to the composing range, as
are deletions via backspace and the delete key. This patch adds two new
private convenience methods, `editing_range` and `text_range`. The
former returns the range for which editing is currently active -- the
composing range, if composing, otherwise the full range of the text. The
latter, returns a range from position 0 (inclusive) to `text_.length()`
exclusive.

Windows IME support revolves around two main UI windows: the composition window
and the candidate window. The composition window is a system window overlaid
within the current window bounds which renders the composing string. Flutter
already renders this string itself, so we request that this window be hidden.
The candidate window is a system-rendered dropdown that displays all possible
conversions for the text in the composing region.  Since the contents of this
window are specific to the particular IME in use, and because the user may have
installed one or more third-party IMEs, Flutter does not attempt to render this
as a widget itself, but rather delegates to the system-rendered window.

The lifecycle of IME composing begins follows the following event order:
1. WM_IME_SETCONTEXT: on window creation this event is received. We strip the
   ISC_SHOWUICOMPOSITIONWINDOW bit from the event lparam before passing it to
   DefWindowProc() in order to hide the composition window, which Flutter
   already renders itself.
2. WM_IME_STARTCOMPOSITION: triggered whenever the user begins inputting new
   text. We use this event to set Flutter's TextInputModel into composing mode.
3. WM_IME_COMPOSITION: triggered on each keypress as the user adds, replaces,
   or deletes text in the composing region, navigates with their cursor within
   the composing region, or selects a new conversion candidate from the
   candidates list.
4. WM_IME_ENDCOMPOSITION: triggered when the user has finished editing the text
   in the composing region and decides to commit or abort the composition.

Additionally, the following IME-related events are emitted but not yet handled:
* WM_INPUTLANGCHANGE: triggered whenever the user selects a new language using
  the system language selection menu. Since there some language-specific
  behaviours to IMEs, we may want to make use of this in the future.
* WM_IME_NOTIFY: triggered to notify of various status events such as opening
  or closing the candidate window, setting the conversion mode, etc. None of
  these are relevant to Flutter at the moment.
* WM_IME_REQUEST: triggered to notify of various commands/requests such as
  triggering reconversion of text, which should begin composition mode, insert
  the selected text into the composing region, and allow the user to select new
  alternative candidates for the text in question before re-committing their
  new selection. This patch doesn't support this feature, but it's an important
  feature that we should support in future.
2021-01-24 12:56:08 -08:00

332 lines
13 KiB
C++

// Copyright 2013 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
#include "flutter/shell/platform/windows/text_input_plugin.h"
#include <windows.h>
#include <cstdint>
#include <iostream>
#include "flutter/shell/platform/common/cpp/json_method_codec.h"
#include "flutter/shell/platform/windows/flutter_windows_view.h"
static constexpr char kSetEditingStateMethod[] = "TextInput.setEditingState";
static constexpr char kClearClientMethod[] = "TextInput.clearClient";
static constexpr char kSetClientMethod[] = "TextInput.setClient";
static constexpr char kShowMethod[] = "TextInput.show";
static constexpr char kHideMethod[] = "TextInput.hide";
static constexpr char kSetMarkedTextRect[] = "TextInput.setMarkedTextRect";
static constexpr char kSetEditableSizeAndTransform[] =
"TextInput.setEditableSizeAndTransform";
static constexpr char kMultilineInputType[] = "TextInputType.multiline";
static constexpr char kUpdateEditingStateMethod[] =
"TextInputClient.updateEditingState";
static constexpr char kPerformActionMethod[] = "TextInputClient.performAction";
static constexpr char kTextInputAction[] = "inputAction";
static constexpr char kTextInputType[] = "inputType";
static constexpr char kTextInputTypeName[] = "name";
static constexpr char kComposingBaseKey[] = "composingBase";
static constexpr char kComposingExtentKey[] = "composingExtent";
static constexpr char kSelectionAffinityKey[] = "selectionAffinity";
static constexpr char kAffinityDownstream[] = "TextAffinity.downstream";
static constexpr char kSelectionBaseKey[] = "selectionBase";
static constexpr char kSelectionExtentKey[] = "selectionExtent";
static constexpr char kSelectionIsDirectionalKey[] = "selectionIsDirectional";
static constexpr char kTextKey[] = "text";
static constexpr char kXKey[] = "x";
static constexpr char kYKey[] = "y";
static constexpr char kWidthKey[] = "width";
static constexpr char kHeightKey[] = "height";
static constexpr char kTransformKey[] = "transform";
static constexpr char kChannelName[] = "flutter/textinput";
static constexpr char kBadArgumentError[] = "Bad Arguments";
static constexpr char kInternalConsistencyError[] =
"Internal Consistency Error";
namespace flutter {
void TextInputPlugin::TextHook(FlutterWindowsView* view,
const std::u16string& text) {
if (active_model_ == nullptr) {
return;
}
active_model_->AddText(text);
SendStateUpdate(*active_model_);
}
bool TextInputPlugin::KeyboardHook(FlutterWindowsView* view,
int key,
int scancode,
int action,
char32_t character,
bool extended) {
if (active_model_ == nullptr) {
return false;
}
if (action == WM_KEYDOWN) {
// Most editing keys (arrow keys, backspace, delete, etc.) are handled in
// the framework, so don't need to be handled at this layer.
switch (key) {
case VK_RETURN:
EnterPressed(active_model_.get());
break;
default:
break;
}
}
return false;
}
TextInputPlugin::TextInputPlugin(flutter::BinaryMessenger* messenger,
TextInputPluginDelegate* delegate)
: channel_(std::make_unique<flutter::MethodChannel<rapidjson::Document>>(
messenger,
kChannelName,
&flutter::JsonMethodCodec::GetInstance())),
delegate_(delegate),
active_model_(nullptr) {
channel_->SetMethodCallHandler(
[this](
const flutter::MethodCall<rapidjson::Document>& call,
std::unique_ptr<flutter::MethodResult<rapidjson::Document>> result) {
HandleMethodCall(call, std::move(result));
});
}
TextInputPlugin::~TextInputPlugin() = default;
void TextInputPlugin::ComposeBeginHook() {
active_model_->BeginComposing();
SendStateUpdate(*active_model_);
}
void TextInputPlugin::ComposeEndHook() {
active_model_->CommitComposing();
active_model_->EndComposing();
SendStateUpdate(*active_model_);
}
void TextInputPlugin::ComposeChangeHook(const std::u16string& text,
int cursor_pos) {
active_model_->AddText(text);
cursor_pos += active_model_->composing_range().base();
active_model_->UpdateComposingText(text);
active_model_->SetSelection(TextRange(cursor_pos, cursor_pos));
SendStateUpdate(*active_model_);
}
void TextInputPlugin::HandleMethodCall(
const flutter::MethodCall<rapidjson::Document>& method_call,
std::unique_ptr<flutter::MethodResult<rapidjson::Document>> result) {
const std::string& method = method_call.method_name();
if (method.compare(kShowMethod) == 0 || method.compare(kHideMethod) == 0) {
// These methods are no-ops.
} else if (method.compare(kClearClientMethod) == 0) {
active_model_ = nullptr;
} else if (method.compare(kSetClientMethod) == 0) {
if (!method_call.arguments() || method_call.arguments()->IsNull()) {
result->Error(kBadArgumentError, "Method invoked without args");
return;
}
const rapidjson::Document& args = *method_call.arguments();
const rapidjson::Value& client_id_json = args[0];
const rapidjson::Value& client_config = args[1];
if (client_id_json.IsNull()) {
result->Error(kBadArgumentError, "Could not set client, ID is null.");
return;
}
if (client_config.IsNull()) {
result->Error(kBadArgumentError,
"Could not set client, missing arguments.");
return;
}
client_id_ = client_id_json.GetInt();
input_action_ = "";
auto input_action_json = client_config.FindMember(kTextInputAction);
if (input_action_json != client_config.MemberEnd() &&
input_action_json->value.IsString()) {
input_action_ = input_action_json->value.GetString();
}
input_type_ = "";
auto input_type_info_json = client_config.FindMember(kTextInputType);
if (input_type_info_json != client_config.MemberEnd() &&
input_type_info_json->value.IsObject()) {
auto input_type_json =
input_type_info_json->value.FindMember(kTextInputTypeName);
if (input_type_json != input_type_info_json->value.MemberEnd() &&
input_type_json->value.IsString()) {
input_type_ = input_type_json->value.GetString();
}
}
active_model_ = std::make_unique<TextInputModel>();
} else if (method.compare(kSetEditingStateMethod) == 0) {
if (!method_call.arguments() || method_call.arguments()->IsNull()) {
result->Error(kBadArgumentError, "Method invoked without args");
return;
}
const rapidjson::Document& args = *method_call.arguments();
if (active_model_ == nullptr) {
result->Error(
kInternalConsistencyError,
"Set editing state has been invoked, but no client is set.");
return;
}
auto text = args.FindMember(kTextKey);
if (text == args.MemberEnd() || text->value.IsNull()) {
result->Error(kBadArgumentError,
"Set editing state has been invoked, but without text.");
return;
}
auto base = args.FindMember(kSelectionBaseKey);
auto extent = args.FindMember(kSelectionExtentKey);
if (base == args.MemberEnd() || base->value.IsNull() ||
extent == args.MemberEnd() || extent->value.IsNull()) {
result->Error(kInternalConsistencyError,
"Selection base/extent values invalid.");
return;
}
// Flutter uses -1/-1 for invalid; translate that to 0/0 for the model.
int selection_base = base->value.GetInt();
int selection_extent = extent->value.GetInt();
if (selection_base == -1 && selection_extent == -1) {
selection_base = selection_extent = 0;
}
active_model_->SetText(text->value.GetString());
active_model_->SetSelection(TextRange(selection_base, selection_extent));
base = args.FindMember(kComposingBaseKey);
extent = args.FindMember(kComposingExtentKey);
if (base == args.MemberEnd() || base->value.IsNull() ||
extent == args.MemberEnd() || extent->value.IsNull()) {
result->Error(kInternalConsistencyError,
"Composing base/extent values invalid.");
return;
}
int composing_base = base->value.GetInt();
int composing_extent = base->value.GetInt();
if (composing_base == -1 && composing_extent == -1) {
active_model_->EndComposing();
} else {
int composing_start = std::min(composing_base, composing_extent);
int cursor_offset = selection_base - composing_start;
active_model_->SetComposingRange(
TextRange(composing_base, composing_extent), cursor_offset);
}
} else if (method.compare(kSetMarkedTextRect) == 0) {
if (!method_call.arguments() || method_call.arguments()->IsNull()) {
result->Error(kBadArgumentError, "Method invoked without args");
return;
}
const rapidjson::Document& args = *method_call.arguments();
auto x = args.FindMember(kXKey);
auto y = args.FindMember(kYKey);
auto width = args.FindMember(kWidthKey);
auto height = args.FindMember(kHeightKey);
if (x == args.MemberEnd() || x->value.IsNull() || //
y == args.MemberEnd() || y->value.IsNull() || //
width == args.MemberEnd() || width->value.IsNull() || //
height == args.MemberEnd() || height->value.IsNull()) {
result->Error(kInternalConsistencyError,
"Composing rect values invalid.");
return;
}
composing_rect_ = {{x->value.GetDouble(), y->value.GetDouble()},
{width->value.GetDouble(), height->value.GetDouble()}};
Rect transformed_rect = GetCursorRect();
delegate_->OnCursorRectUpdated(transformed_rect);
} else if (method.compare(kSetEditableSizeAndTransform) == 0) {
if (!method_call.arguments() || method_call.arguments()->IsNull()) {
result->Error(kBadArgumentError, "Method invoked without args");
return;
}
const rapidjson::Document& args = *method_call.arguments();
auto transform = args.FindMember(kTransformKey);
if (transform == args.MemberEnd() || transform->value.IsNull() ||
!transform->value.IsArray() || transform->value.Size() != 16) {
result->Error(kInternalConsistencyError,
"EditableText transform invalid.");
return;
}
size_t i = 0;
for (auto& entry : transform->value.GetArray()) {
if (entry.IsNull()) {
result->Error(kInternalConsistencyError,
"EditableText transform contains null value.");
return;
}
editabletext_transform_[i / 4][i % 4] = entry.GetDouble();
++i;
}
Rect transformed_rect = GetCursorRect();
delegate_->OnCursorRectUpdated(transformed_rect);
} else {
result->NotImplemented();
return;
}
// All error conditions return early, so if nothing has gone wrong indicate
// success.
result->Success();
}
Rect TextInputPlugin::GetCursorRect() const {
Point transformed_point = {
composing_rect_.left() * editabletext_transform_[0][0] +
composing_rect_.top() * editabletext_transform_[1][0] +
editabletext_transform_[3][0] + composing_rect_.width(),
composing_rect_.left() * editabletext_transform_[0][1] +
composing_rect_.top() * editabletext_transform_[1][1] +
editabletext_transform_[3][1] + composing_rect_.height()};
return {transformed_point, composing_rect_.size()};
}
void TextInputPlugin::SendStateUpdate(const TextInputModel& model) {
auto args = std::make_unique<rapidjson::Document>(rapidjson::kArrayType);
auto& allocator = args->GetAllocator();
args->PushBack(client_id_, allocator);
TextRange selection = model.selection();
rapidjson::Value editing_state(rapidjson::kObjectType);
editing_state.AddMember(kSelectionAffinityKey, kAffinityDownstream,
allocator);
editing_state.AddMember(kSelectionBaseKey, selection.base(), allocator);
editing_state.AddMember(kSelectionExtentKey, selection.extent(), allocator);
editing_state.AddMember(kSelectionIsDirectionalKey, false, allocator);
int composing_base = model.composing() ? model.composing_range().base() : -1;
int composing_extent =
model.composing() ? model.composing_range().extent() : -1;
editing_state.AddMember(kComposingBaseKey, composing_base, allocator);
editing_state.AddMember(kComposingExtentKey, composing_extent, allocator);
editing_state.AddMember(
kTextKey, rapidjson::Value(model.GetText(), allocator).Move(), allocator);
args->PushBack(editing_state, allocator);
channel_->InvokeMethod(kUpdateEditingStateMethod, std::move(args));
}
void TextInputPlugin::EnterPressed(TextInputModel* model) {
if (input_type_ == kMultilineInputType) {
model->AddText(std::u16string({u'\n'}));
SendStateUpdate(*model);
}
auto args = std::make_unique<rapidjson::Document>(rapidjson::kArrayType);
auto& allocator = args->GetAllocator();
args->PushBack(client_id_, allocator);
args->PushBack(rapidjson::Value(input_action_, allocator).Move(), allocator);
channel_->InvokeMethod(kPerformActionMethod, std::move(args));
}
} // namespace flutter