This updates the Win32 desktop embedder to support input method (abbreviated IM
or IME) composing regions.
In contrast to languages such as English, where keyboard input is
managed keystroke-by-keystroke, languages such as Japanese require a
multi-step input process wherein the user begins a composing sequence,
during which point their keystrokes are captured by a system input
method and converted into a text sequence. During composing, the user is
able to edit the composing range and manage the conversion from keyboard
input to text before eventually committing the text to the underlying
text input field.
To illustrate this, in Japanese, this sequence might look something like
the following:
1. User types 'k'. The character 'k' is added to the composing region.
Typically, the text 'k' will be inserted inline into the underlying
text field but the composing range will be highlighted in some manner,
frequently with a highlight or underline.
2. User types 'a'. The composing range is replaced with the phonetic
kana character 'か' (ka). The composing range continues to be
highlighted.
3. User types 'k'. The character 'k' is appended to the composing
range such that the highlighted text is now 'かｋ'
4. User types 'u'. The trailing 'k' is replaced with the phonetic kana
character 'く' (ku) such that the composing range now reads 'かく'
The composing range continues to be highlighted.
5. The user presses the space bar to convert the kana characters to
kanji. The composing range is replaced with '書く' (kaku: to write).
6. The user presses the space bar again to show other conversions. The
user's configured input method (for example, ibus) pops up a
completions menu populated with alternatives such as 各 (kaku:
every), 描く (kaku: to draw), 核 (kaku: pit of a fruit, nucleus), 角
(kaku: angle), etc.
7. The user uses the arrow keys to navigate the completions menu and
select the alternative to input. As they do, the inline composing
region in the text field is updated. It continues to be highlighted
or underlined.
8. The user hits enter to commit the composing region. The text is
committed to the underlying text field and the visual highlighting is
removed.
9. If the user presses another key, a new composing sequence begins.
If a selection is present when composing begins, it is preserved until
the first keypress of input is received, at which point the selection is
deleted. If a composing sequence is aborted before the first keypress,
the selection is preserved. Creating a new selection (with the mouse,
for example) aborts composing and the composing region is automatically
committed. A composing range and selection, both with an extent, are
not permitted to co-exist.
During composing, keyboard navigation via the arrow keys, or home and
end (or equivalent shortcuts) is restricted to the composing range, as
are deletions via backspace and the delete key. This patch adds two new
private convenience methods, `editing_range` and `text_range`. The
former returns the range for which editing is currently active -- the
composing range, if composing, otherwise the full range of the text. The
latter, returns a range from position 0 (inclusive) to `text_.length()`
exclusive.
Windows IME support revolves around two main UI windows: the composition window
and the candidate window. The composition window is a system window overlaid
within the current window bounds which renders the composing string. Flutter
already renders this string itself, so we request that this window be hidden.
The candidate window is a system-rendered dropdown that displays all possible
conversions for the text in the composing region. Since the contents of this
window are specific to the particular IME in use, and because the user may have
installed one or more third-party IMEs, Flutter does not attempt to render this
as a widget itself, but rather delegates to the system-rendered window.
The lifecycle of IME composing begins follows the following event order:
1. WM_IME_SETCONTEXT: on window creation this event is received. We strip the
ISC_SHOWUICOMPOSITIONWINDOW bit from the event lparam before passing it to
DefWindowProc() in order to hide the composition window, which Flutter
already renders itself.
2. WM_IME_STARTCOMPOSITION: triggered whenever the user begins inputting new
text. We use this event to set Flutter's TextInputModel into composing mode.
3. WM_IME_COMPOSITION: triggered on each keypress as the user adds, replaces,
or deletes text in the composing region, navigates with their cursor within
the composing region, or selects a new conversion candidate from the
candidates list.
4. WM_IME_ENDCOMPOSITION: triggered when the user has finished editing the text
in the composing region and decides to commit or abort the composition.
Additionally, the following IME-related events are emitted but not yet handled:
* WM_INPUTLANGCHANGE: triggered whenever the user selects a new language using
the system language selection menu. Since there some language-specific
behaviours to IMEs, we may want to make use of this in the future.
* WM_IME_NOTIFY: triggered to notify of various status events such as opening
or closing the candidate window, setting the conversion mode, etc. None of
these are relevant to Flutter at the moment.
* WM_IME_REQUEST: triggered to notify of various commands/requests such as
triggering reconversion of text, which should begin composition mode, insert
the selected text into the composing region, and allow the user to select new
alternative candidates for the text in question before re-committing their
new selection. This patch doesn't support this feature, but it's an important
feature that we should support in future.
* Add multi-step IME support to TextInputModel
This updates the platform-independent TextInputModel to add support for
input method (abbreviated IM or IME) composing regions.
In contrast to languages such as English, where keyboard input is
managed keystroke-by-keystroke, languages such as Japanese require a
multi-step input process wherein the user begins a composing sequence,
during which point their keystrokes are captured by a system input
method and converted into a text sequence. During composing, the user is
able to edit the composing range and manage the conversion from keyboard
input to text before eventually committing the text to the underlying
text input field.
To illustrate this, in Japanese, this sequence might look something like
the following:
1. User types 'k'. The character 'k' is added to the composing region.
Typically, the text 'k' will be inserted inline into the underlying
text field but the composing range will be highlighted in some manner,
frequently with a highlight or underline.
2. User types 'a'. The composing range is replaced with the phonetic
kana character 'か' (ka). The composing range continues to be
highlighted.
3. User types 'k'. The character 'k' is appended to the composing
range such that the highlighted text is now 'かｋ'
4. User types 'u'. The trailing 'k' is replaced with the phonetic kana
character 'く' (ku) such that the composing range now reads 'かく'
The composing range continues to be highlighted.
5. The user presses the space bar to convert the kana characters to
kanji. The composing range is replaced with '書く' (kaku: to write).
6. The user presses the space bar again to show other conversions. The
user's configured input method (for example, ibus) pops up a
completions menu populated with alternatives such as 各 (kaku:
every), 描く (kaku: to draw), 核 (kaku: pit of a fruit, nucleus), 角
(kaku: angle), etc.
7. The user uses the arrow keys to navigate the completions menu and
select the alternative to input. As they do, the inline composing
region in the text field is updated. It continues to be highlighted
or underlined.
8. The user hits enter to commit the composing region. The text is
committed to the underlying text field and the visual highlighting is
removed.
9. If the user presses another key, a new composing sequence begins.
If a selection is present when composing begins, it is preserved until
the first keypress of input is received, at which point the selection is
deleted. If a composing sequence is aborted before the first keypress,
the selection is preserved. Creating a new selection (with the mouse,
for example) aborts composing and the composing region is automatically
committed. A composing range and selection, both with an extent, are
not permitted to co-exist.
During composing, keyboard navigation via the arrow keys, or home and
end (or equivalent shortcuts) is restricted to the composing range, as
are deletions via backspace and the delete key. This patch adds two new
private convenience methods, `editing_range` and `text_range`. The
former returns the range for which editing is currently active -- the
composing range, if composing, otherwise the full range of the text. The
latter, returns a range from position 0 (inclusive) to `text_.length()`
exclusive.
* Move SetComposingLength to TextRange::set_*
Adds set_base, set_extent, set_start, set_end methods to TextRange.
Replaces selection_base() and selection_extent() with selection() and
SetSelection(int, int) with SetSelection(range).
This also adds the following convenience methods to TextRange:
* reversed()
* Contains(size_t position)
* Contains(const TextRange& range)
as well as operator== for use in unit tests. When Flutter migrates to
C++20, we can replace that method with a default declaration.
Extracts a TextRange class with a base and extent, and start(), end(),
collapsed(), and length() getters.
The possibility of reversed base and extent in selections and composing
ranges makes reasoning about them complex and increases the chances of
errors in the code. This change migrates most uses of base and extent in
the text model to start()/end() or position(). The position() method is
intended purely as an aid to readability to indicate that a collapsed
selection is expected at the call site; it also enforces a debug-time
assertion that this is the case.
Previously, the selection base and extent were stored internally as
iterators over text_. Since iterators must be treated as invalidated
whenever the underlying container changes, this requires that
selection_base_ and selection_extent_ be re-assigned after every change
to text_.
This is not currently particularly problematic, but once we add fields
to track the base and extent of the composing region for multi-step
input method support, as well as support for the sub-range within the
composing region to which edits/completions apply, we end up having to
regenerate a lot of iterators with each change, many of which are
logically unchanged in position.
A side benefit is that this simplifies inspection of these fields when
debugging.
Previously, TextInputModel's SetEditingState method was a 1:1 mapping of
the underlying protocol used on the text input channel between the
framework and the engine. This breaks it up into two methods, which
allows the selection to be updated independently of the text, and avoids
tying the API the the underlying protocol.
This will become more important when we add additional state to support
composing regions for multi-step input methods such as those used for
Japanese.
SetText resets the selection rather than making a best-efforts attempt
to preserve it. This choice was primarily to keep the code simple and
make the API easier to reason about. An alternative would have been to
make a best-effort attempt to preserve the selection, potentially
clamping one or both to the end of the new string. In all cases where an
embedder resets the string, it is expected that they also have the
selection, so can call SetSelection with an updated selection if needed.
The C++ text input model used by Windows and Linux currently uses UTF-32. The intention was to facilitate handling of arrow keys, backspace/delete, etc., however since part of what is synchronized with the engine is cursor+selection offsets, and those offsets are defined in terms of UTF-16 code units, this causes very bad interactions with the framework-side model.
This converts to using UTF-16, rather than UTF-32, so that the offsets align with the framework. It also adds surrogate pair handling to the operations that adjust indexes, to avoid breaking surrogate pairs. (Arbitrary grapheme cluster handling is out of scope for this PR; while definitely desirable in the long term, surrogate pair handling is much more critical since improper handling yields invalid UTF-16, which breaks the text field).
This partially fixes https://github.com/flutter/flutter/issues/55014. A framework-side fix is also necessary (since currently both the engine and the framework attempt to handle arrow keys, which is another out-of-scope-for-this-PR issue), but even without the framework fix this dramatically improves the cursor behavior on Windows when there are surrogate pairs somewhere in the string since at least the two sides agree on what indexes mean.
Includes minor plumbing changes to the text input plumbing on Windows so that we're not pointlessly converting from UTF-16 to UTF-32 and then back to UTF-16.
Fixes a few issues with Windows text input:
- Filters out ASCII control characters
- Filters out lead surrogates, which aren't valid UTF-16 on their own so will cause assertion failures if sent to Flutter
- Adds a bandaid fix for a crash due to mismatches in indexing in the C++ and Dart text models. (A better fix would be to use UTF-16 and add surrogate pair handling to deletion and forward/back; that will be a later PR since it has a larger scope.)
Fixes https://github.com/flutter/flutter/issues/54879
Significantly improves the behavior of non-ASCII text input on Windows. Correctly
processes incoming character events as UTF-16, and for now uses UTF-32 for
the text model so that the existing index-based logic will work much more often.
Future work is still needed, but this will handle far more cases correctly.
Changes include:
- File structure
- Header guards
- Include paths
- Namespaces
- Integration with the engine's GN build
- Conversion from jsoncpp to rapidjson
- Style and clang-format adjustment to match engine repository
