This reverts commit b5aedb3 and relands #17712.
Fixesflutter/flutter#53288 and flutter/flutter#41654.
Together with #17791, this reland addresses some of Jim's concerns in the original PR #17712.
The major part of this PR is still the same as the original PR, and the performance / golden image impacts should be the same.