For the slight cost of counting code points when converting between
encodings and a teeny bit of memory, this commit adds a fast path for
all-happy utf-16 substrings and code point operations.
This seems to be a significant chunk of time spent in many regex
benchmarks.
When the cursor was positioned at the end of text,
attempting to move it left(using the left arrow key)
would fail because align_boundary() was rejecting
the end-of-text position as a valid boundary.
In the UTF-8 implementation, this prevents out-of-bounds access of the
underlying text data, as the ICU macro would essentially do something
akin to `text[text.length()]`.
The UTF-16 implementation already checks for out-of-bounds, but would
previously return 0. We now return an empty Optional in both impls. This
doesn't affect LibJS (the user of the UTF-16 impl), as it already does
bounds checking before invoking LibUnicode APIs.
LibLocale was split off from LibUnicode a couple years ago to reduce the
number of applications on SerenityOS that depend on CLDR data. Now that
we use ICU, both LibUnicode and LibLocale are actually linking in this
data. And since vcpkg gives us static libraries, both libraries are over
30MB in size.
This patch reverts the separation and merges LibLocale into LibUnicode
again. We now have just one library that includes the ICU data.
Further, this will let LibUnicode share the locale cache that previously
would only exist in LibLocale.
2024-06-23 19:52:45 +02:00
Renamed from Tests/LibLocale/TestSegmenter.cpp (Browse further)
