Previously, _Utf_view accepted any input_range, including reference-to-array
types like char(&)[2], and stored it as the _M_base member. In such cases,
_Utf_view was not assignable, failing the requirements of view concept.
This patch addresses the issue by adding the ranges::view constraint to the
second template parameter of _Utf_view, and for clarity renaming it from
_Range to _View. The constructor is also adjusted to accept its argument
by value (views must be O(1) move-constructible). This prevents implicitly
generated CTAD from deducing a reference type.
This makes _Utf_view consistent with both other standard views and the
wording from P2728R8: Unicode in the Library, Part 1: UTF Transcoding [1].
The explicit CTAD from viewable_range is not defined for _Utf_view because
it depends on views::all_t, views::ref_view, and views::owning_view,
which are declared in <ranges>. Consequently, users must explicitly cast
the argument to a view or specify it as a template parameter.
[1] https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2025/p2728r8.html
libstdc++-v3/ChangeLog:
* include/bits/unicode.h (_Utf_view): Rename the template parameter
from _Range to _View and constrain it with ranges::view.
(_Utf_view::_Utf_view): Accept by value instead of rvalue reference.
* include/std/format (__format::__write_padded): Replace _Utf_view
over const char32_t(&)[1] with span<const char32_t, 1>.
* testsuite/ext/unicode/view.cc: Add checks if specialization
of _Utf_view satisfy view. Wrap arrays into std::span before
constructing _Utf_view.
Reviewed-by: Jonathan Wakely <jwakely@redhat.com>
Signed-off-by: Tomasz Kamiński <tkaminsk@redhat.com>
When iterating over a range of char16_t in reverse the _Utf_view was
incorrectly treating U+DC00 as a valid high surrogate that can precede
the low surrogate. But U+DC00 is a low surrogate, and so should not be
allowed before another low surrogate. The check should be u2 >= 0xDC00
rather than u2 > 0xDC00.
libstdc++-v3/ChangeLog:
* include/bits/unicode.h (_Utf_view::_M_read_reverse_utf16):
Fix check for high surrogate preceding low surrogate.
* testsuite/ext/unicode/view.cc: Check unpaired low surrogates.
Reviewed-by: Tomasz Kamiński <tkaminsk@redhat.com>
This should check for c <= 0x7f not x < 0x7f, because 0x7f is an ASCII
character (DEL).
libstdc++-v3/ChangeLog:
* include/bits/unicode.h (__is_single_code_unit): Fix check for
7-bit ASCII characters.
Reviewed-by: Tomasz Kamiński <tkaminsk@redhat.com>
This implements the missing functions in _Utf_iterator to support
reverse iteration. All existing tests pass when the view is reversed, so
that the same code units are seen when iterating forwards or backwards.
libstdc++-v3/ChangeLog:
* include/bits/unicode.h (_Utf_iterator::operator--): Reorder
conditions and update position after reading a code unit.
(_Utf_iterator::_M_read_reverse): Define.
(_Utf_iterator::_M_read_utf8): Return extracted code point.
(_Utf_iterator::_M_read_reverse_utf8): Define.
(_Utf_iterator::_M_read_reverse_utf16): Define.
(_Utf_iterator::_M_read_reverse_utf32): Define.
* testsuite/ext/unicode/view.cc: Add checks for reversed views
and reverse iteration.
Reviewed-by: Tomasz Kamiński <tkaminsk@redhat.com>
This reorders the data members of _Utf_iterator to avoid padding bytes
between members due to alignment requirements. For x86_64 the previous
layout had padding after _M_buf and after _M_to_increment for the common
case where the iterators and sentinel types are pointers, so the size
shrinks from 40 bytes to 32 bytes. (For i686 there's no change, it's
still 20 bytes).
We could compress the three uint8_t members into one byte by using
bit-fields:
uint8_t _M_buf_index : 2; // [0,3]
uint8_t _M_buf_last : 3; // [0,4]
uint8_t _M_to_increment : 3; // [0,4]
But there doesn't seem to be any point, because it will just be slower
to access them and there will be tail padding so the size isn't any
smaller. We could also reduce _M_buf_last and _M_to_increment to 2 bits
because the 0 value is only used for a default constructed iterator, and
we don't actually care about the values in that case. Again, this
doesn't seem worth doing.
libstdc++-v3/ChangeLog:
* include/bits/unicode.h (_Utf_iterator): Reorder data members
to be more compact.
Reviewed-by: Tomasz Kamiński <tkaminsk@redhat.com>
Add comments documenting what it does and how it does it.
Also reorder the if-else in operator++ so that we check whether to
iterate over code units in the local buffer before checking whether to
refill that buffer. That seems the more natural way to structure the
function.
libstdc++-v3/ChangeLog:
* include/bits/unicode.h (__unicode::_Utf_iterator): Add
comments.
(__unicode:_Utf_iterator::operator++()): Check whether to
iterate over the buffer first.
Reviewed-by: Tomasz Kamiński <tkaminsk@redhat.com>
This patch implements part P2286R8 that specified debug (escaped)
format for the strings and characters sequences. This include both
handling of the '?' format specifier and set_debug_format member.
To indicate partial support we define __glibcxx_format_ranges macro
value 1, without defining __cpp_lib_format_ranges.
We provide two separate escaping routines depending on the literal
encoding for the corresponding character types. If the character
encoding is Unicode, we follow the specification for the standard
(__format::__write_escaped_unicode).
For other encodings, we escape only characters in range [0x00, 0x80),
interpreting them as ASCII values: [0x00, 0x20), 0x7f and '\t', '\r',
'\n', '\\', '"', '\'' are escaped. We assume every character outside
this range is printable (__format::_write_escaped_ascii).
In particular we do not yet implement special handling of shift
sequences.
For Unicode escaping a new __unicode::__escape_edges table is introduced,
that encodes information if character belongs to General_Category that is
escaped by the standard (Control or Other). This table is generated from
DerivedGeneralCategory.txt provided by Unicode. Only boolean flag is
preserved to reduce the number of entries. The additional rules for escaping
are handled by __format::__should_escape_unicode.
When width or precision is specified, we emit escaped string to the temporary
buffer and format the resulting string according to the format spec.
For characters use a fixed size stack buffer, for which a new _Fixedbuf_sink is
introduced. For strings, we use _Str_sink and to avoid allocations,
we compute the estimated size of (possibly truncated) input, and if it is
larger than width field we print directly.
PR libstdc++/109162
contrib/ChangeLog:
* unicode/README: Mentioned DerivedGeneralCategory.txt.
* unicode/gen_libstdcxx_unicode_data.py: Generation __escape_edges
table from DerivedGeneralCategory.txt. Update file name in comments.
* unicode/DerivedGeneralCategory.txt: Copy of file distributed by
Unicode Consortium.
libstdc++-v3/ChangeLog:
* include/bits/chrono_io.h (__detail::_Widen): Moved to std/format file.
* include/bits/unicode-data.h: Regnerate.
* include/bits/unicode.h (__unicode::_Utf_iterator::_M_units)
(__unicode::__should_escape_category): Define.
* include/std/format (_GLIBCXX_WIDEN_, _GLIBCXX_WIDEN): Copied from
include/bits/chrono_io.h.
(__format::_Widen): Moved from include/bits/chrono_io.h.
(__format::_Term_char, __format::_Escapes, __format::_Separators)
(__format::__should_escape_ascii, __format::__should_escape_unicode)
(__format::__write_escape_seq, __format::__write_escaped_char)
(__format::__write_escaped_acii, __format::__write_escaped_unicode)
(__format::__write_escaped): Define.
(__formatter_str::_S_trunc): Extracted truncation of character
sequences.
(__formatter_str::format): Handle _Pres_esc.
(__formatter_int::_M_do_parse) [__glibcxx_format_ranges]: Parse '?'.
(__formatter_int::_M_format_character_escaped): Define.
(formatter<_CharT, _CharT>::format, formatter<char, wchar_t>::format):
Handle _Pres_esc.
(__formatter_str::set_debug_format, formatter<...>::set_debug_format)
Guard with __glibcxx_format_ranges.
(__format::_Fixedbuf_sink): Define.
* testsuite/23_containers/vector/bool/format.cc: Use __format::_Widen
and remove unnecessary <chrono> include.
* testsuite/std/format/debug.cc: New test.
* testsuite/std/format/debug_nonunicode.cc: New test.
* testsuite/std/format/parse_ctx.cc (escaped_strings_supported): Define
to true if __glibcxx_format_ranges is defined.
* testsuite/std/format/string.cc (escaped_strings_supported): Define to
true if __glibcxx_format_ranges is defined.
Reviewed-by: Jonathan Wakely <jwakely@redhat.com>
Signed-off-by: Tomasz Kamiński <tkaminsk@redhat.com>
This patch corrects handling of UTF-32LE and UTF32-BE in
__unicode::__literal_encoding_is_unicode<_CharT>, so they are
recognized as unicode and functions produces correct result for wchar_t.
Use `__unicode::__field_width` to compute the estimated witdh
of the charcter for unicode wide encoding.
PR libstdc++/119593
libstdc++-v3/ChangeLog:
* include/bits/unicode.h
(__unicode::__literal_encoding_is_unicode<_CharT>):
Corrected handing for UTF-16 and UTF-32 with "LE" or "BE" suffix.
* include/std/format (__formatter_str::_S_character_width):
Define.
(__formatter_str::_S_character_width): Updated passed char
length.
* testsuite/std/format/functions/format.cc: Test for wchar_t.
Reviewed-by: Jonathan Wakely <jwakely@redhat.com>
Signed-off-by: Tomasz Kamiński <tkaminsk@redhat.com>
It is autumn again and there is a new Unicode version 16.0.
The following patch updates our Unicode stuff in contrib, libcpp and
libstdc++ from that Unicode version.
2024-10-08 Jakub Jelinek <jakub@redhat.com>
contrib/
* unicode/README: Update glibc git commit hash, replace
Unicode 15 or 15.1 versions with 16.
* unicode/gen_libstdcxx_unicode_data.py: Use 160000 instead of
150100 in _GLIBCXX_GET_UNICODE_DATA test.
* unicode/from_glibc/utf8_gen.py: Updated from glibc
064c708c78cc2a6b5802dce73108fc0c1c6bfc80 commit.
* unicode/DerivedCoreProperties.txt: Updated from Unicode 16.0.
* unicode/emoji-data.txt: Likewise.
* unicode/PropList.txt: Likewise.
* unicode/GraphemeBreakProperty.txt: Likewise.
* unicode/DerivedNormalizationProps.txt: Likewise.
* unicode/NameAliases.txt: Likewise.
* unicode/UnicodeData.txt: Likewise.
* unicode/EastAsianWidth.txt: Likewise.
gcc/testsuite/
* c-c++-common/cpp/named-universal-char-escape-1.c: Add tests
for some Unicode 16.0 characters, both normal and generated.
libcpp/
* makeucnid.cc (write_copyright): Update Unicode Copyright years.
* makeuname2c.cc (generated_ranges): Adjust Unicode version from 15.1
to 16.0. Add EGYPTIAN HIEROGLYPH- generated range, adjust indexes in
following entries.
(write_copyright): Update Unicode Copyright years.
* generated_cpp_wcwidth.h: Regenerated.
* ucnid.h: Regenerated.
* uname2c.h: Regenerated.
libstdc++-v3/
* include/bits/unicode.h (std::__unicode::__v15_1_0): Rename inline
namespace to ...
(std::__unicode::__v16_0_0): ... this.
(_GLIBCXX_GET_UNICODE_DATA): Change from 150100 to 160000.
* include/bits/unicode-data.h: Regenerated.
* testsuite/ext/unicode/properties.cc: Check for _Gcb_SpacingMark
on U+11F03 rather than U+1D16D as the latter lost SpacingMark property
in Unicode 16.0.
Eddie Nolan reported to me that _Unicode_view was not correctly
implementing the substitution of ill-formed subsequences with U+FFFD,
due to failing to increment the counter when the iterator reaches the
end of the sequence before a multibyte sequence is complete. As a
result, the incomplete sequence was not completely consumed, and then
the remaining character was treated as another ill-formed sequence,
giving two U+FFFD characters instead of one.
To avoid similar mistakes in future, this change introduces a lambda
that increments the iterator and the counter together. This ensures the
counter is always incremented when the iterator is incremented, so that
we always know how many characters have been consumed.
libstdc++-v3/ChangeLog:
* include/bits/unicode.h (_Unicode_view::_M_read_utf8): Ensure
count of characters consumed is correct when the end of the
input is reached unexpectedly.
* testsuite/ext/unicode/view.cc: Test incomplete UTF-8
sequences.
Instead of just omitting the definition of __unicode::_Utf8_view when
char8_t is disabled, we can make it use char instead.
libstdc++-v3/ChangeLog:
PR libstdc++/114519
* include/bits/unicode.h (_Utf8_view) [!__cpp_char8_t]: Define
using char instead of char8_t.
* testsuite/ext/unicode/view.cc: Use u8""sv literals to create
string views, instead of std::u8string_view.
libstdc++-v3/ChangeLog:
PR libstdc++/114519
* include/bits/unicode.h (_Utf8_view): Guard with check for
char8_t being enabled.
(__literal_encoding_is_unicode): Guard use of char8_t with check
for it being enabled.
* testsuite/std/format/functions/114519.cc: New test.
These variables are only read from if we haven't reached the end of
either range, in which case they're guaranteed to be initialized to the
next alphanumeric character. But we can just initialize them to make the
compiler happy.
libstdc++-v3/ChangeLog:
* include/bits/unicode.h (__charset_alias_match): Initialize
__var_a and __var_b.
This is another C++26 change, approved in Varna 2023. We require a new
static array of data that is extracted from the IANA Character Sets
database. A new Python script to generate a header from the IANA CSV
file is added.
The text_encoding class is basically just a pointer to an {ID,name} pair
in the static array. The aliases view is also just the same pointer (or
empty), and the view's iterator moves forwards and backwards in the
array while the array elements have the same ID (or to one element
further, for a past-the-end iterator).
Because those iterators refer to a global array that never goes out of
scope, there's no reason they should every produce undefined behaviour
or indeterminate values. They should either have well-defined
behaviour, or abort. The overhead of ensuring those properties is pretty
low, so seems worth it.
This means that an aliases_view iterator should never be able to access
out-of-bounds. A non-value-initialized iterator always points to an
element of the static array even when not dereferenceable (the array has
unreachable entries at the start and end, which means that even a
past-the-end iterator for the last encoding in the array still points to
valid memory). Dereferencing an iterator can always return a valid
array element, or "" for a non-dereferenceable iterator (but doing so
will abort when assertions are enabled). In the language being proposed
for C++26, dereferencing an invalid iterator erroneously returns "".
Attempting to increment/decrement past the last/first element in the
view is erroneously a no-op, so aborts when assertions are enabled, and
doesn't change value otherwise.
Similarly, constructing a std::text_encoding with an invalid id (one
that doesn't have the value of an enumerator) erroneously behaves the
same as constructing with id::unknown, or aborts with assertions
enabled.
libstdc++-v3/ChangeLog:
PR libstdc++/113318
* acinclude.m4 (GLIBCXX_CONFIGURE): Add c++26 directory.
(GLIBCXX_CHECK_TEXT_ENCODING): Define.
* config.h.in: Regenerate.
* configure: Regenerate.
* configure.ac: Use GLIBCXX_CHECK_TEXT_ENCODING.
* include/Makefile.am: Add new headers.
* include/Makefile.in: Regenerate.
* include/bits/locale_classes.h (locale::encoding): Declare new
member function.
* include/bits/unicode.h (__charset_alias_match): New function.
* include/bits/text_encoding-data.h: New file.
* include/bits/version.def (text_encoding): Define.
* include/bits/version.h: Regenerate.
* include/std/text_encoding: New file.
* src/Makefile.am: Add new subdirectory.
* src/Makefile.in: Regenerate.
* src/c++26/Makefile.am: New file.
* src/c++26/Makefile.in: New file.
* src/c++26/text_encoding.cc: New file.
* src/experimental/Makefile.am: Include c++26 convenience
library.
* src/experimental/Makefile.in: Regenerate.
* python/libstdcxx/v6/printers.py (StdTextEncodingPrinter): New
printer.
* scripts/gen_text_encoding_data.py: New file.
* testsuite/22_locale/locale/encoding.cc: New test.
* testsuite/ext/unicode/charset_alias_match.cc: New test.
* testsuite/std/text_encoding/cons.cc: New test.
* testsuite/std/text_encoding/members.cc: New test.
* testsuite/std/text_encoding/requirements.cc: New test.
Reviewed-by: Ulrich Drepper <drepper.fsp@gmail.com>
Reviewed-by: Patrick Palka <ppalka@redhat.com>
We were relying on P1814R0 (CTAD for alias templates) which isn't
supported by Clang. We can just not use CTAD and provide an explicit
template argument list for _Utf32_view.
Ideally we'd define a deduction guide for _Grapheme_cluster_view that
uses views::all_t to properly convert non-views to views, but all_t is
defined in <ranges> and we don't want to include all of that in
<bits/unicode.h>. So just make it require a view for now, which can be
cheaply copied.
Although it's not needed yet, it would also be more correct to
specialize enable_borrowed_range for the views in <bits/unicode.h>.
libstdc++-v3/ChangeLog:
* include/bits/unicode.h (_Grapheme_cluster_view): Require view.
Do not use CTAD for _Utf32_view.
(__format_width, __truncate): Do not use CTAD.
(enable_borrowed_range<_Utf_view<T, R>>): Define specialization.
(enable_borrowed_range<_Grapheme_cluster_view<R>>): Likewise.
Fix some copy & pasted logic in __is_extended_pictographic. This
function should yield false for the values before the first edge, not
true. Also add a missing boundary condition check in __incb_property.
Also Fix an off-by-one error in _Utf_iterator::operator++() that would
make dereferencing a past-the-end iterator undefined (where the intended
design is that the iterator is always incrementable and dereferenceable,
for better memory safety).
Also simplify the grapheme view iterator, which still contained some
remnants of an earlier design I was experimenting with.
Slightly tweak the gen_libstdcxx_unicode_data.py script so that the
_Gcb_property enumerators are in the order we encounter them in the data
file, instead of sorting them alphabetically. Start with the "Other"
property at value 0, because that's the default property for anything
not in the file. This makes no practical difference, but seems cleaner.
It causes the values in the __gcb_edges table to change, so can only be
done now before anybody is using this code yet. The enumerator values
and table entries become ABI artefacts for the function using them.
contrib/ChangeLog:
* unicode/gen_libstdcxx_unicode_data.py: Print out Gcb_property
enumerators in the order they're seen, not alphabetical order.
libstdc++-v3/ChangeLog:
* include/bits/unicode-data.h: Regenerate.
* include/bits/unicode.h (_Utf_iterator::operator++()): Fix off
by one error.
(__incb_property): Add missing check for values before the
first edge.
(__is_extended_pictographic): Invert return values to fix
copy&pasted logic.
(_Grapheme_cluster_view::_Iterator): Remove second iterator
member and find end of cluster lazily.
* testsuite/ext/unicode/grapheme_view.cc: New test.
* testsuite/ext/unicode/properties.cc: New test.
* testsuite/ext/unicode/view.cc: New test.
The name __null_sentinel is defined as a macro by newlib, so we can't
use it as an identifier. That variable is not actually used by
libstdc++, it was added because P2728R6 proposes std::uc::null_sentinel.
Since we don't need it and it breaks bootstrap for newlib targets, just
remove it. A null sentinel can still be used by constructing a
_Null_sentinel_t object as needed, rather than having a named object of
that type predefined.
libstdc++-v3/ChangeLog:
* include/bits/unicode.h (__null_sentinel): Remove.
* testsuite/17_intro/names.cc: Add __null_sentinel.
This implements the requirements in the following proposals, which
dictate how std::format deals with non-ASCII strings:
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1868r1.htmlhttps://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2572r1.htmlhttps://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2675r1.pdf
There are two parts to this. The width estimation for strings must only
count the width of the first character in an extended grapheme cluster.
That requires implementing the algorithm for detecting cluster breaks,
which requires a number of lookup tables of the grapheme cluster break
properties (and Indic_Conjunct_Break and Extended_Pictographic
properties) of every code point. Additionally, some characters have a
field width of 2, which requires another lookup table of field widths
for every code point. The tables added in this commit do not contain
entries for every code point from 0 to 0x10FFFF as that would be very
inefficient and use too much memory. Instead the tables only contain the
code points that form an "edge" for a property, omitting all the code
points that have the same property as the preceding one. We can use a
binary search to find the closest code point in the table that is not
greater than the one we're looking for.
The tables are generated by a new Python script added to the
contrib/unicode directory, and a new data file downloaded from the
Unicode Consortium website.
The rules for extended grapheme cluster breaking are implemented for the
latest Unicode standard, version 15.1.0.
libstdc++-v3/ChangeLog:
* include/Makefile.am: Add new headers.
* include/Makefile.in: Regenerate.
* include/bits/unicode.h: New file.
* include/bits/unicode-data.h: New file.
* include/std/format: Include <bits/unicode.h>.
(__literal_encoding_is_utf8): Move to <bits/unicode.h>.
(_Spec::_M_fill): Change type to char32_t.
(_Spec::_M_parse_fill_and_align): Read a Unicode scalar value
instead of a single character.
(__write_padded): Change __fill_char parameter to char32_t and
encode it into the output.
(__formatter_str::format): Use new __unicode::__field_width and
__unicode::__truncate functions.
* include/std/ostream: Adjust namespace qualification for
__literal_encoding_is_utf8.
* include/std/print: Likewise.
* src/c++23/print.cc: Add [[unlikely]] attribute to error path.
* testsuite/ext/unicode/view.cc: New test.
* testsuite/std/format/functions/format.cc: Add missing examples
from the standard demonstrating alignment with non-ASCII
characters. Add examples checking correct handling of extended
grapheme clusters.
contrib/ChangeLog:
* unicode/README: Add notes about generating libstdc++ tables.
* unicode/GraphemeBreakProperty.txt: New file.
* unicode/emoji-data.txt: New file.
* unicode/gen_libstdcxx_unicode_data.py: New file.
Tomasz Kamiński