openRuyi-Project/gcc
3
0
Fork 0
mirror of https://github.com/openRuyi-Project/gcc.git synced 2026-09-05 13:31:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
227,552 Commits 2 Branches 0 Tags
master
Commit Graph
7 Commits
Author SHA1 Message Date
Jakub Jelinek 0c0847158c Update to Unicode 17.0.0 
The following patch updates GCC from Unicode 16.0.0 to 17.0.0.

I've followed what the README says and updated also one script from
glibc, but that needed another Unicode file - HangulSyllableType.txt -
around as well, so I'm adding it.
I've added one new test to named-universal-char-escape-1.c for
randomly chosen character from new CJK block.
Note, Unicode 17.0.0 authors forgot to adjust the 4-8 table, I've filed
bugreports about that but the UnicodeData.txt changes for the range ends
and the new range seems to match e.g. what is in the glyph tables, so
the patch follows UnicodeData.txt and not 4-8 table here.

Another thing was that makeuname2c.cc didn't handle correctly when
the size of the generated string table modulo 77 was 76 or 77, in which
case it forgot to emit a semicolon after the string literal and so failed
to compile.

And as can be seen in the emoji-data.txt diff, some properties like
Extended_Pictographic have been removed from certain characters, e.g.
from the Mahjong cards characters except U+1F004, and one libstdc++
test was testing that property exactly on U+1F000.  Dunno why that was
changed, but U+1F004 is the only colored one among tons of black and white
ones.

2025-10-08  Jakub Jelinek  <jakub@redhat.com>

contrib/
	* unicode/README: Add HangulSyllableType.txt file to the
	list as newest utf8_gen.py from glibc now needs it.  Adjust
	git commit hash and change unicode 16 version to 17.
	* unicode/from_glibc/utf8_gen.py: Updated from glibc.
	* unicode/DerivedCoreProperties.txt: Updated from Unicode 17.0.0.
	* unicode/emoji-data.txt: Likewise.
	* unicode/PropList.txt: Likewise.
	* unicode/GraphemeBreakProperty.txt: Likewise.
	* unicode/DerivedNormalizationProps.txt: Likewise.
	* unicode/NameAliases.txt: Likewise.
	* unicode/UnicodeData.txt: Likewise.
	* unicode/EastAsianWidth.txt: Likewise.
	* unicode/DerivedGeneralCategory.txt: Likewise.
	* unicode/HangulSyllableType.txt: New file.
gcc/testsuite/
	* c-c++-common/cpp/named-universal-char-escape-1.c: Add test for
	\N{CJK UNIFIED IDEOGRAPH-3340E}.
libcpp/
	* makeucnid.cc (write_copyright): Adjust copyright year.
	* makeuname2c.cc (generated_ranges): Adjust end points for a couple
	of ranges based on UnicodeData.txt Last changes and add a whole new
	CJK UNIFIED IDEOGRAPH- entry.  None of these changes are in the 4-8
	table, but clearly it has just been forgotten.
	(write_copyright): Adjust copyright year.
	(write_dict): Fix up condition when to print semicolon.
	* generated_cpp_wcwidth.h: Regenerate.
	* ucnid.h: Regenerate.
	* uname2c.h: Regenerate.
libstdc++-v3/
	* include/bits/unicode-data.h: Regenerate.
	* testsuite/ext/unicode/properties.cc: Test __is_extended_pictographic
	on U+1F004 rather than U+1F000.
 2025-10-08 18:02:39 +02:00
Tomasz Kamiński 3b33d792cf libstdc++: Implement debug format for strings and characters formatters [PR109162] 
This patch implements part P2286R8 that specified debug (escaped)
format for the strings and characters sequences. This include both
handling of the '?' format specifier and set_debug_format member.

To indicate partial support we define __glibcxx_format_ranges macro
value 1, without defining __cpp_lib_format_ranges.

We provide two separate escaping routines depending on the literal
encoding for the corresponding character types. If the character
encoding is Unicode, we follow the specification for the standard
(__format::__write_escaped_unicode).
For other encodings, we escape only characters in range [0x00, 0x80),
interpreting them as ASCII values: [0x00, 0x20), 0x7f and  '\t', '\r',
'\n', '\\', '"', '\'' are escaped. We assume every character outside
this range is printable (__format::_write_escaped_ascii).
In particular we do not yet implement special handling of shift
sequences.

For Unicode escaping a new __unicode::__escape_edges table is introduced,
that encodes information if character belongs to General_Category that is
escaped by the standard (Control or Other). This table is generated from
DerivedGeneralCategory.txt provided by Unicode. Only boolean flag is
preserved to reduce the number of entries. The additional rules for escaping
are handled by __format::__should_escape_unicode.

When width or precision is specified, we emit escaped string to the temporary
buffer and format the resulting string according to the format spec.
For characters use a fixed size stack buffer, for which a new _Fixedbuf_sink is
introduced. For strings, we use _Str_sink and to avoid allocations,
we compute the estimated size of (possibly truncated) input, and if it is
larger than width field we print directly.

	PR libstdc++/109162

contrib/ChangeLog:

	* unicode/README: Mentioned DerivedGeneralCategory.txt.
	* unicode/gen_libstdcxx_unicode_data.py: Generation __escape_edges
	table from DerivedGeneralCategory.txt. Update file name in comments.
	* unicode/DerivedGeneralCategory.txt: Copy of file distributed by
	Unicode Consortium.

libstdc++-v3/ChangeLog:

	* include/bits/chrono_io.h (__detail::_Widen): Moved to std/format file.
	* include/bits/unicode-data.h: Regnerate.
	* include/bits/unicode.h (__unicode::_Utf_iterator::_M_units)
	(__unicode::__should_escape_category): Define.
	* include/std/format (_GLIBCXX_WIDEN_, _GLIBCXX_WIDEN):	Copied from
	include/bits/chrono_io.h.
	(__format::_Widen): Moved from include/bits/chrono_io.h.
	(__format::_Term_char, __format::_Escapes, __format::_Separators)
	(__format::__should_escape_ascii, __format::__should_escape_unicode)
	(__format::__write_escape_seq, __format::__write_escaped_char)
	(__format::__write_escaped_acii, __format::__write_escaped_unicode)
	(__format::__write_escaped): Define.
	(__formatter_str::_S_trunc): Extracted truncation of character
	sequences.
	(__formatter_str::format): Handle _Pres_esc.
	(__formatter_int::_M_do_parse) [__glibcxx_format_ranges]: Parse '?'.
	(__formatter_int::_M_format_character_escaped): Define.
	(formatter<_CharT, _CharT>::format, formatter<char, wchar_t>::format):
	Handle _Pres_esc.
	(__formatter_str::set_debug_format, formatter<...>::set_debug_format)
	Guard with __glibcxx_format_ranges.
	(__format::_Fixedbuf_sink): Define.
	* testsuite/23_containers/vector/bool/format.cc: Use __format::_Widen
	and remove unnecessary <chrono> include.
	* testsuite/std/format/debug.cc: New test.
	* testsuite/std/format/debug_nonunicode.cc: New test.
	* testsuite/std/format/parse_ctx.cc (escaped_strings_supported): Define
	to true if __glibcxx_format_ranges is defined.
	* testsuite/std/format/string.cc (escaped_strings_supported): Define to
	true if __glibcxx_format_ranges is defined.

Reviewed-by: Jonathan Wakely <jwakely@redhat.com>
Signed-off-by: Tomasz Kamiński <tkaminsk@redhat.com>
 2025-04-11 08:43:50 +02:00
Jakub Jelinek d0e8f58b81 contrib, libcpp, libstdc++: Update to Unicode 16.0 
It is autumn again and there is a new Unicode version 16.0.

The following patch updates our Unicode stuff in contrib, libcpp and
libstdc++ from that Unicode version.

2024-10-08  Jakub Jelinek  <jakub@redhat.com>

contrib/
	* unicode/README: Update glibc git commit hash, replace
	Unicode 15 or 15.1 versions with 16.
	* unicode/gen_libstdcxx_unicode_data.py: Use 160000 instead of
	150100 in _GLIBCXX_GET_UNICODE_DATA test.
	* unicode/from_glibc/utf8_gen.py: Updated from glibc
	064c708c78cc2a6b5802dce73108fc0c1c6bfc80 commit.
	* unicode/DerivedCoreProperties.txt: Updated from Unicode 16.0.
	* unicode/emoji-data.txt: Likewise.
	* unicode/PropList.txt: Likewise.
	* unicode/GraphemeBreakProperty.txt: Likewise.
	* unicode/DerivedNormalizationProps.txt: Likewise.
	* unicode/NameAliases.txt: Likewise.
	* unicode/UnicodeData.txt: Likewise.
	* unicode/EastAsianWidth.txt: Likewise.
gcc/testsuite/
	* c-c++-common/cpp/named-universal-char-escape-1.c: Add tests
	for some Unicode 16.0 characters, both normal and generated.
libcpp/
	* makeucnid.cc (write_copyright): Update Unicode Copyright years.
	* makeuname2c.cc (generated_ranges): Adjust Unicode version from 15.1
	to 16.0.  Add EGYPTIAN HIEROGLYPH- generated range, adjust indexes in
	following entries.
	(write_copyright): Update Unicode Copyright years.
	* generated_cpp_wcwidth.h: Regenerated.
	* ucnid.h: Regenerated.
	* uname2c.h: Regenerated.
libstdc++-v3/
	* include/bits/unicode.h (std::__unicode::__v15_1_0): Rename inline
	namespace to ...
	(std::__unicode::__v16_0_0): ... this.
	(_GLIBCXX_GET_UNICODE_DATA): Change from 150100 to 160000.
	* include/bits/unicode-data.h: Regenerated.
	* testsuite/ext/unicode/properties.cc: Check for _Gcb_SpacingMark
	on U+11F03 rather than U+1D16D as the latter lost SpacingMark property
	in Unicode 16.0.
 2024-10-08 10:01:47 +02:00
Jonathan Wakely ef2efc53fd libstdc++: Fix Python scripts to output the correct filename 
These scripts both print "generated by $file, do not edit" header but
one of them prints the wrong filename. Use the built-in __file__
attribute to ensure it's correct.

contrib/ChangeLog:

	* unicode/gen_libstdcxx_unicode_data.py: Fix header of generated
	file to name the correct script.

libstdc++-v3/ChangeLog:

	* include/bits/text_encoding-data.h: Regenerate.
	* include/bits/unicode-data.h: Regenerate.
	* scripts/gen_text_encoding_data.py: Fix header of generated
	file to name the correct script.
 2024-03-19 15:20:07 +00:00
Jonathan Wakely e99d9607f0 libstdc++: Add copyright and license text to new generated headers 
contrib/ChangeLog:

	* unicode/gen_libstdcxx_unicode_data.py: Add copyright and
	license text to the output.

libstdc++-v3/ChangeLog:

	* include/bits/text_encoding-data.h: Regenerate.
	* include/bits/unicode-data.h: Regenerate.
	* scripts/gen_text_encoding_data.py: Add copyright and license
	text to the output.
 2024-02-04 21:40:23 +00:00
Jonathan Wakely ea314ccd62 libstdc++: Fix Unicode property detection functions 
Fix some copy & pasted logic in __is_extended_pictographic. This
function should yield false for the values before the first edge, not
true. Also add a missing boundary condition check in __incb_property.

Also Fix an off-by-one error in _Utf_iterator::operator++() that would
make dereferencing a past-the-end iterator undefined (where the intended
design is that the iterator is always incrementable and dereferenceable,
for better memory safety).

Also simplify the grapheme view iterator, which still contained some
remnants of an earlier design I was experimenting with.

Slightly tweak the gen_libstdcxx_unicode_data.py script so that the
_Gcb_property enumerators are in the order we encounter them in the data
file, instead of sorting them alphabetically. Start with the "Other"
property at value 0, because that's the default property for anything
not in the file. This makes no practical difference, but seems cleaner.
It causes the values in the __gcb_edges table to change, so can only be
done now before anybody is using this code yet. The enumerator values
and table entries become ABI artefacts for the function using them.

contrib/ChangeLog:

	* unicode/gen_libstdcxx_unicode_data.py: Print out Gcb_property
	enumerators in the order they're seen, not alphabetical order.

libstdc++-v3/ChangeLog:

	* include/bits/unicode-data.h: Regenerate.
	* include/bits/unicode.h (_Utf_iterator::operator++()): Fix off
	by one error.
	(__incb_property): Add missing check for values before the
	first edge.
	(__is_extended_pictographic): Invert return values to fix
	copy&pasted logic.
	(_Grapheme_cluster_view::_Iterator): Remove second iterator
	member and find end of cluster lazily.
	* testsuite/ext/unicode/grapheme_view.cc: New test.
	* testsuite/ext/unicode/properties.cc: New test.
	* testsuite/ext/unicode/view.cc: New test.
 2024-01-09 23:42:59 +00:00
Jonathan Wakely 37a4c5c23a libstdc++: Add Unicode-aware width estimation for std::format 
This implements the requirements in the following proposals, which
dictate how std::format deals with non-ASCII strings:
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1868r1.html
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2572r1.html
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2675r1.pdf

There are two parts to this. The width estimation for strings must only
count the width of the first character in an extended grapheme cluster.
That requires implementing the algorithm for detecting cluster breaks,
which requires a number of lookup tables of the grapheme cluster break
properties (and Indic_Conjunct_Break and Extended_Pictographic
properties) of every code point. Additionally, some characters have a
field width of 2, which requires another lookup table of field widths
for every code point.  The tables added in this commit do not contain
entries for every code point from 0 to 0x10FFFF as that would be very
inefficient and use too much memory. Instead the tables only contain the
code points that form an "edge" for a property, omitting all the code
points that have the same property as the preceding one. We can use a
binary search to find the closest code point in the table that is not
greater than the one we're looking for.

The tables are generated by a new Python script added to the
contrib/unicode directory, and a new data file downloaded from the
Unicode Consortium website.

The rules for extended grapheme cluster breaking are implemented for the
latest Unicode standard, version 15.1.0.

libstdc++-v3/ChangeLog:

	* include/Makefile.am: Add new headers.
	* include/Makefile.in: Regenerate.
	* include/bits/unicode.h: New file.
	* include/bits/unicode-data.h: New file.
	* include/std/format: Include <bits/unicode.h>.
	(__literal_encoding_is_utf8): Move to <bits/unicode.h>.
	(_Spec::_M_fill): Change type to char32_t.
	(_Spec::_M_parse_fill_and_align): Read a Unicode scalar value
	instead of a single character.
	(__write_padded): Change __fill_char parameter to char32_t and
	encode it into the output.
	(__formatter_str::format): Use new __unicode::__field_width and
	__unicode::__truncate functions.
	* include/std/ostream: Adjust namespace qualification for
	__literal_encoding_is_utf8.
	* include/std/print: Likewise.
	* src/c++23/print.cc: Add [[unlikely]] attribute to error path.
	* testsuite/ext/unicode/view.cc: New test.
	* testsuite/std/format/functions/format.cc: Add missing examples
	from the standard demonstrating alignment with non-ASCII
	characters. Add examples checking correct handling of extended
	grapheme clusters.

contrib/ChangeLog:

	* unicode/README: Add notes about generating libstdc++ tables.
	* unicode/GraphemeBreakProperty.txt: New file.
	* unicode/emoji-data.txt: New file.
	* unicode/gen_libstdcxx_unicode_data.py: New file.
 2024-01-08 01:14:50 +00:00