If a character is not a valid Unicode codepoint, i.e. one of
the code points reserved for surrogate pairs or a code point
above 0x10FFFF, don't add it to a wmem_strbuf when converting
from other encodings but add a replacement character instead, by
using a new wmem_strbuf_append_unichar_validated() function.
Now we produce valid UTF-8 in various situations where UCS-2 or UTF-32
can encode unpaired surrogate codepoints. Consolidate some related
checks that are now redundant.
Also add a replacement character to the end of invalid UCS-2 strings
with an odd number of bytes, as done with UTF-16 and UTF-32.
Fix#18508
When decoding UCS-4/UTF-32, map Unicode code points above
0x10FFFFF to REPLACEMENT CHARACTER, as they are not legal,
and would create invalid UTF-8.
Also if the number of bytes given is not a multiple of 4,
insert a replacement character at the end as well.
This is two long standing todos. Fixes#18435.
Add support internally to using iconv (always present with glib) to convert
strings from various encodings to UTF-8 (using REPLACEMENT CHARACTER as
recommended), and use that to support GB 18030 and EUC-KR. Replace call
directly to iconv in ANSI 637 for EUC-KR to new API. Update comments
and documentation around character encodings. It is possible to replace
the calls to iconv with an internal decoder later. Tested on Linux and
on Windows (including with illegal characters). Closes#16630.
Implement the Unicode Standard "best practices" for replacing ill-formed
sequences with the Unicode REPLACEMENT CHARACTER. Add wmem_strbuf_append_len
for appending strings with embedded null characters. Clarify why
wmem_strbuf_grow() doesn't always ensure that there's enough room for
a new string, and short-circuit some tests there. Related to #14948
Add an encoding for "unpacked" 3GPP TS 23.038 7-bit strings, in which
each code position is in a byte of its own, rather than with the code
positions packed into 7 bits. Rename the packed encoding to explicitly
indicate that it's packed.
Add an encoding for ETSI TS 102 221 Annex A strings.
Use the new encodings.
Change all wireshark.org URLs to use https.
Fix some broken links while we're at it.
Change-Id: I161bf8eeca43b8027605acea666032da86f5ea1c
Reviewed-on: https://code.wireshark.org/review/34089
Reviewed-by: Guy Harris <guy@alum.mit.edu>
The "Basic code table" in ISO 646 is mostly ASCII, but some code points
either 1) have more than one glyph that can be assigned to them or 2)
have no glyph assigned to them. National versions choose one of the two
glyphs for the code points in group 1) and assign specific glyphs to the
code points in group 2); the International Reference Version assigns the
same glyphs to those code points as does ASCII.
For the "Basic code table" encoding, we map the code points in groups 1)
and 2) to a REPLACEMENT CHARACTER; additional encodings can be added for
the national versions.
Add ENC_ISO_646_IRV (International Reference Version) as an alias for
ENC_ASCII.
Expand some comments, and add some comments, while we're at it.
Change-Id: I4f1b5e426ec193775e919731c5cae1224dc65115
Reviewed-on: https://code.wireshark.org/review/33941
Petri-Dish: Guy Harris <guy@alum.mit.edu>
Tested-by: Petri Dish Buildbot
Reviewed-by: Guy Harris <guy@alum.mit.edu>
While we're at it, add the Euro to code page 1251, expand the comments
for 1250 and 1251 and some DOS code pages, and add support for code page
1251 to tvb_get_stringz_enc().
Change-Id: I053d58f87cac26ad7c109e2f1cd8807ffec0622d
Reviewed-on: https://code.wireshark.org/review/33342
Petri-Dish: Guy Harris <guy@alum.mit.edu>
Tested-by: Petri Dish Buildbot
Reviewed-by: Guy Harris <guy@alum.mit.edu>
Those routines can handle any single-byte character set whose characters
map to characters in the Basic Multilingual Plane; it could be used for
extended ASCII, but we have another routine for that, mapping only
characters with code points > 0x7f, so we just say "nonascii" rather
than "ebcdic".
Change-Id: I3d55b5d58e3e7ab08f3dfbfdb57a0301a30e71d4
Reviewed-on: https://code.wireshark.org/review/19214
Reviewed-by: Guy Harris <guy@alum.mit.edu>
Have a routine that takes a 256-element translation table and uses it to
map various flavors of EBCDIC to Unicode. Have separate translation
tables for "common" EBCDIC (everything that's the same in all EBCDIC
code pages that include the original EBCDIC characters) and EBCDIC code
page 037. Add ENC_EBCDIC_CP037 for code page 037.
Change-Id: Ia882b3c0abef9e30eb54cd47396e6fa0d6342044
Reviewed-on: https://code.wireshark.org/review/19212
Reviewed-by: Guy Harris <guy@alum.mit.edu>
Bug: 13032
Change-Id: I6bf2cc2c43a6262d899a304df6576d9831115966
Reviewed-on: https://code.wireshark.org/review/18350
Petri-Dish: Michael Mann <mmann78@netscape.net>
Tested-by: Petri Dish Buildbot <buildbot-no-reply@wireshark.org>
Reviewed-by: Michael Mann <mmann78@netscape.net>
Use it in epan/tvbuff.c.
Do some other cleanups while we're at it.
Change-Id: I7aed37a568373b896aacfd23f986d445b58b77b7
Reviewed-on: https://code.wireshark.org/review/1342
Reviewed-by: Guy Harris <guy@alum.mit.edu>
This moves a bunch of character set knowledge into epan/charsets.c.
Change-Id: Ieb79dcaac9753c77703af756b666ad2ca9385d9e
Reviewed-on: https://code.wireshark.org/review/1339
Reviewed-by: Guy Harris <guy@alum.mit.edu>
charsets.c is already place with huge number of conversion tables.
Also make gsm_default_alphabet gunichar2, all values fits in 2 bytes.
Change-Id: Ia5ab6c176b4fec21ec76b06513c1d00794ba10ef
Reviewed-on: https://code.wireshark.org/review/1328
Reviewed-by: Anders Broman <a.broman58@gmail.com>
(Using sed : sed -i '/^ \* \$Id\$/,+1 d')
Fix manually some typo (in export_object_dicom.c and crc16-plain.c)
Change-Id: I4c1ae68d1c4afeace8cb195b53c715cf9e1227a8
Reviewed-on: https://code.wireshark.org/review/497
Reviewed-by: Anders Broman <a.broman58@gmail.com>
support DVB-SI character tables (EN 300 468) in a generic way
From me
move things to charsets.c/.h
distinguish between single and multi byte encoding for some tables
(so that the highlighted bytes match the displayed value)
no character table byte -> length 0, use default table
svn path=/trunk/; revision=53886
Bernhard Dick