[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kopete-devel
Subject:    [Kopete-devel] [Bug 72917] UTF8 and other cause XML parsing errors,
From:       Thiago Macieira <thiagom () mail ! com>
Date:       2004-01-26 20:29:11
Message-ID: 20040126202911.1612.qmail () ktown ! kde ! org
[Download RAW message or body]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
      
http://bugs.kde.org/show_bug.cgi?id=72917      
thiagom@mail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |thiagom@mail.com



------- Additional Comments From thiagom@mail.com  2004-01-26 21:29 -------
> I don't know KDE/QT that well. How come utf cannot fail. Utf-8 is designed
> so that it is unlikely that a non-utf-8 string can recognized as utf-8. If
> the UTF-decoder cannot fail, then what does it do when it encounters an
> illegal sequence? 
 
That's actually the very reason we're getting this problem, Robin.

Since Qt 3.2.x, TrollTech introduced a modification to its UTF-8 decoder in response to a bug report from \
us. The original problem was that files whose names or paths were not encodable in the user's selected \
locale could not be opened by KDE applications nor renamed in Konqueror. We had proposed a solution, but \
TrollTech chose instead to accept any input as valid UTF-8: when it sees an invalid sequence, it encodes \
the bytes as a pair of UTF-16 surrogates. The decoder then restores the original byte.

This renders the operation ToUTF8(FromUTF8(any_string)) == any_string true in every case. The \
side-effect: Latin1 and other kinds of strings are accepted in Qt as valid UTF-8, but other programs \
don't accept them (our XML parser being one of those).

> Perhaps Latin9 (ISO-8859-15) should be attempted instead of Latin1. The
> difference is that a few characters that were "never" used were replaces by
> some that actually are used. 

That's ok in principle, but not so from the technical point of view. The Latin1-to-Unicode conversion is \
very simple and fast, since all Latin1's 256 codepoints map 1:1 to Unicode's first 256 codepoints. For \
Latin9 and any other encoding, a non-trivial conversion through table lookups must be performed.

> if( userCodec == QTextCodec::codecForName("utf") ) 
 
Please don't write that. That requires a codec lookup internally by QTextCodec. Instead, use \
userCodec->mibEnum() == 106 to detect the UTF-8 encoder.

A couple more opinions from me:
- trying UTF-8 before the user's locale:
Makes sense, since we may catch UTF-8 being used. The probability of someone writing valid text in \
another encoding and it being valid UTF-8 is very low.

- the user-selected codec fails decoding:
Decode as Latin1, but let the user know about this fact (a non-intrusive warning or an "bug" icon like \
Konqueror's for JavaScript errors).

Important: KStringHandler::isUtf8 rejects control characters, including ASCII 3 used by mIRC-colouring in \
IRC. _______________________________________________
Kopete-devel mailing list
Kopete-devel@kde.org
https://mail.kde.org/mailman/listinfo/kopete-devel


[prev in list] [next in list] [prev in thread] [next in thread]