When will support UTF-8 editor?

[takeshimiya]

Let's see: TiniXml loads files in UTF-8, it can't load any other Unicode encoding (neither from a file or in memory).

In memory it stores the UTF-8 encoded strings as an array of chars. Each byte IS NOT a character (coincidentaly only in english a byte=character).

However, wxWidgets or Windows for the matter, handle Unicode in other encoding (in memory): UTF-16 <multibyte encoding>.

So if we want the Unicode data from TiniXml, we must convert UTF-8->UTF-16. And if we want to talk from wxWidgets to TiniXml, UTF16->UTF-8.

The reason mentioned above by thomas, that it appears to work "somehow", can be because wxWidgets uses (I think) wxMBConv classes to do this conversion by default (it assumes UTF-8 if you don't specify another encoding) when compiled in Unicode mode.

[thomas]

UTF-8 is a way to write Unicode in a backwards-compatible way on media that support 8bit per character tokens. It is a variable length format which uses between 1 octet (for ANSI characters) to 6 octets. Most languages, except the really exotic ones can usually be represented with sequences of 1-2 octets per character.

Unicode is a family of standards (I know at least two different standards) which represent characters in words of 16 bits or 32 bits. Maybe there are even more standards which I do not know about, but that does not matter. The characters that UTF-8 encodes are really words of 16 or 32 bits.

If you are to represent Unicode text in a wxString, this is done by using wchar_t characters. On Windows, these are 16 bits, on my Linux box, these are 32 bits. Whatever size it is, sizeof(wchar_t) != sizeof(char), because if you pass "ABC" then you do not really pass 0x41, 0x42, 0x43 -- in reality, you pass two (four) times as much data, so for example 0x41, 0x00, 0x42, 0x00, 0x43, 0x00. (In fact I have no idea about the actual encoding -- what matters though, is that these are 16/32 bit values).

So obviously it cannot work reliably if you hand this data to some library which expects characters to be octets. It may work for a while, and then fail randomly due to a thing as simple as calling strlen() on a character string that happens to have 0x00 as the upper byte somewhere.

[kagerato]

Let's see: TiniXml loads files in UTF-8, it can't load any other Unicode encoding (neither from a file or in memory).

There's no reason why anyone should require support of multiple Unicode encodings.  People may prefer UTF-16 or some other structure, but it is perfectly possible to convert losslessly.  In any case, this is a tangental discussion irrelevant from my original point.

So obviously it cannot work reliably if you hand this data to some library which expects characters to be octets.

A library that properly supports UTF-8 does not expect each character to be eight bits.  UTF-8 is a variable-width representation.  Characters "wider" than 8 bits come into play when the MSB is set.  (Of course, this only occurs when the encoding is not strict ASCII.)

If the code assumes a particular width or byte alignment when it does not exist (as is clearly the case with UTF-8), then it is an improper implementation -- to say the least.  The claim that TinyXML supports UTF-8 would therefore be false.

Understand what I meant now?

[takeshimiya]

TinyXml supports UTF-8.
How are you supposed to store in memory UTF-8 encoded in memory then...?

[thomas]

We can probably work around it. When first writing the ConfigManager, I used mb_str() a few times when passing data to tinyXML, and c_str() in other places. Actually I don't remember the reason I did that in the first place, any more. I think it was because certain things were ANSI anyway.... Either way, Yiannis was nice enough to change most of them to mb_str() while I wasn't looking, and that was really a good idea. It means that now we are feeding tinyXML octet streams (except for very few exceptions), so it should really work.

The reason it still does not work 100% is because the CRC calculation for the layout is not good and because we may have missed one or two spots. But I expect it to work reliably once that has been adressed.

So.... hopefully no reason to worry about tinyXML any more.

[kagerato]

TinyXml supports UTF-8.
How are you supposed to store in memory UTF-8 encoded in memory then...?

What encoding do you use to store your text in RAM, you mean?  I see two optimal ways:

1.) As UTF-8 (which, once again, is variable-width)
2.) As UTF-16 (Windows and other systems seem to accept Unicode data most often using this encoding)

#1 makes it a simple matter to read and write data between disk and RAM, since you'll very likely be using UTF-8 for both.  The latter option is better if you're commonly calling functions from system or third-party libraries that require UTF-16.  The alternative to #2 in the same situation is multiple copies of the text in different encodings, which is not only messy, tedious, and a potential source of bugs, but also a misuse of RAM and processing.

In any case, thomas sounds like he knows how to manage whatever the problem is/was.  I still do not completely understand the nature of the problem; hence why I asked.

[thomas]

The problem is that we store all text in UTF-16 using wchar[], and we do not have a choice to do otherwise. tinyXML does not support wchar. Therefore, we convert to UTF-8 just before passing the data to tinyXML.

Also, wxScintilla might not be completely Unicode-safe. This is only a suspicion, not necessarily true. While browsing the sources, I have spotted several places where they use chars as indices or compare against const char values. Unless these are only applied on text fragments which have been converted to UTF-8 (which I don't know, maybe they are?), this may be an issue. In that case, we will have another problem which is not easily solved.

[takeshimiya]

Regarding Scintilla, I once asked the SciTE developers if support for Unicode filenames was feasible, and they answered that someone once started working on that, but it wasn't an easy task and requiered a rather major rewrite.
Anyways, Unicode text in Scintilla seems to work ok, but we always can expect bugs because they have their own string class, and I noticed some const chars* around the code too, so I'm not sure if it supports fully Unicode.

[dbtsai]

Well, I would like to try utf-8 version C::B,

but I can not compile it well....

Could anyone release an utf-8 compiled version??

And let people to try what's going wrong!!

^_^

[thomas]

Well, I would like to try utf-8 version C::B,

but I can not compile it well....

http://forums.codeblocks.org/index.php?topic=1701.0