Unicode conversion (attention all devs)

[takeshimiya]

I tried to compile VERSION_1_0 CVS in UNICODE, wx2.6 CVS, win32, MinGW, but I get lots of undefined references


Switching to target: sdk
Linking dynamic library: devel\codeblocks.dll
Creating library file: devel\libcodeblocks.a
.objs\2.6\sdk\autodetectcompilers.o(.text+0x10d): In function `ZN19AutoDetectCompilersC2EP8wxWindow':
D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:19: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'
.objs\2.6\sdk\autodetectcompilers.o(.text+0x663):D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:49: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'
.objs\2.6\sdk\autodetectcompilers.o(.text+0x7eb): In function `ZN19AutoDetectCompilersC1EP8wxWindow':
D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:19: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'
.objs\2.6\sdk\autodetectcompilers.o(.text+0xd41):D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:49: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'
...
Process terminated with status 1 (0 minutes, 22 seconds)
52 errors, 0 warnings


These errors are expected, or anyone compiled all ok with these settings?
VERSION_1_0 is wx2.6 compatible??

[rickg22]

Takeshi Miya:
You need to compile the XRC and STC libraries, too. I'm sure you got something wrong in your compilation. Anyway, unicode is _NOT_ ready for primetime yet. Only try to compile C::B in unicode if you're willing to assist in the conversion. (like handling of files, etc).

[takeshimiya]

Yes, I want to compile VERSION_1_0 to hunt those bugs (normally I would be using HEAD).

But I compiled STC and XRC=1 as options, must be something else.

[rickg22]

Did you compile them with wxUSE_UNICODE, too?

[takeshimiya]

No, what I have done is:

Code

cd <WXWIN>\build\msw
mingw32-make -f makefile.gcc clean
mingw32-make -f makefile.gcc USE_XRC=1 SHARED=1 MONOLITHIC=1 BUILD=release UNICODE=1 VENDOR=cb

cd ..\..\contrib\build\stc
mingw32-make -f makefile.gcc clean
mingw32-make -f makefile.gcc USE_XRC=1 SHARED=1 MONOLITHIC=1 BUILD=release UNICODE=1 VENDOR=cb

I could be wrong, but from what I understand, UNICODE=1 already defines wxUSE_UNICODE 1.

And then, in the C::B project in the "Compiler defines" I've added __UNICODE__ and wxUSE_UNICODE 1 (along with HAVE_W32API_H, __WXMSW__ and WXUSINGDLL). I'm not sure if I'ts necessary or not.

[me22]

Yay!  Success on Linux! ( after a few changes )I did a `cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/codeblocks co -r VERSION_1_0 codeblocks`, then `make -f Makefile.unix` then, after fixing everything, `./update`.  It even runs, although it doesn't open files.

Summary of changes:
- added #include <cctype> to a few files that needed it
- fixed some default parameters in headers
- changed the link libs to the unicode ones.

Here's my diff:

Code

Index: src/Makefile.unix
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/Makefile.unix,v
retrieving revision 1.21.2.1
diff -u -r1.21.2.1 Makefile.unix
--- src/Makefile.unix	25 Jul 2005 19:05:21 -0000	1.21.2.1
+++ src/Makefile.unix	13 Aug 2005 19:13:38 -0000
@@ -7,8 +7,10 @@
 # Compiler used:    GNU GCC Compiler
 
 ### Variables used in this Makefile
-LIB_WXXRC=wx_gtk2_xrc-2.4
-LIB_WXSTC=wx_gtk2_stc-2.4
+#LIB_WXXRC=wx_gtk2_xrc-2.4
+#LIB_WXSTC=wx_gtk2_stc-2.4
+LIB_WXXRC=wx_gtk2u_xrc-2.4
+LIB_WXSTC=wx_gtk2u_stc-2.4
 tinyXML_CC=gcc
 tinyXML_CPP=g++
 tinyXML_LD=g++
Index: src/plugins/codecompletion/nativeparser.cpp
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/plugins/codecompletion/nativeparser.cpp,v
retrieving revision 1.20.2.6
diff -u -r1.20.2.6 nativeparser.cpp
--- src/plugins/codecompletion/nativeparser.cpp	7 Aug 2005 20:55:56 -0000	1.20.2.6
+++ src/plugins/codecompletion/nativeparser.cpp	13 Aug 2005 19:13:38 -0000
@@ -38,6 +38,8 @@
 #include "parser/parser.h"
 #include <compilerfactory.h>
 
+#include <cctype>
+
 BEGIN_EVENT_TABLE(NativeParser, wxEvtHandler)
 	EVT_MENU(THREAD_START, NativeParser::OnThreadStart)
 	EVT_MENU(THREAD_END, NativeParser::OnThreadEnd)
Index: src/plugins/codecompletion/parser/parserthread.cpp
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/plugins/codecompletion/parser/parserthread.cpp,v
retrieving revision 1.16.2.2
diff -u -r1.16.2.2 parserthread.cpp
--- src/plugins/codecompletion/parser/parserthread.cpp	7 Aug 2005 20:55:56 -0000	1.16.2.2
+++ src/plugins/codecompletion/parser/parserthread.cpp	13 Aug 2005 19:13:39 -0000
@@ -29,6 +29,8 @@
 #include <wx/msgdlg.h>
 #include <globals.h>
 
+#include <cctype>
+
 int THREAD_START = wxNewId();
 int THREAD_END = wxNewId();
 int NEW_TOKEN = wxNewId();
Index: src/plugins/codecompletion/parser/tokenizer.cpp
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/plugins/codecompletion/parser/tokenizer.cpp,v
retrieving revision 1.3.2.1
diff -u -r1.3.2.1 tokenizer.cpp
--- src/plugins/codecompletion/parser/tokenizer.cpp	7 Aug 2005 20:55:56 -0000	1.3.2.1
+++ src/plugins/codecompletion/parser/tokenizer.cpp	13 Aug 2005 19:13:39 -0000
@@ -28,6 +28,8 @@
 #include <wx/file.h>
 #include <wx/msgdlg.h>
 
+#include <cctype>
+
 Tokenizer::Tokenizer(const wxString& filename)
 	: m_Filename(filename),
 	m_BufferLen(0),
Index: src/plugins/compilergcc/makefilegenerator.h
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/plugins/compilergcc/makefilegenerator.h,v
retrieving revision 1.12.2.3
diff -u -r1.12.2.3 makefilegenerator.h
--- src/plugins/compilergcc/makefilegenerator.h	22 Jul 2005 08:51:58 -0000	1.12.2.3
+++ src/plugins/compilergcc/makefilegenerator.h	13 Aug 2005 19:13:39 -0000
@@ -36,9 +36,9 @@
         void DoAppendCompilerOptions(wxString& cmd, ProjectBuildTarget* target = 0L, bool useGlobalOptions = false);
         void DoAppendLinkerOptions(wxString& cmd, ProjectBuildTarget* target = 0L, bool useGlobalOptions = false);
         void DoAppendLinkerLibs(wxString& cmd, ProjectBuildTarget* target = 0L, bool useGlobalOptions = false);
-        void DoAppendIncludeDirs(wxString& cmd, ProjectBuildTarget* target = 0L, const wxString& prefix = "-I", bool useGlobalOptions = false);
-        void DoAppendResourceIncludeDirs(wxString& cmd, ProjectBuildTarget* target = 0L, const wxString& prefix = "-I", bool useGlobalOptions = false);
-        void DoAppendLibDirs(wxString& cmd, ProjectBuildTarget* target = 0L, const wxString& prefix = "-L", bool useGlobalOptions = false);
+        void DoAppendIncludeDirs(wxString& cmd, ProjectBuildTarget* target = 0L, const wxString& prefix = _T("-I"), bool useGlobalOptions = false);
+        void DoAppendResourceIncludeDirs(wxString& cmd, ProjectBuildTarget* target = 0L, const wxString& prefix = _T("-I"), bool useGlobalOptions = false);
+        void DoAppendLibDirs(wxString& cmd, ProjectBuildTarget* target = 0L, const wxString& prefix = _T("-L"), bool useGlobalOptions = false);
         void DoAddVarsSet(wxString& buffer, CustomVars& vars);
         void DoAddMakefileVars(wxString& buffer);
 #ifdef __WXMSW__
Index: src/plugins/pluginwizard/pluginwizarddlg.h
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/plugins/pluginwizard/pluginwizarddlg.h,v
retrieving revision 1.3
diff -u -r1.3 pluginwizarddlg.h
--- src/plugins/pluginwizard/pluginwizarddlg.h	14 Mar 2005 09:47:04 -0000	1.3
+++ src/plugins/pluginwizard/pluginwizarddlg.h	13 Aug 2005 19:13:39 -0000
@@ -19,7 +19,7 @@
 		void OnOKClick(wxCommandEvent& event);
 		void OnNameChange(wxCommandEvent& event);
 		void DoGuardBlock();
-		void DoAddHeaderOption(wxString& buffer, bool has, const wxString& retVal = " -1");
+		void DoAddHeaderOption(wxString& buffer, bool has, const wxString& retVal = _T(" -1"));
 		void DoAddHeaderTool(wxString& buffer);
 		void DoAddHeaderCompiler(wxString& buffer);
 		void DoAddHeaderDebugger(wxString& buffer);
Index: src/sdk/projectloader.h
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/sdk/projectloader.h,v
retrieving revision 1.5.2.2
diff -u -r1.5.2.2 projectloader.h
--- src/sdk/projectloader.h	6 Aug 2005 22:13:05 -0000	1.5.2.2
+++ src/sdk/projectloader.h	13 Aug 2005 19:13:39 -0000
@@ -38,11 +38,11 @@
         void DoUnitOptions(TiXmlElement* parentNode, ProjectFile* file);
         
         void BeginOptionSection(wxString& buffer, const wxString& sectionName, int nrOfTabs);
-        bool DoOptionSection(wxString& buffer, const wxArrayString& array, int nrOfTabs, const wxString& optionName = "option");
+        bool DoOptionSection(wxString& buffer, const wxArrayString& array, int nrOfTabs, const wxString& optionName = _("option"));
         void EndOptionSection(wxString& buffer, const wxString& sectionName, int nrOfTabs);
 
         // shortcut that calls BeginOptionSection(), DoOptionSection() and EndOptionSection()
-        void SaveOptions(wxString& buffer, const wxArrayString& array, const wxString& sectionName, int nrOfTabs, const wxString& optionName = "option", const wxString& extra = "");
+        void SaveOptions(wxString& buffer, const wxArrayString& array, const wxString& sectionName, int nrOfTabs, const wxString& optionName = _("option"), const wxString& extra = wxEmptyString);
 	private:
         void SaveCompilerOptions(wxString& buffer, CompileOptionsBase* object, int nrOfTabs);
         void SaveResourceCompilerOptions(wxString& buffer, CompileOptionsBase* object, int nrOfTabs);

[cyberkoa]

I tried to compile VERSION_1_0 CVS in UNICODE, wx2.6 CVS, win32, MinGW, but I get lots of undefined references


Switching to target: sdk
Linking dynamic library: devel\codeblocks.dll
Creating library file: devel\libcodeblocks.a
.objs\2.6\sdk\autodetectcompilers.o(.text+0x10d): In function `ZN19AutoDetectCompilersC2EP8wxWindow':
D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:19: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'
.objs\2.6\sdk\autodetectcompilers.o(.text+0x663):/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:49: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'
.objs\2.6\sdk\autodetectcompilers.o(.text+0x7eb): In function `ZN19AutoDetectCompilersC1EP8wxWindow':
D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:19: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'
.objs\2.6\sdk\autodetectcompilers.o(.text+0xd41):/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:49: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'
...
Process terminated with status 1 (0 minutes, 22 seconds)
52 errors, 0 warnings


These errors are expected, or anyone compiled all ok with these settings?
VERSION_1_0 is wx2.6 compatible??

takeshimiya, I get exactly the error message you got .

Later , I have tried the following,
1. Rebuild , result fail

2. I suddenly remember that the unicode compilation of wxwidgets actually genarated a setup.h with UNICODE=0 !!!! It was my experience last time.

Therefore , I go to the unicode version of wxwidgets , and get the corresponding setup.h (default is under gcc_dllu\mswu\ if I am not mistaken)

I do a clean , dist clean , and recompile .

The link error message gone !!!! and left only 3 warning message ..

[me22]

Now Saving/Loading Files Works Too!Seems the problem was a simple case of a void* buffer being treaded as a wxChar* buffer instead of a char* buffer.

Here's the diff:

Code

Index: src/sdk/cbeditor.cpp
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/sdk/cbeditor.cpp,v
retrieving revision 1.37.2.7
diff -u -r1.37.2.7 cbeditor.cpp
--- src/sdk/cbeditor.cpp	10 Aug 2005 16:56:47 -0000	1.37.2.7
+++ src/sdk/cbeditor.cpp	15 Aug 2005 05:21:13 -0000
@@ -41,6 +41,8 @@
 #include "cbplugin.h"
 #include "cbeditorprintout.h"
 
+#include <cstring> // std::strlen
+
 /* This struct holds private data for the cbEditor class.
  * It's a paradigm to avoid rebuilding the entire project (as cbEditor is a basic dependency)
  * for just adding a private var or method.
@@ -600,13 +602,12 @@
     if (!file.IsOpened())
         return false;
 
-    wxChar* buff = st.GetWriteBuf(file.Length());
+    void* buff = st.GetWriteBuf(file.Length());
     file.Read(buff, file.Length());
     file.Close();
-    st.UngetWriteBuf();
-
-    m_pControl->InsertText(0, st);
+    m_pControl->InsertText(0, _U( (char const *)buff ));
     m_pControl->EmptyUndoBuffer();
+    st.UngetWriteBuf();
 
     // mark the file read-only, if applicable
     bool read_only = !wxFile::Access(m_Filename.c_str(), wxFile::write);
@@ -647,8 +648,9 @@
     }
     
     wxFile file(m_Filename, wxFile::write);
-    if (file.Write(m_pControl->GetText().c_str(), m_pControl->GetTextLength()) == 0 &&
-        m_pControl->GetTextLength() != 0)
+    wxWX2MBbuf utf_text = m_pControl->GetText().mb_str(wxConvUTF8);
+    size_t utf_text_length = std::strlen( utf_text );
+    if (file.Write( utf_text, utf_text_length ) == 0 && utf_text_length != 0)
     {
         return false; // failed; file is read-only?
     }

[edit] Note that this only does files -- workspaces don't seem to save/load ( dunno which ) properly, and I haven't tried projects.

[rickg22]

if you're using void, why not just use *char ? And comment the line, saying "this is a byte-buffer, don't replace with wxChar" or something.

[me22]

The point of the void* was to emphasize that it's not a wxChar*.

I could just use unsigned char *, but then it would show up if someone searched in all files for char .

[zieQ]

Help, unicode men have modified the parsing of MSVC  :lol:

For example this line:

Code

if (line.StartsWith(_T("Project_Dep_Name"))) {

Does _T means that we do nothing to the string (i.e. keep a char* representation, no unicode conversion)? I'm quite lost with _X() macros, can someone sum up them?

[mandrav]

Help, unicode men have modified the parsing of MSVC  :lol:

For example this line:

Code

if (line.StartsWith(_T("Project_Dep_Name"))) {

Does _T means that we do nothing to the string (i.e. keep a char* representation, no unicode conversion)? I'm quite lost with _X() macros, can someone sum up them?

wxT(), _T(): unicode string, not-translatable (IIRC just puts L before the quote, i.e. L"text")
_(): unicode string, translatable (use it for english messages and stuff)

There are also some conversion macros in sdk/settings.h:
_U(), _UU(), _C(), _CC()
for the conversion between char* and unicode.

Yiannis.

[takeshimiya]

About char*, that's why I suggested to use byte* as typedef.

It will make clear that you are using a byte buffer instead of a character string.

char typedefed as byte or wxByte would be good.

[tiwag]

from sdk/settings.h

Code

#if wxUSE_UNICODE
    #define _UU(x,y) wxString((x),(y))
    #define _CC(x,y) (x).mb_str((y))
#else
    #define _UU(x,y) (x)
    #define _CC(x,y) (x)
#endif

#define _U(x) _UU((x),wxConvUTF8)
#define _C(x) _CC((x),wxConvUTF8)

from wxWidgets-Help

Code

--------------------------------------------------------------------------------

wxT
wxChar wxT(char ch)

const wxChar * wxT(const char *s)

wxT() is a macro which can be used with character and string literals (in other words, 'x' or "foo") to automatically convert them to Unicode in Unicode build configuration. Please see the Unicode overview for more information.

This macro is simply returns the value passed to it without changes in ASCII build. In fact, its definition is: 

#ifdef UNICODE
#define wxT(x) L ## x
#else // !Unicode
#define wxT(x) x
#endif

--------------------------------------------------------------------------------

_
const wxChar * _(const char *s)

This macro expands into a call to wxGetTranslation function, so it marks the message for the extraction by xgettext just as wxTRANSLATE does, but also returns the translation of the string for the current locale during execution.

Don't confuse this macro with _T()!



--------------------------------------------------------------------------------

wxPLURAL
const wxChar * wxPLURAL(const char *sing, const char *plur, size_tn)

This macro is identical to _() but for the plural variant of wxGetTranslation.



--------------------------------------------------------------------------------

_T
wxChar _T(char ch)

const wxChar * _T(const wxChar ch)

This macro is exactly the same as wxT and is defined in wxWidgets simply because it may be more intuitive for Windows programmers as the standard Win32 headers also define it (as well as yet another name for the same macro which is _TEXT()).

Don't confuse this macro with _()!

--------------------------------------------------------------------------------

summary     (please someone might correct if i'm wrong or add missing points)

Code

_()

- text's which might be translated to other user-languages

Code

_T()
wxT()

- fixed text's - like XRC resources object names

Code

_U()

- conversion from char* to wxString

Code

_C()

- multibyte C string  see wxhelp (wxMBConv classes overview)


HTH

[tiwag]

..char typedefed as byte or wxByte would be good.

i always used  "byte"  as typedef'ed  "unsigned char"