Author Topic: Unicode conversion (attention all devs)  (Read 67170 times)

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: Unicode conversion (attention all devs)
« Reply #90 on: August 10, 2005, 06:25:00 am »
I tried to compile VERSION_1_0 CVS in UNICODE, wx2.6 CVS, win32, MinGW, but I get lots of undefined references


Switching to target: sdk
Linking dynamic library: devel\codeblocks.dll
Creating library file: devel\libcodeblocks.a
.objs\2.6\sdk\autodetectcompilers.o(.text+0x10d): In function `ZN19AutoDetectCompilersC2EP8wxWindow':
D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:19: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'
.objs\2.6\sdk\autodetectcompilers.o(.text+0x663):D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:49: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'

.objs\2.6\sdk\autodetectcompilers.o(.text+0x7eb): In function `ZN19AutoDetectCompilersC1EP8wxWindow':
D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:19: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'
.objs\2.6\sdk\autodetectcompilers.o(.text+0xd41):D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:49: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'

...
Process terminated with status 1 (0 minutes, 22 seconds)
52 errors, 0 warnings


These errors are expected, or anyone compiled all ok with these settings?
VERSION_1_0 is wx2.6 compatible??
« Last Edit: August 10, 2005, 06:28:32 am by takeshimiya »

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Unicode conversion (attention all devs)
« Reply #91 on: August 10, 2005, 07:46:30 am »
Takeshi Miya:
You need to compile the XRC and STC libraries, too. I'm sure you got something wrong in your compilation. Anyway, unicode is _NOT_ ready for primetime yet. Only try to compile C::B in unicode if you're willing to assist in the conversion. (like handling of files, etc).

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: Unicode conversion (attention all devs)
« Reply #92 on: August 10, 2005, 07:27:16 pm »
Yes, I want to compile VERSION_1_0 to hunt those bugs (normally I would be using HEAD).

But I compiled STC and XRC=1 as options, must be something else.

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Unicode conversion (attention all devs)
« Reply #93 on: August 10, 2005, 07:33:31 pm »
Did you compile them with wxUSE_UNICODE, too?

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: Unicode conversion (attention all devs)
« Reply #94 on: August 10, 2005, 07:40:37 pm »
No, what I have done is:

Code
cd <WXWIN>\build\msw
mingw32-make -f makefile.gcc clean
mingw32-make -f makefile.gcc USE_XRC=1 SHARED=1 MONOLITHIC=1 BUILD=release UNICODE=1 VENDOR=cb

cd ..\..\contrib\build\stc
mingw32-make -f makefile.gcc clean
mingw32-make -f makefile.gcc USE_XRC=1 SHARED=1 MONOLITHIC=1 BUILD=release UNICODE=1 VENDOR=cb

I could be wrong, but from what I understand, UNICODE=1 already defines wxUSE_UNICODE 1.

And then, in the C::B project in the "Compiler defines" I've added __UNICODE__ and wxUSE_UNICODE 1 (along with HAVE_W32API_H, __WXMSW__ and WXUSINGDLL). I'm not sure if I'ts necessary or not.
« Last Edit: August 10, 2005, 08:27:39 pm by takeshimiya »

Offline me22

  • Official tester
  • Multiple posting newcomer
  • ***
  • Posts: 53
    • TA Universe
Re: Unicode conversion (attention all devs)
« Reply #95 on: August 13, 2005, 09:14:59 pm »
Yay!  Success on Linux! ( after a few changes )I did a `cvs -z3 -d:pserver:[email protected]:/cvsroot/codeblocks co -r VERSION_1_0 codeblocks`, then `make -f Makefile.unix` then, after fixing everything, `./update`.  It even runs, although it doesn't open files.

Summary of changes:
- added #include <cctype> to a few files that needed it
- fixed some default parameters in headers
- changed the link libs to the unicode ones.

Here's my diff:
Code
Index: src/Makefile.unix
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/Makefile.unix,v
retrieving revision 1.21.2.1
diff -u -r1.21.2.1 Makefile.unix
--- src/Makefile.unix 25 Jul 2005 19:05:21 -0000 1.21.2.1
+++ src/Makefile.unix 13 Aug 2005 19:13:38 -0000
@@ -7,8 +7,10 @@
 # Compiler used:    GNU GCC Compiler
 
 ### Variables used in this Makefile
-LIB_WXXRC=wx_gtk2_xrc-2.4
-LIB_WXSTC=wx_gtk2_stc-2.4
+#LIB_WXXRC=wx_gtk2_xrc-2.4
+#LIB_WXSTC=wx_gtk2_stc-2.4
+LIB_WXXRC=wx_gtk2u_xrc-2.4
+LIB_WXSTC=wx_gtk2u_stc-2.4
 tinyXML_CC=gcc
 tinyXML_CPP=g++
 tinyXML_LD=g++
Index: src/plugins/codecompletion/nativeparser.cpp
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/plugins/codecompletion/nativeparser.cpp,v
retrieving revision 1.20.2.6
diff -u -r1.20.2.6 nativeparser.cpp
--- src/plugins/codecompletion/nativeparser.cpp 7 Aug 2005 20:55:56 -0000 1.20.2.6
+++ src/plugins/codecompletion/nativeparser.cpp 13 Aug 2005 19:13:38 -0000
@@ -38,6 +38,8 @@
 #include "parser/parser.h"
 #include <compilerfactory.h>
 
+#include <cctype>
+
 BEGIN_EVENT_TABLE(NativeParser, wxEvtHandler)
  EVT_MENU(THREAD_START, NativeParser::OnThreadStart)
  EVT_MENU(THREAD_END, NativeParser::OnThreadEnd)
Index: src/plugins/codecompletion/parser/parserthread.cpp
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/plugins/codecompletion/parser/parserthread.cpp,v
retrieving revision 1.16.2.2
diff -u -r1.16.2.2 parserthread.cpp
--- src/plugins/codecompletion/parser/parserthread.cpp 7 Aug 2005 20:55:56 -0000 1.16.2.2
+++ src/plugins/codecompletion/parser/parserthread.cpp 13 Aug 2005 19:13:39 -0000
@@ -29,6 +29,8 @@
 #include <wx/msgdlg.h>
 #include <globals.h>
 
+#include <cctype>
+
 int THREAD_START = wxNewId();
 int THREAD_END = wxNewId();
 int NEW_TOKEN = wxNewId();
Index: src/plugins/codecompletion/parser/tokenizer.cpp
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/plugins/codecompletion/parser/tokenizer.cpp,v
retrieving revision 1.3.2.1
diff -u -r1.3.2.1 tokenizer.cpp
--- src/plugins/codecompletion/parser/tokenizer.cpp 7 Aug 2005 20:55:56 -0000 1.3.2.1
+++ src/plugins/codecompletion/parser/tokenizer.cpp 13 Aug 2005 19:13:39 -0000
@@ -28,6 +28,8 @@
 #include <wx/file.h>
 #include <wx/msgdlg.h>
 
+#include <cctype>
+
 Tokenizer::Tokenizer(const wxString& filename)
  : m_Filename(filename),
  m_BufferLen(0),
Index: src/plugins/compilergcc/makefilegenerator.h
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/plugins/compilergcc/makefilegenerator.h,v
retrieving revision 1.12.2.3
diff -u -r1.12.2.3 makefilegenerator.h
--- src/plugins/compilergcc/makefilegenerator.h 22 Jul 2005 08:51:58 -0000 1.12.2.3
+++ src/plugins/compilergcc/makefilegenerator.h 13 Aug 2005 19:13:39 -0000
@@ -36,9 +36,9 @@
         void DoAppendCompilerOptions(wxString& cmd, ProjectBuildTarget* target = 0L, bool useGlobalOptions = false);
         void DoAppendLinkerOptions(wxString& cmd, ProjectBuildTarget* target = 0L, bool useGlobalOptions = false);
         void DoAppendLinkerLibs(wxString& cmd, ProjectBuildTarget* target = 0L, bool useGlobalOptions = false);
-        void DoAppendIncludeDirs(wxString& cmd, ProjectBuildTarget* target = 0L, const wxString& prefix = "-I", bool useGlobalOptions = false);
-        void DoAppendResourceIncludeDirs(wxString& cmd, ProjectBuildTarget* target = 0L, const wxString& prefix = "-I", bool useGlobalOptions = false);
-        void DoAppendLibDirs(wxString& cmd, ProjectBuildTarget* target = 0L, const wxString& prefix = "-L", bool useGlobalOptions = false);
+        void DoAppendIncludeDirs(wxString& cmd, ProjectBuildTarget* target = 0L, const wxString& prefix = _T("-I"), bool useGlobalOptions = false);
+        void DoAppendResourceIncludeDirs(wxString& cmd, ProjectBuildTarget* target = 0L, const wxString& prefix = _T("-I"), bool useGlobalOptions = false);
+        void DoAppendLibDirs(wxString& cmd, ProjectBuildTarget* target = 0L, const wxString& prefix = _T("-L"), bool useGlobalOptions = false);
         void DoAddVarsSet(wxString& buffer, CustomVars& vars);
         void DoAddMakefileVars(wxString& buffer);
 #ifdef __WXMSW__
Index: src/plugins/pluginwizard/pluginwizarddlg.h
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/plugins/pluginwizard/pluginwizarddlg.h,v
retrieving revision 1.3
diff -u -r1.3 pluginwizarddlg.h
--- src/plugins/pluginwizard/pluginwizarddlg.h 14 Mar 2005 09:47:04 -0000 1.3
+++ src/plugins/pluginwizard/pluginwizarddlg.h 13 Aug 2005 19:13:39 -0000
@@ -19,7 +19,7 @@
  void OnOKClick(wxCommandEvent& event);
  void OnNameChange(wxCommandEvent& event);
  void DoGuardBlock();
- void DoAddHeaderOption(wxString& buffer, bool has, const wxString& retVal = " -1");
+ void DoAddHeaderOption(wxString& buffer, bool has, const wxString& retVal = _T(" -1"));
  void DoAddHeaderTool(wxString& buffer);
  void DoAddHeaderCompiler(wxString& buffer);
  void DoAddHeaderDebugger(wxString& buffer);
Index: src/sdk/projectloader.h
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/sdk/projectloader.h,v
retrieving revision 1.5.2.2
diff -u -r1.5.2.2 projectloader.h
--- src/sdk/projectloader.h 6 Aug 2005 22:13:05 -0000 1.5.2.2
+++ src/sdk/projectloader.h 13 Aug 2005 19:13:39 -0000
@@ -38,11 +38,11 @@
         void DoUnitOptions(TiXmlElement* parentNode, ProjectFile* file);
         
         void BeginOptionSection(wxString& buffer, const wxString& sectionName, int nrOfTabs);
-        bool DoOptionSection(wxString& buffer, const wxArrayString& array, int nrOfTabs, const wxString& optionName = "option");
+        bool DoOptionSection(wxString& buffer, const wxArrayString& array, int nrOfTabs, const wxString& optionName = _("option"));
         void EndOptionSection(wxString& buffer, const wxString& sectionName, int nrOfTabs);
 
         // shortcut that calls BeginOptionSection(), DoOptionSection() and EndOptionSection()
-        void SaveOptions(wxString& buffer, const wxArrayString& array, const wxString& sectionName, int nrOfTabs, const wxString& optionName = "option", const wxString& extra = "");
+        void SaveOptions(wxString& buffer, const wxArrayString& array, const wxString& sectionName, int nrOfTabs, const wxString& optionName = _("option"), const wxString& extra = wxEmptyString);
  private:
         void SaveCompilerOptions(wxString& buffer, CompileOptionsBase* object, int nrOfTabs);
         void SaveResourceCompilerOptions(wxString& buffer, CompileOptionsBase* object, int nrOfTabs);
« Last Edit: August 13, 2005, 09:19:11 pm by me22 »

Offline cyberkoa

  • Plugin developer
  • Almost regular
  • ****
  • Posts: 145
    • http://
Re: Unicode conversion (attention all devs)
« Reply #96 on: August 13, 2005, 11:23:06 pm »
I tried to compile VERSION_1_0 CVS in UNICODE, wx2.6 CVS, win32, MinGW, but I get lots of undefined references


Switching to target: sdk
Linking dynamic library: devel\codeblocks.dll
Creating library file: devel\libcodeblocks.a
.objs\2.6\sdk\autodetectcompilers.o(.text+0x10d): In function `ZN19AutoDetectCompilersC2EP8wxWindow':
D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:19: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'
.objs\2.6\sdk\autodetectcompilers.o(.text+0x663):D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:49: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'

.objs\2.6\sdk\autodetectcompilers.o(.text+0x7eb): In function `ZN19AutoDetectCompilersC1EP8wxWindow':
D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:19: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'
.objs\2.6\sdk\autodetectcompilers.o(.text+0xd41):D:/Develop/src/codeblocks-VERSION_1_0/codeblocks/src/sdk/autodetectcompilers.cpp:49: undefined reference to `_imp___ZN13wxXmlResource8GetXRCIDEPKc'

...
Process terminated with status 1 (0 minutes, 22 seconds)
52 errors, 0 warnings


These errors are expected, or anyone compiled all ok with these settings?
VERSION_1_0 is wx2.6 compatible??

takeshimiya, I get exactly the error message you got .

Later , I have tried the following,
1. Rebuild , result fail

2. I suddenly remember that the unicode compilation of wxwidgets actually genarated a setup.h with UNICODE=0 !!!! It was my experience last time.

Therefore , I go to the unicode version of wxwidgets , and get the corresponding setup.h (default is under gcc_dllu\mswu\ if I am not mistaken)

I do a clean , dist clean , and recompile .

The link error message gone !!!! and left only 3 warning message ..





Offline me22

  • Official tester
  • Multiple posting newcomer
  • ***
  • Posts: 53
    • TA Universe
Re: Unicode conversion (attention all devs)
« Reply #97 on: August 15, 2005, 07:25:24 am »
Now Saving/Loading Files Works Too!Seems the problem was a simple case of a void* buffer being treaded as a wxChar* buffer instead of a char* buffer.

Here's the diff:
Code
Index: src/sdk/cbeditor.cpp
===================================================================
RCS file: /cvsroot/codeblocks/codeblocks/src/sdk/cbeditor.cpp,v
retrieving revision 1.37.2.7
diff -u -r1.37.2.7 cbeditor.cpp
--- src/sdk/cbeditor.cpp 10 Aug 2005 16:56:47 -0000 1.37.2.7
+++ src/sdk/cbeditor.cpp 15 Aug 2005 05:21:13 -0000
@@ -41,6 +41,8 @@
 #include "cbplugin.h"
 #include "cbeditorprintout.h"
 
+#include <cstring> // std::strlen
+
 /* This struct holds private data for the cbEditor class.
  * It's a paradigm to avoid rebuilding the entire project (as cbEditor is a basic dependency)
  * for just adding a private var or method.
@@ -600,13 +602,12 @@
     if (!file.IsOpened())
         return false;
 
-    wxChar* buff = st.GetWriteBuf(file.Length());
+    void* buff = st.GetWriteBuf(file.Length());
     file.Read(buff, file.Length());
     file.Close();
-    st.UngetWriteBuf();
-
-    m_pControl->InsertText(0, st);
+    m_pControl->InsertText(0, _U( (char const *)buff ));
     m_pControl->EmptyUndoBuffer();
+    st.UngetWriteBuf();
 
     // mark the file read-only, if applicable
     bool read_only = !wxFile::Access(m_Filename.c_str(), wxFile::write);
@@ -647,8 +648,9 @@
     }
     
     wxFile file(m_Filename, wxFile::write);
-    if (file.Write(m_pControl->GetText().c_str(), m_pControl->GetTextLength()) == 0 &&
-        m_pControl->GetTextLength() != 0)
+    wxWX2MBbuf utf_text = m_pControl->GetText().mb_str(wxConvUTF8);
+    size_t utf_text_length = std::strlen( utf_text );
+    if (file.Write( utf_text, utf_text_length ) == 0 && utf_text_length != 0)
     {
         return false; // failed; file is read-only?
     }

[edit] Note that this only does files -- workspaces don't seem to save/load ( dunno which ) properly, and I haven't tried projects.
« Last Edit: August 15, 2005, 07:35:12 am by me22 »

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Unicode conversion (attention all devs)
« Reply #98 on: August 15, 2005, 09:27:44 am »
if you're using void, why not just use *char ? And comment the line, saying "this is a byte-buffer, don't replace with wxChar" or something.

Offline me22

  • Official tester
  • Multiple posting newcomer
  • ***
  • Posts: 53
    • TA Universe
Re: Unicode conversion (attention all devs)
« Reply #99 on: August 15, 2005, 05:29:43 pm »
The point of the void* was to emphasize that it's not a wxChar*.

I could just use unsigned char *, but then it would show up if someone searched in all files for char .

Offline zieQ

  • Almost regular
  • **
  • Posts: 155
Re: Unicode conversion (attention all devs)
« Reply #100 on: August 23, 2005, 09:46:32 am »
Help, unicode men have modified the parsing of MSVC  :lol:

For example this line:
Code
if (line.StartsWith(_T("Project_Dep_Name"))) {

Does _T means that we do nothing to the string (i.e. keep a char* representation, no unicode conversion)? I'm quite lost with _X() macros, can someone sum up them?

Offline mandrav

  • Project Leader
  • Administrator
  • Lives here!
  • *****
  • Posts: 4315
    • Code::Blocks IDE
Re: Unicode conversion (attention all devs)
« Reply #101 on: August 23, 2005, 10:09:47 am »
Help, unicode men have modified the parsing of MSVC  :lol:

For example this line:
Code
if (line.StartsWith(_T("Project_Dep_Name"))) {

Does _T means that we do nothing to the string (i.e. keep a char* representation, no unicode conversion)? I'm quite lost with _X() macros, can someone sum up them?

wxT(), _T(): unicode string, not-translatable (IIRC just puts L before the quote, i.e. L"text")
_(): unicode string, translatable (use it for english messages and stuff)

There are also some conversion macros in sdk/settings.h:
_U(), _UU(), _C(), _CC()
for the conversion between char* and unicode.

Yiannis.
Be patient!
This bug will be fixed soon...

Offline takeshi miya

  • Lives here!
  • ****
  • Posts: 1487
Re: Unicode conversion (attention all devs)
« Reply #102 on: August 23, 2005, 10:13:37 am »
About char*, that's why I suggested to use byte* as typedef.

It will make clear that you are using a byte buffer instead of a character string.

char typedefed as byte or wxByte would be good.
« Last Edit: August 23, 2005, 10:15:49 am by takeshimiya »

Offline tiwag

  • Developer
  • Lives here!
  • *****
  • Posts: 1196
  • sailing away ...
    • tiwag.cb
Re: Unicode conversion (attention all devs)
« Reply #103 on: August 23, 2005, 10:27:09 am »
from sdk/settings.h
Code
#if wxUSE_UNICODE
    #define _UU(x,y) wxString((x),(y))
    #define _CC(x,y) (x).mb_str((y))
#else
    #define _UU(x,y) (x)
    #define _CC(x,y) (x)
#endif

#define _U(x) _UU((x),wxConvUTF8)
#define _C(x) _CC((x),wxConvUTF8)


from wxWidgets-Help
Code
--------------------------------------------------------------------------------

wxT
wxChar wxT(char ch)

const wxChar * wxT(const char *s)

wxT() is a macro which can be used with character and string literals (in other words, 'x' or "foo") to automatically convert them to Unicode in Unicode build configuration. Please see the Unicode overview for more information.

This macro is simply returns the value passed to it without changes in ASCII build. In fact, its definition is:

#ifdef UNICODE
#define wxT(x) L ## x
#else // !Unicode
#define wxT(x) x
#endif

--------------------------------------------------------------------------------

_
const wxChar * _(const char *s)

This macro expands into a call to wxGetTranslation function, so it marks the message for the extraction by xgettext just as wxTRANSLATE does, but also returns the translation of the string for the current locale during execution.

Don't confuse this macro with _T()!



--------------------------------------------------------------------------------

wxPLURAL
const wxChar * wxPLURAL(const char *sing, const char *plur, size_tn)

This macro is identical to _() but for the plural variant of wxGetTranslation.



--------------------------------------------------------------------------------

_T
wxChar _T(char ch)

const wxChar * _T(const wxChar ch)

This macro is exactly the same as wxT and is defined in wxWidgets simply because it may be more intuitive for Windows programmers as the standard Win32 headers also define it (as well as yet another name for the same macro which is _TEXT()).

Don't confuse this macro with _()!

--------------------------------------------------------------------------------


summary     (please someone might correct if i'm wrong or add missing points)

Code
_()
- text's which might be translated to other user-languages
Code
_T()
wxT()
- fixed text's - like XRC resources object names

Code
_U()
- conversion from char* to wxString

Code
_C()
- multibyte C string  see wxhelp (wxMBConv classes overview)


HTH

Offline tiwag

  • Developer
  • Lives here!
  • *****
  • Posts: 1196
  • sailing away ...
    • tiwag.cb
Re: Unicode conversion (attention all devs)
« Reply #104 on: August 23, 2005, 10:29:07 am »
..char typedefed as byte or wxByte would be good.

i always used  "byte"  as typedef'ed  "unsigned char"