Author Topic: Code::Blocks very slow to open long files  (Read 12452 times)

Offline Dahman

  • Multiple posting newcomer
  • *
  • Posts: 16
Code::Blocks very slow to open long files
« on: May 05, 2008, 10:22:20 am »
I've noticed that code::Blocks is very slow to open files (.txt) with a lot of lines (ex. 50.000).
Often it takes more than 30 seconds.
Is it just me?

Thank for any advice

Dahman

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Code::Blocks very slow to open long files
« Reply #1 on: May 05, 2008, 01:10:29 pm »
Created a file with >60k lines (6.6 MiB) by concatenating ntbtlog.txt a dozen times, takes under a second to load on my old Athlon64.

So, to answer your question: yes.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9496
Re: Code::Blocks very slow to open long files
« Reply #2 on: May 05, 2008, 01:14:27 pm »
Often it takes more than 30 seconds.
When you say "often" this obviously means "not always". So why don't you provide us with an example so we can reproduce?! ;-)
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline Dahman

  • Multiple posting newcomer
  • *
  • Posts: 16
Re: Code::Blocks very slow to open long files
« Reply #3 on: May 05, 2008, 04:31:50 pm »
Thanks for your replay.

My file is a sort of a log file witch is produced by writing some traces of my project.
When I said "often" I meant the file often exceed 50-60 K of lines and in that case code::Blocks become slow (very slow).
As soon as I go home I will send you an example.

By the way, I am using Ubuntu 7.04 and Code::Blocks 8.02.

Thank you again

Offline Dahman

  • Multiple posting newcomer
  • *
  • Posts: 16
Re: Code::Blocks very slow to open long files
« Reply #4 on: May 06, 2008, 03:25:42 pm »
Hello,
Here is an attached example (nothing special, it is just a zipped text file).
To open this file, Code::blocks take exactly 16 seconds.
The same file is opened in 1 second with gEdit.

Thanks for any tip.

Dahman

[attachment deleted by admin]

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Code::Blocks very slow to open long files
« Reply #5 on: May 06, 2008, 05:05:08 pm »
Roughly 1 second here (maybe a bit less, who can tell), rev 5028 on Windows XP.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: Code::Blocks very slow to open long files
« Reply #6 on: May 06, 2008, 05:17:48 pm »
Hello,
Here is an attached example (nothing special, it is just a zipped text file).
To open this file, Code::blocks take exactly 16 seconds.
The same file is opened in 1 second with gEdit.

Thanks for any tip.

Dahman

try:
1. removing the file from any open projects that it might be in
2. switching off code completion plugin

does either of these make a difference?

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9496
Re: Code::Blocks very slow to open long files
« Reply #7 on: May 06, 2008, 06:02:38 pm »
Roughly 1 second here (maybe a bit less, who can tell), rev 5028 on Windows XP.
Same here (same revision, too), for sure less than a second. Works very fast - as expected. :-(
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline manowar

  • Multiple posting newcomer
  • *
  • Posts: 16
Re: Code::Blocks very slow to open long files
« Reply #8 on: May 06, 2008, 09:55:32 pm »
I can confirm the issue here. I opened the file on hardy using codeblocks 5020 (from LGP repository) and it took about 12-15 seconds to open it. It is quite slow indeed. My machine is a dual core athlon 3800. I tried to open it with geany and gedit, it takes less than a second. There is definitely something wrong on linux platform. (Disabling the codecompletion does not improve anything)
« Last Edit: May 06, 2008, 09:57:04 pm by manowar »

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: Code::Blocks very slow to open long files
« Reply #9 on: May 07, 2008, 12:27:29 am »
I can confirm the issue on Ubuntu 8.04 with all plugins turned off. I remember making some patches many months ago that fixed file opening delays on win32, but those problems were nothing like this. Scite opens the same file instantaneously

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: Code::Blocks very slow to open long files
« Reply #10 on: May 07, 2008, 02:05:19 am »
i just ran through a few debug sessions. seems like the bottleneck is in the encoder. specifically,

line 93 EncodingDetector.cpp  (see *** CHOKES HERE **** line below)
Code: [Select]
bool EncodingDetector::ConvertToWxStr(const wxByte* buffer, size_t size)
{
    if (!buffer || size == 0)
        return false;

    if (m_BOMSizeInBytes > 0)
    {
        for (int i = 0; i < m_BOMSizeInBytes; ++i)
            *buffer++;
    }

    size_t outlen = 0;
    wxCSConv conv(m_Encoding);

/* NOTE (Biplab#5#): FileManager returns a buffer with 4 extra NULL chars appended.
    But the buffer size is returned sans the NULL chars */
    wxWCharBuffer wideBuff = conv.cMB2WC((char*)buffer, size + 4 - m_BOMSizeInBytes, &outlen);  ****CHOKES HERE*****
    m_ConvStr = wxString(wideBuff);


strangely, this code gets called twice every file open (and is slow each time)

UNRELATED OBSERVATION: when i remove breakpoints in an active debug session, the breakpoints remain (this is a pretty longstanding issue with debugger plugin in cb)

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Code::Blocks very slow to open long files
« Reply #11 on: May 07, 2008, 09:25:15 am »
One possible cause might be this:
Quote from: wx/strconv.h
// inLen is the length of the buffer including trailing NUL if any: if the
// last 4 bytes of the buffer are all NULs, these functions are more
// efficient as they avoid copying the string, but otherwise a copy is made
// internally which could be quite bad for (very) long strings.
Of course the documentation doesn't have any mention of that, you have to dig through the headers to find it.

However, FileManager already adds 4 gratious null bytes to every loaded file to prevent some other crap wxWidgets string function from crashing in UTF-16/UTF-32 mode, so actually this situation should not be possible.
Maybe the string conversion function is counting them from the wrong location though, who knows? Even though copying 500 kB of data shouldn't take 15 seconds... but who knows, maybe they make 20,000 copies? :)
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline afb

  • Developer
  • Lives here!
  • *****
  • Posts: 884
Re: Code::Blocks very slow to open long files
« Reply #12 on: May 07, 2008, 09:44:18 am »
Opens up directly on Mac OS X, for both wxMac and wxGTK versions of Code::Blocks.
Though the GTK+ version uses ANSI, so it probably didn't need to convert very much.

So looks like it is something specific to Unicode or Linux, or maybe even just Ubuntu ?

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Code::Blocks very slow to open long files
« Reply #13 on: May 07, 2008, 10:49:55 am »
If I've followed the never ending chain of macros, functions, and macros renaming functions correctly, it's
mbsrtowcs which finally gets called in Unicode build (ANSI build copies the data to a new buffer using strncpy).
So, is there a way to trace mbsrtowcs to confirm or reject the "it's an Ubuntu thing" hypothesis? Something similar to strace maybe...?
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline mandrav

  • Project Leader
  • Administrator
  • Lives here!
  • *****
  • Posts: 4291
    • Code::Blocks IDE
Re: Code::Blocks very slow to open long files
« Reply #14 on: May 07, 2008, 01:47:27 pm »
Yes, I can confirm it's the call to mbsrtowcs that's causing the slow down. Unfortunately, I 'm not sure we can do anything about that... :(

On the bright side, I found out why the encoding detection was called twice and fixed it. This means that the slow-down time is now cut in half. That's something, at least.
Be patient!
This bug will be fixed soon...

Offline Dahman

  • Multiple posting newcomer
  • *
  • Posts: 16
Re: Code::Blocks very slow to open long files
« Reply #15 on: May 07, 2008, 02:10:53 pm »
Quote
This means that the slow-down time is now cut in half. That's something, at least.

Yes, if you know how many times I must open my log file, that is more than "something"
Thanks and I hope you can fix definitely this problem.

Dahman

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Code::Blocks very slow to open long files
« Reply #16 on: May 07, 2008, 02:15:51 pm »
Unfortunately, I 'm not sure we can do anything about that... :(

Try compiling with -DwxNEED_WX_MBSTOWCS.
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline mandrav

  • Project Leader
  • Administrator
  • Lives here!
  • *****
  • Posts: 4291
    • Code::Blocks IDE
Re: Code::Blocks very slow to open long files
« Reply #17 on: May 07, 2008, 02:31:59 pm »
Unfortunately, I 'm not sure we can do anything about that... :(

Try compiling with -DwxNEED_WX_MBSTOWCS.

I 'd be happy to but, even if that fixed it, that would mean that each user would have to do the same (instead of using a package from a repository). Not a nice thing to do.
Be patient!
This bug will be fixed soon...

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: Code::Blocks very slow to open long files
« Reply #18 on: May 07, 2008, 02:35:30 pm »
Well, I meant rather something like "put a line into the makefile that adds this switch for Ubuntu" (supposed that it actually works) :)

I'm not even sure it will work, but there's an #ifdef in the code which suggests that it might...
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline Biplab

  • Developer
  • Lives here!
  • *****
  • Posts: 1874
    • Biplab's Blog
Re: Code::Blocks very slow to open long files
« Reply #19 on: May 07, 2008, 05:13:30 pm »
Yes, I can confirm it's the call to mbsrtowcs that's causing the slow down. Unfortunately, I 'm not sure we can do anything about that... :(

Unfortunately mbsrtowcs (defined in wchar.h ) is not the cause. It's the mbstowcs (defined in stdlib.h ) which is causing this slowdown. Though I don't know the difference.

In <wx/wxcrtbase.h> file (line:580):
Code: [Select]
#ifdef wxNEED_WX_MBSTOWCS
    /* even though they are defined and "implemented", they are bad and just
       stubs so we need our own - we need these even in ANSI builds!! */
    WXDLLIMPEXP_BASE size_t wxMbstowcs(wchar_t *, const char *, size_t);
    WXDLLIMPEXP_BASE size_t wxWcstombs(char *, const wchar_t *, size_t);
#else
    #define wxMbstowcs mbstowcs
    #define wxWcstombs wcstombs
#endif


In src/common/wxcrt.cpp file (line:88):
Code: [Select]
#ifdef HAVE_WCSRTOMBS
    return mbsrtowcs(buf, &psz, n, &mbstate);
#else
    return wxMbstowcs(buf, psz, n);
#endif

And this HAVE_WCSRTOMBS macro is defined only for Metroworks compiler on Mac.

I'm not sure why they have used mbsrtowcs() only for Mac when the wxchar.h header is available on different platforms?? :?
« Last Edit: May 07, 2008, 05:15:22 pm by Biplab »
Be a part of the solution, not a part of the problem.

Online oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 11937
    • Travis build status
Re: Code::Blocks very slow to open long files
« Reply #20 on: May 08, 2008, 01:43:12 am »
Hello,
I can confirm that it is not an Ubuntu only issue.
It is slow on my machine at home: gentoo amd64 opteron 2.0ghz c::b svn 5028 wx 2.8.7

Best regards
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline Deschamps

  • Multiple posting newcomer
  • *
  • Posts: 120
Re: Code::Blocks very slow to open long files
« Reply #21 on: May 08, 2008, 11:01:25 am »
I can confirm that it is not an Ubuntu only issue.
It is slow on my machine at home: gentoo amd64 opteron 2.0ghz c::b svn 5028 wx 2.8.7

The same issue here. Almost 5 seconds to open the TXT file.

openSUSE 10.3 x86_64 on a Core2Duo T7500 laptop w/3GB RAM and SATA HD
C::B svn 5013 compiled against wxGTK 2.8.7 Unicode

Regards.
Misquotations are the only quotations that are never misquoted

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: Code::Blocks very slow to open long files
« Reply #22 on: May 08, 2008, 04:31:34 pm »
It probably doesn't make any difference, but why do we use

Code: [Select]
    wxWCharBuffer wideBuff = conv.cMB2WC((char*)buffer, size + 4 - m_BOMSizeInBytes, &outlen);
    m_ConvStr = wxString(wideBuff);

instead of

Code: [Select]
    m_ConvStr = wxString((char*)buffer,conv);
    outlen = m_ConvStr.Len();

Offline Biplab

  • Developer
  • Lives here!
  • *****
  • Posts: 1874
    • Biplab's Blog
Re: Code::Blocks very slow to open long files
« Reply #23 on: May 08, 2008, 06:54:44 pm »
It probably doesn't make any difference, but why do we use

I don't remember exactly why I used such a workaround. Probably due to some crash. :)



I've found out a better solution for this problem.  :D

If we use iconv based implementation, the conversion routine becomes faster. Just apply the following patch and see the difference.

Code: [Select]
Index: src/sdk/encodingdetector.cpp
===================================================================
--- src/sdk/encodingdetector.cpp (revision 5040)
+++ src/sdk/encodingdetector.cpp (working copy)
@@ -19,6 +19,8 @@
 
 #include "encodingdetector.h"
 #include "filemanager.h"
+#include <wx/stopwatch.h>
+#include <iconv.h>
 
 
 EncodingDetector::EncodingDetector(const wxString& filename)
@@ -106,9 +108,25 @@
 
 /* NOTE (Biplab#5#): FileManager returns a buffer with 4 extra NULL chars appended.
     But the buffer size is returned sans the NULL chars */
+    /*wxStopWatch sw;
+    sw.Start();
     wxWCharBuffer wideBuff = conv.cMB2WC((char*)buffer, size + 4 - m_BOMSizeInBytes, &outlen);
     m_ConvStr = wxString(wideBuff);
+    sw.Pause();
+    Manager::Get()->GetLogManager()->DebugLog(wxString::Format(_T("Time taken: %d milliseconds\n"), sw.Time() ));*/
 
+    wxStopWatch sw;
+    iconv_t cd = iconv_open("ISO-8859-1", "UTF-8");
+    size_t inbytesleft = 0, outbytesleft = 0;
+    char* outbuf = new char[size + 4 - m_BOMSizeInBytes];
+    sw.Start();
+    iconv(cd, (char **)&buffer, &inbytesleft, &outbuf, &outbytesleft);
+    sw.Pause();
+    m_ConvStr = wxString((wchar_t *)outbuf);
+    delete [] outbuf;
+    iconv_close(cd);
+    Manager::Get()->GetLogManager()->DebugLog(wxString::Format(_T("Time taken: %d milliseconds\n"), sw.Time() ));
+
     if (outlen == 0)
     {
         // Possibly the conversion has failed. Let's try with System-default encoding

Using wx based implementation:
Quote
Time taken: 6201 milliseconds

Using iconv based implementation:
Quote
Time taken: 0 millisecond

The patch contains commented old code. You can easily compare the performance by commenting out new / old code.

Please note: The iconv based implementation code is NOT generic one. It would work only with the log file.

PS: Please bear with my late replies. :)
« Last Edit: May 08, 2008, 07:06:41 pm by Biplab »
Be a part of the solution, not a part of the problem.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9496
Re: Code::Blocks very slow to open long files
« Reply #24 on: May 08, 2008, 07:34:00 pm »
If we use iconv based implementation, the conversion routine becomes faster. [...]
Although this is a very nice idea anyways, but you are aware, that this massively "complicates" the compilation on Windows, right? Because MinGW does not ship with this lib so we would have to bundle iconv with the C::B sources on Windows. I'm not sure on how many libs iconv itself depends but I would guess it's not a "stand-alone". So the question is either we find another solution or probably (as this seems to be a Linux only thing) we make this implementation on Linux only... Any thoughts?
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: http://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: http://wiki.codeblocks.org/index.php?title=FAQ

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: Code::Blocks very slow to open long files
« Reply #25 on: May 08, 2008, 09:14:22 pm »
i would assume that this would be a temporary linux-only change. someone should file a bug with the wx guys (perhaps with a minimal sample to demonstrate the problem)

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: Code::Blocks very slow to open long files
« Reply #26 on: May 09, 2008, 01:27:13 am »
I've found out a better solution for this problem.  :D

Code: [Select]
+    iconv_t cd = iconv_open("ISO-8859-1", "UTF-8");


correct me if I'm wrong, biplab, but all you need to do to make this operational is to add a function that maps the wxFontEncoding typedef to the strings that iconv can handle?

Offline Biplab

  • Developer
  • Lives here!
  • *****
  • Posts: 1874
    • Biplab's Blog
Re: Code::Blocks very slow to open long files
« Reply #27 on: May 09, 2008, 05:22:33 am »
Although this is a very nice idea anyways, but you are aware, that this massively "complicates" the compilation on Windows, right?

Not really. GTK+ team uses win_iconv on Windows which is purely written in Win32 API and is very easy to compile on Windows. Visit it's homepage (You need to translate it) :
Quote
http://yukihiro.nakadaira.googlepages.com/

Or just download the compiled package from the following page.
Quote
http://www.gtk.org/download-windows.html


I'm not sure on how many libs iconv itself depends but I would guess it's not a "stand-alone".

It has no external chain of dependency. So rest be assured that it'll not increase the overhead.

So the question is either we find another solution or probably (as this seems to be a Linux only thing) we make this implementation on Linux only... Any thoughts?

IMO, we can make the changes for Linux only where libiconv.so is available.


But my real concern is not the compilation of iconv on Windows or so. My major concern is to see that the iconv and wxWidgets work together without creating nuisance. That's why I thought of wrapping iconv as a separate plugin so that it doesn't affect the core code.

I've already noticed problems with some encodings with the new patch. wx fails to accept the passed wchar_t pointer and the conversion looses several paragraphs of characters. I guess it's about more number of NULL chars at the end confuses wx.

correct me if I'm wrong, biplab, but all you need to do to make this operational is to add a function that maps the wxFontEncoding typedef to the strings that iconv can handle?

Yes it is. IIRC, iconv supports more encodings than wx handles. That's one reason I wanted to implement it as a separate plugin. Also read the problem I wrote in previous para.
Be a part of the solution, not a part of the problem.

Offline Biplab

  • Developer
  • Lives here!
  • *****
  • Posts: 1874
    • Biplab's Blog
Re: Code::Blocks very slow to open long files
« Reply #28 on: May 18, 2008, 06:37:16 pm »
Well folks, I need to say that the earlier patch was incorrect and it doesn't solve the problem. Sorry for the noise. :oops:

Actually the delay is because C::B tries to convert the file from UTF-8. Then this conversion fails as the file is in different encoding (This process is consuming time). Then the fallack encoding converts it properly.

The file is converted in just 8 milliseconds if correct encoding is used.

I'm looking for a solution to this problem.
Be a part of the solution, not a part of the problem.

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: Code::Blocks very slow to open long files
« Reply #29 on: February 13, 2009, 03:33:59 pm »
sorry to resurrect -- has this been solved? I still seem to be seeing longer than necessary file open times on my linux boxes...