Author Topic: Forrtran - Add Files Recursively is Slow  (Read 830 times)

Offline FEA

  • Multiple posting newcomer
  • *
  • Posts: 19
Forrtran - Add Files Recursively is Slow
« on: May 20, 2020, 01:37:06 am »
Hi, I noticed that if the project has about 200 files, "Add Files Recursively.." only takes a few seconds. If I increase that number to 600, it takes about 5 min. If I increase that to 1800 files, it takes about 40 minutes. So it appears that there is an exponential increase in time to read the files. All of those times are faster on another computer, but that was a baseline for a laptop.

Is there any way to reduce the time it takes to read files in? I think splitting it into multiple directories can work, but I am wondering if there is something that can be done at a more fundamental level? This is a big bottleneck for the project.
« Last Edit: May 21, 2020, 12:34:05 am by FEA »

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 2730
Re: Add Files Recursively is Slow
« Reply #1 on: May 20, 2020, 09:10:48 am »
Do you have to import this many files so often?

Do you have the possibility to write the paths to the files in a text file (i have a script, that imports files from a text file, i would like test it)

Have you disabled plugins and tried again (specially code completion)?

Offline FEA

  • Multiple posting newcomer
  • *
  • Posts: 19
Re: Add Files Recursively is Slow
« Reply #2 on: May 20, 2020, 06:23:36 pm »
In my case, there have been a lot of issues due to the large number of files. So in attempt to resolve that, I have had to go through the process dozens of times (including crashes, etc.). But after its resolved, it is less of a problem. The major problem is the project is to be worked on by multiple people and some of them may be turned off by this.

Yes, I have a VBA program that can write the full path of filenames to a text file. So I would be interested in your script. That may be the best solution if it is quick.

I disabled the code completion plugin, but it did not help. I tried to disable the plugins one-by-one, but it crashed about half way through. Then I removed the plugins from the plugins folder, but could not start a project to test it. Do you know what the minimum required plugins are required to start a project?

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 2730
Re: Add Files Recursively is Slow
« Reply #3 on: May 20, 2020, 06:37:16 pm »
This is the script http://wiki.codeblocks.org/index.php/Some_examples#Add_files_from_a_text_file_to_the_active_project

I think the compiler plugin and the sriptedWizard plugin are probably the most basic two...

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12625
    • Travis build status
Re: Add Files Recursively is Slow
« Reply #4 on: May 20, 2020, 06:53:52 pm »
I doubt disabling plugins would help.
It would be best if you can make a project which is shareable and which is similar to the project you are working on.
I could profile it and try to improve the performance. I doubt it would be hard.

p.s. It will be good if you could report any crashes you encounter.
p.p.s. Don't disable the ThreadSearch plugin. It is known to cause crashes.
 
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 2730
Re: Add Files Recursively is Slow
« Reply #5 on: May 20, 2020, 08:13:53 pm »
Also if you want a really quick fix, create the Project file by yourself. It is a simple xml file... Open it with a text editor and check the content. It is quite verbose and pretty easy to create.

Of course it would be better to fix codeblocks, and we will do it if we can reproduce it.
Are this some special files?

Offline FEA

  • Multiple posting newcomer
  • *
  • Posts: 19
Re: Add Files Recursively is Slow
« Reply #6 on: May 20, 2020, 09:14:39 pm »
I removed all plugins from the plugin folder except compiler.dll and scriptedwizard.dll. Then I created a project and added the source files. It read them in within seconds, so it appears that one of the plugins is bogging it down.

EDIT: I isolated the problem plugin to just FortranProject.dll. With that plugin disabled, it reads in within seconds. This is a Fortran project, but I am not sure how necessary FortranProject.dll is to the project. I will have to better understand the value of this plugin.
« Last Edit: May 20, 2020, 09:37:08 pm by FEA »

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 2730
Re: Add Files Recursively is Slow
« Reply #7 on: May 20, 2020, 09:49:39 pm »
Yes, it is the fortran symbol browser.
You can deactivate it in Settings->Editor->Fortran Project->FSymbols browser->Enable symbols browser

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 2730
Re: Add Files Recursively is Slow
« Reply #8 on: May 20, 2020, 09:53:04 pm »
After importing and saving the project you can close the project and codeblocks, re start codeblocks and re enable it and then load the project and it should work fine.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12625
    • Travis build status
Re: Add Files Recursively is Slow
« Reply #9 on: May 20, 2020, 11:54:06 pm »
The problem happens because the symbol browser is updated on every added file in FortranProject::OnProjectFileAdded.
This is wrong and it should be change to happen in response to a cbEVT_PROJECT_END_ADD_FILES event.
This is something which should be adjusted in the fortran plugin.
The fortran plugin is an external project we just ship for convenience.
So you either report this directly to the project here http://cbfortran.sourceforge.net/ or wait for darmar to see this topic and fix the plugin. :)

p.s. There is a smaller problem which makes this take some time, but after I fix it it will happen almost instantaneous. :)
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline FEA

  • Multiple posting newcomer
  • *
  • Posts: 19
Re: Forrtran - Add Files Recursively is Slow
« Reply #10 on: May 21, 2020, 12:37:41 am »
Great detective work guys! Now that I know the specific issue, I can work around it.

It was really concerning since the command line limit length resolution appeared to also take a very long time via the script from BlueHazzard (twice the time to read the files). So even if BlueHazzard made the patch, Code::Blocks would probably not be practical.

Anyway, now that it has been identified and there is a workaround, I am really looking forward to BlueHazzard's patch. That seems to be the last piece of the puzzle.

Thanks again!

Offline darmar

  • Multiple posting newcomer
  • *
  • Posts: 59
Re: Forrtran - Add Files Recursively is Slow
« Reply #11 on: May 22, 2020, 08:16:08 am »
I updated FortranProject plugin. Changes in r294.

Thanks to oBFusCATed for showing where is the problem.

Just a note: my self I could not repeat the problem. Even adding >700 files with >500k lines of code it took ~10-20s. A new version takes now shorter.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12625
    • Travis build status
Re: Forrtran - Add Files Recursively is Slow
« Reply #12 on: May 22, 2020, 07:55:52 pm »
The project provided by the user has this behaviour without the fortran plugin:
Code: [Select]
ProjectManager::AddMultipleFilesToProject took: 0.115 seconds for 1873 files.
And with fortran plugin:
Code: [Select]
ProjectManager::AddMultipleFilesToProject took: 1.845 seconds for 1873 files.

This same problem seems to happen if you do remove files from project. FortranProject::OnProjectFileRemoved seems to be really expensive.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12625
    • Travis build status
Re: Forrtran - Add Files Recursively is Slow
« Reply #13 on: May 22, 2020, 08:02:20 pm »
Here is a flamegraph for the add with the fixed version: https://imgur.com/a/IkWLMzx

@darmar: It seems that you're parsing every file on at the moment it is added. I suppose it is better just to schedule the file for parsing and do this in another thread or onidle.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline FEA

  • Multiple posting newcomer
  • *
  • Posts: 19
Re: Forrtran - Add Files Recursively is Slow
« Reply #14 on: May 22, 2020, 08:39:56 pm »
Looks like things are on the right track. For 1873 files, I had 10 minutes, 40 minutes, and 60 minutes on 3 different computers. When disabling the Symbols Browser, it took about 5 seconds on the fastest computer.

On the fastest computer, it takes about a minute to read in 600 files (so not dramatically different from Darmar). The issue is that it seems to be exponential and 1900 files takes about 10X that of 600 files. It seems to really show on slower laptops.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12625
    • Travis build status
Re: Forrtran - Add Files Recursively is Slow
« Reply #15 on: May 22, 2020, 10:40:58 pm »
Can you list machine specs and times, so we have some kind of a reference?
My tests are done on a AMD Ryzen 7 1700, which isn't the fastest cpu on the planet, but it isn't slow either :)
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline FEA

  • Multiple posting newcomer
  • *
  • Posts: 19
Re: Forrtran - Add Files Recursively is Slow
« Reply #16 on: May 22, 2020, 11:19:01 pm »
From what I can see, the processor was not maxed out, but it may still be the bottlneck?

Fastest = Intel Core i7 870 2.93 Ghz, 8GB RAM
Slowest = Intel i5 M460 2.53 Ghz, 6 GB RAM

Maybe these are ancient by today's standards? I tend to need faster disk speed and CPU power isn't a high priority for me. That said, I have been thinking about upgrading both desktop and laptop. The setup time on a new computer takes such a long time (may configurations) that I try to avoid this until necessary.
« Last Edit: May 22, 2020, 11:20:40 pm by FEA »

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12625
    • Travis build status
Re: Forrtran - Add Files Recursively is Slow
« Reply #17 on: May 22, 2020, 11:48:04 pm »
I wanted the specs just for reference.

p.s. What OS and C::B version are you running?
« Last Edit: May 22, 2020, 11:49:49 pm by oBFusCATed »
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline FEA

  • Multiple posting newcomer
  • *
  • Posts: 19
Re: Forrtran - Add Files Recursively is Slow
« Reply #18 on: May 23, 2020, 12:15:28 am »
Windows 10 and C::B 20.03 for both.

Offline darmar

  • Multiple posting newcomer
  • *
  • Posts: 59
Re: Forrtran - Add Files Recursively is Slow
« Reply #19 on: May 23, 2020, 06:10:39 pm »
@darmar: It seems that you're parsing every file on at the moment it is added. I suppose it is better just to schedule the file for parsing and do this in another thread or onidle.

@oBFusCATed: you are right. I will think about how to move this parsing onto a separate thread.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 12625
    • Travis build status
Re: Forrtran - Add Files Recursively is Slow
« Reply #20 on: May 23, 2020, 08:03:47 pm »
You don't need to move the parsing on a separate thread.
You can do it on the main thread, but just do it on an idle even.
To limit its impact you can time limit it - say you do parsing for 10ms and then stop and wait for the next idle event.
This will be a lot simpler than going the separate thread. It seems parsing fortran is really fast :)
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]