Author Topic: Cannot increase the number of parallel build processes beyond 16  (Read 3455 times)

Offline alexchen

  • Multiple posting newcomer
  • *
  • Posts: 84
I am using the latest Code::Blocks 20.  I try to increase the number of parallel builds to 32 but it can only go up to 16. 
It will no go up even if I change the setting <PARALLEL_PROCESSES int="32" /> in $HOME/.config/codeblocks/default.conf and restart Code::Blocks. 
What is the factor that caps this value?

Offline BlueHazzard

  • Developer
  • Lives here!
  • *****
  • Posts: 3353
Re: Cannot increase the number of parallel build processes beyond 16
« Reply #1 on: July 08, 2020, 08:15:06 am »
I think it is locked to the maximum count of (virtual) cpus your pc has, because it makes no sense to increase the number past this number

Offline alexchen

  • Multiple posting newcomer
  • *
  • Posts: 84
Re: Cannot increase the number of parallel build processes beyond 16
« Reply #2 on: July 09, 2020, 08:04:45 pm »
I don't think that's the case.  My CentOS VM starts with 4 CPU on a 12 CPU MacPro (the cylindrical model) and I can have set up to 16 parallel builds.  Even if I bump the VM CPU to 8 and eventually to 12 CPU with 32 GB memory and 128 GB HD.
But  CodeBlocks still max out at 16 parallel builds.

I assume Parallel builds is at thread level to launch multiple compiler processes. 
Number of CPU can affect the performance but should not be the limiting factor.

Offline sodev

  • Regular
  • ***
  • Posts: 497
Re: Cannot increase the number of parallel build processes beyond 16
« Reply #3 on: July 09, 2020, 09:05:05 pm »
I think that value is hardcoded and its maximum got increased in the past, sadly i can't find the commit. Im my build (r12183) i can increase the number to 65K.

Offline alexchen

  • Multiple posting newcomer
  • *
  • Posts: 84
Re: Cannot increase the number of parallel build processes beyond 16
« Reply #4 on: July 09, 2020, 09:08:01 pm »
That's what I suspect.  Maybe CodeBlocks cannot handle more than 16 concurrent builds due to some architectural limitation or unresolved issues?

Offline Miguel Gimenez

  • Developer
  • Lives here!
  • *****
  • Posts: 1553
Re: Cannot increase the number of parallel build processes beyond 16
« Reply #5 on: July 09, 2020, 09:17:19 pm »
Revision 10822:

https://sourceforge.net/p/codeblocks/code/10822/

EDIT: if you select 0 then std::max(1, wxThread::GetCPUCount()) is used.
« Last Edit: July 09, 2020, 09:20:07 pm by Miguel Gimenez »

Offline alexchen

  • Multiple posting newcomer
  • *
  • Posts: 84
Re: Cannot increase the number of parallel build processes beyond 16
« Reply #6 on: July 09, 2020, 09:52:17 pm »
Currently I can set up to 16 parallel builds on a VM with 12 CPU.  If I set to 0 according the link mentioned, wouldn't it mean that I can have only 12 parallel builds? That is even worse.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Cannot increase the number of parallel build processes beyond 16
« Reply #7 on: July 10, 2020, 12:16:40 am »
But why do you need it more than 12 in the first place? What do you think it will happen if you set it to 32 or 64 or some other high number? (I know the limitation is not really good idea)
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline alexchen

  • Multiple posting newcomer
  • *
  • Posts: 84
Re: Cannot increase the number of parallel build processes beyond 16
« Reply #8 on: July 10, 2020, 12:20:19 am »
I do not need to bump it up that much really.  I just want to know whether it is a hard limit regardless how powerful your machine is or some other issues.  When I do a big project build wit Code::Blocks command line tool in a batch build, it can sometime take several hours to finish.

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Cannot increase the number of parallel build processes beyond 16
« Reply #9 on: July 10, 2020, 12:29:14 am »
Actually the limit has been removed in r10822.
Are you sure you have a clean install of 20.03?
On my machine I can set it to 32 no problem.

Most of the time this limit is not what is limiting scalability/parallelism in cb projects/workspaces.
The main problem is that targets/projects cannot be run in parallel.
So if you have many small targets they are run in serial almost mode with very little use of CPU resources.

Currently the only solution is to switch to cmake+ninja or something similar.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline alexchen

  • Multiple posting newcomer
  • *
  • Posts: 84
Re: Cannot increase the number of parallel build processes beyond 16
« Reply #10 on: July 10, 2020, 12:34:38 am »
This is my version.  I downloaded it from CB's download site directly

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: Cannot increase the number of parallel build processes beyond 16
« Reply #11 on: July 10, 2020, 01:55:27 am »
Obviously you have to install the latest and greatest to get this feature.
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline sodev

  • Regular
  • ***
  • Posts: 497
Re: Cannot increase the number of parallel build processes beyond 16
« Reply #12 on: July 10, 2020, 07:02:39 am »
So you said you are using CodeBlocks 20 which i assumed is 20.03, but this version was build 2016 and is the release from 2013... scary things happening here :o ::)