Thomas, I have a request... are you sure you can't make a way to free VERY LARGE unused blocks of memory with your blockallocator? i.e. we have tokens allocated in groups of 10,000. Could you make it so that when all the tokens of a specific block have been freed, the block would be released automatically? This would spare us having 600Megs of unused memory making up space after we close the contribs workspace... ¬¬
sizeof(Token) == 140, thus 600 MB == 4,491,264
Tokens. If you
really allocate that many tokens, then you should think about looking for a serious bug in the parser

But now seriously: we already talked about this idea of releasing blocks, and I already told you once: this is a really bad idea. Tokens, or their allocation is not the problem we're trying to solve. Starting to tweak the allocator will get you absolutely nowhere, it will only add to overhead.
Parsing
CodeBlocks.cbp produces 79,000 tokens with "everything turned on". The block allocator allocates 80,000 tokens (10.68 MB) for these.
However, if you look at the actual memory consumption, you get figures like this:
Code::Blocks flat: 11.0 MB
Code::Blocks loading CodeBlocks.cbp with code completion turned off: 11.6 MB
Code::Blocks loading CodeBlocks.cbp with code completion turned on: 85.6 MB
Code completion uses up 74 MB, but only 10.68 MB are assigned to tokens. So where do the other 63.62 MB go? Do you see my point?
Adding the ability to release free blocks would make the allocator a lot more complicated and would mean a noticeable increase (read "noticeable " as "about 100%") in execution time for allocations and deallocations, and there is really no reason why we need it. The memory consumed by tokens is
very acceptable compared to other structures. Also, the memory is not "lost", but reused when more objects are allocated.