I'm thinking that the code:
bool Tokenizer::ReplaceBufferText(const wxString& target, bool updatePeekToken)
{
...
...
// Fix token index
m_TokenIndex -= bufferLen;
// Reset undo token
m_SavedTokenIndex = m_UndoTokenIndex = m_TokenIndex;
m_SavedLineNumber = m_UndoLineNumber = m_LineNumber;
m_SavedNestingLevel = m_UndoNestLevel = m_NestLevel;
// Update the peek token
if (m_PeekAvailable && updatePeekToken)
{
m_PeekAvailable = false;
// we set the m_PeekAvailable after calling DoGetToken() function inside the PeekToken()
// to prevent unnecessary recursive call of PeekToken() function.
PeekToken();
}
return true;
}
Should be changed to
bool Tokenizer::ReplaceBufferText(const wxString& target)
{
...
...
// Fix token index
m_TokenIndex -= bufferLen;
// Reset undo token
m_SavedTokenIndex = m_UndoTokenIndex = m_TokenIndex;
m_SavedLineNumber = m_UndoLineNumber = m_LineNumber;
m_SavedNestingLevel = m_UndoNestLevel = m_NestLevel;
// Update the peek token
if (m_PeekAvailable)
m_PeekAvailable = false;
return true;
}
Reason:
1, After some debugging, I see that once the buffer has changed, peek should be invalid, the following "PeekToken() or GetToken()" function call will update the peek token, but not inside the ReplaceBufferText() function.
2, I even think the last parameter of "bool ReplaceFunctionLikeMacro(const Token* tk, bool updatePeekToken = true);" should be removed.
Those kinds of functions just replace some piece of text in the buffer, no need to update peek token, the caller can explicitly call PeekToken or GetToken functions
EDIT1: when using git blame, I found that the second parameter of this function was added in
* * cc_branch: applied patch with rewritten function-like macro handle (v5)
git-svn-id: http://svn.code.sf.net/p/codeblocks/code/branches/codecompletion_refactoring@6436
It is used to expand function-like macro usage.
EDIT2, the second parameter of ReplaceFunctionLikeMacro() as added in
* * cc_branch: improved function-like macro parsing, a little improved for avoid the endless loop
git-svn-id: http://svn.code.sf.net/p/codeblocks/code/branches/codecompletion_refactoring@6691
EDIT3It looks like all the function call with second parameter value == true is used for handling macro expansion. (HandleMacro)
Here is my analysis:
Rev6436
The first place of ReplaceBufferForReparse() with second parameter value == true:
+wxString Tokenizer::GetActualContextForMacro(Token* tk)
+{
+ if (!tk)
+ return wxEmptyString;
+
+ // 1. break the args into substring with "," and store them in normals
+ wxArrayString normalArgs;
+ if (!tk->m_Args.IsEmpty())
+ {
+ ReplaceBufferForReparse(tk->m_Args);
+ SpliteArguments(normalArgs);
+ }
Here, not sure why this need to run peek again?
Since ReplaceBufferForReparse() just put a new piece of text in the buffer(a macro definition's formal parameter), and move back the m_TokenIndex a bit, what is the reason to internally call PeekToken() function?
The second place of ReplaceBufferForReparse() with second parameter value == true:
+ TRACE(_T("HandleMacro() : Adding token '%s' (peek='%s')"), tk->m_Name.wx_str(), peek.wx_str());
+ DoAddToken(tkMacro, tk->m_Name, m_Tokenizer.GetLineNumber(), 0, 0, peek);
+ m_Tokenizer.ReplaceBufferForReparse(m_Tokenizer.GetActualContextForMacro(tk));
Which is inside the ParserThread::HandleMacro() function (handling macro usage) body, but I still don't see a forced PeekToken() call is necessary.
---------------
Rev6691
First place of the ReplaceMacroActualContext() function call with second parameter value == true
void ParserThread::HandleMacro(int id, const wxString &peek)
{
Token* tk = m_TokensTree->at(id);
if (tk)
{
TRACE(_T("HandleMacro() : Adding token '%s' (peek='%s')"), tk->m_Name.wx_str(), peek.wx_str());
DoAddToken(tkMacro, tk->m_Name, m_Tokenizer.GetLineNumber(), 0, 0, peek);
if (m_Options.parseComplexMacros)
m_Tokenizer.ReplaceMacroActualContext(tk);
}
}
I think it is not necessary to run PeekToken() inside ReplaceMacroActualContext().