Ok.. about the issue with ReplaceBufferForReparse(), I think it will be more clear with an example.
The goal is to get UngetToken() to work reliably when macro expansions are involved.
As an example, we have this code:
#define MYMACRO X=0 /*Alternatively, user might have a replacement rule for MYMACRO*/
int X;
MYMACRO;
X = 1;
^ denotes the token index.
_ underlines the current token string.
EOL char is ignored for our example.
CASE I: when ReplaceBufferForReparse() is called in GetToken() / used directly before calling GetToken().
GetToken() Begin:
int X;MYMACRO;X = 1;
Token: _^
Undo: ^
Step 1: Update undo token and get next token.
int X;MYMACRO;X = 1;
Token: _______^
Undo: ^
Step 2: ReplaceBufferForReparse(): Replace the macro and reset token index.
BUG HERE:
Undo token is still pointing to bad location! int X;MYMAX=0;X = 1;
Token: ^
Undo: ^
FIX:
Move the undo index to correct location. int X;MYMAX=0;X = 1;
Token: ^
Undo: ^
Step 3: Get the next token now after replace.
int X;MYMAX=0;X = 1;
Token: _^
Undo: ^
CASE II: when ReplaceBufferForReparse() is called in PeekToken(), we have an additional problem.
PeekToken() Begin:
Step 1: Save the token index.
int X;MYMACRO;X = 1;
Token: _^
Save: ^
Step 2: DoGetToken():
- Get next token.
int X;MYMACRO;X = 1;
Token: _______^
Save: ^
- Replace the macro and reset token index.
int X;MYMAX=0;X = 1;
Token: ^
Save: ^
- Get the next token now after replace.
int X;MYMAX=0;X = 1;
Token: _^
Save: ^
Step 3: Assign Token to Peek, then restore Token from Save.
BUG HERE:
Save is still pointing to bad location! int X;MYMAX=0;X = 1;
Peek: _^
Token: _^
FIX:
Reset the token index to correct location. int X;MYMAX=0;X = 1;
Peek: _^
Token: _ ^
Here is an updated patch with fixes for both cases.
Index: src/plugins/codecompletion/parser/tokenizer.cpp
===================================================================
--- src/plugins/codecompletion/parser/tokenizer.cpp (revision 9271)
+++ src/plugins/codecompletion/parser/tokenizer.cpp (working copy)
@@ -1135,6 +1146,8 @@
unsigned int savedLineNumber = m_LineNumber;
unsigned int savedNestLevel = m_NestLevel;
+ int savedReplaceCount = m_IsReplaceParsing ? m_RepeatReplaceCount : -1;
+
if (SkipUnwanted())
m_PeekToken = DoGetToken();
else
@@ -1144,9 +1157,20 @@
m_PeekLineNumber = m_LineNumber;
m_PeekNestLevel = m_NestLevel;
- m_TokenIndex = savedTokenIndex;
- m_LineNumber = savedLineNumber;
- m_NestLevel = savedNestLevel;
+ // Check whether a ReplaceBufferForReparse() was done in DoGetToken().
+ // We assume m_Undo... have already been reset in ReplaceBufferForReparse().
+ if (m_IsReplaceParsing && savedReplaceCount != (int)m_RepeatReplaceCount)
+ {
+ m_TokenIndex = m_UndoTokenIndex;
+ m_LineNumber = m_UndoLineNumber;
+ m_NestLevel = m_UndoNestLevel;
+ }
+ else
+ {
+ m_TokenIndex = savedTokenIndex;
+ m_LineNumber = savedLineNumber;
+ m_NestLevel = savedNestLevel;
+ }
}
return m_PeekToken;
@@ -1818,6 +1871,11 @@
// Fix token index
m_TokenIndex -= bufferLen;
+ // Fix undo position
+ m_UndoTokenIndex = m_TokenIndex;
+ m_UndoLineNumber = m_LineNumber;
+ m_UndoNestLevel = m_NestLevel;
+
// Update the peek token
if (m_PeekAvailable && updatePeekToken)
{