Author Topic: vector<int> is OK, but string or wstring no-work.  (Read 90994 times)

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: vector<int> is OK, but string or wstring no-work.
« Reply #30 on: January 11, 2010, 04:17:58 pm »
Code
        if (token == ParserConsts::kw_template)
        {
            // There are some template defintions that are not working like
            // within gcc headers (NB: This syntax is a GNU extension):
            // extern template
            //    const codecvt<char, char, mbstate_t>&
            //    use_facet<codecvt<char, char, mbstate_t> >(const locale&);
            m_Tokenizer.SetState(tsTemplateArgument);
            wxString args = m_Tokenizer.GetToken();
            token = m_Tokenizer.GetToken();
            TRACE(_T("DoParse() : Template argument='%s', token ='%s'"), args.wx_str(), token.wx_str());
            if (token==ParserConsts::kw_class)
            {
                m_Str.Clear();
                if (m_Options.handleClasses)
                    HandleClass(ctClass, args);
                else
                    SkipToOneOfChars(ParserConsts::semicolonclbrace, true);
            }
            else if (token==ParserConsts::kw_struct)
            {
                m_Str.Clear();
                if (m_Options.handleClasses)
                    HandleClass(ctStructure, args);
                else
                    SkipToOneOfChars(ParserConsts::semicolonclbrace, true);
            }
            else
            {
                SkipToOneOfChars(ParserConsts::semicolonclbrace, true);
            }
        }
We should rethink the code above, for example, there are some template declaration like below:
Code
template<typename _CharT, typename _Traits, typename _Alloc>
    inline bool
    operator<=(const basic_string<_CharT, _Traits, _Alloc>& __lhs,
      const basic_string<_CharT, _Traits, _Alloc>& __rhs)
    { return __lhs.compare(__rhs) <= 0; }

at this time, the "token string" behind the template argument is not "class" nor "struct", but a general function declaration. So, I think we should use a variable like "template argument" to specify the current context".

For example:

When the parser eat these strings "template<typename _CharT, typename _Traits, typename _Alloc>", we can set the variable "template argument", then we continue our DoParse Loop. next we add a Token, we should attach this "template argument" context to the Token. So, This variable is just like other variables like: m_Str( store the type information) or m_EncounteredTypeNamespaces (store the namespace in data types).

Any way, checking the next token after the template argument is not a correct way.
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: vector<int> is OK, but string or wstring no-work.
« Reply #31 on: January 11, 2010, 06:29:01 pm »
Any way, checking the next token after the template argument is not a correct way.
True, but I still wonder why it works with vector and alike and why with any revision before applying your and blueshake's patch it used to work. There must be another error.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline Loaden

  • Lives here!
  • ****
  • Posts: 1014
Re: vector<int> is OK, but string or wstring no-work.
« Reply #32 on: January 12, 2010, 01:24:56 am »
Will it be to replace the rules of the problem? This is CodeLite replacement rules:
Code
EXPORT
WXDLLIMPEXP_CORE
WXDLLIMPEXP_BASE
WXDLLIMPEXP_XML
WXDLLIMPEXP_XRC
WXDLLIMPEXP_ADV
WXDLLIMPEXP_AUI
WXDLLIMPEXP_CL
WXDLLIMPEXP_LE_SDK
WXDLLIMPEXP_SQLITE3
WXMAKINGDLL
WXUSINGDLL
_CRTIMP
__CRT_INLINE
__cdecl
__stdcall
WXDLLEXPORT
WXDLLIMPORT
wxT
wxTopLevelWindowNative=wxTopLevelWindowMSW
wxWindow=wxWindowMSW
wxStatusBar=wxStatusBarBase
WXUNUSED
wxDEPRECATED
_T
ATTRIBUTE_PRINTF_1
ATTRIBUTE_PRINTF_2
WXDLLIMPEXP_FWD_BASE
WXDLLIMPEXP_FWD_CORE
DLLIMPORT
DECLARE_INSTANCE_TYPE
emit
Q_OBJECT
Q_PACKED
Q_GADGET
QT_BEGIN_HEADER
QT_END_HEADER
Q_REQUIRED_RESULT
Q_INLINE_TEMPLATE
Q_OUTOFLINE_TEMPLATE
_GLIBCXX_BEGIN_NAMESPACE(std)=namespace std{
_GLIBCXX_END_NAMESPACE=}
_GLIBCXX_BEGIN_NESTED_NAMESPACE(std, _GLIBCXX_STD)=namespace std{
_GLIBCXX_BEGIN_NESTED_NAMESPACE(std, _GLIBCXX_STD_D)=namespace std{
_GLIBCXX_END_NESTED_NAMESPACE=}
_GLIBCXX_STD=std
WXDLLIMPEXP_SCI
__const=const
__restrict
__THROW
__wur
_STD_BEGIN=namespace std{
_STD_END=}
__CLRCALL_OR_CDECL
_CRTIMP2_PURE
Among them, through the following replacement rules, can support VC9 header file parsing:
Code
_STD_BEGIN=namespace std{
_STD_END=}
__CLRCALL_OR_CDECL
« Last Edit: January 12, 2010, 01:28:34 am by Loaden »

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: vector<int> is OK, but string or wstring no-work.
« Reply #33 on: January 12, 2010, 02:28:38 am »
hi,guys:

see what the template handle do,
Code
        if (token == ParserConsts::kw_template)
        {
            m_Tokenizer.SetState(tsTemplateArgument);
            wxString args = m_Tokenizer.GetToken();
            token = m_Tokenizer.GetToken();
            TRACE(_T("DoParse() : template argument='%s', token ='%s'"), args.wx_str(), token.wx_str());
            if (token==ParserConsts::kw_class)
            {
                m_Str.Clear();
                if (m_Options.handleClasses)
                    HandleClass(ctClass,args);
                else
                    SkipToOneOfChars(ParserConsts::semicolonclbrace, true);
            }
            else if (token==ParserConsts::kw_struct)
            {
                m_Str.Clear();
                if (m_Options.handleClasses)
                    HandleClass(ctStructure,args);
                else
                    SkipToOneOfChars(ParserConsts::semicolonclbrace, true);
            }
            else
            {
                SkipToOneOfChars(ParserConsts::semicolonclbrace, true);
            }
        }

just as ollydbg said,when the the next token is not class or struct,the parser will skip to }/;,I think it is wrong here.
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: vector<int> is OK, but string or wstring no-work.
« Reply #34 on: January 12, 2010, 02:51:53 am »
@blueshake:
Yes, the code is added by me.
Code
            else
            {
                SkipToOneOfChars(ParserConsts::semicolonclbrace, true);
            }
I nearly forget what's in the visualfc's patch, he use a different logic. So, I will check it.

@macfly
I will exam it, it seems the code before is not correct either, but it do show some code completion list.
« Last Edit: January 12, 2010, 02:53:41 am by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: vector<int> is OK, but string or wstring no-work.
« Reply #35 on: January 12, 2010, 02:58:23 am »
Here is improved testString.txt, I have test in an old rev 5988, it seems the last part of file was skipped.
Eg, You can't find the declaration of this function.
Code
  inline wstring
  to_wstring(long long __val)
  { return __gnu_cxx::__to_xstring<wstring>(&std::vswprintf,
   4 * sizeof(long long),
   L"%lld", __val); }


You can enable PARSERTHREAD_DEBUG_OUTPUT 1 to see the log.

« Last Edit: January 12, 2010, 02:09:54 pm by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: vector<int> is OK, but string or wstring no-work.
« Reply #36 on: January 12, 2010, 03:49:54 am »
I'm planning adding a variable to store the Template argument in the ParserThread.cpp.

Some thing like below:

Code
        if (token == ParserConsts::kw_template)
        {
            m_Tokenizer.SetState(tsTemplateArgument);
            m_TemplateArgument = m_Tokenizer.GetToken();
            m_Str.Clear();
            token = m_Tokenizer.GetToken();
            TRACE(_T("DoParse() : Reading Template argument='%s'"), m_TemplateArgument.wx_str());
            m_Tokenizer.SetState(tsSkipUnwanted);
        }

So, when we are about to add some tokens like "class" or "function", we can check m_TemplateArgument. if it is an wxEmptyString, then the currently token is a "normal" token, if not, the current token is a template token.

Also, I suggest that in the Token.h.

we can add a member to the Token class. like:
Code
class Token : public BlockAllocated<Token, 10000>{
...
wxString m_TemplateArgument;
}

Any comments?
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: vector<int> is OK, but string or wstring no-work.
« Reply #37 on: January 12, 2010, 04:28:40 am »
If you do this, do not forget to change the codes about mouse hover tip too. :D
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: vector<int> is OK, but string or wstring no-work.
« Reply #38 on: January 12, 2010, 06:54:30 am »
I'm planning adding a variable to store the Template argument in the ParserThread.cpp.
[...]
Sounds good.

Also, I suggest that in the Token.h.
we can add a member to the Token class. like:
[...]
For what purpose? Displaying the template arguments?
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: vector<int> is OK, but string or wstring no-work.
« Reply #39 on: January 12, 2010, 08:39:56 am »
Quote
For what purpose? Displaying the template arguments?
Yes, also, if we need to do some template instantiation, we need to store the template argument.

Currently, Further more, if a "class template" Token, we can use the "argument" field to store the template argument. But if it is a "function template" Token, we need another field. :D
« Last Edit: January 12, 2010, 08:53:51 am by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline blueshake

  • Regular
  • ***
  • Posts: 459
Re: vector<int> is OK, but string or wstring no-work.
« Reply #40 on: January 12, 2010, 09:46:26 am »
hi,ollydbg:

I get some codes from the string test file.It seems the typedef can not handle correctly.
Code
class basic_string
{

};
_GLIBCXX_BEGIN_NAMESPACE(std)

  template<typename _Alloc>
    class allocator;

  template<class _CharT>
    struct char_traits;

  template<typename _CharT, typename _Traits = char_traits<_CharT>,
           typename _Alloc = allocator<_CharT> >
    class basic_string;

  template<> struct char_traits<char>;

  typedef basic_string<char>    string;

#ifdef _GLIBCXX_USE_WCHAR_T
  template<> struct char_traits<wchar_t>;

  typedef basic_string<wchar_t> wstring;
#endif

#if (defined(__GXX_EXPERIMENTAL_CXX0X__) \
     && defined(_GLIBCXX_USE_C99_STDINT_TR1))

  template<> struct char_traits<char16_t>;
  template<> struct char_traits<char32_t>;

  typedef basic_string<char16_t> u16string;
  typedef basic_string<char32_t> u32string;

#endif

_GLIBCXX_END_NAMESPACE

//string aaaa;
//aaaa.



you can find that the string loses its ancestor.It work before.
Keep low and hear the sadness of little dog.
I fall in love with a girl,but I don't dare to tell her.What should I do?

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: vector<int> is OK, but string or wstring no-work.
« Reply #41 on: January 12, 2010, 10:11:17 am »
you can find that the string loses its ancestor.It work before.
Exactly. That's what I noticed, too. Still: I don't see why that is. But it's most likely related to Ollydbg's patch. So I was hoping he'd help us...
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: vector<int> is OK, but string or wstring no-work.
« Reply #42 on: January 12, 2010, 10:18:55 am »
you can find that the string loses its ancestor.It work before.
Exactly. That's what I noticed, too. Still: I don't see why that is. But it's most likely related to Ollydbg's patch. So I was hoping he'd help us...
Ok, I will try my best to find the bug :D
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: vector<int> is OK, but string or wstring no-work.
« Reply #43 on: January 12, 2010, 02:09:20 pm »
This is a patch for testing, I just add a member variable to ParserThread and Token. It seems the parser works better. :D.
Also, I refined my testString.txt.

Comments and suggestion are welcome. :D

[attachment deleted by admin]
« Last Edit: January 12, 2010, 02:25:21 pm by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5910
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: vector<int> is OK, but string or wstring no-work.
« Reply #44 on: January 12, 2010, 03:29:51 pm »
I noticed that we have forgot one Macro replacement rule

Code
_GLIBCXX_END_NAMESPACE   to  }

in default.conf file , change to this
Code
		<TOKEN_REPLACEMENTS>
<ssmap>
   <_GLIBCXX_END_NAMESPACE>
<![CDATA[}]]>
</_GLIBCXX_END_NAMESPACE>
<_GLIBCXX_BEGIN_NAMESPACE>
<![CDATA[+namespace]]>
</_GLIBCXX_BEGIN_NAMESPACE>
<_GLIBCXX_BEGIN_NAMESPACE_TR1>
<![CDATA[-namespace tr1 {]]>
</_GLIBCXX_BEGIN_NAMESPACE_TR1>
<_GLIBCXX_BEGIN_NESTED_NAMESPACE>
<![CDATA[+namespace]]>
</_GLIBCXX_BEGIN_NESTED_NAMESPACE>
<_GLIBCXX_END_NAMESPACE_TR1>
<![CDATA[}]]>
</_GLIBCXX_END_NAMESPACE_TR1>
<_GLIBCXX_END_NESTED_NAMESPACE>
<![CDATA[}]]>
</_GLIBCXX_END_NESTED_NAMESPACE>
<_GLIBCXX_STD>
<![CDATA[std]]>
</_GLIBCXX_STD>
</ssmap>
</TOKEN_REPLACEMENTS>

Edit

Also, this rule should be added to this function body
Code
Tokenizer::SetReplacementString(_T("_GLIBCXX_END_NAMESPACE"),          _T("}"));

Code
void CodeCompletion::LoadTokenReplacements()
{
    ConfigManager* cfg = Manager::Get()->GetConfigManager(_T("code_completion"));

    ConfigManagerContainer::StringToStringMap& repl = Tokenizer::GetTokenReplacementsMap();
    repl.clear();

    if (!cfg->Exists(_T("token_replacements")))
    {
        // first run; add default replacements string
        Tokenizer::SetReplacementString(_T("_GLIBCXX_STD"),                    _T("std"));
        Tokenizer::SetReplacementString(_T("_GLIBCXX_BEGIN_NESTED_NAMESPACE"), _T("+namespace"));
        Tokenizer::SetReplacementString(_T("_GLIBCXX_END_NESTED_NAMESPACE"),   _T("}"));
        Tokenizer::SetReplacementString(_T("_GLIBCXX_BEGIN_NAMESPACE"),        _T("+namespace"));
        Tokenizer::SetReplacementString(_T("_GLIBCXX_END_NAMESPACE_TR1"),      _T("}"));
        Tokenizer::SetReplacementString(_T("_GLIBCXX_BEGIN_NAMESPACE_TR1"),    _T("-namespace tr1 {"));
    }
    else
        cfg->Read(_T("token_replacements"), &repl);
}
By the way, I found it's hard to enter in the CC's configure dialog.
« Last Edit: January 12, 2010, 03:45:05 pm by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.