Code::Blocks Forums

Developer forums (C::B DEVELOPMENT STRICTLY!) => Development => CodeCompletion redesign => Topic started by: ollydbg on September 30, 2017, 08:33:27 am

Title: std::string_view is a good string candicate for Token class in our Lexer
Post by: ollydbg on September 30, 2017, 08:33:27 am
I once asked a question How to pass token kind with its associated information from lexer to preprocessor, then to parser (https://stackoverflow.com/questions/42041579/how-to-pass-token-kind-with-its-associated-information-from-lexer-to-preprocesso) in the stackoverflow, but unlucky, there is no answers.

Today, I just see that basic_string_view on cpp-reference (http://en.cppreference.com/w/cpp/string/basic_string_view)(it will comes in C++17) may be a good candidate for such kind of string in the Token class.

I see a project named ninja-build/ninja: a small build system with a focus on speed (https://github.com/ninja-build/ninja), which use a similar class string_piece.h (https://github.com/ninja-build/ninja/blob/master/src/string_piece.h) for it's lexer, which has those comments:
Quote
/// StringPiece represents a slice of a string whose memory is managed
/// externally.  It is useful for reducing the number of std::strings
/// we need to allocate.

Another project(by the author of cppcheck) danmar/simplecpp: C++ preprocessor (https://github.com/danmar/simplecpp), in it's source file, it said:
Code
/**
     * token class.
     * @todo don't use std::string representation - for both memory and performance reasons
     */
    class SIMPLECPP_LIB Token {
    public:
        Token(const TokenString &s, const Location &loc) :
            str(string), location(loc), previous(NULL), next(NULL), string(s) {
            flags();
        }

        Token(const Token &tok) :
            str(string), macro(tok.macro), location(tok.location), previous(NULL), next(NULL), string(tok.str) {
            flags();
}
...

Luckily, there are some back-ported string_view class for c++11, see bitwizeshift/string_view-standalone: A custom implementation of the C++17 'string_view' back-ported to c++11 (https://github.com/bitwizeshift/string_view-standalone), so may be, let's hope some news will happen.  :)