Author Topic: Question about the code completion revamping done around 2010...  (Read 34677 times)

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Question about the code completion revamping done around 2010...
« Reply #15 on: November 07, 2011, 06:41:39 am »
it's just a demo :P I'm only tokenizing with spaces for the test.

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Question about the code completion revamping done around 2010...
« Reply #16 on: November 07, 2011, 06:44:37 am »

We don't have a word "ded"? or "dlength,"?


needed, wordlength. What you're seeing aren't words, but tree vertices.

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Question about the code completion revamping done around 2010...
« Reply #17 on: November 07, 2011, 06:51:05 am »
Okay, more results.

Using basic tokenizing, I've managed to dump the tree info of projectmanager.cpp in 47K.  Half of that are the numeric values for the tree nodes. UNFORTUNATELY, this means we'll have to read the full tree dump and process it, JUST to load the basic (non-suffix) tree in memory to search. :-/

I'm wondering if using a lexer and just dumping the words would be better.

UNLESS... we keep a full dictionary of ALL the keywords in memory (a-la code completion) and store in the files only the indexes for the keywords. That WOULD be a lot faster. I'm wondering if it would be good to keep a dictionary per folder to conserve memory.
« Last Edit: November 07, 2011, 06:55:30 am by rickg22 »

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Question about the code completion revamping done around 2010...
« Reply #18 on: November 07, 2011, 07:41:55 am »
Okay, second test. I wrote the numeric info in base64, and got rid of a few CR's. This is the result, occupying no more than 28K for projectmanager.cpp:

Code
[NODES]
A,A,A,,A;E,Ih,A,C,C;I,By,B,E,E;K,IK,C,C,I;E,EB,D,D,B;C,Z,E,B,B;H,L9,F,D,E;P,Gw,G,C,N;G,Er,H,B,F;D,Cq,I,C,B;D,NG,J,C,B;I,b,K,H,B;F,CX,L,C,D;D,P,M,B,C;G,S,N,B,F;B,A,M,,B;
H,EZ,O,B,F;G,Hv,P,C,E;B,A,N,,B;H,S,Q,,G;H,Bp,R,C,F;B,A,S,,B;F,g,T,C,D;D,BE,U,B,C;D,Bh,V,B,C;B,A,E,,B;D,PM,W,B,B;H,Ki,K,C,F;I,b,X,,B;B,A,K,,B;E,d,Y,,D;C,t,Z,B,B;C,BO,T,B,B;
E,g,a,,C;I,GR,b,C,G;E,A,c,,E;B,A,I,,B;C,k,d,,B;B,A,D,,B;O,B8,e,G,H;D,4,f,B,C;E,A,g,,E;C,r,h,B,B;B,A,h,,B;C,r,i,,B;B,A,Z,,B;C,t,j,,B;D,A,k,,D;I,OX,l,B,H;H,QQ,m,C,F;
L,Dw,n,C,J;H,Ce,o,C,F;K,Hk,p,D,H;H,8,q,B,G;B,A,n,,B;F,ET,r,B,D;B,A,f,,B;K,BU,s,B,I;B,A,F,,B;F,Co,t,C,C;B,A,q,,B;D,8,u,,C;L,BQ,v,D,H;B,A,C,,B;H,B5,w,C,E;G,NW,x,B,E;

(Followed by many more similar lines - skipped for bandwidth reasons)

[LABELS]
This filename isShutdown part of there's CodeBlocksEvent Blocks IDE and licensed under GNU Lesser eneral Public icense version 3 https www gnu rg s gpl 04 ml Revision
7556 d rojectmanager cpp 2011 11 0 7 44Z jenslody HeadURL rickg22@svn berlios descendant svnroot epos odeblocks runk rc dk_precomp nclude fndef B_PRECOMP xNewId
atetime maglist rame menuBar plitter lass's eader events anager nfigmanager bproject ogmanager luginmanager editormanager servarmanager orkspaceloader workspace
editor irectory lobals xception orce assert ndif ctor lgorithm uibook usyinfo hoicdlg dlg gdlg extdlg okenzr tils rementalselectlistdlg groupsandmasks sfilemasksdlg
depsdlg ultiselectdlg filters rmreplacedlg fileoptionsdlg mplate rojectManager Mgr stance ool alse ximum number tems OpenFilesRecursively ith text tatic_cast structor
signed ternal AX_OPEN_WITH_ITEMS keep ync low rray dOpenWithInternal ecial try penWith _CanShutdown e _ProjectManager MenuSetActiveProject OpenFile aveProject File
CloseProject File AddFilePopup sRecursivelyPopup RemoveFolderFilesPopup olderFilesPopup ilePopup nameFile ProjectNotes Properties FileProperties TreeProjectProperties
FileProperties GotoFile ExecParams ViewCategorizePopup UseFoldersPopup HideFolderNamePopup FileMasks NextProject iorProject TreeProps Up Down RenameWorkspace SaveWorkspace
AsWorkspace CloseWorkspace VirtualFolder DeleteVirtualFolder ndFile NB_TabTop Bottom __WXMSW__ Under GTK ave oticed at TreeCtrl ending EVT_COMMAND_RIGHT_CLICK vents
hen ght icking ne ent ea xy ndles is jTree ublic Window ent DefaultPosition Size R_EDIT_LABELS DEFAULT_STYLE MULTIPLE O_BORDER tected oid nRightClick MouseEvent
turn lags itTest Point tX Y EE_HITTEST_ABOVE BELOW NOWHERE alled CommandEvent EVT_COMMAND_RIGHT_CLICK stEvent ParentProject lse Skip DECLARE_EVENT_TABLE EGIN_EVENT_TABLE
RIGHT_DOWN ND_EVENT_TABLE vtHandler TREE_BEGIN_DRAG TreeBeginDrag END_DRAG EndDrag LABEL_EDIT BeginEditNode LABEL_EDIT EndEditNode ITEM_ACTIVATED ProjectFileActivated
RIGHT_CLICK ItemRightClick KEY_DOWN KeyDown AUINOTEBOOK_TAB_RIGHT_UP abContextMenu MENU_RANGE OpenWith Position SetActiveProject enameWorkspace aveWorkspace AsWorkspace
CloseWorkspace AddVirtualFolder DeleteVirtualFolder FileToProject sToProjectRecursively moveFileFromProject FolderFiles File Project File Project File ile Notes
perties GotoFile xecParameters ViewCategorize UseFolders HideFolderName FileMasks FindFile IDLE Idle _pTree Workspace TreeFreezeCounter IsLoadingProject Workspace
ClosingProject Workspace nitialDir T isCheckingForExternallyModifiedProjects CanSendWorkspaceChanged Notebook ewTitle AuiNotebook anager AppWindow AUI_NB_WINDOWLIST_BUTTON
ConfigManager pproves adBool vironment _tabs_bottom etWindowStyleFlag WindowStyleFlag BOTTOM FileName wd ActiveProject L ProjectToActivate s sArray learProjectDependencies
anel anel ID_ANY AB_TRAVERSAL LIP_CHILDREN nitPane FileGroups FilesGroupsAndMasks nfigManager fg manager VisualState tvsNone tegorize_tree Categorize _folders UseFolders
ide_folder_name HideFolderName x valid mbination lders sn't ecedence ~ptvsHideFolderName Write buildTree gistered inks gisterEventSink EVT_APP_STARTUP_DONE ventFunctor
ppDoneStartup vent ing st HE AST ING ctivated rtup structors tructors ways lowing IFO ule ast rst utofdate shEventHandler ProjectManager re ource moved hutting own
ses ready ooked eds rselves moveEventHandler lete unt unt Project's temHasChildren Images estroy appShuttingDown uildTree AddPage orkspaceIconIndex ad_only vsWorkspaceReadOnly
IconIndex ProjectReadOnly olderIconIndex Folder VirtualFolderIconIndex VirtualFolder String gs NOTE Keep VisualState ese lated able es ng Normal issing Missing odified
odified only ReadOnly rol stems c ch ddedFiles cAdded lict Conflict Missing odified OutOfDate ptodate UpToDate quireslock RequiresLock xternal External otlock GotLock
ckstolen LockStolen match match ncontrolled NonControlled irtual _open folder_open mptyString Bitmap mp mageList 6 fix DataPath es ile sEmpty LoadBitmap ITMAP_TYPE_PNG
ImageList reateMenuTreeProps enuBar ODO drav oveProjectUp om in re ostpone ndMenu archId MenuItemCount ppendSeparator oto Alt sert perties SEPARATOR ead s cursively
rams' guments ecution ameters argets otes pup eeprops Ctrl hift pdateProjectFiles own ctivate ior 5 st xt 6 CheckItem ategorize y ypes isplay sked sk de amespace
_categorise use_folders hide_folder_name ke ure uration istent heckForExternallyModifiedProjects nable ditorBase es leaseMenu DefaultPath th fault_path 're ILE_SEP_PATH
DefaultPath ProjectStillOpen ProjectFile fresh ly t apce asActive eProjectName fore Modified temBold rojectNode ItemIdValue id sureVisible fresh PROJECT_ACTIVATE
luginManager ifyPlugins owMenu Ok ption TreeData tdkVirtualFolder ItemData _vfolder Kind lowing ll eck urrently piling f nds lected n able me tions opUpMenuOption
Manager s Project File older unt InTree urProject Title SameAs urrentlyCompilingTarget reak re aveLayout oseProject leteAllItems FileOptionsDlg file File EditorManager
pen FullPath ctiveEditor intf witch Text _str luginsArray mes imeOffers MimePlugin nHandleFile Info fo Info tle known ternal ileState name older keRelativeTo mmonTopLevelPath
rmat yway s skPluginsForModuleMenu tProjectManager ULL ould ast upMenu lFile path def S indows names ensitive ere void ing ame ly fferent per wercase name ower tches
enu sultFiles rame Frame b Bar adProject It Exists eginLoadingProject ry locks op ch ets ed ce remely ful s mature its tead ing ple ints alues os your ust ypeOf
CodeBlocksProject e'll cause reating lows osed ddle ration o thod Loaded rs gin s wn dLoadingProject IMEHandlerForFile ne til Workspace oadProject reezeTree Modified
odified Dependencies oaded pendsOn DependingOnReloaded psMap rator ProjectDeps cond inue dex T_FOUND sh_back iginalPosition adedProject mpty rojectDependency osition
dex ToActivate llptr At freezeTree ewProject essageBox sts Are ly nt VERWRITE rmation YES_NO CON_QUESTION YES RemoveFile ldn't ldpath e The ght rror WARNING jCount
QueryCloseAllProjects ving owever es ' gain Project tsavefiles Files BatchBuild sg ' Do anges witch ANCEL NO CANCEL AllProjects mmented romyth ctiveProject AppShuttingDown
Cwd Changed so ClosingOtherProjects ProjectFromAllDependencies ved ied CLOSE Files storeTreeState ing ProjectAs As ctiveProjectAs llProjects Modified rpAround xProject
ed de ssert lectItem Down Workspace orkspace Title Workspace Workspace K WorkspaceAs Workspace 't ank_workspace d Default Workspace Text Root date con d unction
venience usy man ingOrClosing Project Workspace ClosingProject Workspace aw t me t rge TreeState s_file xists cess rite Root pand AddFileToProject rrayInt BuildTargetsCount
played ions alog ForMultiBuildTargetIndex iveFilename name rmalize ATH_NORM_DOTS ABSOLUTE asePath FileToProject ccessfully t ze_t BuildTarget BuildTarget Project
FILE_ADDED String MultipleFilesToProject String list rogressDialog ess ease it ing AddFiles nown AddFiles BuildTargetIndex etSingleChoiceIndex ices ultiSelectDlg
lg ng aceWindow odal OK SelectedIndices OpenFile asic uff nnot ill sent _file Exists esNotDependOn FileState LogManager gError Header Source ide ditor ailed k stom
s ils y IME b iltinEditor ognized No ldn't SelectedFile reeSelection usesCircularDependency asePath st ther pendenciesForProject w eper ched sibility ircular ency
ies ld HECK_VERSION 9 bugLog s _str Dependency hing rase d 2 ureProjectDependencies DepsDlg uiNotebookEvent Bmenu abs p tom Item ng yle wxAUI_NB_BOTTOM d bout bs
ition Event ions DraggingSelection at ragging a sociated llow DragNode ed llow id m To To From From deDragged vent pand ggled ckwards xpanded llapse Group Target
parator MousePosition creenToClient ell RightClickItem oint set ing ser ybe veral ed TreeItemIds ually TextFromUser er irExists ooseDirectory n nerated llFiles IR_FILES
DIRS ability sons ies card ies ll CMs tains VS d GroupsCount Masks Strings ally Dialog Filters terString D_OPEN MULTIPLE FILE_MUST_EXIST atibility HideReadonly terIndex
ndexForFilterAll ths Files okie rstChild NextChild sCount ain re se DEFAULT RemoveFiles e rmally omes ing FileFromProject lculateCommonTopLevelPath RemoveFiles file
gle irtual sRecursively Deleted XUNUSED se Index Notes upTitle Options t d build priate oltips b otebook Tip geCount ame sWith place abToolTip perties tinActiveEditor
ruct RelativePathCmp ActiveProject or 1 2 mp vate AbsolutePathCmp Equal def d s _index rt ique rator crementalSelectIterator wProject Files ShowProject isplayItem
ListDlg hecked Categorize UseFolders FileMasksDlg Found arching und lectAll INFORMATION d Added ing eto Renamed abel UpdateUI UpdateUIEvent dleEvent et ished it'ing
et's 's utoGeneratedBy 't name uto GET_VOLUME SEPARATOR Name extEntryDialog OK ENTRE Value _name verwrite t nameFile LL ditions 're cess ing pecifically O lication
en ialized WORKSPACE_CHANGED inimizeFreeSpace tex er nna bugging gr All py ers ssing h der Pointers Project ateTime cationTime aterThan stModificationTime side se
aved ReplaceDlg All Yes Cancel No ouch valid ually cessary t REMOVED _id ild hildrenCount PluginByName piler activating er st wise If nd down ing Addition Layout
oot ll fter fy letion icks arly erceived OPEN cessEvent es howNotesOnLoad hile yout us sy rs ined tch UserVariableManager rogate dn't Item RootItem ing Notes ce
ing s ing ther KeyEvent y_event eyEvent Code K_DELETE NUMPAD_DELETE mand


The advantage over just posting the keywords would be the tree construction speed.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9699
Re: Question about the code completion revamping done around 2010...
« Reply #19 on: November 07, 2011, 08:17:26 am »
About the wxString or std::string, I think using std::string is enough. Non-anis (code point value > 128) code should only be appeared in comments.
You might have missed that, but we had this discussion - but we do have users using non-ascii codes in statements and variables (not comments).
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 6035
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: Question about the code completion revamping done around 2010...
« Reply #20 on: November 07, 2011, 08:20:05 am »

We don't have a word "ded"? or "dlength,"?


needed, wordlength. What you're seeing aren't words, but tree vertices.
Sorry, I still no follow you, e.g. see the tree below:
Code
- "" (0)

      [skip..]

      +- "d" (158)
      |         +- "ed" (267)
      |         +- "i" (299)
      |         |         +- "fication" (159)
      |         |         \- "ng" (300)
      |         +- "length," (253)
      |         \- "o" (11)
A full key is the string that accumulate all the labels from the root to the leaf, here the "d"(158) is a child of the root ""(0), so I can see the full string is "" + "d" + "ed" => "ded".

Am I right?
« Last Edit: November 07, 2011, 08:25:05 am by ollydbg »
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9699
Re: Question about the code completion revamping done around 2010...
« Reply #21 on: November 07, 2011, 08:33:46 am »
Am I right?
If it would be the tree's root, yes. But I guess Rick showed the tree only partially and there are simply missing some parent nodes.
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline rickg22

  • Lives here!
  • ****
  • Posts: 2283
Re: Question about the code completion revamping done around 2010...
« Reply #22 on: November 07, 2011, 05:18:27 pm »
Oh crap, I think i screwed the tree algorithm. I need to recopy the old version then :P

EDIT: Oh, no. I saw what happened. The tree I displayed is a SUFFIX tree. It contains all the suffixes of the dictionary for faster lookup. Anyway, I hadn't implemented it well and it got on an endless loop after processing a big file. I'm rewriting the algorithm to use a less complicated edge labeling scheme, and possibly make some other modifications.
« Last Edit: November 13, 2011, 01:14:39 am by rickg22 »