« previous next »
Pages: 1 2 [All]   Go Down

Author Topic: EOL issue, the EOL should be automatically detected, and make consistent  (Read 32829 times)

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
EOL issue, the EOL should be automatically detected, and make consistent
« on: August 24, 2012, 08:07:23 am »
I just want to have a feature request:
When I open a LF file, when I hit enters, it should add many lines of LF.
When I open a CRLF file, when I hit enters, it should add many lines of CRLF.
But currently, I see that I can't do that, the EOL is fixed, and either I result a mixed EOF file, or I need to set the EOL in the editor when I open each projects.

This is really annoying.

I have see such report 6 years ago, see: Feature Request: Auto EOL Mode, but I don't think it was implemented.

BTW: I see that Notepad++ have such feature. Will C::B finally implement it?

EDIT: I also find on in the feature request:
Leave EOL format be

EDIT: I just tested the SciTE 3.2.1, and it works as what I want "leave as it is".
« Last Edit: August 24, 2012, 11:31:21 am by ollydbg »
Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #1 on: August 24, 2012, 11:05:29 am »
I dig the code a little, and see that the EOL is set in the editor before the file is loaded.

sdk\cbeditor.cpp

Code
// static
void cbEditor::InternalSetEditorStyleBeforeFileOpen(cbStyledTextCtrl* control)
{
    if (!control)
        return;

    ......
    ......


    // NOTE: duplicate line in editorconfigurationdlg.cpp (ctor)
    control->SetEOLMode(                  mgr->ReadInt(_T("/eol/eolmode"),                   platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF)); // Windows takes CR+LF, other platforms LF only
    control->SetScrollWidthTracking(      mgr->ReadBool(_T("/margin/scroll_width_tracking"), false));
    control->SetMultipleSelection(        mgr->ReadBool(_T("/selection/multi_select"),       false));
    control->SetAdditionalSelectionTyping(mgr->ReadBool(_T("/selection/multi_typing"),       false));
    if (mgr->ReadBool(_T("/selection/use_vspace"), false))
        control->SetVirtualSpaceOptions(wxSCI_SCVS_RECTANGULARSELECTION | wxSCI_SCVS_USERACCESSIBLE);
    else
        control->SetVirtualSpaceOptions(wxSCI_SCVS_NONE);
}
Here, it just read the "eol/eolmode" from the configure file.

So, my question is: If we have add an option like "leave the eol as it is", then we need to Set the EOL after the file is loaded, right?

If I remove the line "control->SetEOLMode....", I think the control should have a eol which is consistent to the opened file?
« Last Edit: August 24, 2012, 11:07:08 am by ollydbg »
Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #2 on: August 24, 2012, 11:54:02 am »
Hi, guys, I dig it further, and I found that the SciTE use this code:

In its source: C:\scite321\scite\src\SciTEIO.cxx

Code
void SciTEBase::DiscoverEOLSetting() {
	SetEol();
	if (props.GetInt("eol.auto")) {
		int linesCR;
		int linesLF;
		int linesCRLF;
		CountLineEnds(linesCR, linesLF, linesCRLF);
		if (((linesLF >= linesCR) && (linesLF > linesCRLF)) || ((linesLF > linesCR) && (linesLF >= linesCRLF)))
			wEditor.Call(SCI_SETEOLMODE, SC_EOL_LF);
		else if (((linesCR >= linesLF) && (linesCR > linesCRLF)) || ((linesCR > linesLF) && (linesCR >= linesCRLF)))
			wEditor.Call(SCI_SETEOLMODE, SC_EOL_CR);
		else if (((linesCRLF >= linesLF) && (linesCRLF > linesCR)) || ((linesCRLF > linesLF) && (linesCRLF >= linesCR)))
			wEditor.Call(SCI_SETEOLMODE, SC_EOL_CRLF);
	}
}

So, it change the EOL dynamically.

BTW: currently, by default, under Windows, it use CRLF, see:
[scintilla] default eol mode on OS X - Google Groups
Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #3 on: August 24, 2012, 03:29:45 pm »
I'm confused (though I think you want the correct thing, but brain is not working well during the last weeks, so I'm not sure).

To clarify, with "add many LF" do you mean "change editor's EOL mode to file's EOL mode" or do you mean "change all line endings in the file"? You basically want anything typed into a file to be whatever the file was before, correct?

Also, what if the file, like many files from OSS projects is mixed already?
Logged
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #4 on: August 24, 2012, 04:13:31 pm »
Quote from: thomas on August 24, 2012, 03:29:45 pm
I'm confused (though I think you want the correct thing, but brain is not working well during the last weeks, so I'm not sure).
Hi, thomas, sorry about the confusion, I'm not a English native speaker, so my English may be poor.

Quote
To clarify, with "add many LF" do you mean "change editor's EOL mode to file's EOL mode" or do you mean "change all line endings in the file"?
I say "add many LF" which means when I hit the enter key many times, the editor add many lines which have LF styled EOL.

Here is an example, when I open a file, which has LF styled EOL, by default, if I have the configuration with "CR LF" in the editor setting, the new added line in the editor will have "CR LF" ending, so, the file becomes "LF" and "CR LF" mixed.

See the image below: (The new added empty line have CRLF styled EOL, but the original file have all LF styled EOL)


On the other side, if you set the editor as "LF" format, then you will have the same issue that an original CRLF styled file will contains many LF styled lines.
Quote
You basically want anything typed into a file to be whatever the file was before, correct?
Yes, What I want is to let the editor automatically detect which EOL is used in the original file, then the new added line should still use same type of EOL.

Currently, both SciTE and NotePad++ has this feature, so I believe C::B should have this feature too.
Quote
Also, what if the file, like many files from OSS projects is mixed already?
For how to detect a files EOL, I think it is simple, just scan the first lines of the file, and set the EOL of the whole file as it is, I think SciTe use this kind of detection.

OK about my brain?  :)

EDIT:
It looks like to determine which EOL is used for a mixed EOL files, SciTe use a voting
Code
void SciTEBase::CountLineEnds(int &linesCR, int &linesLF, int &linesCRLF) {
	linesCR = 0;
	linesLF = 0;
	linesCRLF = 0;
	int lengthDoc = LengthDocument();
	char chPrev = ' ';
	TextReader acc(wEditor);
	char chNext = acc.SafeGetCharAt(0);
	for (int i = 0; i < lengthDoc; i++) {
		char ch = chNext;
		chNext = acc.SafeGetCharAt(i + 1);
		if (ch == '\r') {
			if (chNext == '\n')
				linesCRLF++;
			else
				linesCR++;
		} else if (ch == '\n') {
			if (chPrev != '\r') {
				linesLF++;
			}
		} else if (i > 1000000) {
			return;
		}
		chPrev = ch;
	}
}


« Last Edit: August 24, 2012, 04:43:49 pm by ollydbg »
Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #5 on: August 24, 2012, 04:42:34 pm »
Looks like you are on the right track. Why don't you make a patch? For mixed files, voting makes sense.

Do we have room on the status bar for the EOL mode? (Or is it there already) Some sort of warning, especially for mixed sources would be nice (though not everyone likes warnings and they shouldn't be intrusive).
Logged

Offline thomas

  • Administrator
  • Lives here!
  • *****
  • Posts: 3979
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #6 on: August 24, 2012, 05:25:01 pm »
Quote from: ollydbg on August 24, 2012, 04:13:31 pm
OK about my brain?  :)
Yours is hopefully ok, it's mine that isn't good.

The approach sounds very good, though I would like to also reserve a way to always have a particular encoding and line ending (for example I'm using UTF-8 and LF everywhere, for everything, even under Windows -- don't want to have the editor interfere with this in any way, even hypothetically).

Maybe these options would be a good solution:
  • use system default
  • use XYZ line ending
  • use file's line ending (and fall back to system default)

To detect the encoding, I'd just scan over all lines (does not really take that much longer!) and take whichever encoding occurs more often. On a tie, fall back to system default.
Logged
"We should forget about small efficiencies, say about 97% of the time: Premature quotation is the root of public humiliation."

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #7 on: August 24, 2012, 05:41:10 pm »
Thank you thomas and dmoore for the constructive suggestion.

OK, I will take the job to implement this, but before that, I need to firstly comprehend what the current way do. (Now, I even do not understand what does "Ensure consistent EOLs" used for, so it will take some time to read the source)
Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #8 on: August 24, 2012, 07:01:59 pm »
Quote from: ollydbg on August 24, 2012, 05:41:10 pm
(Now, I even do not understand what does "Ensure consistent EOLs" used for, so it will take some time to read the source)

Converts all EOLs in the doc to whatever is the currently set EOL mode of that editor when you save.

Note that I put in some features related to EOLs in the editortweaks plugin so users could temporarly change from the default manually, per editor. Would be nice not to break that, but I suspect what you are doing will be fine. (If installed, those options are in Edit->Editor tweaks)
Logged

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #9 on: August 25, 2012, 05:24:48 pm »
Here is the draft patch, I add an item "AUTO" in the editor setting, this has the value "3".
The changed file are: src\sdk\cbeditor.cpp and src\sdk\resources\editor_configuration.xrc
Note, I have problems to generate the patch file, because wxsmith change the resource file a lot(strange, it convert the spaces to tabs, I manually change them back to spaces), so I add the xrc file. Both of the files were generated by tortoiseGit, which have LF format.
I added a static function, which I think is not a good idea, right?
Comments are welcome. Thanks.

EDIT: I remove the old attachment, and the patch against the latest svn revision is below:
Code
Index: cbeditor.cpp
===================================================================
--- cbeditor.cpp	(revision 8249)
+++ cbeditor.cpp	(working copy)
@@ -731,6 +731,39 @@
 
 END_EVENT_TABLE()
 
+
+// Count the EOL in the opened file
+static void CountLineEnds(cbStyledTextCtrl* control, int &linesCR, int &linesLF, int &linesCRLF)
+{
+    linesCR = 0;
+    linesLF = 0;
+    linesCRLF = 0;
+
+    //int GetCharAt(int pos)
+    //int GetLength() const;
+    int lengthDoc = control->GetLength();
+    char chPrev = ' ';
+
+    char chNext = control->GetCharAt(0);
+    for (int i = 0; i < lengthDoc; i++) {
+        char ch = chNext;
+        chNext = control->GetCharAt(i + 1);
+        if (ch == '\r') {
+            if (chNext == '\n')
+                linesCRLF++;
+            else
+                linesCR++;
+        } else if (ch == '\n') {
+            if (chPrev != '\r') {
+                linesLF++;
+            }
+        } else if (i > 1000000) {
+            return;
+        }
+        chPrev = ch;
+    }
+}
+
 // class constructor
 cbEditor::cbEditor(wxWindow* parent, const wxString& filename, EditorColourSet* theme)
     : EditorBase(parent, filename),
@@ -1557,7 +1590,6 @@
         control->SetMarginWidth(C_CHANGEBAR_MARGIN, 0);
 
     // NOTE: duplicate line in editorconfigurationdlg.cpp (ctor)
-    control->SetEOLMode(                  mgr->ReadInt(_T("/eol/eolmode"),                   platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF)); // Windows takes CR+LF, other platforms LF only
     control->SetScrollWidthTracking(      mgr->ReadBool(_T("/margin/scroll_width_tracking"), false));
     control->SetMultipleSelection(        mgr->ReadBool(_T("/selection/multi_select"),       false));
     control->SetAdditionalSelectionTyping(mgr->ReadBool(_T("/selection/multi_typing"),       false));
@@ -1575,6 +1607,23 @@
 
     ConfigManager* mgr = Manager::Get()->GetConfigManager(_T("editor"));
 
+    // set the EOL, fall back value: Windows takes CR+LF, other platforms LF only
+    int eolMode = mgr->ReadInt(_T("/eol/eolmode"), platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF);
+    if (eolMode == 3) //auto detect the eol
+    {
+        int linesCR;
+        int linesLF;
+        int linesCRLF;
+        CountLineEnds(control, linesCR, linesLF, linesCRLF);
+        if (((linesLF >= linesCR) && (linesLF > linesCRLF)) || ((linesLF > linesCR) && (linesLF >= linesCRLF)))
+            eolMode = wxSCI_EOL_LF;
+        else if (((linesCR >= linesLF) && (linesCR > linesCRLF)) || ((linesCR > linesLF) && (linesCR >= linesCRLF)))
+            eolMode = wxSCI_EOL_CR;
+        else if (((linesCRLF >= linesLF) && (linesCRLF > linesCR)) || ((linesCRLF > linesLF) && (linesCRLF >= linesCR)))
+            eolMode = wxSCI_EOL_CRLF;
+    }
+    control->SetEOLMode(eolMode);
+
     // Interpret #if/#else/#endif to grey out code that is not active
     control->SetProperty(_T("lexer.cpp.track.preprocessor"), mgr->ReadBool(_T("/track_preprocessor"), true) ? _T("1") : _T("0"));
 
Index: resources/editor_configuration.xrc
===================================================================
--- resources/editor_configuration.xrc	(revision 8249)
+++ resources/editor_configuration.xrc	(working copy)
@@ -200,6 +200,7 @@
                                                 <item>CR LF</item>
                                                 <item>CR</item>
                                                 <item>LF</item>
+                                                <item>AUTO</item>
                                               </content>
                                               <style>wxCB_READONLY</style>
                                             </object>


« Last Edit: August 27, 2012, 04:59:40 pm by ollydbg »
Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #10 on: August 27, 2012, 11:38:32 am »
Damn, I found is that my local git clone was merged around 2012-08-10 (it is the code base my git patch was against on), and after that day( in 2012-08-14) there is a new svn commit 8230 in the xrc file, see:
WebSVN - codeblocks - Path Comparison - /trunk/src/sdk/resources/editor_configuration.xrc Rev 8142 and /trunk/src/sdk/resources/editor_configuration.xrc Rev 8230

So, I think lots of the "extra change"(I believed the bug of wxsmith) in my attachment file in the previous post did the same thing as the commit 8230.

Sorry about the noise. :-[
Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ironhead

  • Almost regular
  • **
  • Posts: 210
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #11 on: August 27, 2012, 08:20:43 pm »
I've manually applied the patch against r.8250 (some of the offsets were wrong) and it's working really well!  Thank you for implementing this feature, I hope to see it applied to the official repository. ;)
Logged

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #12 on: August 28, 2012, 05:03:06 am »
I don't have time to test right now, but the code looks good. I have one small suggestion to use the info window to alert the user to the outcome of the voting logic if there are mixed line endings. In your logic that sets the line ending, you could modify it as follows:

Code
        if (((linesLF >= linesCR) && (linesLF > linesCRLF)) || ((linesLF > linesCR) && (linesLF >= linesCRLF)))
        {
            eolMode = wxSCI_EOL_LF;
            if((linesCR>0) || (linesCRLF>0))
            {
                wxBell();
                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"LF\""), 1000);
            }
        }
        else if (((linesCR >= linesLF) && (linesCR > linesCRLF)) || ((linesCR > linesLF) && (linesCR >= linesCRLF)))
        {
            eolMode = wxSCI_EOL_CR;
            if((linesLF>0) || (linesCRLF>0))
            {
                wxBell();
                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"CR\""), 1000);
            }
        }
        else if (((linesCRLF >= linesLF) && (linesCRLF > linesCR)) || ((linesCRLF > linesLF) && (linesCRLF >= linesCR)))
        {
            eolMode = wxSCI_EOL_CRLF;
            if((linesCR>0) || (linesLF>0))
            {
                wxBell();
                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"CR-LF\""), 1000);
            }
        }
« Last Edit: August 28, 2012, 05:09:37 am by dmoore »
Logged

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #13 on: August 28, 2012, 07:59:13 am »
Thanks ironhead for the test.
Thanks dmoore for the suggestion, below is the refined patch. (combined your suggestion, code style and comments refined)
Code
Index: cbeditor.cpp
===================================================================
--- cbeditor.cpp	(revision 8249)
+++ cbeditor.cpp	(working copy)
@@ -731,6 +731,42 @@
 
 END_EVENT_TABLE()
 
+// Count lines of EOL style in the opened file
+static void CountLineEnds(cbStyledTextCtrl* control, int &linesCR, int &linesLF, int &linesCRLF)
+{
+    linesCR = 0;
+    linesLF = 0;
+    linesCRLF = 0;
+
+    int lengthDoc = control->GetLength();
+    const int maxLineNumber = 1000000;
+    char chPrev = ' ';
+    char chNext = control->GetCharAt(0);
+    for (int i = 0; i < lengthDoc; i++)
+    {
+        char ch = chNext;
+        chNext = control->GetCharAt(i + 1);
+        if (ch == '\r')
+        {
+            if (chNext == '\n')
+                linesCRLF++;
+            else
+                linesCR++;
+        }
+        else if (ch == '\n')
+        {
+            if (chPrev != '\r')
+            {
+                linesLF++;
+            }
+        }
+        else if (i > maxLineNumber)     // stop the loop if the file contains too many lines
+            return;
+
+        chPrev = ch;
+    }
+}
+
 // class constructor
 cbEditor::cbEditor(wxWindow* parent, const wxString& filename, EditorColourSet* theme)
     : EditorBase(parent, filename),
@@ -1557,7 +1593,6 @@
         control->SetMarginWidth(C_CHANGEBAR_MARGIN, 0);
 
     // NOTE: duplicate line in editorconfigurationdlg.cpp (ctor)
-    control->SetEOLMode(                  mgr->ReadInt(_T("/eol/eolmode"),                   platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF)); // Windows takes CR+LF, other platforms LF only
     control->SetScrollWidthTracking(      mgr->ReadBool(_T("/margin/scroll_width_tracking"), false));
     control->SetMultipleSelection(        mgr->ReadBool(_T("/selection/multi_select"),       false));
     control->SetAdditionalSelectionTyping(mgr->ReadBool(_T("/selection/multi_typing"),       false));
@@ -1575,6 +1610,53 @@
 
     ConfigManager* mgr = Manager::Get()->GetConfigManager(_T("editor"));
 
+    // set the EOL, fall back value: Windows takes CR+LF, other platforms LF only
+    int eolMode = mgr->ReadInt(_T("/eol/eolmode"), platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF);
+
+    // The code snippet of auto detect EOL is copied from SciTE's source, CountLineEnds() function
+    // scans the current source file, and counts lines of each EOL style, then finally sets the EOL
+    // by voting logic. In the case of mixed EOL files, we give the user a beep and InfoWindow notification.
+    if (eolMode == 3) //auto detect the EOL
+    {
+        int linesCR;
+        int linesLF;
+        int linesCRLF;
+
+        // count lines of each EOL style
+        CountLineEnds(control, linesCR, linesLF, linesCRLF);
+
+        //voting logic
+        unsigned int delay = 2000;
+        if (((linesLF >= linesCR) && (linesLF > linesCRLF)) || ((linesLF > linesCR) && (linesLF >= linesCRLF)))
+        {
+            eolMode = wxSCI_EOL_LF;
+            if((linesCR>0) || (linesCRLF>0))
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"LF\""), delay);
+            }
+        }
+        else if (((linesCR >= linesLF) && (linesCR > linesCRLF)) || ((linesCR > linesLF) && (linesCR >= linesCRLF)))
+        {
+            eolMode = wxSCI_EOL_CR;
+            if((linesLF>0) || (linesCRLF>0))
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"CR\""), delay);
+            }
+        }
+        else if (((linesCRLF >= linesLF) && (linesCRLF > linesCR)) || ((linesCRLF > linesLF) && (linesCRLF >= linesCR)))
+        {
+            eolMode = wxSCI_EOL_CRLF;
+            if((linesCR>0) || (linesLF>0))
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"CR-LF\""), delay);
+            }
+        }
+    }
+    control->SetEOLMode(eolMode);
+
     // Interpret #if/#else/#endif to grey out code that is not active
     control->SetProperty(_T("lexer.cpp.track.preprocessor"), mgr->ReadBool(_T("/track_preprocessor"), true) ? _T("1") : _T("0"));
 
Index: resources/editor_configuration.xrc
===================================================================
--- resources/editor_configuration.xrc	(revision 8249)
+++ resources/editor_configuration.xrc	(working copy)
@@ -200,6 +200,7 @@
                                                 <item>CR LF</item>
                                                 <item>CR</item>
                                                 <item>LF</item>
+                                                <item>AUTO</item>
                                               </content>
                                               <style>wxCB_READONLY</style>
                                             </object>


I would like to show the filename of the control in the InfoWindow::Display(), I see cbEditor(editorbase) have a member function:
Code
virtual const wxString& GetShortName() const { return m_Shortname; }
What a pity is that
Code
// static
void cbEditor::InternalSetEditorStyleAfterFileOpen(cbStyledTextCtrl* control)
This is a static member function, so no access to the filename.  :(
Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #14 on: August 28, 2012, 08:02:50 am »
... you shouldn't not forget to do nothing (!) if the file dos not contain a line-feed, btw. For this to work, you can return true/false in the method you've introduced.
Logged
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #15 on: August 28, 2012, 09:47:14 am »
Quote from: MortenMacFly on August 28, 2012, 08:02:50 am
... you shouldn't not forget to do nothing (!) if the file dos not contain a line-feed, btw. For this to work, you can return true/false in the method you've introduced.
Thanks, here is an improved patch, which handle them in the "else" clause.

Code
Index: cbeditor.cpp
===================================================================
--- cbeditor.cpp	(revision 8249)
+++ cbeditor.cpp	(working copy)
@@ -731,6 +731,42 @@
 
 END_EVENT_TABLE()
 
+// Count lines of EOL style in the opened file
+static void CountLineEnds(cbStyledTextCtrl* control, int &linesCR, int &linesLF, int &linesCRLF)
+{
+    linesCR = 0;
+    linesLF = 0;
+    linesCRLF = 0;
+
+    int lengthDoc = control->GetLength();
+    const int maxLengthDoc = 1000000;
+    char chPrev = ' ';
+    char chNext = control->GetCharAt(0);
+    for (int i = 0; i < lengthDoc; i++)
+    {
+        char ch = chNext;
+        chNext = control->GetCharAt(i + 1);
+        if (ch == '\r')
+        {
+            if (chNext == '\n')
+                linesCRLF++;
+            else
+                linesCR++;
+        }
+        else if (ch == '\n')
+        {
+            if (chPrev != '\r')
+            {
+                linesLF++;
+            }
+        }
+        else if (i > maxLengthDoc)     // stop the loop if the file contains too many characters
+            return;
+
+        chPrev = ch;
+    }
+}
+
 // class constructor
 cbEditor::cbEditor(wxWindow* parent, const wxString& filename, EditorColourSet* theme)
     : EditorBase(parent, filename),
@@ -1557,7 +1593,6 @@
         control->SetMarginWidth(C_CHANGEBAR_MARGIN, 0);
 
     // NOTE: duplicate line in editorconfigurationdlg.cpp (ctor)
-    control->SetEOLMode(                  mgr->ReadInt(_T("/eol/eolmode"),                   platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF)); // Windows takes CR+LF, other platforms LF only
     control->SetScrollWidthTracking(      mgr->ReadBool(_T("/margin/scroll_width_tracking"), false));
     control->SetMultipleSelection(        mgr->ReadBool(_T("/selection/multi_select"),       false));
     control->SetAdditionalSelectionTyping(mgr->ReadBool(_T("/selection/multi_typing"),       false));
@@ -1575,6 +1610,79 @@
 
     ConfigManager* mgr = Manager::Get()->GetConfigManager(_T("editor"));
 
+    // set the EOL, fall back value: Windows takes CR+LF, other platforms LF only
+    int eolMode = mgr->ReadInt(_T("/eol/eolmode"), platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF);
+
+    // The code snippet of auto detect EOL is copied from SciTE's source, CountLineEnds() function
+    // scans the current source file, and counts lines of each EOL style, then finally sets the EOL
+    // by voting logic. In the case of mixed EOL files, we give the user a beep and InfoWindow notification.
+    if (eolMode == 3) //auto detect the EOL
+    {
+        int linesCR;
+        int linesLF;
+        int linesCRLF;
+
+        // count lines of each EOL style
+        CountLineEnds(control, linesCR, linesLF, linesCRLF);
+
+        //voting logic
+        unsigned int delay = 2000;
+        if (((linesLF >= linesCR) && (linesLF > linesCRLF)) || ((linesLF > linesCR) && (linesLF >= linesCRLF)))
+        {
+            eolMode = wxSCI_EOL_LF;
+            if((linesCR>0) || (linesCRLF>0))
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"LF\""), delay);
+            }
+        }
+        else if (((linesCR >= linesLF) && (linesCR > linesCRLF)) || ((linesCR > linesLF) && (linesCR >= linesCRLF)))
+        {
+            eolMode = wxSCI_EOL_CR;
+            if((linesLF>0) || (linesCRLF>0))
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"CR\""), delay);
+            }
+        }
+        else if (((linesCRLF >= linesLF) && (linesCRLF > linesCR)) || ((linesCRLF > linesLF) && (linesCRLF >= linesCR)))
+        {
+            eolMode = wxSCI_EOL_CRLF;
+            if((linesCR>0) || (linesLF>0))
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"CR-LF\""), delay);
+            }
+        }
+        else // in the case of the file does not contain any line-feed, or the file have equally counts of EOL styles
+        {
+            //set to the fall back mode
+            wxString eolModeStr;
+            if (platform::windows)
+            {
+                eolMode =  wxSCI_EOL_CRLF;
+                eolModeStr = _T("\"CR-LF\"");
+            }
+            else
+            {
+                eolMode =  wxSCI_EOL_LF;
+                eolModeStr = _T("\"LF\"");
+            }
+
+            if((linesCR>0) || (linesLF>0) || (linesCRLF>0)) //equal counts
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode ") + eolModeStr, delay);
+            }
+            else //no line-feed found
+            {
+                wxBell();
+                InfoWindow::Display(_("No Line Endings"), _("No Line Endings found, setting mode ") + eolModeStr, delay);
+            }
+        }
+    }
+    control->SetEOLMode(eolMode);
+
     // Interpret #if/#else/#endif to grey out code that is not active
     control->SetProperty(_T("lexer.cpp.track.preprocessor"), mgr->ReadBool(_T("/track_preprocessor"), true) ? _T("1") : _T("0"));
 
Index: resources/editor_configuration.xrc
===================================================================
--- resources/editor_configuration.xrc	(revision 8249)
+++ resources/editor_configuration.xrc	(working copy)
@@ -200,6 +200,7 @@
                                                 <item>CR LF</item>
                                                 <item>CR</item>
                                                 <item>LF</item>
+                                                <item>AUTO</item>
                                               </content>
                                               <style>wxCB_READONLY</style>
                                             </object>


I have tested several cases, the only minor issue can be: e.g. Under Windows, a file have: linesCR = 0; linesLF = 5; linesCRLF = 5; --------> this will set the eol mode to "LF"......Logicially, I think it should set the eol mode to "CRLF", because under Windows, the CRLF take precedence. Any good idea about the voting logic? I can't find a better, simple and clean way. ;)
Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #16 on: August 28, 2012, 10:13:56 am »
Quote from: ollydbg on August 28, 2012, 09:47:14 am
Logicially, I think it should set the eol mode to "CRLF", because under Windows, the CRLF take precedence. Any good idea about the voting logic? I can't find a better, simple and clean way. ;)
Well you can initialise eolMode according the platform and then only change it if you are sure you counted enough different EOL's (so not >=, but only >).
Logged
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #17 on: August 28, 2012, 10:14:30 am »
Is it possible to put the new code in a separate function?
Logged
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #18 on: August 28, 2012, 10:19:31 am »
Quote from: oBFusCATed on August 28, 2012, 10:14:30 am
Is it possible to put the new code in a separate function?
Maybe even rename CountLineEnds to DetectLineEnds (or alike) and have have it return what eolMode should be set.

Which makes me wonder: Could it be, that scintilla has such a function, too?
Logged
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #19 on: August 28, 2012, 10:29:53 am »
Quote from: MortenMacFly on August 28, 2012, 10:19:31 am
Quote from: oBFusCATed on August 28, 2012, 10:14:30 am
Is it possible to put the new code in a separate function?
Maybe even rename CountLineEnds to DetectLineEnds (or alike) and have have it return what eolMode should be set.

Which makes me wonder: Could it be, that scintilla has such a function, too?
Scintilla does not have such feature, but SciTE does....
Scintilla should keep as simple as it can. (From my point of view), so it leave this function to client.

Quote from: oBFusCATed on August 28, 2012, 10:14:30 am
Is it possible to put the new code in a separate function?
OK, I will did this in the next patch.
Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #20 on: August 28, 2012, 04:43:16 pm »
Quote from: ollydbg on August 28, 2012, 10:29:53 am
OK, I will did this in the next patch.
Ok, see this patch. Any further comments?
Code
Index: cbeditor.cpp
===================================================================
--- cbeditor.cpp	(revision 8249)
+++ cbeditor.cpp	(working copy)
@@ -731,6 +731,95 @@
 
 END_EVENT_TABLE()
 
+// Count lines of EOL style in the opened file
+static void CountLineEnds(cbStyledTextCtrl* control, int &linesCR, int &linesLF, int &linesCRLF)
+{
+    linesCR = 0;
+    linesLF = 0;
+    linesCRLF = 0;
+
+    int lengthDoc = control->GetLength();
+    const int maxLengthDoc = 1000000;
+    char chPrev = ' ';
+    char chNext = control->GetCharAt(0);
+    for (int i = 0; i < lengthDoc; i++)
+    {
+        char ch = chNext;
+        chNext = control->GetCharAt(i + 1);
+        if (ch == '\r')
+        {
+            if (chNext == '\n')
+                linesCRLF++;
+            else
+                linesCR++;
+        }
+        else if (ch == '\n')
+        {
+            if (chPrev != '\r')
+            {
+                linesLF++;
+            }
+        }
+        else if (i > maxLengthDoc)     // stop the loop if the file contains too many characters
+            return;
+
+        chPrev = ch;
+    }
+}
+
+static int DetectLineEnds(cbStyledTextCtrl* control)
+{
+    int eolMode;
+    wxString eolModeStr;
+    // initial EOL mode depend on OS
+    if (platform::windows)
+    {
+        eolMode =  wxSCI_EOL_CRLF;
+        eolModeStr = _T("\"CR-LF\"");
+    }
+    else
+    {
+        eolMode =  wxSCI_EOL_LF;
+        eolModeStr = _T("\"LF\"");
+    }
+
+    int linesCR;
+    int linesLF;
+    int linesCRLF;
+    // count lines of each EOL style
+    CountLineEnds(control, linesCR, linesLF, linesCRLF);
+
+    // voting logic
+    // if the file does not contain any line-feed or the most largest counts are equal( e.g.: linesLF=5,
+    // linesCRLF=5, linesCR=0 ), then we will use the initial EOL mode
+    if ( (linesLF > linesCR) && (linesLF > linesCRLF) )
+    {
+        eolMode = wxSCI_EOL_LF;
+        eolModeStr = _T("\"LF\"");
+    }
+    else if ( (linesCR > linesLF) && (linesCR > linesCRLF) )
+    {
+        eolMode = wxSCI_EOL_CR;
+        eolModeStr = _T("\"CR\"");
+    }
+    else if ( (linesCRLF > linesLF) && (linesCRLF > linesCR))
+    {
+        eolMode = wxSCI_EOL_CRLF;
+        eolModeStr = _T("\"CR-LF\"");
+    }
+
+    unsigned int delay = 2000;
+    if (  ( (linesCR>0) && (linesCRLF>0) )
+       || ( (linesLF>0) && (linesCRLF>0) )
+       || ( (linesCR>0) && (linesLF>0) ) )
+    {
+        //In mixed EOL file, give the user a beep and InfoWindow notification.
+        wxBell();
+        InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode ") + eolModeStr, delay);
+    }
+    return eolMode;
+}
+
 // class constructor
 cbEditor::cbEditor(wxWindow* parent, const wxString& filename, EditorColourSet* theme)
     : EditorBase(parent, filename),
@@ -1557,7 +1646,6 @@
         control->SetMarginWidth(C_CHANGEBAR_MARGIN, 0);
 
     // NOTE: duplicate line in editorconfigurationdlg.cpp (ctor)
-    control->SetEOLMode(                  mgr->ReadInt(_T("/eol/eolmode"),                   platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF)); // Windows takes CR+LF, other platforms LF only
     control->SetScrollWidthTracking(      mgr->ReadBool(_T("/margin/scroll_width_tracking"), false));
     control->SetMultipleSelection(        mgr->ReadBool(_T("/selection/multi_select"),       false));
     control->SetAdditionalSelectionTyping(mgr->ReadBool(_T("/selection/multi_typing"),       false));
@@ -1575,6 +1663,14 @@
 
     ConfigManager* mgr = Manager::Get()->GetConfigManager(_T("editor"));
 
+    // set the EOL, fall back value: Windows takes CR+LF, other platforms LF only
+    int eolMode = mgr->ReadInt(_T("/eol/eolmode"), platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF);
+
+    if (eolMode == 3) //auto detect the EOL
+        eolMode = DetectLineEnds(control);
+
+    control->SetEOLMode(eolMode);
+
     // Interpret #if/#else/#endif to grey out code that is not active
     control->SetProperty(_T("lexer.cpp.track.preprocessor"), mgr->ReadBool(_T("/track_preprocessor"), true) ? _T("1") : _T("0"));
 
Index: resources/editor_configuration.xrc
===================================================================
--- resources/editor_configuration.xrc	(revision 8249)
+++ resources/editor_configuration.xrc	(working copy)
@@ -200,6 +200,7 @@
                                                 <item>CR LF</item>
                                                 <item>CR</item>
                                                 <item>LF</item>
+                                                <item>AUTO</item>
                                               </content>
                                               <style>wxCB_READONLY</style>
                                             </object>

Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline Jenna

  • Administrator
  • Lives here!
  • *****
  • Posts: 7255
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #21 on: August 28, 2012, 05:17:13 pm »
Quote from: ollydbg on August 28, 2012, 04:43:16 pm
Code
+    // set the EOL, fall back value: Windows takes CR+LF, other platforms LF only

The default line-ending on Mac is CR as far as I know.

Sorry to make it a little more complicated.
Logged

Offline MortenMacFly

  • Administrator
  • Lives here!
  • *****
  • Posts: 9694
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #22 on: August 28, 2012, 07:12:07 pm »
Quote from: jens on August 28, 2012, 05:17:13 pm
The default line-ending on Mac is CR as far as I know.
Sorry to make it a little more complicated.
EOL's and he 21st century... still feeling like stone-age... ::)
Logged
Compiler logging: Settings->Compiler & Debugger->tab "Other"->Compiler logging="Full command line"
C::B Manual: https://www.codeblocks.org/docs/main_codeblocks_en.html
C::B FAQ: https://wiki.codeblocks.org/index.php?title=FAQ

Offline oBFusCATed

  • Developer
  • Lives here!
  • *****
  • Posts: 13413
    • Travis build status
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #23 on: August 28, 2012, 09:06:04 pm »
I think CR or LF are supported on MacOSX, prior to 10.xx, the CR was the default and now with the FreeBSD heritage it is the LF.
But I think most softwares support both and users have files with both endings.
Logged
(most of the time I ignore long posts)
[strangers don't send me private messages, I'll ignore them; post a topic in the forum, but first read the rules!]

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #24 on: August 29, 2012, 02:49:35 am »
Quote from: oBFusCATed on August 28, 2012, 09:06:04 pm
I think CR or LF are supported on MacOSX, prior to 10.xx, the CR was the default and now with the FreeBSD heritage it is the LF.
But I think most softwares support both and users have files with both endings.


Quote from: jens on August 28, 2012, 05:17:13 pm
The default line-ending on Mac is CR as far as I know.

Adding the fall-back on Mac as CR is quite easy. But I'm mostly follow what Scintilla did and a discussion on their forum:
See:[scintilla] default eol mode on OS X - Google Groups

And in the source code: Document.cxx
Code
Document::Document() {
	refCount = 0;
#ifdef _WIN32
	eolMode = SC_EOL_CRLF;
#else
	eolMode = SC_EOL_LF;
#endif
...
...
So, I believe Mac can support LF without any problem. :)

Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline dmoore

  • Developer
  • Lives here!
  • *****
  • Posts: 1576
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #25 on: August 29, 2012, 03:25:56 am »
Don't we set a CR for wxMAC in a few places? (E.g. the default when the user first uses C::B on a mac?)I think whatever behavior we have should be consistent in all places.
Logged

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #26 on: August 29, 2012, 03:37:57 am »
Quote from: dmoore on August 29, 2012, 03:25:56 am
Don't we set a CR for wxMAC in a few places? (E.g. the default when the user first uses C::B on a mac?)I think whatever behavior we have should be consistent in all places.
This is the old behavior in cbEditor.cpp
Code
control->SetEOLMode(                  mgr->ReadInt(_T("/eol/eolmode"),                   platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF)); // Windows takes CR+LF, other platforms LF only
If the user does not select an eol mode, or read eol from configure manager failed, we use either CRLF(windows) or LF(non-windows). So, I think my new patch is just consistent with the old behavior.

EDIT:
I just search the C::B source, and it looks like the CR is the old mac EOL style, see:
plugins\astyle\astyle\astyle_main.h
Code
int eolWindows;        // number of Windows line endings, CRLF
int eolLinux;          // number of Linux line endings, LF
int eolMacOld;         // number of old Mac line endings. CR
eolMacOld

« Last Edit: August 29, 2012, 03:40:39 am by ollydbg »
Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.

Offline ollydbg

  • Developer
  • Lives here!
  • *****
  • Posts: 5916
  • OpenCV and Robotics
    • Chinese OpenCV forum moderator
Re: EOL issue, the EOL should be automatically detected, and make consistent
« Reply #27 on: September 03, 2012, 04:37:34 am »
Committed in trunk rev 8332, thanks for everyone's suggestion.
Logged
If some piece of memory should be reused, turn them to variables (or const variables).
If some piece of operations should be reused, turn them to functions.
If they happened together, then turn them to classes.
Pages: 1 2 [All]   Go Up
« previous next »