Code::Blocks Forums

User forums => Help => Topic started by: ollydbg on August 24, 2012, 08:07:23 am

Title: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on August 24, 2012, 08:07:23 am
I just want to have a feature request:
When I open a LF file, when I hit enters, it should add many lines of LF.
When I open a CRLF file, when I hit enters, it should add many lines of CRLF.
But currently, I see that I can't do that, the EOL is fixed, and either I result a mixed EOF file, or I need to set the EOL in the editor when I open each projects.

This is really annoying.

I have see such report 6 years ago, see: Feature Request: Auto EOL Mode (http://forums.codeblocks.org/index.php/topic,2130.msg16760.html#msg16760), but I don't think it was implemented.

BTW: I see that Notepad++ have such feature. Will C::B finally implement it?

EDIT: I also find on in the feature request:
Leave EOL format be (https://developer.berlios.de/feature/index.php?func=detailfeature&feature_id=3969&group_id=5358)

EDIT: I just tested the SciTE 3.2.1, and it works as what I want "leave as it is".
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on August 24, 2012, 11:05:29 am
I dig the code a little, and see that the EOL is set in the editor before the file is loaded.

sdk\cbeditor.cpp

Code
// static
void cbEditor::InternalSetEditorStyleBeforeFileOpen(cbStyledTextCtrl* control)
{
    if (!control)
        return;

    ......
    ......


    // NOTE: duplicate line in editorconfigurationdlg.cpp (ctor)
    control->SetEOLMode(                  mgr->ReadInt(_T("/eol/eolmode"),                   platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF)); // Windows takes CR+LF, other platforms LF only
    control->SetScrollWidthTracking(      mgr->ReadBool(_T("/margin/scroll_width_tracking"), false));
    control->SetMultipleSelection(        mgr->ReadBool(_T("/selection/multi_select"),       false));
    control->SetAdditionalSelectionTyping(mgr->ReadBool(_T("/selection/multi_typing"),       false));
    if (mgr->ReadBool(_T("/selection/use_vspace"), false))
        control->SetVirtualSpaceOptions(wxSCI_SCVS_RECTANGULARSELECTION | wxSCI_SCVS_USERACCESSIBLE);
    else
        control->SetVirtualSpaceOptions(wxSCI_SCVS_NONE);
}
Here, it just read the "eol/eolmode" from the configure file.

So, my question is: If we have add an option like "leave the eol as it is", then we need to Set the EOL after the file is loaded, right?

If I remove the line "control->SetEOLMode....", I think the control should have a eol which is consistent to the opened file?
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on August 24, 2012, 11:54:02 am
Hi, guys, I dig it further, and I found that the SciTE use this code:

In its source: C:\scite321\scite\src\SciTEIO.cxx

Code
void SciTEBase::DiscoverEOLSetting() {
SetEol();
if (props.GetInt("eol.auto")) {
int linesCR;
int linesLF;
int linesCRLF;
CountLineEnds(linesCR, linesLF, linesCRLF);
if (((linesLF >= linesCR) && (linesLF > linesCRLF)) || ((linesLF > linesCR) && (linesLF >= linesCRLF)))
wEditor.Call(SCI_SETEOLMODE, SC_EOL_LF);
else if (((linesCR >= linesLF) && (linesCR > linesCRLF)) || ((linesCR > linesLF) && (linesCR >= linesCRLF)))
wEditor.Call(SCI_SETEOLMODE, SC_EOL_CR);
else if (((linesCRLF >= linesLF) && (linesCRLF > linesCR)) || ((linesCRLF > linesLF) && (linesCRLF >= linesCR)))
wEditor.Call(SCI_SETEOLMODE, SC_EOL_CRLF);
}
}

So, it change the EOL dynamically.

BTW: currently, by default, under Windows, it use CRLF, see:
[scintilla] default eol mode on OS X - Google Groups (https://groups.google.com/forum/#!searchin/scintilla-interest/eol/scintilla-interest/7oKCtl_-I_0/rBtb1dUY6bIJ)
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: thomas on August 24, 2012, 03:29:45 pm
I'm confused (though I think you want the correct thing, but brain is not working well during the last weeks, so I'm not sure).

To clarify, with "add many LF" do you mean "change editor's EOL mode to file's EOL mode" or do you mean "change all line endings in the file"? You basically want anything typed into a file to be whatever the file was before, correct?

Also, what if the file, like many files from OSS projects is mixed already?
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on August 24, 2012, 04:13:31 pm
I'm confused (though I think you want the correct thing, but brain is not working well during the last weeks, so I'm not sure).
Hi, thomas, sorry about the confusion, I'm not a English native speaker, so my English may be poor.

Quote
To clarify, with "add many LF" do you mean "change editor's EOL mode to file's EOL mode" or do you mean "change all line endings in the file"?
I say "add many LF" which means when I hit the enter key many times, the editor add many lines which have LF styled EOL.

Here is an example, when I open a file, which has LF styled EOL, by default, if I have the configuration with "CR LF" in the editor setting, the new added line in the editor will have "CR LF" ending, so, the file becomes "LF" and "CR LF" mixed.

See the image below: (The new added empty line have CRLF styled EOL, but the original file have all LF styled EOL)
(http://i683.photobucket.com/albums/vv194/ollydbg_cb/2012-08-24220636.png)

On the other side, if you set the editor as "LF" format, then you will have the same issue that an original CRLF styled file will contains many LF styled lines.
Quote
You basically want anything typed into a file to be whatever the file was before, correct?
Yes, What I want is to let the editor automatically detect which EOL is used in the original file, then the new added line should still use same type of EOL.

Currently, both SciTE and NotePad++ has this feature, so I believe C::B should have this feature too.
Quote
Also, what if the file, like many files from OSS projects is mixed already?
For how to detect a files EOL, I think it is simple, just scan the first lines of the file, and set the EOL of the whole file as it is, I think SciTe use this kind of detection.

OK about my brain?  :)

EDIT:
It looks like to determine which EOL is used for a mixed EOL files, SciTe use a voting
Code
void SciTEBase::CountLineEnds(int &linesCR, int &linesLF, int &linesCRLF) {
linesCR = 0;
linesLF = 0;
linesCRLF = 0;
int lengthDoc = LengthDocument();
char chPrev = ' ';
TextReader acc(wEditor);
char chNext = acc.SafeGetCharAt(0);
for (int i = 0; i < lengthDoc; i++) {
char ch = chNext;
chNext = acc.SafeGetCharAt(i + 1);
if (ch == '\r') {
if (chNext == '\n')
linesCRLF++;
else
linesCR++;
} else if (ch == '\n') {
if (chPrev != '\r') {
linesLF++;
}
} else if (i > 1000000) {
return;
}
chPrev = ch;
}
}


Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: dmoore on August 24, 2012, 04:42:34 pm
Looks like you are on the right track. Why don't you make a patch? For mixed files, voting makes sense.

Do we have room on the status bar for the EOL mode? (Or is it there already) Some sort of warning, especially for mixed sources would be nice (though not everyone likes warnings and they shouldn't be intrusive).
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: thomas on August 24, 2012, 05:25:01 pm
OK about my brain?  :)
Yours is hopefully ok, it's mine that isn't good.

The approach sounds very good, though I would like to also reserve a way to always have a particular encoding and line ending (for example I'm using UTF-8 and LF everywhere, for everything, even under Windows -- don't want to have the editor interfere with this in any way, even hypothetically).

Maybe these options would be a good solution:

To detect the encoding, I'd just scan over all lines (does not really take that much longer!) and take whichever encoding occurs more often. On a tie, fall back to system default.
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on August 24, 2012, 05:41:10 pm
Thank you thomas and dmoore for the constructive suggestion.

OK, I will take the job to implement this, but before that, I need to firstly comprehend what the current way do. (Now, I even do not understand what does "Ensure consistent EOLs" used for, so it will take some time to read the source)
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: dmoore on August 24, 2012, 07:01:59 pm
(Now, I even do not understand what does "Ensure consistent EOLs" used for, so it will take some time to read the source)

Converts all EOLs in the doc to whatever is the currently set EOL mode of that editor when you save.

Note that I put in some features related to EOLs in the editortweaks plugin so users could temporarly change from the default manually, per editor. Would be nice not to break that, but I suspect what you are doing will be fine. (If installed, those options are in Edit->Editor tweaks)
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on August 25, 2012, 05:24:48 pm
Here is the draft patch, I add an item "AUTO" in the editor setting, this has the value "3".
The changed file are: src\sdk\cbeditor.cpp and src\sdk\resources\editor_configuration.xrc
Note, I have problems to generate the patch file, because wxsmith change the resource file a lot(strange, it convert the spaces to tabs, I manually change them back to spaces), so I add the xrc file. Both of the files were generated by tortoiseGit, which have LF format.
I added a static function, which I think is not a good idea, right?
Comments are welcome. Thanks.

EDIT: I remove the old attachment, and the patch against the latest svn revision is below:
Code
Index: cbeditor.cpp
===================================================================
--- cbeditor.cpp (revision 8249)
+++ cbeditor.cpp (working copy)
@@ -731,6 +731,39 @@
 
 END_EVENT_TABLE()
 
+
+// Count the EOL in the opened file
+static void CountLineEnds(cbStyledTextCtrl* control, int &linesCR, int &linesLF, int &linesCRLF)
+{
+    linesCR = 0;
+    linesLF = 0;
+    linesCRLF = 0;
+
+    //int GetCharAt(int pos)
+    //int GetLength() const;
+    int lengthDoc = control->GetLength();
+    char chPrev = ' ';
+
+    char chNext = control->GetCharAt(0);
+    for (int i = 0; i < lengthDoc; i++) {
+        char ch = chNext;
+        chNext = control->GetCharAt(i + 1);
+        if (ch == '\r') {
+            if (chNext == '\n')
+                linesCRLF++;
+            else
+                linesCR++;
+        } else if (ch == '\n') {
+            if (chPrev != '\r') {
+                linesLF++;
+            }
+        } else if (i > 1000000) {
+            return;
+        }
+        chPrev = ch;
+    }
+}
+
 // class constructor
 cbEditor::cbEditor(wxWindow* parent, const wxString& filename, EditorColourSet* theme)
     : EditorBase(parent, filename),
@@ -1557,7 +1590,6 @@
         control->SetMarginWidth(C_CHANGEBAR_MARGIN, 0);
 
     // NOTE: duplicate line in editorconfigurationdlg.cpp (ctor)
-    control->SetEOLMode(                  mgr->ReadInt(_T("/eol/eolmode"),                   platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF)); // Windows takes CR+LF, other platforms LF only
     control->SetScrollWidthTracking(      mgr->ReadBool(_T("/margin/scroll_width_tracking"), false));
     control->SetMultipleSelection(        mgr->ReadBool(_T("/selection/multi_select"),       false));
     control->SetAdditionalSelectionTyping(mgr->ReadBool(_T("/selection/multi_typing"),       false));
@@ -1575,6 +1607,23 @@
 
     ConfigManager* mgr = Manager::Get()->GetConfigManager(_T("editor"));
 
+    // set the EOL, fall back value: Windows takes CR+LF, other platforms LF only
+    int eolMode = mgr->ReadInt(_T("/eol/eolmode"), platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF);
+    if (eolMode == 3) //auto detect the eol
+    {
+        int linesCR;
+        int linesLF;
+        int linesCRLF;
+        CountLineEnds(control, linesCR, linesLF, linesCRLF);
+        if (((linesLF >= linesCR) && (linesLF > linesCRLF)) || ((linesLF > linesCR) && (linesLF >= linesCRLF)))
+            eolMode = wxSCI_EOL_LF;
+        else if (((linesCR >= linesLF) && (linesCR > linesCRLF)) || ((linesCR > linesLF) && (linesCR >= linesCRLF)))
+            eolMode = wxSCI_EOL_CR;
+        else if (((linesCRLF >= linesLF) && (linesCRLF > linesCR)) || ((linesCRLF > linesLF) && (linesCRLF >= linesCR)))
+            eolMode = wxSCI_EOL_CRLF;
+    }
+    control->SetEOLMode(eolMode);
+
     // Interpret #if/#else/#endif to grey out code that is not active
     control->SetProperty(_T("lexer.cpp.track.preprocessor"), mgr->ReadBool(_T("/track_preprocessor"), true) ? _T("1") : _T("0"));
 
Index: resources/editor_configuration.xrc
===================================================================
--- resources/editor_configuration.xrc (revision 8249)
+++ resources/editor_configuration.xrc (working copy)
@@ -200,6 +200,7 @@
                                                 <item>CR LF</item>
                                                 <item>CR</item>
                                                 <item>LF</item>
+                                                <item>AUTO</item>
                                               </content>
                                               <style>wxCB_READONLY</style>
                                             </object>


Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on August 27, 2012, 11:38:32 am
Damn, I found is that my local git clone was merged around 2012-08-10 (it is the code base my git patch was against on), and after that day( in 2012-08-14) there is a new svn commit 8230 in the xrc file, see:
WebSVN - codeblocks - Path Comparison - /trunk/src/sdk/resources/editor_configuration.xrc Rev 8142 and /trunk/src/sdk/resources/editor_configuration.xrc Rev 8230 (http://svn.berlios.de/wsvn/codeblocks?op=comp&compare%5B%5D=%2Ftrunk%2Fsrc%2Fsdk%2Fresources%2Feditor_configuration.xrc%408230&compare%5B%5D=%2Ftrunk%2Fsrc%2Fsdk%2Fresources%2Feditor_configuration.xrc%408142)

So, I think lots of the "extra change"(I believed the bug of wxsmith) in my attachment file in the previous post did the same thing as the commit 8230.

Sorry about the noise. :-[
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ironhead on August 27, 2012, 08:20:43 pm
I've manually applied the patch against r.8250 (some of the offsets were wrong) and it's working really well!  Thank you for implementing this feature, I hope to see it applied to the official repository. ;)
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: dmoore on August 28, 2012, 05:03:06 am
I don't have time to test right now, but the code looks good. I have one small suggestion to use the info window to alert the user to the outcome of the voting logic if there are mixed line endings. In your logic that sets the line ending, you could modify it as follows:

Code
        if (((linesLF >= linesCR) && (linesLF > linesCRLF)) || ((linesLF > linesCR) && (linesLF >= linesCRLF)))
        {
            eolMode = wxSCI_EOL_LF;
            if((linesCR>0) || (linesCRLF>0))
            {
                wxBell();
                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"LF\""), 1000);
            }
        }
        else if (((linesCR >= linesLF) && (linesCR > linesCRLF)) || ((linesCR > linesLF) && (linesCR >= linesCRLF)))
        {
            eolMode = wxSCI_EOL_CR;
            if((linesLF>0) || (linesCRLF>0))
            {
                wxBell();
                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"CR\""), 1000);
            }
        }
        else if (((linesCRLF >= linesLF) && (linesCRLF > linesCR)) || ((linesCRLF > linesLF) && (linesCRLF >= linesCR)))
        {
            eolMode = wxSCI_EOL_CRLF;
            if((linesCR>0) || (linesLF>0))
            {
                wxBell();
                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"CR-LF\""), 1000);
            }
        }
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on August 28, 2012, 07:59:13 am
Thanks ironhead for the test.
Thanks dmoore for the suggestion, below is the refined patch. (combined your suggestion, code style and comments refined)
Code
Index: cbeditor.cpp
===================================================================
--- cbeditor.cpp (revision 8249)
+++ cbeditor.cpp (working copy)
@@ -731,6 +731,42 @@
 
 END_EVENT_TABLE()
 
+// Count lines of EOL style in the opened file
+static void CountLineEnds(cbStyledTextCtrl* control, int &linesCR, int &linesLF, int &linesCRLF)
+{
+    linesCR = 0;
+    linesLF = 0;
+    linesCRLF = 0;
+
+    int lengthDoc = control->GetLength();
+    const int maxLineNumber = 1000000;
+    char chPrev = ' ';
+    char chNext = control->GetCharAt(0);
+    for (int i = 0; i < lengthDoc; i++)
+    {
+        char ch = chNext;
+        chNext = control->GetCharAt(i + 1);
+        if (ch == '\r')
+        {
+            if (chNext == '\n')
+                linesCRLF++;
+            else
+                linesCR++;
+        }
+        else if (ch == '\n')
+        {
+            if (chPrev != '\r')
+            {
+                linesLF++;
+            }
+        }
+        else if (i > maxLineNumber)     // stop the loop if the file contains too many lines
+            return;
+
+        chPrev = ch;
+    }
+}
+
 // class constructor
 cbEditor::cbEditor(wxWindow* parent, const wxString& filename, EditorColourSet* theme)
     : EditorBase(parent, filename),
@@ -1557,7 +1593,6 @@
         control->SetMarginWidth(C_CHANGEBAR_MARGIN, 0);
 
     // NOTE: duplicate line in editorconfigurationdlg.cpp (ctor)
-    control->SetEOLMode(                  mgr->ReadInt(_T("/eol/eolmode"),                   platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF)); // Windows takes CR+LF, other platforms LF only
     control->SetScrollWidthTracking(      mgr->ReadBool(_T("/margin/scroll_width_tracking"), false));
     control->SetMultipleSelection(        mgr->ReadBool(_T("/selection/multi_select"),       false));
     control->SetAdditionalSelectionTyping(mgr->ReadBool(_T("/selection/multi_typing"),       false));
@@ -1575,6 +1610,53 @@
 
     ConfigManager* mgr = Manager::Get()->GetConfigManager(_T("editor"));
 
+    // set the EOL, fall back value: Windows takes CR+LF, other platforms LF only
+    int eolMode = mgr->ReadInt(_T("/eol/eolmode"), platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF);
+
+    // The code snippet of auto detect EOL is copied from SciTE's source, CountLineEnds() function
+    // scans the current source file, and counts lines of each EOL style, then finally sets the EOL
+    // by voting logic. In the case of mixed EOL files, we give the user a beep and InfoWindow notification.
+    if (eolMode == 3) //auto detect the EOL
+    {
+        int linesCR;
+        int linesLF;
+        int linesCRLF;
+
+        // count lines of each EOL style
+        CountLineEnds(control, linesCR, linesLF, linesCRLF);
+
+        //voting logic
+        unsigned int delay = 2000;
+        if (((linesLF >= linesCR) && (linesLF > linesCRLF)) || ((linesLF > linesCR) && (linesLF >= linesCRLF)))
+        {
+            eolMode = wxSCI_EOL_LF;
+            if((linesCR>0) || (linesCRLF>0))
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"LF\""), delay);
+            }
+        }
+        else if (((linesCR >= linesLF) && (linesCR > linesCRLF)) || ((linesCR > linesLF) && (linesCR >= linesCRLF)))
+        {
+            eolMode = wxSCI_EOL_CR;
+            if((linesLF>0) || (linesCRLF>0))
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"CR\""), delay);
+            }
+        }
+        else if (((linesCRLF >= linesLF) && (linesCRLF > linesCR)) || ((linesCRLF > linesLF) && (linesCRLF >= linesCR)))
+        {
+            eolMode = wxSCI_EOL_CRLF;
+            if((linesCR>0) || (linesLF>0))
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"CR-LF\""), delay);
+            }
+        }
+    }
+    control->SetEOLMode(eolMode);
+
     // Interpret #if/#else/#endif to grey out code that is not active
     control->SetProperty(_T("lexer.cpp.track.preprocessor"), mgr->ReadBool(_T("/track_preprocessor"), true) ? _T("1") : _T("0"));
 
Index: resources/editor_configuration.xrc
===================================================================
--- resources/editor_configuration.xrc (revision 8249)
+++ resources/editor_configuration.xrc (working copy)
@@ -200,6 +200,7 @@
                                                 <item>CR LF</item>
                                                 <item>CR</item>
                                                 <item>LF</item>
+                                                <item>AUTO</item>
                                               </content>
                                               <style>wxCB_READONLY</style>
                                             </object>


I would like to show the filename of the control in the InfoWindow::Display(), I see cbEditor(editorbase) have a member function:
Code
virtual const wxString& GetShortName() const { return m_Shortname; }
What a pity is that
Code
// static
void cbEditor::InternalSetEditorStyleAfterFileOpen(cbStyledTextCtrl* control)
This is a static member function, so no access to the filename.  :(
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: MortenMacFly on August 28, 2012, 08:02:50 am
... you shouldn't not forget to do nothing (!) if the file dos not contain a line-feed, btw. For this to work, you can return true/false in the method you've introduced.
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on August 28, 2012, 09:47:14 am
... you shouldn't not forget to do nothing (!) if the file dos not contain a line-feed, btw. For this to work, you can return true/false in the method you've introduced.
Thanks, here is an improved patch, which handle them in the "else" clause.

Code
Index: cbeditor.cpp
===================================================================
--- cbeditor.cpp (revision 8249)
+++ cbeditor.cpp (working copy)
@@ -731,6 +731,42 @@
 
 END_EVENT_TABLE()
 
+// Count lines of EOL style in the opened file
+static void CountLineEnds(cbStyledTextCtrl* control, int &linesCR, int &linesLF, int &linesCRLF)
+{
+    linesCR = 0;
+    linesLF = 0;
+    linesCRLF = 0;
+
+    int lengthDoc = control->GetLength();
+    const int maxLengthDoc = 1000000;
+    char chPrev = ' ';
+    char chNext = control->GetCharAt(0);
+    for (int i = 0; i < lengthDoc; i++)
+    {
+        char ch = chNext;
+        chNext = control->GetCharAt(i + 1);
+        if (ch == '\r')
+        {
+            if (chNext == '\n')
+                linesCRLF++;
+            else
+                linesCR++;
+        }
+        else if (ch == '\n')
+        {
+            if (chPrev != '\r')
+            {
+                linesLF++;
+            }
+        }
+        else if (i > maxLengthDoc)     // stop the loop if the file contains too many characters
+            return;
+
+        chPrev = ch;
+    }
+}
+
 // class constructor
 cbEditor::cbEditor(wxWindow* parent, const wxString& filename, EditorColourSet* theme)
     : EditorBase(parent, filename),
@@ -1557,7 +1593,6 @@
         control->SetMarginWidth(C_CHANGEBAR_MARGIN, 0);
 
     // NOTE: duplicate line in editorconfigurationdlg.cpp (ctor)
-    control->SetEOLMode(                  mgr->ReadInt(_T("/eol/eolmode"),                   platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF)); // Windows takes CR+LF, other platforms LF only
     control->SetScrollWidthTracking(      mgr->ReadBool(_T("/margin/scroll_width_tracking"), false));
     control->SetMultipleSelection(        mgr->ReadBool(_T("/selection/multi_select"),       false));
     control->SetAdditionalSelectionTyping(mgr->ReadBool(_T("/selection/multi_typing"),       false));
@@ -1575,6 +1610,79 @@
 
     ConfigManager* mgr = Manager::Get()->GetConfigManager(_T("editor"));
 
+    // set the EOL, fall back value: Windows takes CR+LF, other platforms LF only
+    int eolMode = mgr->ReadInt(_T("/eol/eolmode"), platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF);
+
+    // The code snippet of auto detect EOL is copied from SciTE's source, CountLineEnds() function
+    // scans the current source file, and counts lines of each EOL style, then finally sets the EOL
+    // by voting logic. In the case of mixed EOL files, we give the user a beep and InfoWindow notification.
+    if (eolMode == 3) //auto detect the EOL
+    {
+        int linesCR;
+        int linesLF;
+        int linesCRLF;
+
+        // count lines of each EOL style
+        CountLineEnds(control, linesCR, linesLF, linesCRLF);
+
+        //voting logic
+        unsigned int delay = 2000;
+        if (((linesLF >= linesCR) && (linesLF > linesCRLF)) || ((linesLF > linesCR) && (linesLF >= linesCRLF)))
+        {
+            eolMode = wxSCI_EOL_LF;
+            if((linesCR>0) || (linesCRLF>0))
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"LF\""), delay);
+            }
+        }
+        else if (((linesCR >= linesLF) && (linesCR > linesCRLF)) || ((linesCR > linesLF) && (linesCR >= linesCRLF)))
+        {
+            eolMode = wxSCI_EOL_CR;
+            if((linesLF>0) || (linesCRLF>0))
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"CR\""), delay);
+            }
+        }
+        else if (((linesCRLF >= linesLF) && (linesCRLF > linesCR)) || ((linesCRLF > linesLF) && (linesCRLF >= linesCR)))
+        {
+            eolMode = wxSCI_EOL_CRLF;
+            if((linesCR>0) || (linesLF>0))
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode \"CR-LF\""), delay);
+            }
+        }
+        else // in the case of the file does not contain any line-feed, or the file have equally counts of EOL styles
+        {
+            //set to the fall back mode
+            wxString eolModeStr;
+            if (platform::windows)
+            {
+                eolMode =  wxSCI_EOL_CRLF;
+                eolModeStr = _T("\"CR-LF\"");
+            }
+            else
+            {
+                eolMode =  wxSCI_EOL_LF;
+                eolModeStr = _T("\"LF\"");
+            }
+
+            if((linesCR>0) || (linesLF>0) || (linesCRLF>0)) //equal counts
+            {
+                wxBell();
+                InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode ") + eolModeStr, delay);
+            }
+            else //no line-feed found
+            {
+                wxBell();
+                InfoWindow::Display(_("No Line Endings"), _("No Line Endings found, setting mode ") + eolModeStr, delay);
+            }
+        }
+    }
+    control->SetEOLMode(eolMode);
+
     // Interpret #if/#else/#endif to grey out code that is not active
     control->SetProperty(_T("lexer.cpp.track.preprocessor"), mgr->ReadBool(_T("/track_preprocessor"), true) ? _T("1") : _T("0"));
 
Index: resources/editor_configuration.xrc
===================================================================
--- resources/editor_configuration.xrc (revision 8249)
+++ resources/editor_configuration.xrc (working copy)
@@ -200,6 +200,7 @@
                                                 <item>CR LF</item>
                                                 <item>CR</item>
                                                 <item>LF</item>
+                                                <item>AUTO</item>
                                               </content>
                                               <style>wxCB_READONLY</style>
                                             </object>


I have tested several cases, the only minor issue can be: e.g. Under Windows, a file have: linesCR = 0; linesLF = 5; linesCRLF = 5; --------> this will set the eol mode to "LF"......Logicially, I think it should set the eol mode to "CRLF", because under Windows, the CRLF take precedence. Any good idea about the voting logic? I can't find a better, simple and clean way. ;)
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: MortenMacFly on August 28, 2012, 10:13:56 am
Logicially, I think it should set the eol mode to "CRLF", because under Windows, the CRLF take precedence. Any good idea about the voting logic? I can't find a better, simple and clean way. ;)
Well you can initialise eolMode according the platform and then only change it if you are sure you counted enough different EOL's (so not >=, but only >).
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: oBFusCATed on August 28, 2012, 10:14:30 am
Is it possible to put the new code in a separate function?
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: MortenMacFly on August 28, 2012, 10:19:31 am
Is it possible to put the new code in a separate function?
Maybe even rename CountLineEnds to DetectLineEnds (or alike) and have have it return what eolMode should be set.

Which makes me wonder: Could it be, that scintilla has such a function, too?
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on August 28, 2012, 10:29:53 am
Is it possible to put the new code in a separate function?
Maybe even rename CountLineEnds to DetectLineEnds (or alike) and have have it return what eolMode should be set.

Which makes me wonder: Could it be, that scintilla has such a function, too?
Scintilla does not have such feature, but SciTE does....
Scintilla should keep as simple as it can. (From my point of view), so it leave this function to client.

Is it possible to put the new code in a separate function?
OK, I will did this in the next patch.
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on August 28, 2012, 04:43:16 pm
OK, I will did this in the next patch.
Ok, see this patch. Any further comments?
Code
Index: cbeditor.cpp
===================================================================
--- cbeditor.cpp (revision 8249)
+++ cbeditor.cpp (working copy)
@@ -731,6 +731,95 @@
 
 END_EVENT_TABLE()
 
+// Count lines of EOL style in the opened file
+static void CountLineEnds(cbStyledTextCtrl* control, int &linesCR, int &linesLF, int &linesCRLF)
+{
+    linesCR = 0;
+    linesLF = 0;
+    linesCRLF = 0;
+
+    int lengthDoc = control->GetLength();
+    const int maxLengthDoc = 1000000;
+    char chPrev = ' ';
+    char chNext = control->GetCharAt(0);
+    for (int i = 0; i < lengthDoc; i++)
+    {
+        char ch = chNext;
+        chNext = control->GetCharAt(i + 1);
+        if (ch == '\r')
+        {
+            if (chNext == '\n')
+                linesCRLF++;
+            else
+                linesCR++;
+        }
+        else if (ch == '\n')
+        {
+            if (chPrev != '\r')
+            {
+                linesLF++;
+            }
+        }
+        else if (i > maxLengthDoc)     // stop the loop if the file contains too many characters
+            return;
+
+        chPrev = ch;
+    }
+}
+
+static int DetectLineEnds(cbStyledTextCtrl* control)
+{
+    int eolMode;
+    wxString eolModeStr;
+    // initial EOL mode depend on OS
+    if (platform::windows)
+    {
+        eolMode =  wxSCI_EOL_CRLF;
+        eolModeStr = _T("\"CR-LF\"");
+    }
+    else
+    {
+        eolMode =  wxSCI_EOL_LF;
+        eolModeStr = _T("\"LF\"");
+    }
+
+    int linesCR;
+    int linesLF;
+    int linesCRLF;
+    // count lines of each EOL style
+    CountLineEnds(control, linesCR, linesLF, linesCRLF);
+
+    // voting logic
+    // if the file does not contain any line-feed or the most largest counts are equal( e.g.: linesLF=5,
+    // linesCRLF=5, linesCR=0 ), then we will use the initial EOL mode
+    if ( (linesLF > linesCR) && (linesLF > linesCRLF) )
+    {
+        eolMode = wxSCI_EOL_LF;
+        eolModeStr = _T("\"LF\"");
+    }
+    else if ( (linesCR > linesLF) && (linesCR > linesCRLF) )
+    {
+        eolMode = wxSCI_EOL_CR;
+        eolModeStr = _T("\"CR\"");
+    }
+    else if ( (linesCRLF > linesLF) && (linesCRLF > linesCR))
+    {
+        eolMode = wxSCI_EOL_CRLF;
+        eolModeStr = _T("\"CR-LF\"");
+    }
+
+    unsigned int delay = 2000;
+    if (  ( (linesCR>0) && (linesCRLF>0) )
+       || ( (linesLF>0) && (linesCRLF>0) )
+       || ( (linesCR>0) && (linesLF>0) ) )
+    {
+        //In mixed EOL file, give the user a beep and InfoWindow notification.
+        wxBell();
+        InfoWindow::Display(_("Mixed Line Endings"), _("Mixed line endings found, setting mode ") + eolModeStr, delay);
+    }
+    return eolMode;
+}
+
 // class constructor
 cbEditor::cbEditor(wxWindow* parent, const wxString& filename, EditorColourSet* theme)
     : EditorBase(parent, filename),
@@ -1557,7 +1646,6 @@
         control->SetMarginWidth(C_CHANGEBAR_MARGIN, 0);
 
     // NOTE: duplicate line in editorconfigurationdlg.cpp (ctor)
-    control->SetEOLMode(                  mgr->ReadInt(_T("/eol/eolmode"),                   platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF)); // Windows takes CR+LF, other platforms LF only
     control->SetScrollWidthTracking(      mgr->ReadBool(_T("/margin/scroll_width_tracking"), false));
     control->SetMultipleSelection(        mgr->ReadBool(_T("/selection/multi_select"),       false));
     control->SetAdditionalSelectionTyping(mgr->ReadBool(_T("/selection/multi_typing"),       false));
@@ -1575,6 +1663,14 @@
 
     ConfigManager* mgr = Manager::Get()->GetConfigManager(_T("editor"));
 
+    // set the EOL, fall back value: Windows takes CR+LF, other platforms LF only
+    int eolMode = mgr->ReadInt(_T("/eol/eolmode"), platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF);
+
+    if (eolMode == 3) //auto detect the EOL
+        eolMode = DetectLineEnds(control);
+
+    control->SetEOLMode(eolMode);
+
     // Interpret #if/#else/#endif to grey out code that is not active
     control->SetProperty(_T("lexer.cpp.track.preprocessor"), mgr->ReadBool(_T("/track_preprocessor"), true) ? _T("1") : _T("0"));
 
Index: resources/editor_configuration.xrc
===================================================================
--- resources/editor_configuration.xrc (revision 8249)
+++ resources/editor_configuration.xrc (working copy)
@@ -200,6 +200,7 @@
                                                 <item>CR LF</item>
                                                 <item>CR</item>
                                                 <item>LF</item>
+                                                <item>AUTO</item>
                                               </content>
                                               <style>wxCB_READONLY</style>
                                             </object>

Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: Jenna on August 28, 2012, 05:17:13 pm
Code
+    // set the EOL, fall back value: Windows takes CR+LF, other platforms LF only

The default line-ending on Mac is CR as far as I know.

Sorry to make it a little more complicated.
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: MortenMacFly on August 28, 2012, 07:12:07 pm
The default line-ending on Mac is CR as far as I know.
Sorry to make it a little more complicated.
EOL's and he 21st century... still feeling like stone-age... ::)
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: oBFusCATed on August 28, 2012, 09:06:04 pm
I think CR or LF are supported on MacOSX, prior to 10.xx, the CR was the default and now with the FreeBSD heritage it is the LF.
But I think most softwares support both and users have files with both endings.
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on August 29, 2012, 02:49:35 am
I think CR or LF are supported on MacOSX, prior to 10.xx, the CR was the default and now with the FreeBSD heritage it is the LF.
But I think most softwares support both and users have files with both endings.


The default line-ending on Mac is CR as far as I know.

Adding the fall-back on Mac as CR is quite easy. But I'm mostly follow what Scintilla did and a discussion on their forum:
See:[scintilla] default eol mode on OS X - Google Groups (https://groups.google.com/forum/#!searchin/scintilla-interest/eol/scintilla-interest/7oKCtl_-I_0/rBtb1dUY6bIJ)

And in the source code: Document.cxx
Code
Document::Document() {
refCount = 0;
#ifdef _WIN32
eolMode = SC_EOL_CRLF;
#else
eolMode = SC_EOL_LF;
#endif
...
...
So, I believe Mac can support LF without any problem. :)

Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: dmoore on August 29, 2012, 03:25:56 am
Don't we set a CR for wxMAC in a few places? (E.g. the default when the user first uses C::B on a mac?)I think whatever behavior we have should be consistent in all places.
Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on August 29, 2012, 03:37:57 am
Don't we set a CR for wxMAC in a few places? (E.g. the default when the user first uses C::B on a mac?)I think whatever behavior we have should be consistent in all places.
This is the old behavior in cbEditor.cpp
Code
control->SetEOLMode(                  mgr->ReadInt(_T("/eol/eolmode"),                   platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF)); // Windows takes CR+LF, other platforms LF only
If the user does not select an eol mode, or read eol from configure manager failed, we use either CRLF(windows) or LF(non-windows). So, I think my new patch is just consistent with the old behavior.

EDIT:
I just search the C::B source, and it looks like the CR is the old mac EOL style, see:
plugins\astyle\astyle\astyle_main.h
Code
int eolWindows;        // number of Windows line endings, CRLF
int eolLinux;          // number of Linux line endings, LF
int eolMacOld;         // number of old Mac line endings. CR
eolMacOld

Title: Re: EOL issue, the EOL should be automatically detected, and make consistent
Post by: ollydbg on September 03, 2012, 04:37:34 am
Committed in trunk rev 8332, thanks for everyone's suggestion.