Author Topic: Multiline Search & Replace (Read 36628 times)

rickg22 · « **on:** November 08, 2009, 03:03:32 am »

Hi guys! I'm modifying my fresh SVN copy of Code::Blocks to support Multi-line search and replace (I'm needing it to refactor a 400-files ASP project, and we need to save as much time as possible).

Currently I've only been able to modify the replace dialog (the search dialog is not as critical), and only the XRC (but I'm about to start with the C++ code). See the attachment for a screenshot.

What do you think?

[attachment deleted by admin]

rickg22 · « **Reply #1 on:** November 08, 2009, 05:58:01 am »

I've managed to make the changes in replacedlg.cpp.

In some files, replacing works, but in others, it doesn't. I've realized it's a line-endings issue. The wxTextCtrl interprets a line feed as LF, while the files can vary their endings in LF, CR or CRLF.

To solve this, I need to make the following steps:

If the search string does NOT contains a CR or LF, search normally.

If it does,
1) Get the EOL mode from C::B.
2) Convert the "find" string to this EOL mode.
3) Convert the file to be searched (the in-memory buffer, i mean) to this EOL mode so line endings are consistent. We're going to replace in files, anyway.
4) Convert the "replace" string to this EOL mode.
5) Find and replace the string as we'd do with any other strings.

This should be done in EditorManager::Replace(cbStyledTextCtrl* control, cbFindReplaceData* data).

I'll keep posting my progress.

BTW, regex search hasn't been tested yet. Wish me luck!

blueshake · « **Reply #2 on:** November 08, 2009, 07:42:14 am »

Seems cool,waiting for your good news.

rickg22 · « **Reply #3 on:** November 08, 2009, 07:45:30 am »

Progress update.

Sun, Nov 8, 2009.

The Multi-line Search and replace algorithm has been finished and tested successfully, in an unsaved file, with default EOL mode set to CRLF.

The EOL conversion of the search and replace expressions is done inside replacedlg.cpp. Inside EditorManager::Replace and EditorManager::ReplaceInFiles we check the search expression for LFs and CRs. If there's a match, the whole file is converted to the current EOL mode to avoid false positives/negatives. Then the search takes place normally.

Search & Replace in a single file works, for both normal and regex searches.

Extended Regular Expressions is enabled.

Before:

Code

<html>
<body>
hola mundo!
</body>
</html>

Search Expression:

Code

<html>
<body>
(.*)
</body>
</html>

Replace expression:

Code

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Hello world!</title>
</head>
<body>
\1
</body>
</html>

After:

Code

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Hello world!</title>
</head>
<body>
hola mundo!
</body>
</html>

There are still some quirks in the UI that need to be worked on.

Multi-line S&R in files has not been tested yet.

Will continue to report tomorrow.
Good night!

mariocup · « **Reply #4 on:** November 08, 2009, 10:49:17 am »

seems very very cool.

Jenna · « **Reply #5 on:** November 08, 2009, 11:06:09 am »

Quote from: mariocup on November 08, 2009, 10:49:17 am

seems very very cool.

Absolutely correct.

Does it also work with mixed line-edings ?

rickg22 · « **Reply #6 on:** November 08, 2009, 10:58:12 pm »

Quote from: jens on November 08, 2009, 11:06:09 am

Quote from: mariocup on November 08, 2009, 10:49:17 am
seems very very cool.

Absolutely correct.

Does it also work with mixed line-edings ?

It didn't work with mixed line-endings at first. I needed to convert the line endings of both the search/replace expressions AND of files. What I do is checking if the search/replace expressions contain a LF or CR. In that case, we convert the current file to the configured EOL mode so line endings aren't mixed anymore. We're going to search/replace, so it's not a problem if the file is modified. Of course, this is done INSIDE the undo action, so that if the match isn't found, the file is undo'ed. The undoing is already done with existing code inside EditorManager, i only had to move the begin-undo-action before the line conversion.

Here's the patch for EditorManager:

Code

int EditorManager::Replace(cbStyledTextCtrl* control, cbFindReplaceData* data)
{
    if (!control || !data)
        return -1;

    if (control->GetReadOnly())
    {
        cbMessageBox(_("This file is read-only.\nReplacing in a read-only file is not possible."),
                     _("Warning"), wxICON_EXCLAMATION);
        return -1;
    }

    bool AdvRegex=false;
    int replacecount=0;
    int foundcount=0;
    int flags = 0;

    control->BeginUndoAction(); // The undo is set at this point in case we need to convert the EOLs.

    if((data->findText.Find(_T("\n")) != wxNOT_FOUND) || (data->findText.Find(_T("\r")) != wxNOT_FOUND))
    {
        // If it's a multi-line S&R, We need to convert the EOLs, or the find may be inaccurate.
        // This will slow down processing as the whole file needs to be preprocessed, but it's a small price
        // compared to searching and replacing multiple lines by hand.
        static const int default_eol = platform::windows ? wxSCI_EOL_CRLF : wxSCI_EOL_LF;
        int eolMode = Manager::Get()->GetConfigManager(_T("editor"))->ReadInt(_T("/eol/eolmode"), default_eol);
        control->SetEOLMode(eolMode);
        control->ConvertEOLs(eolMode);
    }
    CalculateFindReplaceStartEnd(control, data);
    // ... the rest of the code is kept as it was.

For find in files, it's a similar code.

This check is done only for the find string, since having a multiline replace string isn't a problem. It's the find string that gives us all the problems.

Algorithm aside, my real problem right now is getting the ctrl-tab key to work. I don't want the tabs to change between the single-line and multiple-line tabs, but between the Replace and Replace in files. How do I disable a wxNotebook control from accepting ctrl-tab keys?

Just in case, I'm going to change the wxNotebook for a wxChoicebook. I hope ctrl-tab isn't affected this way. (And I think it looks prettier and less bulky).

rickg22 · « **Reply #7 on:** November 09, 2009, 01:50:39 am »

Ugh, this sucks!

wxNotebook interferes with my tabs and I find no way to make it ignore keys.
wxChoicebook completely deletes the tabs.

So I've decided to use a pair of radio buttons and emulate a wxNotebook by hiding and showing panels. This way the ctrl-tab isn't messed with, and the radio buttons are what I wanted in the first place.
It may take me a couple of hours to make it work.

Wish me luck!

rickg22 · « **Reply #8 on:** November 09, 2009, 07:31:25 am »

:x ARGH! It took me four friggin' hours to get the dialog to work. It's incredible that the GUI was much more difficult to work on than the algorithm itself.

Okay, progress report:

The UI is done. Yay!

Replace in files hangs C::B. Not good. I guess I introduced a bug in there, or I didn't think well the replace in files part. It might take me another hour to get it to work.

Wish me luck!

rickg22 · « **Reply #9 on:** November 09, 2009, 07:56:06 am »

Well well well. What do you know? The original XRC file had the "Project files" and "Open files" scope swapped. No wonder it was hanging. It was searching on ALL the files. I'll keep debugging.

rickg22 · « **Reply #10 on:** November 09, 2009, 08:40:05 am »

It seems I've found another "feature". The confirmation dialog is only shown when you select "all". I wonder if this is wise, due to the fact that finding the *FIRST* match might take a while when doing multiline S&R.

But taking that aside, I've managed to complete the algorithm succesfully.

There's a minor problem: Converting the line endings of ALL files to the search expression is completely overkill. It's much better to just convert the search and replace expressions to match the files' EOL mode.

This is easy. Unfortunately, files with mixed line endings might get some matches skipped. I'm afraid there's no easy way around this.

rickg22 · « **Reply #11 on:** November 09, 2009, 08:54:28 am »

FINISHED!

I've just mailed Jens the patch file for the multiline S&R.

Note that the patch only affects Find-and-Replace (CTRL-R), not Find (CTRL-F).

MortenMacFly · « **Reply #12 on:** November 09, 2009, 09:43:52 am »

Quote from: rickg22 on November 09, 2009, 07:56:06 am

The original XRC file had the "Project files" and "Open files" scope swapped.

Notice that this was inconsistent. That was modified in a patch lately. So hopefully you left the XRC file untouched and corrected the code (although I was quite sure that I did it already in SVN...?!).

rickg22 · « **Reply #13 on:** November 09, 2009, 02:20:26 pm »

Quote from: MortenMacFly on November 09, 2009, 09:43:52 am

Quote from: rickg22 on November 09, 2009, 07:56:06 am
The original XRC file had the "Project files" and "Open files" scope swapped.
Notice that this was inconsistent. That was modified in a patch lately. So hopefully you left the XRC file untouched and corrected the code (although I was quite sure that I did it already in SVN...?!).

At first I thought that I had swapped those lines accidentally while editing the XRC, so I diffed against HEAD. The version in the repository was unfixed. And to fix it... um... no, it was the XRC that I modified. I added like a ton of lines, why not just swap two lines near the end?

Sorry.

Okay, I modified my local copy to swap again the XRC, and now I fixed the editormanager code to reflect the changes.
Hey, guess what. I still have the SVN authentication. Should I commit to head now?

MortenMacFly · « **Reply #14 on:** November 10, 2009, 06:28:23 am »

Quote from: rickg22 on November 09, 2009, 02:20:26 pm

Should I commit to head now?

Not head, but the scintilla branch would be fine. :-)
(Unless Jens has serious doubts.)

Jenna · « **Reply #15 on:** November 10, 2009, 07:14:48 am »

Sorry for the late reply.

Yes some problems:

if I do a "normal" search and replace it works, but if I try to use a regex (replace a part of the search string with \(.*\) )it does not find any matches.

More weired: if I change the line-endings to CR in one of my files and run s&r then, it also does not find anything, but changes all line-endings, that might be okay, because of the conversion to unique line-endings, but if I now open any of my project-files, all the line-endings are changed on first view of the file (if the editor gets active), I did not change the global line-ending settings, only the settings of the file in the edit-menu.

EDIT:

The last thing also happens in clean C::B, but only seems to happen, if the file I come from (last active editor) has CR-line-endings.
I think I should look into this issue at another place.

@Rick:
is it okay for you to push the patch on my server, so others can test it too ?

rickg22 · « **Reply #16 on:** November 10, 2009, 02:47:15 pm »

Okay, here are my tests so far.

Normal, single-line S&R: Works.
Regex, single-line S&R: Works.
Normal, multi-line S&R: Works.
Regex, multi-line S&R: Works.

Did you check your Editor settings? I've tried with both "Extended Regex search" checked and unchecked, and handling files with both LF and CRLF endings. This on both Linux and Windows. (I think CR-only line endings are quite obsolete now. I don't think C::B has been fully tested in this case, mac OS now uses LF ending as default since the Freebsd adoption).

I've also noticed some issues, both in C::B and in my patch.

1) (C::B) In single-line S&R, (.*) INCLUDES the EOL chars. I need to use ([^\r\n])* to do a good find&replace. Perhaps we should add another option to EXCLUDE the CR and LF from regex search. Just a thought.

2) (My patch) The S&R in files uses a different approach than the single file S&R to fix line endings. The single-file changes the file's EOL according to the global editor settings (or at least to the value of Manager::Get()->GetConfigManager(_T("editor"))->ReadInt(_T("/eol/eolmode"). The multiple file S&R changes only the search and replace expression's EOL mode. I think that in single-file, we should convert the file's line endings not to the global line endings, but to the file's line endings (just to change as few lines as possible before searching) and still achieve an effective S&R.

3) (My patch) This problem happens in the S&R dialog. First, if you enter a very large regex expression in the combos, the dialog becomes HUGE because it adjusts the width AFTER loading the data, not BEFORE. I need to fix that.

4) (My patch) Another problem I've noticed is that tab navigation doesn't work correctly on Windows. It jumps from "search from:" to the options, and i have to press shift-tab to jump to "replace to:". This doesn't happen under Linux. WTH??

5) I'm also considering making a change in cbFindReplaceData and give it methods for EOL conversion. I plan to add it an EOLMode variable and methods for conversion, like SetEOLMode(int mode). This will clean up EOL conversion in replacedlg.cpp and will prepare for a multiline Find-only dialog in the future. I'll work on it tonight.

So, how do I commit to Jens' server? I want to share my patch and allow others to help me fixing the UI problems. Jens, I might need you to give me an authentication so we can all fix this patch's issues.

Jens, please pm me with the data on your repo. Meanwhile, feel free to commit my patch on your server, but I might change some code again.

Thanks!

rickg22 · « **Reply #17 on:** November 11, 2009, 03:28:34 pm »

I did some modifications to EditorManager to clean up my changes:

1) Files are checked for consistent EOLs, but if no replacements were
made (and if the file was not open), files are closed correctly.
2) I'm using the file's EOL mode to search, and not the global EOL mode.
3) The EOL conversion is not shown in the change history (the colored
bar at the left of the file).
4) For a find-only search, I convert the cbReplaceData's strings to
the current file's EOL mode.

I submitted the patch to Jens. Since this patch seems to be permanent, I hope it will be submitted to trunk.

I'll still be working on the replace dialog to make it more user friendly, and hopefully, fix the tabbing quirks on win32.

MortenMacFly · « **Reply #18 on:** November 11, 2009, 03:38:49 pm »

Quote from: rickg22 on November 11, 2009, 03:28:34 pm

1) Files are checked for consistent EOLs, but if no replacements were
made (and if the file was not open), files are closed correctly.

Are you aware that we have an option concerning the handling EOL's? Does that still work after that modification? I would not recommend to change it "silently". Because you cause a non-functional difference in the files on every platform in the worst case scenario.

rickg22 · « **Reply #19 on:** November 12, 2009, 05:30:13 am »

Quote from: MortenMacFly on November 11, 2009, 03:38:49 pm

Quote from: rickg22 on November 11, 2009, 03:28:34 pm
1) Files are checked for consistent EOLs, but if no replacements were
made (and if the file was not open), files are closed correctly.
Are you aware that we have an option concerning the handling EOL's? Does that still work after that modification? I would not recommend to change it "silently". Because you cause a non-functional difference in the files on every platform in the worst case scenario.

Yes, this is only done for multi-line S&R. If the search string has no eol's (current behavior), no conversion is done. Also, he files' EOLs are not changed to the current user's but the file's (just the same as "ensure consistent EOL's menu option).

But you're absolutely right. The user must have complete control of the situation. I'm going to add an option to let the user choose whether to preprocess EOL's before searching.

MortenMacFly · « **Reply #20 on:** November 12, 2009, 06:23:01 am »

Quote from: rickg22 on November 11, 2009, 03:28:34 pm

I submitted the patch to Jens. Since this patch seems to be permanent, I hope it will be submitted to trunk.

Reminds me: Why not to the patch tracker? If you do that we all can try from day one basically. Jens seems rather busy atm (so am I)...

rickg22 · « **Reply #21 on:** November 13, 2009, 04:47:26 am »

Ah, right. I need to relogin to berlios.

rickg22 · « **Reply #22 on:** November 13, 2009, 06:56:01 am »

Okay, I've submitted the patch to berlios. The patch can be found at http://developer.berlios.de/patch/download.php?id=2849

WARNING! The patch _IS_ buggy (at least on Windows: the UI is having tab problems). Windows users are welcome to test it and fix it. It's got to do with the pseudo-notebook panels (single-line / multiline). Also, the multiline search tab for only-files makes the input boxes too small because of the scrollbars, assigning them a bigger size will fix it.

I've added some internal features to editormanager, namely, the ability to search only the current file (useful for listing ALL matches in the file, in find-in-files), the ability to search for beginning-of-file matches, and the option to fix mixed EOLs while searching. None of these options are active unless the finddlg and replacedlg add them to the UI.

I'm planning to change the UI and make it wider so that the extra options can fit in the dialog, but I have to go to sleep now.

Good night!

rickg22 · « **Reply #23 on:** November 18, 2009, 03:54:21 pm »

I've submitted a revised patch to berlios. It looks prettier and not that bulky, but still needs a bit of win32 tweaking. Unfortunately, I've been unable to fix the weird tabbing bug on windows. Why does it only happen on windows?

Can the patch still be applied with this usability problem?

blueshake · « **Reply #24 on:** November 19, 2009, 02:53:18 am »

Quote from: rickg22 on November 18, 2009, 03:54:21 pm

I've submitted a revised patch to berlios. It looks prettier and not that bulky, but still needs a bit of win32 tweaking. Unfortunately, I've been unable to fix the weird tabbing bug on windows. Why does it only happen on windows?

Can the patch still be applied with this usability problem?

Is something relatived to this?http://forums.codeblocks.org/index.php/topic,11519.0.html

rickg22 · « **Reply #25 on:** November 19, 2009, 04:30:41 am »

Quote from: blueshake on November 19, 2009, 02:53:18 am

Quote from: rickg22 on November 18, 2009, 03:54:21 pm
I've submitted a revised patch to berlios. It looks prettier and not that bulky, but still needs a bit of win32 tweaking. Unfortunately, I've been unable to fix the weird tabbing bug on windows. Why does it only happen on windows?

Can the patch still be applied with this usability problem?

Is something relatived to this?http://forums.codeblocks.org/index.php/topic,11519.0.html

No, this is on a dialog using 100% native wxWidgets controls. Thanks for trying, tho

rickg22 · « **Reply #26 on:** November 19, 2009, 04:07:35 pm »

I've been having inconsistent behavior with my windows build of C::B, I think my windows development environment is somewhat faulty. Therefore I don't even know whether it's actually my code that's failing. I need a windows user to test my multiline S&R patch.

Thank you.

UPDATE: It's fixed!!!! The tabbing problem was finally fixed. It was a style badly set in the xrc. Tonight, maybe tomorrow, I'll fix some minor visual details and upload the final version of the patch.

rickg22 · « **Reply #27 on:** November 20, 2009, 07:13:48 am »

The final patch for the Multiline S&R has been uploaded!

I mailed Jens so he can commit it to trunk. Cross your fingers! ;-)

MortenMacFly · « **Reply #28 on:** November 20, 2009, 09:01:10 am »

Quote from: rickg22 on November 20, 2009, 07:13:48 am

The final patch for the Multiline S&R has been uploaded!

Good work, Rick! I'll apply this now in my local copy for testing. I am looking forward to it... :-)

ironhead · « **Reply #29 on:** December 01, 2009, 02:41:00 pm »

Quote from: MortenMacFly on November 20, 2009, 09:01:10 am

Quote from: rickg22 on November 20, 2009, 07:13:48 am
The final patch for the Multiline S&R has been uploaded!
Good work, Rick! I'll apply this now in my local copy for testing. I am looking forward to it... :-)

Any update?

MortenMacFly · « **Reply #30 on:** December 01, 2009, 02:44:29 pm »

Quote from: ironhead on December 01, 2009, 02:41:00 pm

Any update?

I am testing.

gygabyte017 · « **Reply #31 on:** May 08, 2010, 12:12:55 pm »

Hi, I really need this patch! Is this patch ready? If so, how to download and add it to my svn build?
Thanks!

oBFusCATed · « **Reply #32 on:** May 08, 2010, 01:41:38 pm »

Download the patch:
http://developer.berlios.de/patch/?func=detailpatch&patch_id=2849&group_id=5358

Then:
1. Open a cmd
2. goto the trunk directory
3. patch -p0 < multiline.patch

You'll need the patch utility another option is to use TortoiseSVN and it feature "Apply patch"

Loaden · « **Reply #33 on:** May 08, 2010, 01:59:43 pm »

Quote from: oBFusCATed on May 08, 2010, 01:41:38 pm

Download the patch:
http://developer.berlios.de/patch/?func=detailpatch&patch_id=2849&group_id=5358

Then:
1. Open a cmd
2. goto the trunk directory
3. patch -p0 < multiline.patch

You'll need the patch utility another option is to use TortoiseSVN and it feature "Apply patch"

cool!

gygabyte017 · « **Reply #34 on:** May 08, 2010, 06:03:18 pm »

Quote from: oBFusCATed on May 08, 2010, 01:41:38 pm

Download the patch:
http://developer.berlios.de/patch/?func=detailpatch&patch_id=2849&group_id=5358

Then:
1. Open a cmd
2. goto the trunk directory
3. patch -p0 < multiline.patch

You'll need the patch utility another option is to use TortoiseSVN and it feature "Apply patch"

Thank you! But there are some conflicts in replacedlg.cpp... Indeed, my replacedlg.cpp is at svn revision 6104 (I've updated svn right now), but the patched one is at 5102.... It seems ok for editormanager.cpp, finddlg.cpp, the headers, and the xrc file, even if the patched version is at 5102, but replacedlg.cpp has been edited heavy since 5102...

Is it a way to update the patch?

Edit: I've modified manually the conflicts, and I've rebuilt the code. Compilation is ok, and after few tests everything seems ok, the multiline search and replace works (at least on windows).

So, I've attached the new patch! Please test it to see if everything is ok! This is a great feature...

[attachment deleted by admin]

rickg22 · « **Reply #35 on:** November 08, 2010, 03:26:25 pm »

Question - is this patch applied to head now?

Loaden · « **Reply #36 on:** November 08, 2010, 03:43:25 pm »

Quote from: rickg22 on November 08, 2010, 03:26:25 pm

Question - is this patch applied to head now?

Yes, it's in HEAD now.
:lol:

ironhead · « **Reply #37 on:** November 08, 2010, 07:03:15 pm »

Quote from: Loaden on November 08, 2010, 03:43:25 pm

Quote from: rickg22 on November 08, 2010, 03:26:25 pm
Question - is this patch applied to head now?
Yes, it's in HEAD now.
:lol:

How do I enable it? As it stands find / replace in the 6843 build only has a single line for search / replace.

Edit: Never mind, found it. I didn't realize that it's limited to the 'Replace' dialogue only. Any chance it could be extended to support 'Search' as well?

Jenna · « **Reply #38 on:** November 08, 2010, 07:07:14 pm »

In "Replace" or "Replace in files" dialog there is a checkbox "Muliti-line Search -> Enabled" .

rickg22 · « **Reply #39 on:** November 09, 2010, 03:24:41 pm »

ironhead: At the time I wrote it (around a year ago), I was desperately needing a multiline replace. Actually I think Code::Blocks is already potentially capable of multiline searching - but we need a GUI to tell it to do so

News:

Author Topic: Multiline Search & Replace (Read 36628 times)

mariocup