Don't you want to use the debug build of the library to be able to debug the state of the objects that the library exports? You always get more information, which you can use.
More information, yes, but useful information? For me, most of the time not. I rarely have need to step into a foreign library; my normal use case is to only need to identify my program's entry and exit points/conditions to libraries.
The distributed SFML package including binaries also has the binaries for the debug build. I doubt they would be distributed unless the authors suspected users might find them useful.
Building the source by default only created the release binaries. Using
apt-get by default does not acquire the debug files.
You also use the corresponding debug build for wxWidgets in its project wizard.
However, the wxWidgets wizard also checks if the files exist, and gives the option to link against release binaries if the debug ones are not available.
As a habit where I work, we link to debug builds for any third party libraries that provide it and/or the source. For the ones that don't... nothing we can do there.
I do not yet program in the professional workplace, so perhaps I am lacking on some practical experience...