I don't know what firefox uses, but creating a named mutex is another common technique to allow only one running instance.
This is exactly what Code::Blocks does (
wxSingleInstanceChecker does just that).
But: this does not solve the problem, it really makes it worse, it leads to the "another instance is already running, quitting now" issue.
The problem with multiple instances is that we need a bulletproof way to hand the commandline from a new instance to the already running instance. Unluckily, this is not precisely trivial to implement cross-platform.
A named pipe would do, but Linux pipes are not named. A message queue on Linux would do, too, but Windows has no such thing. TCP could be used, but that would make all those Windows application firewalls go frenzy. Gcc allows shared variables between processes, but that is compiler-specific and OS-specific in addition. A shared memory page could be used, but again, this is very OS-dependent. Tempfiles have been discussed, but were turned down due to race conditions...
It is not so simple really.
DDE is a fat, clumsy pig (like most MS technologies), but at least it works 90% of the time, except for the occasional "cannot find file" error. And since we are using the wxWidgets thingie for DDE, it should run on Unix, too (honestly, I don't know, but the docs claim that it does).