Lots of packages depend on the latest version of libcairo, so I'll restrain from downgrading -- I don't want to break my main desktop...
I'm not sure if it was just libcairo's fault, because others like you use older versions without having the slowness problem. Anyway, it was solved for me by upgrading, so hopefully it helps others, too.
Jabber