Yes, I think that in such cases casts could be seen as legitimate (if the needed precautions are taken) and I do not see for the moment a better solution. Keeping a copy of the buffer could be useful for different reasons, but if available memory is a constraint, then keeping a copy of the buffer would be not a so good idea.
Michael