Change of_string_utf8_decode() API

It now returns <= 0 on error, with negative values being the number of
bytes it would have needed * -1.

This can be used to detect cut off and how many bytes are missing.