I have a solution. With jQuery. Whenever you are on a page and someone PM's you. You get a notification hover box on one corner of the screen. That is a good idea.
Facebook's implementation is fabulous, would be happy to see anything that's even in that direction. But I mean, I don't feel the need for live chatting at all, I just think the PM system needs to be slicker and less intrusive to browsing (modals!!)
Imho modals are as intrusive as it gets (At least ones that cover the entire screen that you have to click to dismiss). If you're opening a modal you could just as well be opening up your inbox.