I am seeing this message on vmware.log , this is a windows 2008 R2 x64 VM running with hardware version 7. Same VM console is just fine on ESX 5.0 base but with ESX5.1 base its not.
vmware.log
2013-07-09T18:13:57.599Z| mks| I120: SOCKET 30972 (14) recv error 104: Connection reset by peer
2013-07-09T18:13:57.599Z| mks| W110: SOCKET 30972 (14) error 1 during VNC negotiation!
2013-07-09T18:14:15.181Z| mks| I120: SOCKET 30973 (55) recv error 104: Connection reset by peer
2013-07-09T18:14:15.182Z| mks| W110: SOCKET 30973 (55) error 1 during VNC negotiation!
2013-07-09T18:14:19.733Z| mks| I120: SOCKET 30974 (55) recv error 104: Connection reset by peer
2013-07-09T18:14:19.733Z| mks| W110: SOCKET 30974 (55) error 1 during VNC negotiation!
2013-07-09T18:14:31.756Z| mks| I120: SOCKET 30975 (14) recv error 104: Connection reset by peer
2013-07-09T18:14:31.756Z| mks| W110: SOCKET 30975 (14) error 1 during VNC negotiation!
2013-07-09T18:14:41.893Z| mks| I120: SOCKET 30976 (14) recv error 104: Connection reset by peer
2013-07-09T18:14:41.893Z| mks| W110: SOCKET 30976 (14) error 1 during VNC negotiation!