Do not use server.PeerCount() when getting peers right after receiving add/remove peer event as it does not contain updated info.
Adam Babik