IsErrEOF returns false when it should return true in a couple of cases:
1. if the error has been wrapped in another error (for example, if EOF
is wrapped in an RPC error)
2. if the error has been created from an Error field in an RPC response
(as it is the case in CallWithCodec in the net-rpc-msgpackrpc package
for example)
* Adds client-side retry for no leader errors.
This paves over the case where the client was connected to the leader
when it loses leadership.
* Adds a configurable server RPC drain time and a fail-fast path for RPCs.
When a server leaves it gets removed from the Raft configuration, so it will
never know who the new leader server ends up being. Without this we'd be
doomed to wait out the RPC hold timeout and then fail. This makes things fail
a little quicker while a sever is draining, and since we added a client retry
AND since the server doing this has already shut down and left the Serf LAN,
clients should retry against some other server.
* Makes the RPC hold timeout configurable.
* Reorders struct members.
* Sets the RPC hold timeout default for test servers.
* Bumps the leave drain time up to 5 seconds.
* Robustifies retries with a simpler client-side RPC hold.
* Reverts untended delete.
Dhia Ayachi
Pierre Cauchois
James Phillips