Thursday, May 08, 2014

NFS extremely slow in VMWare Solaris guest

I had to investigate an issue related to slow NFS writes from a VMWare Solaris VM.

To debug protocols issues you need to use of course TCP packet sniffers. So I started with the following test for Solaris:
mkfile 5M /tmp/5MegSol
sudo snoop -t a host nfs.sample.com > /tmp/capture &
cp /tmp/5MegSol /mnt/nfs/
ps -ef|grep snoop|grep -v grep |awk '{print $2}'|xargs sudo kill 2>/dev/null
less /tmp/capture
view raw gistfile1.txt hosted with ❤ by GitHub
Basically we create a 5MB file and transfer it via NFS. The file was taking two minutes to be transferred. The result from /tmp/capture uncovered a lot of DUP ACKs:
13:49:23.32024 solaris.sample.com -> nfs.sample.com TCP D=2049 S=1016 Push Ack=2575691166 Seq=2647130780 Len=1460 Win
=49640
13:49:23.32056 nfs.sample.com -> solaris.sample.com TCP D=1016 S=2049 Ack=2647108880 Seq=2575691166 Len=0 Win=501
13:49:23.32058 solaris.sample.com -> nfs.sample.com TCP D=2049 S=1016 Ack=2575691166 Seq=2647132240 Len=1460 Win=4964
0
13:49:23.32059 solaris.sample.com -> nfs.sample.com TCP D=2049 S=1016 Push Ack=2575691166 Seq=2647133700 Len=1460 Win
=49640
13:49:23.32175 nfs.sample.com -> solaris.sample.com TCP D=1016 S=2049 Ack=2647111800 Seq=2575691166 Len=0 Win=501
13:49:23.32179 solaris.sample.com -> nfs.sample.com TCP D=2049 S=1016 Ack=2575691166 Seq=2647135160 Len=1460 Win=4964
0
13:49:23.32180 solaris.sample.com -> nfs.sample.com TCP D=2049 S=1016 Push Ack=2575691166 Seq=2647136620 Len=1460 Win
=49640
13:49:23.32194 nfs.sample.com -> solaris.sample.com RPC R XID=1086274887 Success
view raw gistfile1.txt hosted with ❤ by GitHub
From a Linux box we then run something similar:
fallocate -l 5M /tmp/5Meg
sudo tcpdump -s0 -S -i eth0 host nfs.sample.com > /tmp/capture &
cp /tmp//5Meg /mnt/nfs/
sudo killall tcpdump
less /tmp/capture
view raw gistfile1.txt hosted with ❤ by GitHub
And then I confirmed it the write went fast and with no DUP ACK. After we shipped the issue to Infrastructure they found out the culprint to be the usage of a conflictive network adapter in VMWare. Using vmxnet3 network adapter looks to be the right option when it comes to supporting NFS traffic. No DUP ACK anymore.

No comments:

Followers