I am running FreeBSD 12-RELEASE on our local NFS server. Previously it was working fine, but as of a few days ago some of our tools that require file locking stopped working. Locking does work on our NFS clients for other servers.
Here is some information I think might be relevant:
On our FreeBSD server, san6:
root@san6:~ # lsof | grep rpc.lockd
lsof: WARNING: compiled for FreeBSD release 12.0-RELEASE-p5; this is 12.0-RELEASE.
rpc.lockd 1063 root cwd VDIR 2,16 1024 2 /
rpc.lockd 1063 root rtd VDIR 2,16 1024 2 /
rpc.lockd 1063 root txt VREG 2,16 73176 31059299 /usr/sbin/rpc.lockd
rpc.lockd 1063 root txt VREG 2,16 148320 12279169 /libexec/ld-elf.so.1
rpc.lockd 1063 root txt VREG 2,16 49560 31070185 /usr/lib/librpcsvc.so.5
rpc.lockd 1063 root txt VREG 2,16 1949672 46066959 /lib/libc.so.7
rpc.lockd 1063 root 0u VCHR 2,18 0t1091 530 /dev/pts/0
rpc.lockd 1063 root 1u VCHR 2,18 0t1091 530 /dev/pts/0
rpc.lockd 1063 root 2u VCHR 2,18 0t1091 530 /dev/pts/0
rpc.lockd 1063 root 3u unix 0xfffff8013dc176d0 0t0 ->0xfffff8013dec6a38
root@san6:~ # ps aux | grep lockd
root 1063 0.0 0.0 48016 13692 - Ds 11:54 0:00.43 rpc.lockd -d 2
root 3102 0.0 0.0 11256 2736 1 S+ 05:53 0:00.00 grep lockd
root@san6:/usr/ports # rpcinfo -p canada
program vers proto port service
100000 4 tcp 111 rpcbind
100000 3 tcp 111 rpcbind
100000 2 tcp 111 rpcbind
100000 4 udp 111 rpcbind
100000 3 udp 111 rpcbind
100000 2 udp 111 rpcbind
100024 1 udp 58683 status
100024 1 tcp 36401 status
100021 1 udp 33875 nlockmgr
100021 3 udp 33875 nlockmgr
100021 4 udp 33875 nlockmgr
100021 1 tcp 33610 nlockmgr
100021 3 tcp 33610 nlockmgr
100021 4 tcp 33610 nlockmgr
root@san6:~ # dmesg | tail -n 1
NLM: failed to contact remote rpcbind, stat = 5, port = 28416
root@san6:~ # mount | grep nfs
data0 on /data0 (zfs, NFS exported, local, noatime, nfsv4acls)
And on one of our clients, "canada", running Centos 7.6.1810:
[root@canada ~]# dmesg | tail -n 10
[ 24.205026] virbr0: port 1(virbr0-nic) entered disabled state
[ 588.552140] FS-Cache: Loaded
[ 588.594282] FS-Cache: Netfs 'nfs' registered for caching
[ 888.710226] lockd: server san6 not responding, still trying
[ 1265.871541] lockd: server san6 OK
[ 1265.871551] lockd: unexpected unlock status: 7
[ 1415.519506] nfs: server san6 not responding, still trying
[ 1439.592184] nfs: server san6 OK
[ 2137.130503] nfs: server san6 not responding, still trying
[ 2622.383586] nfs: server san6 OK
[root@canada ~]# mount | grep /projects
auto.direct on /projects type autofs (rw,relatime,fd=29,pgrp=15535,timeout=300,minproto=5,maxproto=5,direct,pipe_ino=87674)
san6:/data0/projects on /projects type nfs (rw,nosuid,noatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.252.48.106,mountvers=3,mountport=816,mountproto=udp,local_lock=none,addr=10.252.48.106)
Note that lockd is in "Ds" state, I'm not sure which file it is waiting for disk io from. It seems to enter this state when I try to kill/restart it and also after running for awhile. I've tried restarting the server with debug flags turned on (the -d2 in the ps log) but didn't see any more information in the syslog.
I've also tried adding the server to the trusted zone of one of our Centos 7 clients to see if that helped the "NLM: failed to contact remote rpcbind, stat = 5, port = 28416" but it did not seem to, no change to whether that client could lock files or not.
I greatly appreciate any help with this issue!
UPDATE: I've reproduced the issue on another server, still the same symptoms. I'm still not sure what is going on here.
