Netapp: High CPU/NFS Ops – low network input

High CPU and NFS operations, however network input is low

Symptoms

High CPU and NFS operations, however network input is low

Cause of this problem

Numerous NFS clients continuously send the filer NFS operation requests, but the clients/users may not have permission to access the filer’s data.

Solution

If the output of the filer’s sysstat command displays higher than normal NFS operations per second and higher than normal CPU utilization, and sysstat displays low network input, determine if NFS clients are sending invalid requests to the filer that are being denied. To determine which NFS clients are sending numerous packets to the filer, complete the following steps:

Note: Enabling NFS per client statistics may cause a slight performance impact.

1. Enable NFS per client statistics:

   options nfs.per_client_stats.enable on

2. To zero the statistics, enter:

   nfsstat -z

3. To display the usage of all NFS clients that mount the filer, enter

   nfsstat -l

4. Investigate the client(s) with the highest usage. Sample nfsstat -l output:

   filer> nfsstat -l
   10.10.10.1 client1 NFSOPS = 49 (1%)
   10.10.10.2 client2 NFSOPS = 47 (1%)
   10.10.10.3 client3 NFSOPS = 745 (96%)
   10.10.10.4 client4 NFSOPS = 48 (1%)
   10.10.10.2 client5 NFSOPS = 50 (1%)

   In the sample output above, client3 has the highest usage and is most likely causing the packet storm.

5. Turn off NFS per client statistics options nfs.per_client_stats.enable off
6. Investigate NFS clients that show high usage in nfsstat -l output for processes that may be causing the packet storm.