Deploy NFS

When you choose an backup media, we recommend that you use Alibaba Cloud Object Storage Service (OSS) as the destination. As a stateless object storage service, OSS offers higher stability compared with Network File System Version 4 (NFS4), a stateful storage system.

To use NFS as the backup destination, deploy NFS first by referring to this topic. NFS is available in both software and hardware. However, software-based NFS is unstable and and tends to hang, which can cause backup tasks to be stuck or the servers mounting NFS to malfunction. Therefore, we recommend that you use dedicated NFS hardware.

Considerations

• When you use an NFS environment, make sure that you mount the NFS share before you enable backup. If an issue occurs with NFS during backup, stop data backup and log archiving first, and then resolve the issue with NFS.

• The current version of OceanBase Database supports NFS version 3 (NFSv3) and later.

• We recommend that you use SSD-based NFS services to avoid instability caused by insufficient NFS performance.

• When using NFS as a backup medium, ensure that all OBServer nodes are mounted to the same NFS server. Additionally, to ensure the backup proceeds smoothly, you must use the parameters recommended in this document to mount NFS. For more information, see the Deploy the NFS client section in this topic.

• When you restart an OBServer node, start NFS first.

• After adding new servers, ensure that the new servers are successfully mounted to NFS or can back up to other media before starting the OBServer node.

Deploy the NFS server

1. Log in to the NFS server.

2. Run the following command to install NFS using the YUM package manager:

   sudo yum install nfs-utils
   

3. Configure the exports file.

   1. Choose an appropriate directory as the shared directory, taking into account the space and performance requirements for backups.

      For example, in this topic, the shared directory is /data/nfs_server/.

   2. Run the sudo vim /etc/exports command to open the configuration file and set the following parameters.

      /data/nfs_server/ xx.xx.xx.xx/16(rw,sync,all_squash)
      

      Here, xx.xx.xx.xx indicates the IP address range allowed to access the shared directory.

   3. Execute the following command to grant privileges to the nfsnobody user to ensure that it can access the directory specified in the exports file.

      CentOS 7

      CentOS 8

      By default, in CentOS 7, NFS uses nfsnobody as the default anonymous user. The command is as follows:

      sudo chown nfsnobody:nfsnobody -R /data/nfs_server
      

      In CentOS 8 and later, NFS uses nobody as the default anonymous user. The command is as follows:

      sudo chown nobody:nobody -R /data/nfs_server
      

4. Run the following command to restart NFS.

   sudo systemctl restart nfs-server
   

5. Set the slot table.

   1. Run the sudo vim /etc/sysctl.conf command to open the sysctl.conf configuration file and add the following line to the file:

      sunrpc.tcp_max_slot_table_entries=128
      

   2. Run the following command to change the maximum number of concurrent NFS requests to 128.

      sudo sysctl -w sunrpc.tcp_max_slot_table_entries=128
      

      After the command is executed, you can run the cat /proc/sys/sunrpc/tcp_max_slot_table_entries command to check whether the setting takes effect. If the returned value is 128, the modification is successful.

   3. (Optional) Restart the server.

Deploy the NFS client

When you deploy an NFS client, you must operate on all OBServer nodes.

The following example shows the procedure for deploying an NFS client on one OBServer node.

1. Log in to the OBServer node.

2. Run the following command to install NFS using the YUM package manager:

   sudo yum install nfs-utils
   

3. Set the slot table.

   1. Run the sudo vim /etc/sysctl.conf command to open the sysctl.conf configuration file and add the following information in the file:

      sunrpc.tcp_max_slot_table_entries=128
      

   2. Run the following command to change the maximum number of concurrent NFS requests to 128.

      sudo sysctl -w sunrpc.tcp_max_slot_table_entries=128
      

      After the command is executed, you can run the cat /proc/sys/sunrpc/tcp_max_slot_table_entries command to check whether the setting takes effect. If the returned value is 128, the modification is successful.

   3. (Optional) Restart the server.

4. Select an appropriate directory as the mount point and run the following command to mount NFS.

   NFS 3.x

   NFS 4.x

   For example, in this topic, it is assumed that the mount point is the /data/nfs directory. If no suitable directory exists, you can create one.

   sudo mount -tnfs -o rw,nfsvers=3,sync,lookupcache=positive,hard,nolock,timeo=600,wsize=1048576,rsize=1048576,namlen=255 10.10.10.1:/data/nfs_server /data/nfs
   

   For example, in this topic, it is assumed that the mount point is the /data/nfs directory. If no suitable directory exists, you can create one.

   sudo mount -tnfs -o rw,nfsvers=4.1,sync,lookupcache=positive,hard,timeo=600,wsize=1048576,rsize=1048576,namlen=255 10.10.10.1:/data/nfs_server /data/nfs
   

   In the commands above:

   • nfsvers: the NFS version.

   • sync: ensures that data is written synchronously to the server to ensure data consistency.

   • lookupcache=positive: specifies the way the operating system kernel manages the directory cache for a given mount point. Valid values are all, none, pos, and positive. The value positive is used to avoid mistakenly reporting the non-existence of directories or files when they are concurrently accessed, to ensure data consistency.

   • hard: In cases where NFS is unavailable, the system will block application read/write requests to ensure data consistency. The soft option must not be used as it poses a risk of data errors.

   • lock/nolock: specifies whether to use the NLM protocol to lock files on the server. The default value is lock. Since the NFS 4.x protocol integrates file locking, and the NFS 3.x protocol does not inherently support it, it is recommended to use the nolock option when mounting with the NFS 3.x protocol.

   • timeo: specifies the time to wait for a retry. The unit is 0.1s. We recommend that you do not set it to a large value. The recommended value is 600.

   • wsize: specifies the size of the blocks to write. The recommended value is 1048576.

   • rsize: specifies the size of the blocks to read. The recommended value is 1048576.

   • namlen: the recommended value is 255.

   • 10.10.10.1: the IP address of the NFS server.

   Notice

   • When you mount NFS 3/4, make sure that the parameters in the backup mounting environment include the recommended parameters in the example.
   • In a Docker environment, mount NFS on the host machine and then map it to the Docker container. If you mount NFS directly in the Docker container, there may be a risk that the client gets stuck (hangs).

5. Run the following command to verify the performance of NFS after the mounting is completed.

   fio -filename=/data/nfs/fio_test -direct=1  -rw=randwrite -bs=2048K -size=100G  -runtime=300 -group_reporting -name=mytest  -ioengine=libaio -numjobs=1 -iodepth=64 -iodepth_batch=8 -iodepth_low=8 -iodepth_batch_complete=8
   

   For example, the execution result is as follows:

   Run status group 0 (all jobs):
   WRITE: io=322240MB, aggrb=1074.2MB/s, minb=1074.2MB/s, maxb=1074.2MB/s, mint=300006msec, maxt=300006msec
   

How to handle errors when mounting NFS 3

When deploying NFS, if you encounter the error requested NFS version or transport protocol is not supported while mounting NFS 3, you can resolve it as follows:

1. Log in to the NFS server.

2. Check the NFS versions supported by the server.

   sudo cat /proc/fs/nfsd/versions
   

   If you see the following output:

   -2 -3 +4 +4.1 +4.2
   

   Here, -3 means NFS 3 is disabled.

3. Adjust the configuration.

   Take the following steps to enable NFS 3:

   1. Check if the /etc/nfs.conf file exists. If it does, remove the # in front of vers3=y in the /etc/nfs.conf file.

      The modified sample file should look like this:

      [nfsd]
      # debug=0 
      # threads=8
      # host=
      # port=0
      # grace-time=90
      # Tease-time=90
      # udp=y
      # tcp=y
      # vers2=n
         vers3=y
      # vers4=y
      # vers4.0=y
      # vers4.1=y
      # vers4.2=y
      # rdma=n
      ##
      

   2. Check if the /etc/sysconfig/nfs file exists. If it does, look for the RPCNFSDARGS parameter and see if -N 3 is specified. If so, remove -N 3.

      For example, after removing -N 3, the value of the RPCNFSDARGS parameter should look like this:

      RPCNFSDARGS="-N 2 -N 4"
      

4. Run the following command to restart NFS.

   sudo systemctl restart nfs-server
   

5. Check the NFS versions supported by the server again.

   sudo cat /proc/fs/nfsd/versions
   

   If you see the following output:

   -2 +3 -4 -4.1 -4.2
   

   Here, +3 means NFS 3 is now enabled.