Deploy NFS

When you choose a backup medium, we recommend that you use Alibaba Cloud Object Storage Service (OSS) as the backup destination. As a stateless object storage service, OSS offers higher stability compared with Network File System Version 4 (NFS4), a stateful storage system.

To use NFS as the backup destination, you can refer to this section to deploy NFS. NFS is available in both software and hardware. However, software-based NFS is unstable and tends to hang, which can cause backup tasks to be stuck or the machines mounting NFS to malfunction. Therefore, we recommend that you use dedicated NFS hardware.

Considerations

• When you use an NFS environment, make sure that you mount the NFS share before you enable backup. If an issue occurs with NFS during backup, stop data backup and log archiving first, and then resolve the issue with NFS.

• The current version of OceanBase Database supports NFS version 3 (NFSv3) and later.

• We recommend that you use SSD-based NFS services to avoid instability caused by insufficient NFS performance.

• When using NFS as a backup medium, ensure that all OBServer nodes are mounted to the same NFS server. Additionally, to ensure the backup proceeds smoothly, you must use the parameters recommended in this document to mount NFS. For the specific mounting operations, see the Deploy the NFS client section in this document.

• When you restart an OBServer node, start NFS first, then start the OBServer node.

• After adding new machines, ensure that the new machines are successfully mounted to NFS or can back up to other media before starting the OBServer node.

Deploy the NFS software server

1. Log in to the NFS server.

2. Execute the following command to install NFS through the YUM package manager:

   sudo yum install nfs-utils
   

3. Set Exports.

   1. Choose an appropriate directory as the shared directory, taking into account the space and performance requirements for backups.

      For example, in this document, the shared directory is /data/nfs_server/.

   2. Use the sudo vim /etc/exports command to open the configuration file and set the following parameters.

      /data/nfs_server/ xx.xx.xx.xx/16(rw,sync,all_squash)
      

      Here, xx.xx.xx.xx indicates the IP address range allowed to access the shared directory.

   3. Execute the following command to grant privileges to the anonymous user to ensure that it can access the directory specified in the exports file.

      CentOS 7

      CentOS 8

      For CentOS 7, NFS uses nfsnobody as the default anonymous user by default. The command is as follows:

      sudo chown nfsnobody:nfsnobody -R /data/nfs_server
      

      For CentOS 8 and later, NFS uses nobody as the default anonymous user. The command is as follows:

      sudo chown nobody:nobody -R /data/nfs_server
      

4. Execute the following command to restart NFS.

   sudo systemctl restart nfs-server
   

5. Set the Slot Table.

   1. Use the sudo vim /etc/sysctl.conf command to open the sysctl.conf configuration file and add the following line to the file:

      sunrpc.tcp_max_slot_table_entries=128
      

   2. Execute the following command to change the maximum number of concurrent NFS requests to 128.

      sudo sysctl -w sunrpc.tcp_max_slot_table_entries=128
      

      After the command is executed successfully, you can run the cat /proc/sys/sunrpc/tcp_max_slot_table_entries command to verify whether the setting takes effect. If the returned value is 128, the modification is successful.

   3. (Optional) Restart the machine.

Deploy the NFS client

When you deploy the NFS client, you must perform the operations on all OBServer nodes.

The following uses the operations on one OBServer node as an example to provide guidance.

1. Log in to the OBServer node.

2. Execute the following command to install NFS through the YUM package manager:

   sudo yum install nfs-utils
   

3. Set the Slot Table.

   1. Use the sudo vim /etc/sysctl.conf command to open the sysctl.conf configuration file and add the following line to the file:

      sunrpc.tcp_max_slot_table_entries=128
      

   2. Execute the following command to change the maximum number of concurrent NFS requests to 128.

      sudo sysctl -w sunrpc.tcp_max_slot_table_entries=128
      

      After the command is executed successfully, you can run the cat /proc/sys/sunrpc/tcp_max_slot_table_entries command to verify whether the setting takes effect. If the returned value is 128, the modification is successful.

   3. (Optional) Restart the machine.

4. Choose an appropriate directory as the mount point and execute the following command to mount NFS.

   NFS 3.x

   NFS 4.x

   For example, in this document, it is assumed that NFS is mounted to the /data/nfs directory. If no suitable directory exists, you can create one.

   sudo mount -tnfs -o rw,nfsvers=3,sync,lookupcache=positive,hard,nolock,timeo=600,wsize=1048576,rsize=1048576,namlen=255 10.10.10.1:/data/nfs_server /data/nfs
   

   For example, in this document, it is assumed that NFS is mounted to the /data/nfs directory. If no suitable directory exists, you can create one.

   sudo mount -tnfs -o rw,nfsvers=4.1,sync,lookupcache=positive,hard,timeo=600,wsize=1048576,rsize=1048576,namlen=255 10.10.10.1:/data/nfs_server /data/nfs
   

   The parameters are described as follows:

   • nfsvers: the NFS version. When using NFS 4.x, we recommend that you use NFS 4.1 or later. NFS 4.0 has a risk of reading stale files after renaming and is not recommended.

   • sync: uses synchronous write to ensure that data is flushed to the server in time, thus ensuring data consistency.

   • lookupcache=positive: specifies how the operating system kernel manages the directory cache for a given mount point. Valid values are all, none, pos, and positive. The value positive is used to avoid mistakenly reporting that directories or files do not exist when they are accessed concurrently, thus ensuring data consistency.

   • hard: when NFS is unavailable, the system blocks application read/write requests to ensure data consistency. The soft option must not be used as it poses a risk of data errors.

   • lock/nolock: specifies whether to use the NLM protocol to lock files on the server. The default value is lock. Since the NFS 4.x protocol integrates file locking and the NFS 3.x protocol does not support file locking, we recommend that you add the nolock option when mounting with NFS 3.x.

   • timeo: specifies the wait time for retries. The unit is 0.1s. We recommend that you do not set it to a large value. The recommended value is 600.

   • wsize: the size of the data blocks to write. The recommended value is 1048576.

   • rsize: the size of the data blocks to read. The recommended value is 1048576.

   • namlen: the recommended value is 255.

   • 10.10.10.1: the IP address of the NFS server.

   Notice

   • When you mount NFS 3/4, make sure that the parameters in the backup mounting environment include the recommended parameters in the example.
   • In a Docker environment, mount NFS on the host machine and then map it to the Docker container. If you mount NFS directly inside the Docker container, the client may hang.

5. After the mounting is completed, execute the following command to verify the NFS performance.

   fio -filename=/data/nfs/fio_test -direct=1  -rw=randwrite -bs=2048K -size=100G  -runtime=300 -group_reporting -name=mytest  -ioengine=libaio -numjobs=1 -iodepth=64 -iodepth_batch=8 -iodepth_low=8 -iodepth_batch_complete=8
   

   For example, the execution result is as follows:

   Run status group 0 (all jobs):
   WRITE: io=322240MB, aggrb=1074.2MB/s, minb=1074.2MB/s, maxb=1074.2MB/s, mint=300006msec, maxt=300006msec
   

How to handle errors when mounting NFS 3

When deploying NFS, if you encounter the error requested NFS version or transport protocol is not supported while mounting NFS 3, you can resolve it as follows:

1. Log in to the NFS server.

2. Check the NFS versions supported by the server.

   sudo cat /proc/fs/nfsd/versions
   

   The output is as follows:

   -2 -3 +4 +4.1 +4.2
   

   Here, -3 means NFS 3 is disabled.

3. Adjust the configuration.

   Perform the following operations to enable NFS 3:

   1. Check if the /etc/nfs.conf file exists. If it does, remove the # in front of vers3=y in the file.

      The modified sample file is as follows:

      [nfsd]
      # debug=0 
      # threads=8
      # host=
      # port=0
      # grace-time=90
      # lease-time=90
      # udp=y
      # tcp=y
      # vers2=n
         vers3=y
      # vers4=y
      # vers4.0=y
      # vers4.1=y
      # vers4.2=y
      # rdma=n
      ##
      

   2. Check if the /etc/sysconfig/nfs file exists. If it does, check whether -N 3 is specified in the RPCNFSDARGS parameter. If so, remove -N 3.

      For example, after removing -N 3, the value of the RPCNFSDARGS parameter is as follows:

      RPCNFSDARGS="-N 2 -N 4"
      

4. Execute the following command to restart NFS.

   sudo systemctl restart nfs-server
   

5. Check the NFS versions supported by the server again.

   sudo cat /proc/fs/nfsd/versions
   

   The output is as follows:

   -2 +3 -4 -4.1 -4.2
   

   Here, +3 means NFS 3 is enabled.