Using Oracle GI 19c to Provide Highly-Available NFS Services

Introduction

Since Oracle GI version 11.2.0.3, Oracle have provided a clustered file-system called ACFS, based on the underlined technology Oracle ASM. This technology has provided a reliable-enough, although not free from bugs, clustered file-system, while Oracle OCFS2 (which had its own cluster framework: O2CB) has been abandoned and left to die.

ACFS was a good-enough clustered file-system, however, Oracle did not provide network file server cluster services to harness this file-system. While Oracle GI was advocated as a general-purpose cluster, it failed to do so due to the relative complexity of setting custom application services and their dependencies, while the main tasks – Oracle RDBMS services – were so well integrated into the GI framework. To conclude: If you were not a DBA running a database over Oracle GI (in RAC environment) the chances of you using the cluster framework for other applications was extremely low.

Storage Services

Around version 12.1, Oracle have started providing a High Availability Network File Storage, elevating ACFS with their cluster framework, to provide a reliable NFS services. With an always-mounted clustered file-system, fail-over nodes will not need to spend time mounting, and maybe running FSCK while doing so, on shared block devices. All cluster nodes (or all members of that fail-over domain) should have the filesystem already mounted, and the only action required by the agent is to configure the relevant virtual IP (HAVIP) and export the relevant shares, using settings kept in the cluster registry – utilising the built-in Linux NFS server service.

Later versions supported SMB as well as NFS, and the name changed from “High Availability Network File Storage” to “Oracle ACFS NAS Maximum Availability eXtensions“. The general commands have not changed much, and the same principles, especially around NFSv3 remain quite the same.

The cluster handles a virtual IP, active on a single node, with dependencies to an ACFS file-system mount directory, and to a set of NFS export settings. These services rely heavily on the Linux distribution NFS or SMB services, without which – the cluster will not be able to provide this service.

The main advantages of this solution is that the system admin can rely on a certified and supported platform, with a very low impact on the business-critical database platform, without setting up a new highly available platform. It can be considered as an action of “Critical Services Consolidation”, if you will.

Oracle provides a procedure to set up HANFS, however, this procedure is incomplete and lacking. In this document I will provide an easy walk through to setting up HANFS quickly and easily.

This document was tested on Oracle GI 19c version 19.14, and should probably be valid as-is for all 19.x releases.

Prerequisites

• A working Oracle GI 19c version 19.5 and above (tested on 19.14)
• Existing ASM DiskGroup(s), or the knowledge on how to create a new ASM DiskGroup
• A 2+ node cluster. Oracle root-services (HAVIP, VIP, ACFS mount and NFS exports) are not supported on Oracle Restart – a single-node Oracle GI cluster.
• Recommended: Using a dedicated ASM DiskGroup for HANFS purposes can disarm Oracle DBA reluctance to share cluster resources with “foreign” services.
• All cluster nodes must have NFS service enabled and active. To do so, run (on modern Linux systems):

systemctl enable --now nfs-server

• A dedicated IP address for the HANFS services, including a matching DNS name (DNS is mandatory), and is within the cluster public IP address subnet.

Assumptions

In order to perform this configuration, we will assume the following details:

• The name of the ASM DiskGroup in our example will be HANFSDG
• The name of the ADVM volume is NFSVOL
• The ACFS mount point will be /ACFS
• The HAVIP selected will be called (DNS name) nfs-havip (short DNS name is enough), and is fully resolvable.
• The HAVIP resource name will be nfs-srv-havip. This is a service name (label) used by Oracle GI
• The exportfs resource name will be mynfsshare. This is a service name (label) used by Oracle GI
• A list of clients with the IP addresses 192.168.10.10, 192.168.10.11, 192.168.10.12
• All clients share the same NFS export settings, which are “no_root_squash” and “rw“. Other NFS export settings are reasonable defaults provided by Linux, such as “sync” and “secure”, and are omitted in our configuration, because they are the default.

Steps

We will follow the following sequence:

• Create ADVM block device (ASM Volume)
• Format the ADVM block device using ACFS file-system
• Register the ACFS file-system with the GI cluster services using ‘srvctl’ command
• Start the ACFS file-system services using ‘srvctl’ command
• Register a virtual IP (HAVIP) with the GI cluster services using ‘srvctl’ command
• Register an ‘exportfs’ service (share and settings) with the GI cluster services using ‘srvctl’ command
• Starting the ‘exportfs’ services using ‘srvctl’ command
• Verifying the configuration is working correctly.

Create ADVM block device

As the oracle grid user, with correctly defined ORACLE_HOME to point at the GI_HOME and ORACLE_SID to point at the name of the ASM instance (“+ASM1“), run the command:

$ asmcmd volcreate -G HANFSDG -s 1000G NFSVOL

Parameters:

• -G : Name of ASM DiskGroup
• -s : Size (in G, M, T, P) of the ADVM volume

The resulting ADVM volume is located in /dev/asm/nfsvol-436

The number suffix is randomly generated during the ADVM creation, and will change from system to system. In our example, we will use the actual name nfsvol-436

Format ACFS

This command can be run as root or the oracle grid user. Notice that we are using the same ADVM block device as generated by the previous command:

# mkfs.acfs /dev/asm/nfsvol-436

Register the ACFS file-system with the GI cluster

This command can be called by the oracle grid user, or by root, if commands are available in this user’s PATH:

# srvctl add filesystem -device /dev/asm/nfsvol-436 -volume NFSVOL -diskgroup HANFSDG -path /ACFS

The parameters are rather self-explanatory. The dependencies created by this command ensure that the ASM DiskGroup will be mounted before the ADVM volume, and that only afterwards ACFS mount command will be called.

Start the ACFS file-system services

# srvctl start filesystem -device /dev/asm/nfsvol-436

Register HAVIP with the GI cluster

This command can be called as the oracle grid user, or root, if the commands are in the PATH:

# srvctl add havip -id nfs-srv-havip -address nfs-havip

Parameters:

• -id : Name of the GI resource
• -address : resolvable DNS name for the dedicated VIP

Register an ‘exportfs’ service

This command can be called as the oracle grid user, or root, if the commands are in the PATH:

# srvctl add exportfs -id nfs-srv-havip -path /ACFS -name mynfsshare -clients 192.168.10.10,192.168.10.11,192.168.10.12 -options "no_root_squash,rw"

Parameters:

• -id : The name of the HAVIP resource, as defined in the previous section
• -path : The ACFS mount path
• -name : The name of the GI resource
• -clients : A comma separated list of hosts, subnets and anything else complying with Linux NFS exports directives
• -options : A quoted list of comma separated NFS share directives

Starting the ‘exportfs’ services

This command can be called as the oracle grid user, or root, if the commands are in the PATH:

# srvctl start exportfs -name mynfsshare

Verifying the configuration is working correctly

# srvctl status exportfs
export file system mynfsshare is enabled
export file system mynfsshare is exported on node ginode01

Another test can be performed on the active host:

# exportfs
/ACFS        192.168.10.10
/ACFS        192.168.10.11
/ACFS        192.168.10.12

Follow-up

Perform client tests. You can attempt to fail-over the hosts (violently, or gracefully), or relocate the service, using the following command:

srvctl relocate havip -id nfs-srv-havip -f

Conclusion

Setting up HANFS is rather simple, if you understand the basic requirements, and follow up on these directives. Oracle may modify the commands’ syntax from time to time, and while it can be considered a bug by some, this can easily be resolved on the spot by running the parent command with the flag ‘-h’ to get a full list of legal flags. Do not be afraid to use this flag.

All in all – this is a very reliable process, with near-zero impact on any other running services, which enhances this already high-available solution to provide an additional layer of critical compute service – HA NFS – in a consolidated and already maintained and well-cared-of environment.