VxVM System Administrator's Guide
The Volume Manager supports up to sixteen partitions (also called slices) on a physical disk. These partitions are named, in order, 0 through 15. Partition 0 is reserved to indicate the entire disk.
When a partition is placed under Volume Manager control, a VM disk is assigned to that partition. A symbolic name (the disk name or disk media name) such as
disk01 can be established to refer to a VM disk.
A partition is addressed through a physical address (generally referred to as the device name) of the form
c#b#t#d#s#, which is comprised of the following elements:
c#--The number of the controller to which the disk drive is attached.
b#--The corresponding bus.
d#--The target ID and device number that constitute the address of the disk drive on the controller.
s#--The partition number on the disk drive.
c0b0t0d0s7. By convention,
s7is used to refer to the standard partitioning method used by VxVM. The physical disk is identified to the Volume Manager as
c#b#t#d#s#. For many commands, the suffix
s#is normally optional, though display commands usually report devices with an
s#suffix. The Volume Manager
vxdiskaddutilities take device names without the
s7suffix. For example, to specify the second disk attached to the first controller to
vxdiskadd, use the name
The boot disk (which contains the root file system and is used when booting the system) is often identified to the Volume Manager by the device name
A VM disk is typically composed of two regions:
s7, the default type is
A system with the Volume Manager installed will always have the default disk group,
rootdg. By default, operations are directed to the
rootdg disk group. The system administrator can create additional disk groups, as necessary. Many systems do not need to use more than one disk group, unless they have a large number of disks. Disks do not need to be added to disk groups until the disks are needed to create VxVM objects. They can be initialized and reserved to be added to disk groups later. However, at least one disk (partition) must be added to
rootdg in order to perform the VxVM installation procedures.
When a disk is added to a disk group, it is given a name (such as
disk02). This name can be used to identify a disk for volume operations, such as volume creation or mirroring. This name relates directly to the physical disk. If a physical disk is moved to a different target address or to a different controller, the name
disk02 will continue to refer to it. Disks can be replaced by first associating a different physical disk with the name of the disk to be replaced and then recovering any volume data that was stored on the original disk (from mirrors or backup copies).
Having excessively large disk groups may cause the private region to fill. In the case of larger disk groups, disks should be set up with larger private areas to log in. A major portion of a disk's private region consists of space for a disk group configuration database containing configuration records for each VxVM object in that disk group. Since each configuration record takes up 256 bytes (or half a block), the number of configuration records that can be created in a disk group is twice the configuration database copy size (which can be obtained from the output of the command
vxdiskadm--This is the Volume Manager Support Operations menu interface. This utility provides a menu of disk operations. Each entry in the main menu leads you through a particular operation by providing you with information and asking questions. Default answers are provided for many questions so that common answers can be selected easily.
vxdiskadmutility, but does not describe its use in detail (refer to the VERITAS Volume Manager User's Guide for information on how to use
vxdiskadd--This utility is used to add standard disks to the Volume Manager.
vxdiskaddleads you through the process of initializing a new disk by displaying information and asking questions.
vxdisk--This is the command-line utility for administering disk devices.
vxdiskis used to define special disk devices, to initialize information stored on disks that the Volume Manager uses to identify and manage disks, and to perform additional special operations. See the
vxdisk(1M) manual page for complete information on how to use
vxdg--This is the command-line utility for operating on disk groups. This can be used to create new disk groups, to add and remove disks from disk groups, and to enable (import) or disable (deport) access to disk groups. See the
vxdg(1M) manual page for complete information on how to use
vxdiskaddutility and most
vxdiskadmoperations can be used only with standard disk devices.
diskaddcommand to do a media format of any disk.
diskaddcommand typically is needed only if the format becomes severely damaged.
vxdiskadd. This section describes how to use
vxdiskadd. For information on how to use
vxdiskadmto initialize a single disk or all disks on a controller, refer to Chapter 13, "Menu Interface Operations," of the VERITAS Volume Manager User's Guide.
You can use
vxdiskadd to initialize a specific disk. For example, to initialize the second disk on the first controller, use the command:
vxdiskadd c0b0t1d0This will examine your disk to determine whether it has been initialized already and will ask questions based on what it finds.
vxdiskaddwill check for disks that can be encapsulated (described in "Using vxdisk for Special Encapsulations"), for disks that have already been added to the Volume Manager, and for a number of less-likely conditions.
vxdiskadd. Ignore these messages. These messages should not appear after the disk has been fully initialized;
vxdiskadddisplays a success message when the initialization completes.
Add or initialize disks Menu: VolumeManager/Disk/AddDisks
Here is the disk selected. Output format: [Device_Name] c0b0t1d0 Continue operation? [y,n,q,?] (default: y)
If the disk is uninitialized, or if you choose to reinitialize the disk, you will be prompted with the following:
You can choose to add this disk to an existing disk group, a new disk group, or leave the disk available for use by future add or replacement operations. To create a new disk group, select a disk group name that does not yet exist. To leave the disk available for future use, specify a disk group name of "none". Which disk group [<group>,none,list,q,?] (default: rootdg)To add this disk to the default group
rootdg, press Return. To leave the disk free as a replacement disk (not yet added to any disk group), enter
none. After this, you will be asked to select a name for the disk in the disk group:
Use a default disk name for the disk? [y,n,q,?] (default: y) yNormally, you should accept the default disk name (unless you prefer to enter a special disk name).
At the following prompt, enter
n to indicate that this disk should not be used as a hot-relocation spare:
Add disk as a spare disk for rootdg? [y,n,q,?] (default: n) nWhen the following screen appears, press Return to continue with the operation:
The selected disks will be added to the disk group rootdg with default disk names. c0b0t1d0 Continue with operation? [y,n,q,?] (default: y) yIf you are certain that there is no data on this disk that needs to be preserved, enter
nat the following prompt:
The following disk device has a valid VTOC, but does not appear to have been initialized for the Volume Manager. If there is data on the disk that should NOT be destroyed you should encapsulate the existing disk partitions as volumes instead of adding the disk as a new disk. Output format: [Device_Name]
Encapsulate this device? [y,n,q,?] (default: y)When
vxdiskadmasks if you want to initialize the disk instead, enter
Instead of encapsulating, initialize? [y,n,q,?] (default: n) yMessages similar to the following should now confirm that disk
c1b0t0d1is being placed under Volume Manager control. You are also given the option of performing surface analysis.
Initializing device c0b0t1d0. Perform surface analysis (highly recommended) [y,n,q,?] (default: y) n Adding disk device c0b0t1d0 to disk group rootdg with disk name disk33.
vxdg [-gwhere the disk group name need only be specified for a disk group other than the default,
For example, to remove
vxdg rmdisk disk02If the disk has subdisks on it when you try to remove it, the following error message is displayed:
vxdg:Disk diskname is used by one or more subdisksUse the
vxdgto remove device assignment. Using the
-koption allows you to remove the disk in spite of the presence of subdisks. For more information, see the
vxdg(1M) manual page.
Once the disk has been removed from its disk group, you can (optionally) remove it from Volume Manager control completely, as follows:
vxdisk rm devicenameFor example, to remove
c1b0t0d0from Volume Manager control, enter:
vxdisk rm c1b0t0d0s7In some cases, you may want to remove a disk on which some subdisks are defined. For example, you may have three disks on one system and you may want to consolidate all of the volumes onto one disk. If you use
vxdiskadmto remove a disk, you can choose to move volumes off that disk. To do this, run
vxdiskadmand select item
disk) from the main menu.
If the disk is used by some subdisks, then a screen resembling the following is displayed:
The following subdisks currently use part of disk disk02: home usrvol Subdisks must be moved from disk02 before it can be removed. Move subdisks to other disks? [y,n,q,?] (default: n)If you choose
y, then all subdisks will be moved off the disk, if possible. Some subdisks may not be movable. The most common reasons why a subdisk may not be movable are:
vxdiskadmcannot move some subdisks, you may need to remove some plexes from some disks to free more space before proceeding with the disk removal operation. See Chapter 4 for information on how to remove volumes and plexes.
c0b0t3d0(attached with the disk name
disk04) from disk group
rootdgand add it to disk group
mktdg, you could use the commands:
vxdg rmdisk disk04 vxdg -g mktdg adddisk mktdg02=c0b0t3d0
vxdiskadmby selecting item
disk) from the main menu, and then selecting item 1 (
vxrelocd,detects and reacts to Volume Manager events that signify the following types of failures:
vxrelocdinforms the system administrator of the failure and which VxVM objects are affected (via electronic mail).
vxrelocdthen determines which subdisks (if any) can be relocated. If relocation is possible,
vxrelocdfinds suitable relocation space and relocates the subdisks. Hot-relocation space is chosen from the disks reserved for hot-relocation in the disk group where the failure occurred; if no spare disks are available or additional space is needed, free space in the same disk group is used. Once the subdisks are relocated, each relocated subdisk is reattached to its plex. Finally,
vxrelocdinitiates any appropriate recovery procedures (such as mirror resynchronization for mirrored volumes or data recovery for RAID-5 volumes). The system administrator is notified of the hot-relocation and recovery actions taken.
If relocation is not possible, the system administrator is notified and no further action is taken. Relocation is not possible in the following cases:
After a successful relocation occurs, you may need to remove and replace the failed disk (see "Replacing Disks"). Depending on the locations of the relocated subdisks, you may choose to move the relocated subdisks elsewhere after hot-relocation occurs (see "Moving Relocated Subdisks").
vxrelocdis running, hot-relocation is turned on. It is advisable to leave hot-relocation turned on so that you can take advantage of this feature if a failure occurs. However, if you choose to disable this feature for some reason (possibly because you do not want the free space on some of your disks used for relocation), you can do so by preventing
vxrelocdfrom starting at system startup time. You can stop hot-relocation at any time by killing the
vxrelocdprocess (this should not be done while a hot-relocation attempt is in progress).
You can make some minor changes to the way
vxrelocd behaves by either editing the
vxrelocd line in the startup file that invokes
/etc/rc2.d/S95vxvm-recover) or killing the existing
vxrelocd process and restarting it with different options. After making changes to the way
vxrelocd is invoked in the startup file, you need to reboot the system so that the changes will go into effect. If you choose to kill and restart the daemon instead, make sure that hot-relocation is not in progress when you kill the
vxrelocd process. You should also restart the daemon immediately so that hot-relocation can take effect if a failure occurs.
You can alter
vxrelocd's behavior in the following ways:
vxrelocdsends electronic mail to
rootwhen failures are detected and relocation actions are performed. You can instruct
vxrelocdto notify additional users by adding the appropriate user names and invoking
vxrelocd root user_name1 user_name2 &
vxrelocdto increase the delay between the recovery of each region of the volume, as follows:
vxrelocd -o slow[=IOdelay] root &
sparecan be used to display information about all of the spare disks available for relocation. The output of this command lists the following type of information:
GROUP DISK DEVICE TAG OFFSET LENGTH FLAGS rootdg disk02 c0b0t2d0s2 c0b0t2d0 0 658007 sIn this case,
disk02is the only disk designated as a spare. The
LENGTHfield indicates how much spare space is currently available on this disk for relocation.
The following commands can also be used to display information about disks that are currently designated as spares:
list-- lists disk information and displays spare disks with a
vxprint-- lists disk and other information and displays spare disks with a
vxprintcommand. or by using the Visual Administrator to look at the status of the disks. You may also see driver error messages on the console or in the system messages file.
root. If a partial disk failure occurs, the mail should identify the failed plexes. For example, if a disk containing mirrored volumes experiences a failure, you might receive a mail message containing information such as:
To: root Subject: Volume Manager failures on host teal Failures have been detected by the VERITAS Volume Manager: failed plexes: home-02 src-02See "Modifying vxrelocd," for information on how to have the mail sent to users other than
You can determine which disk is causing the failures in the above message by running the command:
vxstat -s -ff home-02 src-02This produces output such as:
FAILED TYP NAME READS WRITES sd disk01-04 0 0 sd disk01-06 0 0 sd disk02-03 1 0 sd disk02-04 1 0This display indicates that the failures are on
disk02(and that subdisks
Hot-relocation should automatically relocate the affected subdisks and initiate any necessary recovery procedures. However, if relocation is not possible or the hot-relocation feature is disabled, you may have to investigate the problem and attempt to recover the plexes. Sometimes these errors are caused by cabling failures, so you should check the cables connecting your disks to your system. If there are any obvious problems, correct them and recover the plexes with the command:
vxrecover -b home srcThis command will start a recovery of the failed plexes in the background (the command will return before the operation is done). If an error message appears later, or if the plexes become detached again and there are no obvious cabling failures, the disk should be replaced (see "Replacing Disks").
To: root Subject: Volume Manager failures on host teal Failures have been detected by the VERITAS Volume Manager: failed disks: disk02 failed plexes: home-02 src-02 mkting-01 failing disks: disk02This message indicates that
disk02was detached by a failure. When a disk is detached, I/O cannot get to that disk. The plexes
mkting-01were also detached (probably because of the failure of the disk).
Again, the problem may be a cabling error. If the problem is not a cabling error, the disk should be replaced (see "Replacing Disks").
vxdiskadmand selecting item
disk) from the main menu. If there are any initialized but unadded disks, you will be able to select one of those disks as a replacement.
vxinfoAny volumes that are listed as
Unstartablemust be restored from backup. For example,
home fsgen Started mkting fsgen Unstartable src fsgen Started standvol gen Started rootvol root Started swapvol swap Started
To restart volume
mkting so that it can be restored from backup, use the command:
vxvol -o bg -f start mkting
-o bg option combination causes any plexes to be resynchronized as a background task.
If failures are starting to occur on a disk, but the disk has not yet failed completely, you should replace the disk. This involves two steps:
vxdiskadmand select item
replacement) from the main menu. If there are initialized disks available as replacements, you can specify the disk as part of this operation. Otherwise, you must specify the replacement disk later by selecting item
disk) from the main menu.
When you select a disk to remove for replacement, all volumes that will be affected by the operation are displayed. For example, the following output might be displayed:
The following volumes will lose mirrors as a result of this operation:
No data on these volumes will be lost.
The following volumes are in use, and will be disabled as a result of this operation:
Any applications using these volumes will fail future accesses. These volumes will require restoration from backup.
Are you sure you want do this? [y,n,q,?] (default: n)
If any volumes are likely to be disabled, you should quit from
vxdiskadm and save the volume. Either back up the volume or move the volume off of the disk. To move the volume
mkting to a disk other than
disk02, use the command:
vxassist move mkting !disk02
After the volume is backed up or moved, run
vxdiskadm again and continue to remove the disk for replacement.
After the disk has been removed for replacement, a replacement disk can be specified by selecting item
vxdiskadm's main menu.
During hot-relocation, one of the electronic mail messages sent to
root should be similar to the following:
To: root Subject: Volume Manager failures on host teal
Attempting to relocate subdisk disk02-03 from plex home-02. Dev_offset 0 length 1164 dm_name disk02 da_name c0b0t5d0s2. The available plex home-01 will be used to recover the data.
This message provides information about the characteristics of the subdisk before relocation and can be used to decide where to move the subdisk after relocation. In addition, a message similar to the following should indicate the new location for the relocated subdisk:
To: root Subject: Attempting VxVM relocation on host teal
Volume home Subdisk disk02-03 relocated to disk05-01, but not yet recovered.
Before moving any relocated subdisks, you should fix or replace the disk that experienced the failure (as described in previous sections). Once this is done, you can move a relocated subdisk back to the original disk. For example, you can move the relocated subdisk
disk05-01 back to
disk02 as follows:
vxassist -g rootdg move home !disk05 disk02
vxdiskadd. If using
vxdiskadd, specify a new disk group name when prompted for a disk group. Disk groups can also be created using the operation
init. To create a disk group with the
vxdgutility, use the command:
vxdg init diskgroup diskname=devicename
For example, to create a disk group named
mktdg on device
c1b0t0d0s7, use the command:
vxdg init mktdg mktdg01=c1b0t0d0
The disk device given to
vxdg must have been initialized already with
vxdiskadd. The disk must not already be added to a disk group.
-goption. For example, to create a volume in disk group
mktdg, you could use the command:
vxassist -g mktdg make mktvol 50m
The (block) volume device for this volume is:
/dev/vx/dsk/mktdg/mktvolIn many cases, the disk group does not have to be specified. Most Volume Manager commands use object names specified on the command line to determine the disk group for the operation. For example, a volume can be created on disk
mktdg01without specifying the disk group name:
vxassist make mktvol 50m mktdg01
This works for many commands as long as two disk groups do not have objects with the same name. For example, the Volume Manager allows you to create volumes named
mktvol in both
rootdg and in
mktdg. If you do this, you must add
-g mktdg to any command where you want to manipulate the volume in the
mktdg disk group.
vxdg deport diskgroup
Deporting a disk group does not actually remove the disk group. It disables use of the disk group by the system. However, disks that are in a deported disk group can be reused, reinitialized, or added to other disk groups.
The following steps are used to move a disk group between systems:
vxdg deport diskgroup
vxdg import diskgroup
vxconfigddaemon will be restarted and will also recognize the new disks. If you do not reboot, use the command
vxconfigdso that VxVM also recognizes the disks.
vxrecover -g diskgroup -sb
vxdg:disk group groupname: import failed: Disk is in use by another host
To clear locks on a specific set of devices, use the command:
vxdisk clearimport devicename ...
It is possible to clear the locks during import by using the command:
vxdg -C import
importcommand on systems that really do have dual-ported disks, as clearing the locks allows those disks to be accessed simultaneously and could result in corrupted data.
importoperation normally fails if some disks for the disk group cannot be found among the disk drives attached to the system. If the
importoperation fails, one of the following error messages will display:
vxdg: Disk group groupname: import failed: Disk group has no valid configuration copies
This message indicates a fatal situation, which requires hardware repair or the creation of a new disk group.
vxdg: Disk group groupname: import failed: Disk for disk group not found
This message indicates a recoverable situation.
If some of the disks in the disk group have failed, you can force the disk group to be imported with the command:
vxdg -f import diskgroup
-foption here, as it can cause the same disk group to be imported twice from disjoint sets of disks, causing the disk group to become inconsistent.
vxdiskadm. To deport a disk group via
vxdiskadm, select menu item
group). To import a disk group, select item
vxdiskadmimport operation checks for host import locks and asks if you want to clear any that are found. It also starts volumes in the disk group.
For instance, since every system running the Volume Manager must have a single
rootdg default disk group, importing or deporting
rootdg across systems presents a problem in that there cannot be two
rootdg disk groups on the same system. This problem can be overcome by renaming the
rootdg disk group during the import or deport.
To give a new name to the disk group during import, use the following command:
vxdg [-t] -n newdg_name import diskgroup
-t option is included, the import is temporary and will not persist across reboots. In this case, the stored name of the disk group remains unchanged on its original host, but the disk group will be known as newdg_name to the importing host. If the
-t option is not used, the name change will be permanent.
As with importing, a disk group can be renamed on deport with the following command:
When renaming on deport, the
-h hostname option can be specified to assign a lock to an alternate host. This ensures that the disk group is automatically imported when the alternate host reboots.
The following steps can be used to temporarily move the
rootdg disk group from one host to another (for repair work on the root volume, for instance) and then move it back:
rootdgdisk group to be imported to the other host:
vxdisk -s list
dgname: rootdg dgid: 774226267.1025.tweety
rootdgdisk group as follows:
vxdg -tC -n newdg_name import diskgroup
-tindicates a temporary import name;
-Cclears import locks;
-nspecifies a temporary name for the
rootdgto be imported (so that it does not conflict with the existing
rootdg); and diskgroup is the disk group ID of the disk group being imported (
774226267.1025.tweety, for example).
rootdg, deport it back to its original host as follows:
vxdg -h hostname deport diskgroup
rootdgis being returned (the system's name can be confirmed with the command
rootdgfrom the importing host and returns locks to its original host. The original host will then autoimport its
rootdgon its next reboot.
VxVM allows the administrator to select a range of minor numbers for a specified disk group. This range is used during the creation of a volume, guaranteeing that each volume will have the same minor number across reboots or reconfigurations. If two disk groups have overlapping ranges, an import collision will be detected and some avoidance or renumbering mechanism will then be needed.
A base volume device minor number can be set for a disk group with the command:
vxdg init diskgroup minor=base_minor devicename
Volume device numbers for a disk group will be chosen to have minor numbers starting at this base_minor number. Minor numbers can range up to 131071. A reasonably sized range should be left at the end for temporary device number remappings (in the event that two device numbers still conflict).
minor operand is specified on the
init command line, the Volume Manager chooses a random number of at least 1000 that is a multiple of 1000, and yields a usable range of 1000 device numbers. This default number is chosen such that it does not overlap within a range of 1000 of any currently imported disk groups, and does not overlap any currently allocated volume device numbers.
vxdg(1M) manual page.
vxdiskfor Special Encapsulations
/etc/vfstabentries are modified so that the file systems are mounted on volumes instead.
The normal disk encapsulation procedure requires that some free space be available on the disk for storing Volume Manager identification and configuration information. This free space cannot be included in any other partitions. (See the Online Data Manager overview and installation topic and the
vxencap(1M) manual page for further information.)
In some cases, you may want to encapsulate a disk that does not have any space that can be used for the Volume Manager private region partition. The
vxdisk utility can be used to encapsulate disks that do not have available space. This is done using special types of disk devices, called
nopriv devices, that do not have private regions. To use this, create a partition on the disk device that maps all parts of the disk that you want to access, then add the partition device for that partition with the command:
vxdisk define partition-device type=nopriv
Here, partition-device is the basename of the device in the
/dev/dsk directory. For example, to use partition
3 of disk device
c0b0t4d0, use the command:
vxdisk define c0b0t4d0s3 type=nopriv
To create volumes for other partitions on the disk drive, add the device to a disk group, determine where those partitions reside within the encapsulation partition, then use
vxassist to create a volume with that offset and length.
A major drawback with using these special encapsulation partition devices is that the Volume Manager cannot track changes in the address or controller of the disk. Normally, the Volume Manager uses identifying information stored in the private region on the physical disk to track changes in the location of a physical disk. Since
nopriv devices do not have private regions and thus have no identifying information stored on the physical disk, this cannot occur.
The best use of special encapsulation partition devices is to encapsulate a disk so that the Volume Manager can be used to move space off the disk. When space is made available on the disk, the special partition device can be removed and the disk can then be encapsulated as a standard disk device.
A disk group cannot be formed entirely from
nopriv devices. This is because
nopriv devices do not provide space for storing disk group configuration information. Configuration information must be stored on at least one disk in the disk group.
vxdiskfor RAM Disks
nopriv devices have a special feature to support RAM disks: a volatile option which indicates to the Volume Manager that the device contents do not survive reboots. Volatile devices are handled specially on system startup. If a volume is mirrored, plexes made from volatile devices are always recovered by copying data from non-volatile plexes.
To use a RAM disk, a device node must be created for the disk in the
/dev/rdsk directories (for example,
/dev/rdsk/ramd0). To define the RAM disk device to the Volume Manager, use the command:
vxdisk define ramd0 type=nopriv volatile
Normally, the Volume Manager will not start volumes that are formed entirely from plexes with volatile subdisks. This is because there is no plex that is guaranteed to contain the most recent volume contents.
Sometimes, RAM disks are used in situations where all volume contents are recreated after reboot. In these situations, you can force volumes formed from RAM disks to be started at reboot using the command:
vxvol set startopts=norecov volume_name
This option can be used only with
gen-type volumes. See
vxvol(1M) for more information on the
set operation and the