Thursday, August 29, 2013

How to determine the appropriate page file size for 64-bit versions of Windows

When you set up a 64-bit version of Microsoft Windows Server 2003 or Microsoft Windows XP, the operating system will create a page file that is one and a half times the amount of RAM that is installed in your computer. However, as the amount of RAM in a computer increases, the need for a page file decreases. The following guidelines and methods will help you determine the appropriate page file size for your system.

Comparison of memory and CPU limits in the 32-bit and 64-bit versions of Windows

When you set up a 32-bit version or a 64-bit version of Windows Server 2003 or Windows XP, a page file is created that is one and a half times the amount of RAM that is installed in the computer provided there is sufficient free space on the system hard disk. However, as more RAM is added to a computer, the need for a page file decreases. If you have enough RAM installed in your computer, you may not require a page file at all, unless one is required by a specific application.

The following chart illustrates the amount of RAM and the number of CPUs that can be installed on a computer depending on the operating system that is installed. RAM and CPU limits are much larger in 64-bit operating systems than in 32-bit operating systems.
General memory limits32-bit64-bit
Total virtual address space4 gigabytes (GB)16 terabytes
Virtual address space per 32-bit process2 GB, 3 GB if the system is booted with the /3GB switch2 GB, 4 GB if the application is compiled with the /LARGEADDRESSAWARE switch
Virtual address space per 64-bit processNot applicable8 terabytes
Paged pool470 megabytes (MB)128 GB
Non-paged pool256 MB128 GB
System cache1 GB1 terabyte
Physical memory and CPU limits32-bit64-bit
Windows XP Professional4 GB / 1-2 CPUs128 GB / 1-2 CPUs
Windows Server 2003, Standard Edition4 GB / 1-4 CPUs32 GB / 1-4 CPUs
Windows Server 2003, Enterprise Edition64 GB / 1-8 CPUs1 terabyte / 1-8 CPUs
Windows Server 2003, Datacenter Edition64 GB / 1-32 CPUs1 terabyte / 1-64 CPUs
Windows Server 2003 SP1, Enterprise Edition64 GB / 1-8 CPUs1 terabyte / 1-8 CPUs
Windows Server 2003 SP1, Datacenter Edition64 GB / 1-32 CPUs1 terabyte / 1-64 CPUs
There is no specific recommendation for page file size. Your requirements will be based on the hardware and software that you use and the load that you put on the computer. To monitor page file usage and requirements, run System Monitor, and gather a log during typical usage conditions. Focus on the following counters.
Counter thresholdSuggested value
Memory\\Available BytesNo less than 4 MB
Memory\\Pages Input/secNo more than 10 pages
Paging File\\% UsageNo more than 70 percent
Paging File\\% Usage PeakNo more than 70 percent
Process\\Page File Bytes PeakNot applicable
Note Page file use should be tracked periodically. When you increase the use or the load on the system, you generally increase the demand for virtual address space and page file space.

The Pages/sec counter

When you use System Monitor, the values that are returned by the Pages/sec counter for the Memory performance object may be more than you expect. These values may not be related to either paging file activity or cache activity. Instead, these values may be caused by an application that is sequentially reading a memory-mapped file.

The Pages/sec counter counts the number of pages that are read from the disk or that are written to the disk to resolve memory references to pages. These pages were not in memory at the time of the reference. The Pages/sec value is the sum of the Pages Input/sec and Pages Output/sec counter values. The Pages/sec counter includes paging traffic on behalf of the system cache to access file data for applications. 

Pages/sec is the primary counter to watch if you are concerned about excessive memory pressure (thrashing) and the excessive paging that may result.

However, the Pages/sec counter also accounts for other activity, such as the sequential reading of memory-mapped files that are either cached or not. Typically, the Pages/sec counter is counting other activity when you see the following:
  • A high value for the Memory: Pages/sec counter.
  • An average value, relative to the system being monitored, or a high value for the Memory: Available Bytes counter.
  • An average or small value for the Paging File: % Usage counter.
  • For a non-cached memory-mapped file, you also see average or low cache activity. (Low cache activity is also known as cache fault activity.)
Therefore, a high value for the Memory: Pages/sec counter does not necessarily indicate memory pressure or a System Monitor reporting error. To gain an accurate reading of your system, you must also monitor other counters.

How to calculate page file size

Use one or more of the following methods to help you calculate page file size.

Method 1: Use performance logs to understand the paging activity on your computer

  1. Click Start, point to Administrative Tools, and then click Performance.
  2. Expand Performance Logs and Alerts, click Counter Logs, right-click the blank space in the right-pane, and then clickNew Log Settings.
  3. In the Name box, type a name for the log, and then click OK.
  4. On the General tab, click Add Counters.
  5. Click Use local computer counters.
  6. In the Performance object list, click Paging File.
  7. Click Select counters from list, click % Usage, and then click Add.
  8. In the Performance object list, click Memory.
  9. In Select counters from list, click Available Bytes, and then click Add.
  10. In Select counters from list, click Pages Input/sec, click Add, and then click Close.
  11. Click OK.
Use the log that you collect during typical computer use to understand the paging activity on your computer. Then, adjust the page file size accordingly.

Method 2: Use the Page File Bytes Peak counter to calculate page file size

  1. Click Start, point to Administrative Tools, and then click Performance.
  2. Click System Monitor.
  3. In the right pane, click + (the Add button).
  4. Click Use local computer counters.
  5. In the Performance object list, click Process.
  6. Click Select counters from list, click Page File Bytes Peak, click Add, and then click Close.
  7. Let the counter run during typical use of your computer.
  8. Note the maximum value for the Page File Bytes Peak counter, and then multiply the value by 0.70. The sum of the equation is the size to set for your page file.

Method 3: Calculate the minimum and maximum page file size

To determine the approximate minimum page file that is required by your system, calculate the sum of peak private bytes that are used by each process on the system. Then, subtract the amount of memory on the system. 

To determine the approximate maximum page file space that is required for your system, calculate the sum of peak private bytes that are used by each process on the system. Then, add a margin of additional space. Do not subtract the amount of memory on the system. The size of the additional margin can be adjusted based on your confidence in the snapshot data that is used to estimate page file requirements. 

Note This estimate is accurate only if the snapshot of data that is used to make the calculations is accurate.

Page file input/output rates

To avoid overloading the system or other disks with page input/output (I/O) activity, use the following guidelines when you set up the page file on your computer:
  • If the page I/O (real disk I/O) rate is more than 10 pages per second, we recommend that you do not put the page file where the I/O activity occurs on the system disk. When the page I/O rate is 10 pages per second or more, we recommend that you dedicate a separate hard disk for paging.
  • If the page I/O rate to a particular disk that is used for paging is more than 60 disk I/O operations per second, use more than one dedicated page hard disk to obtain better performance. To do this, use multiple non-striped disks for paging, or use raid 0 striped disks for paging. Dedicate approximately one I/O hard disk to paging for every 60 pages per second of I/O activity.

    For example, if a system is averaging 150 pages of I/O activity per second, use three individual hard disks, or a three-disk raid 0 stripe set for the page file.

    Note These estimates are for hard disks that run at 7200 revolutions per minute (rpm). If you use a hard disk that runs faster, the I/O rate a disk can handle for page I/O will increase.

    Note If peak performance is critical to your system, use peak I/O rates instead of average I/O rates for these calculations.
Important Supportability Information: This article is specifically for computers that do not need kernel mode or full memory dump analysis. For business-critical servers where business processes require to server to capture physical memory dumps for analysis, the traditional model of the page file should be at least the size of physical ram plus 1 MB, or 1.5 times the default physical RAM. This makes sure that the free disk space of the operating system partition is large enough to hold the OS, hotfixes, installed applications, installed services, a dump file, and the page file. On a server that has 32 GB of memory, drive C may have to be at least 86 GB to 90 GB. This is 32 GB for memory dump, 48 GB for the page file (1.5 times the physical memory), 4 GB for the operating system, and 2 to 4 GB for the applications, the installed services, the temp files, and so on. Remember that a driver or kernel mode service leak could consume all free physical RAM. Therefore, a Windows Server 2003 x64 SP1-based server in 64-bit mode with 32GB of RAM could have a 32 GB kernel memory dump file, where you would expect only a 1 to 2 GB dump file in 32-bit mode. This behavior occurs because of the greatly increased memory pools.

Page File How big is Yours

The Question around ideal sizing of a Page File has been a topic of conversation that Customers have been asking us recently.
I am going to make a stab in this Blog to put together some of the current thinking around this as I think some of the documentation has been around for some time and can be a source for confusion. I have taken as my resources learned colleagues internal to Microsoft who have helped me put this blog together.
SIZING
The sizing of a Page File should not just be arbitrarily set. Some thought and consideration should be put into this. The suggestions I am making below can be considered applicable to  2003 and 2008 platforms 32 bit and 64 bit environment. Although special considerations should be given to 64 bit environment, which is referenced in the kb article list at the bottom of this post.
By default if you have less than 2gb of memory on a machine it will set the page file size to 1.5 times the amount of memory in the box. If the amount of memory exceeds 2gb then by default the page file size will be set  to between 2046-4092MB . This is expected behaviour  However this may not always be the optimum setting.
Perfmon Counters
One of the best recommendations for deciding the pagefile setting is to monitor the following counters in perfmon:-
Counter threshold Suggested value
Memory\\Available Bytes No less than 4 MB
Memory\\Pages Input/sec No more than 10 pages
Paging File\\% Usage No more than 70 percent
Paging File\\% Usage Peak No more than 70 percent
Process\\Page File Bytes Peak Not applicable
Eg: If the Pagefile %usage shows a value of 70% or higher then we need to consider increasing the pagefile or creating another pagefile on a different disk.
In addition the ideal page file size can be determined by looking at Task Manager / Process Explorer:
· ‘Commit Charge Limit’ - Virtual memory without increasing the page file(RAM + total size of all paging files)
· ‘Commit Charge Peak’ - The peak amount of page file space since the system booted that would have been needed if the system had to page out all private committed virtual memory. (Indicates how close to the limit it is)
The aim of this  is to make sure that the Commit charge limit is always larger than the commit charge peak, and the peak never gets close to 70% of the limit.
Process explorer – System Information window provides more information as ‘Percentage of the peak as compared to the limit’ and ‘’the current usage as compared to the limit’ – This is discussed in Windows Internals page 446,447
One other thing to consider is that the Initial and Max pagefile values are the same – this will prevent pagefile expansion which can be a performance killer and can lead to pagefile fragmentation.
Performance can also be increased by splitting the pagefile across disks. -http://support.microsoft.com/?id=197379
Possible suggestion is to start with a 4GB Page File on the root with an additional page file on the data disks and monitor the counters above, tuning accordingly.
Systems with Large Amounts Of Memory
However on systems with larger amount of memory if you have for example a setup such as below
On a  32GB memory  x64 bit system  it would  be advisable  to manually set 8GB pagefile on the Windows drive (min max both set to 8GB). This would then provide plenty of space for paging and plus allows a kernel only dump to be produced (in all but the worse case scenarios)
Mark Russinovich touches upon the subject of pagefile sizing for modern systems
Plus he also mentions;
“The maximum is either three times the size of RAM or 4GB, whichever is larger.”
[although it states that is for Vista  and Server 2008, but seems to apply to Server 2003 as well]
What you must consider is that it is essentially impractical to create a page file on a system which has 32gb of memory to be  32gb x 1.5. Consider the following factors;
1. The amount of time it will take to create it
2.  What practical use will it be
3. The amount of disk space you need to ensure you have on the boot partition
3. How will you transport this huge file to be analysed

How to remove the last disk in a disk group and use it in another disk group in VERITAS Volume Manager 2.x

In order to remove the last disk in a disk group to be used in another disk group, the volumes must first be removed. 

NOTE: The procedure outlined will result in data loss on the disk that is being removed.   

# vxprint -g datadg -ht
DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg datadg       default      default  122000   955057397.1251.kt

dm datadg01     c1t11d0s2    sliced   1089     8921650  -

v  datavol      fsgen        ENABLED  ACTIVE   2097152  SELECT    -
pl datavol-01   datavol      ENABLED  ACTIVE   2097160  CONCAT    -        RW
sd datadg01-01  datavol-01   datadg01 0        2097160  0         c1t11d0  ENA

# vxdisk list
DEVICE       TYPE      DISK         GROUP        STATUS
c0t0d0s2     sliced    rootdisk     rootdg       online
c1t8d0s2     sliced    test04       testdg       online
c1t9d0s2     sliced    test01       testdg       online
c1t10d0s2    sliced    test02       testdg       online
c1t11d0s2    sliced    datadg01     datadg       online

# vxedit -g datadg -rf rm datavol

# vxprint -g datadg -ht
DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg datadg       default      default  122000   955057397.1251.kt

dm datadg01     c1t11d0s2    sliced   1089     8921650  -

NOTE: Trying to remove the disk at this point will result in an error since it is the last disk in the disk group.

# vxdiskadm


Volume Manager Support Operations
Menu: VolumeManager/Disk

1      Add or initialize one or more disks
2      Encapsulate one or more disks
3      Remove a disk
4      Remove a disk for replacement
5      Replace a failed or removed disk
6      Mirror volumes on a disk
7      Move volumes from a disk
8      Enable access to (import) a disk group
9      Remove access to (deport) a disk group
10     Enable (online) a disk device
11     Disable (offline) a disk device
12     Mark a disk as a spare for a disk group
13     Turn off the spare flag on a disk
list   List disk information


?      Display help about menu
??     Display help about the menuing system
q      Exit from menus

Select an operation to perform: 3

Remove a disk
Menu: VolumeManager/Disk/RemoveDisk

Use this operation to remove a disk from a disk group. This operation takes a disk name as input. This is the same name that was given to the disk when the disk was added to the disk group.

Enter disk name [<disk>,list,q,?] list

Disk group: rootdg

DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE

dm rootdisk     c0t0d0s2     sliced   2015     16839647 -

Disk group: datadg

DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE

dm datadg01     c1t11d0s2    sliced   1089     8921650  -

Disk group: testdg

DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE

dm test01       c1t9d0s2     sliced   1089     8921650  -
dm test02       c1t10d0s2    sliced   1089     8921650  -
dm test04       c1t8d0s2     sliced   1089     8921650  -

Enter disk name [<disk>,list,q,?] datadg01

Disk datadg01 cannot be removed from disk group datadg because it is the last disk in the disk group. It is advisable to remove the disk group instead. To remove the disk group, quit this operation and select "Disable access to (deport) a disk group" from the main disk operations menu.

Remove a different disk? [y,n,q,?] (default: n) n

So, in order to remove the disk from the disk group to be used by another disk group, access to the diskgroup must be disabled.  (The disk group must be deported.)

Volume Manager Support Operations
Menu: VolumeManager/Disk

1      Add or initialize one or more disks
2      Encapsulate one or more disks
3      Remove a disk
4      Remove a disk for replacement
5      Replace a failed or removed disk
6      Mirror volumes on a disk
7      Move volumes from a disk
8      Enable access to (import) a disk group
9      Remove access to (deport) a disk group
10     Enable (online) a disk device
11     Disable (offline) a disk device
12     Mark a disk as a spare for a disk group
13     Turn off the spare flag on a disk
list   List disk information


?      Display help about menu
??     Display help about the menuing system
q      Exit from menus

Select an operation to perform: 9

Remove access to (deport) a disk group Menu: VolumeManager/Disk/DeportDiskGroup

Use this menu operation to remove access to a disk group that is currently enabled (imported) by this system. Deport a disk group to move the disks in a disk group to another system. Also, deport a disk group to use all of the disks remaining in a disk group.

A prompt will ask for the name of a disk group. It will also ask if the disks should be disabled (offlined). For removable disk devices on some systems, it is important to disable all access to the disk before removing the disk.

Enter name of disk group [<group>,list,q,?] (default: list) list

GROUP        DISK/VOLUME         DEVICE/STATE LENGTH

datadg       disk   datadg01     c1t11d0s2    8921650

testdg       disk   test01       c1t9d0s2     8921650
testdg       disk   test02       c1t10d0s2    8921650
testdg       disk   test04       c1t8d0s2     8921650
testdg       volume testvol      ENABLED      2097160
testdg       volume vol01        ENABLED      2097152


Enter name of disk group [<group>,list,q,?] (default: list) datadg

The requested operation is to disable access to the removable disk group named datadg. This disk group is stored on the following disks:

       datadg01 on device c1t11d0s2

Access to these disks can be disabled (also known as "offline"). This may be necessary to prevent errors if any of the disks are actually removed from the system.

Disable (offline) the indicated disks? [y,n,q,?] (default: n) y

Continue with operation? [y,n,q,?] (default: y) y

Removal of disk group datadg was successful.

Disable another disk group? [y,n,q,?] (default: n) n

# vxprint -ht
Disk group: rootdg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg rootdg       default      default  0        936308190.1025.dot

dm rootdisk     c0t0d0s2     sliced   2015     16839647 -

sd rootdiskPriv -            rootdisk 16536240 2015     PRIVATE   c0t0d0   ENA

v  rootvol      root         ENABLED  ACTIVE   16536240 ROUND     -
pl rootvol-01   rootvol      ENABLED  ACTIVE   16536240 CONCAT    -        RW
sd rootdisk-B0  rootvol-01   rootdisk 16536239 1        0         c0t0d0   ENA
sd rootdisk-02  rootvol-01   rootdisk 0        16536239 1         c0t0d0   ENA

v  swapvol      swap         ENABLED  ACTIVE   301392   ROUND     -
pl swapvol-01   swapvol      ENABLED  ACTIVE   301392   CONCAT    -        RW
sd rootdisk-01  swapvol-01   rootdisk 16538255 301392   0         c0t0d0   ENA

Disk group: testdg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX

PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg testdg       default      default  10000    952101973.1171.kt

dm test01       c1t9d0s2     sliced   1089     8921650  -
dm test02       c1t10d0s2    sliced   1089     8921650  -
dm test04       c1t8d0s2     sliced   1089     8921650  -

v  testvol      fsgen        ENABLED  ACTIVE   2097160  SELECT    -
pl vol01-02     testvol      ENABLED  ACTIVE   2097160  CONCAT    -        RW
sd test01-01    vol01-02     test01   0        2097160  0         c1t9d0   ENA

v  vol01        fsgen        ENABLED  ACTIVE   2097152  SELECT    -
pl vol01-01     vol01        ENABLED  ACTIVE   2097160  CONCAT    -        RW
sd test04-01    vol01-01     test04   0        2097160  0         c1t8d0   ENA

# vxdisk list
DEVICE       TYPE      DISK         GROUP        STATUS
c0t0d0s2     sliced    rootdisk     rootdg       online
c1t8d0s2     sliced    test04       testdg       online
c1t9d0s2     sliced    test01       testdg       online
c1t10d0s2    sliced    test02       testdg       online
c1t11d0s2    sliced    -            -            offline

The disk can now be used in another disk group as follows:

# vxdiskadm


Volume Manager Support Operations
Menu: VolumeManager/Disk

1      Add or initialize one or more disks
2      Encapsulate one or more disks
3      Remove a disk
4      Remove a disk for replacement
5      Replace a failed or removed disk
6      Mirror volumes on a disk
7      Move volumes from a disk
8      Enable access to (import) a disk group
9      Remove access to (deport) a disk group
10     Enable (online) a disk device
11     Disable (offline) a disk device
12     Mark a disk as a spare for a disk group
13     Turn off the spare flag on a disk
list   List disk information


?      Display help about menu
??     Display help about the menuing system
q      Exit from menus

Select an operation to perform: 1

Add or initialize disks
Menu: VolumeManager/Disk/AddDisks

Use this operation to add one or more disks to a disk group. It is possible to add the selected disks to an existing disk group or to a new disk group that will be created as a part of the operation. The selected disks may also be added to a disk group as spares. The selected disks may also be initialized without adding them to a disk group leaving the disks available for use as replacement disks.

More than one disk or pattern may be entered at the prompt. Here are some disk selection examples:

 all:          all disks
 c3 c4t2:      all disks on both controller 3 and controller 4, target 2
 c3t4d2:       a single disk

Select disk devices to add:                                          
[<pattern-list>,all,list,q,?] list

DEVICE       DISK         GROUP        STATUS
c0t0d0       rootdisk     rootdg       online
c1t8d0       test04       testdg       online
c1t9d0       test01       testdg       online
c1t10d0      test02       testdg       online
c1t11d0      -            -            offline

Select disk devices to add:                                          
[<pattern-list>,all,list,q,?] c1t11d0

Here is the disk selected.  Output format: [Device_Name]

 c1t11d0

Continue operation? [y,n,q,?] (default: y) y

This disk can be added to an existing disk group, a new disk group, or left available for use by future add or replacement operations.  To create a new disk group, select a disk group name that does not yet exist. To leave the disk available for future use, specify a disk group name of "none".

Which disk group [<group>,none,list,q,?] (default: rootdg) testdg

Use a default disk name for the disk? [y,n,q,?] (default: y) n

Add disk as a spare disk for testdg? [y,n,q,?] (default: n) n

The selected disks will be added to the disk group testdg with disk names that you will specify interactively.

 c1t11d0

Continue with operation? [y,n,q,?] (default: y) y

Access is disabled for the following disk.
 Output format: [Device_Name]

 c1t11d0

Enable access for this device? [y,n,q,?] (default: y) y

This disk is listed as added to a disk group. The disk group is not currently enabled (imported). If importing from the disk group, quit from this operation and select "Enable access to (import) a disk group" from the main disk operations menu.
Output format: [Device_Name,Disk_Access_Name,Disk_Group]

 [c1t11d0,c1t11d0s2,datadg]

Use this device? [y,n,q,?] (default: y) y

The following disk selected for use appears to already have been initialized for the volume manager. If the disk has already been initialized for the volume manager, then there is no need to reinitialize the disk device.
Output format: [Device_Name]

 c1t11d0

Reinitialize this device? [y,n,q,?] (default: y) y

 Initializing device c1t11d0.

Enter disk name for c1t11d0 [<name>,q,?] (default: testdg01) test03

 Adding disk device c1t11d0 to disk group testdg with disk
 name test03.

Add or initialize other disks? [y,n,q,?] (default: n) n

# vxprint -ht
Disk group: rootdg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg rootdg       default      default  0        936308190.1025.dot

dm rootdisk     c0t0d0s2     sliced   2015     16839647 -

sd rootdiskPriv -            rootdisk 16536240 2015     PRIVATE   c0t0d0   ENA

v  rootvol      root         ENABLED  ACTIVE   16536240 ROUND     -
pl rootvol-01   rootvol      ENABLED  ACTIVE   16536240 CONCAT    -        RW
sd rootdisk-B0  rootvol-01   rootdisk 16536239 1        0         c0t0d0   ENA
sd rootdisk-02  rootvol-01   rootdisk 0        16536239 1         c0t0d0   ENA

v  swapvol      swap         ENABLED  ACTIVE   301392   ROUND     -
pl swapvol-01   swapvol      ENABLED  ACTIVE   301392   CONCAT    -        RW
sd rootdisk-01  swapvol-01   rootdisk 16538255 301392   0         c0t0d0   ENA

Disk group: testdg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

dg testdg       default      default  10000    952101973.1171.kt

dm test01       c1t9d0s2     sliced   1089     8921650  -
dm test02       c1t10d0s2    sliced   1089     8921650  -
dm test03       c1t11d0s2    sliced   1089     8921650  -
dm test04       c1t8d0s2     sliced   1089     8921650  -

v  testvol      fsgen        ENABLED  ACTIVE   2097160  SELECT    -
pl vol01-02     testvol      ENABLED  ACTIVE   2097160  CONCAT    -        RW
sd test01-01    vol01-02     test01   0        2097160  0         c1t9d0   ENA

v  vol01        fsgen        ENABLED  ACTIVE   2097152  SELECT    -
pl vol01-01     vol01        ENABLED  ACTIVE   2097160  CONCAT    -        RW
sd test04-01    vol01-01     test04   0        2097160  0         c1t8d0   ENA

# vxdisk list
DEVICE       TYPE      DISK         GROUP        STATUS
c0t0d0s2     sliced    rootdisk     rootdg       online
c1t8d0s2     sliced    test04       testdg       online
c1t9d0s2     sliced    test01       testdg       online
c1t10d0s2    sliced    test02       testdg       online
c1t11d0s2    sliced    test03       testdg       online

To do this procedure from the command line:

# vxdisk list
DEVICE       TYPE      DISK         GROUP        STATUS
c0t0d0s2     sliced    rootdisk     rootdg       online
c1t8d0s2     sliced    testdg03     testdg       online
c1t9d0s2     sliced    testdg04     testdg       online
c1t10d0s2    sliced    testdg01     testdg       online
c1t11d0s2    sliced    datadg01     datadg       online

# vxdg deport datadg

# vxdisk list
DEVICE       TYPE      DISK         GROUP        STATUS
c0t0d0s2     sliced    rootdisk     rootdg       online
c1t8d0s2     sliced    testdg03     testdg       online
c1t9d0s2     sliced    testdg04     testdg       online
c1t10d0s2    sliced    testdg01     testdg       online
c1t11d0s2    sliced    -            -            online

# vxdiskunsetup c1t11d0  

# vxdisk list
DEVICE       TYPE      DISK         GROUP        STATUS
c0t0d0s2     sliced    rootdisk     rootdg       online
c1t8d0s2     sliced    testdg03     testdg       online
c1t9d0s2     sliced    testdg04     testdg       online
c1t10d0s2    sliced    testdg01     testdg       online
c1t11d0s2    sliced    -            -            error

# vxdisksetup -i c1t11d0  
# vxdisk list

DEVICE       TYPE      DISK         GROUP        STATUS
c0t0d0s2     sliced    rootdisk     rootdg       online
c1t8d0s2     sliced    testdg03     testdg       online
c1t9d0s2     sliced    testdg04     testdg       online
c1t10d0s2    sliced    testdg01     testdg       online
c1t11d0s2    sliced    -            -            online

# vxdg -g testdg adddisk testdg02=c1t11d0s2

# vxdisk list
DEVICE       TYPE      DISK         GROUP        STATUS
c0t0d0s2     sliced    rootdisk     rootdg       online
c1t8d0s2     sliced    testdg03     testdg       online
c1t9d0s2     sliced    testdg04     testdg       online
c1t10d0s2    sliced    testdg01     testdg       online
c1t11d0s2    sliced    testdg02     testdg       online