Got an issue where the hastatus summary showed the diskgroup as offline failed;
-- SYSTEM STATE
-- System State Frozen
A eupr0001 RUNNING 0
A eupr0002 RUNNING 0
-- GROUP STATE
-- Group System Probed AutoDisabled State
B commonsg eupr0001 Y N ONLINE
B commonsg eupr0002 Y N ONLINE
B nsr01_sg eupr0001 Y N OFFLINE|FAULTED
B nsr01_sg eupr0002 Y N ONLINE
-- RESOURCES FAILED
-- Group Type Resource System
C nsr01_sg Application NetWorker eup
so tried the following;
hares -display
For each resource that is faulted run:
hares -clear resource-name -sys faulted-system
so in this case
hares -clear NetWorker -sys eupr0001
If all of these clear, then run hastatus -summary and make sure that these are clear. You should then see the FAULTED removed, and just be left with the ONLINE status
-------------------------------------------------------------------------
nsr01_proxy eupr0001 ONLINE
nsr01_proxy eupr0002 ONLINE
^Ceupr0001> hastatus -sum
-- SYSTEM STATE
-- System State Frozen
A eupr0001 RUNNING 0
A eupr0002 RUNNING 0
-- GROUP STATE
-- Group System Probed AutoDisabled State
B commonsg eupr0001 Y N ONLINE
B commonsg eupr0002 Y N ONLINE
B nsr01_sg eupr0001 Y N OFFLINE
B nsr01_sg eupr0002 Y N ONLINE
eupr0001>
If some don't clear you MAY be able to clear them on the group level. Only do this as last resort:
hagrp -disableresources groupname
hagrp -flush group -sys sysname
hagrp -enableresources groupname
To get a group to go online:
hagrp -online group -sys desired-system
However, if this was not the issue (because despite doing flushes, clears and onlines, vcs was stating there was not a problem) - it could be something else, like licenses - I did an nsrwatch on the server and could see an issue with networker licenses;
Server: ukprbklas001-mn.emea.abnamro-net.com Wed May 20 19:34:30 2009
Up since: Mon Jul 7 18:43:09 2008 Version: NetWorker nw_7_3_2_jumbo.Build.394 Base enabler disabled
Saves: 0 session(s) Recovers: 0 session(s)
Device type volume
/tmp/fd1 adv_file D.001
1/_AF_readonly adv_file D.001.RO read only
/tmp/fd2 adv_file W.001
2/_AF_readonly adv_file W.001.RO read only
/tmp/fd3 adv_file ukprbklas001.001
3/_AF_readonly adv_file ukprbklas001.001.RO read only
Sessions:
Messages:
Sat 00:13:00 registration info event: Server is disabled: Install base enabler
Sun 00:13:00 registration info: License enabler #none (NetWorker/10 Eval) has expired.
Sun 00:13:00 registration info event: Server is disabled: Install base enabler
Mon 00:13:00 registration info: License enabler #none (NetWorker/10 Eval) has expired.
Mon 00:13:00 registration info event: Server is disabled: Install base enabler
Tue 00:13:00 registration info: License enabler #none (NetWorker/10 Eval) has expired.
Tue 00:13:00 registration info event: Server is disabled: Install base enabler
Wed 00:13:00 registration info: License enabler #none (NetWorker/10 Eval) has expired.
Wed 00:13:00 registration info event: Server is disabled: Install base enabler
Thu 00:13:00 registration info: License enabler #none (NetWorker/10 Eval) has expired.
Thu 00:13:00 registration info event: Server is disabled: Install base enabler
Fri 00:13:00 registration info: License enabler #none (NetWorker/10 Eval) has expired.
Fri 00:13:00 registration info event: Server is disabled: Install base enabler
Sat 00:13:00 registration info: License enabler #none (NetWorker/10 Eval) has expired.
Sat 00:13:00 registration info event: Server is disabled: Install base enabler
Sun 00:13:00 registration info: License enabler #none (NetWorker/10 Eval) has expired.
Sun 00:13:00 registration info event: Server is disabled: Install base enabler
Mon 00:13:00 registration info: License enabler #none (NetWorker/10 Eval) has expired.
Mon 00:13:00 registration info event: Server is disabled: Install base enabler
Tue 00:13:00 registration info: License enabler #none (NetWorker/10 Eval) has expired.
Tue 00:13:00 registration info event: Server is disabled: Install base enabler
Wed 00:13:00 registration info: License enabler #none (NetWorker/10 Eval) has expired.
Wed 00:13:00 registration info event: Server is disabled: Install base enabler
Pending:
Mon 18:43:08 registration info: Server is disabled: Install base enabler
Showing posts with label vcs. Show all posts
Showing posts with label vcs. Show all posts
Wednesday, 20 May 2009
Tuesday, 17 March 2009
VCS cheat sheet
Veritas Cluster Cheat sheet
LLT and GAB Commands | Port Membership | Daemons | Log Files | Dynamic Configuration | Users | Resources | Resource Agents | Service Groups | Clusters | Cluster Status | System Operations | Sevice Group Operations | Resource Operations | Agent Operations | Starting and Stopping
LLT and GRAB
VCS uses two components, LLT and GAB to share data over the private networks among systems.
These components provide the performance and reliability required by VCS.
LLT LLT (Low Latency Transport) provides fast, kernel-to-kernel comms and monitors network connections. The system admin configures the LLT by creating a configuration file (llttab) that describes the systems in the cluster and private network links among them. The LLT runs in layer 2 of the network stack
GAB GAB (Group membership and Atomic Broadcast) provides the global message order required to maintain a synchronised state among the systems, and monitors disk comms such as that required by the VCS heartbeat utility. The system admin configures GAB driver by creating a configuration file ( gabtab).
LLT and GAB files
/etc/llthosts
The file is a database, containing one entry per system, that links the LLT system ID with the hosts name. The file is identical on each server in the cluster.
/etc/llttab
The file contains information that is derived during installation and is used by the utility lltconfig.
/etc/gabtab
The file contains the information needed to configure the GAB driver. This file is used by the gabconfig utility.
/etc/VRTSvcs/conf/config/main.cf
The VCS configuration file. The file contains the information that defines the cluster and its systems.
Gabtab Entries
/sbin/gabdiskconf - i /dev/dsk/c1t2d0s2 -s 16 -S 1123
/sbin/gabdiskconf - i /dev/dsk/c1t2d0s2 -s 144 -S 1124
/sbin/gabdiskhb -a /dev/dsk/c1t2d0s2 -s 16 -p a -s 1123
/sbin/gabdiskhb -a /dev/dsk/c1t2d0s2 -s 144 -p h -s 1124
/sbin/gabconfig -c -n2
gabdiskconf
-i Initialises the disk region
-s Start Block
-S Signature
gabdiskhb (heartbeat disks)
-a Add a gab disk heartbeat resource
-s Start Block
-p Port
-S Signature
gabconfig
-c Configure the driver for use
-n Number of systems in the cluster.
LLT and GAB Commands
Verifying that links are active for LLT lltstat -n
verbose output of the lltstat command lltstat -nvv | more
open ports for LLT lltstat -p
display the values of LLT configuration directives lltstat -c
lists information about each configured LLT link lltstat -l
List all MAC addresses in the cluster lltconfig -a list
stop the LLT running lltconfig -U
start the LLT lltconfig -c
verify that GAB is operating
gabconfig -a
Note: port a indicates that GAB is communicating, port h indicates that VCS is started
stop GAB running gabconfig -U
start the GAB gabconfig -c -n
override the seed values in the gabtab file gabconfig -c -x
GAB Port Memberbership
List Membership
gabconfig -a
Unregister port f /opt/VRTS/bin/fsclustadm cfsdeinit
Port Function a gab driver
b I/O fencing (designed to guarantee data integrity)
d ODM (Oracle Disk Manager)
f CFS (Cluster File System)
h VCS (VERITAS Cluster Server: high availability daemon)
o VCSMM driver (kernel module needed for Oracle and VCS interface)
q QuickLog daemon
v CVM (Cluster Volume Manager)
w vxconfigd (module for cvm)
Cluster daemons
High Availability Daemon had
Companion Daemon hashadow
Resource Agent daemonAgent
Web Console cluster managerment daemon CmdServer
Cluster Log Files
Log Directory /var/VRTSvcs/log
primary log file (engine log file) /var/VRTSvcs/log/engine_A.log
Starting and Stopping the cluster
"-stale" instructs the engine to treat the local config as stale
"-force" instructs the engine to treat a stale config as a valid one
hastart [-stale|-force]
Bring the cluster into running mode from a stale state using the configuration file from a particular server
hasys -force
stop the cluster on the local server but leave the application/s running, do not failover the application/s hastop -local
stop cluster on local server but evacuate (failover) the application/s to another node within the cluster hastop -local -evacuate
stop the cluster on all nodes but leave the application/s running
hastop -all -force
Cluster Status
display cluster summary hastatus -summary
continually monitor cluster hastatus
verify the cluster is operating hasys -display
Cluster Details
information about a cluster haclus -display
value for a specific cluster attribute haclus -value
modify a cluster attribute haclus -modify
Enable LinkMonitoring haclus -enable LinkMonitoring
Disable LinkMonitoring haclus -disable LinkMonitoring
Users
add a user hauser -add
modify a user hauser -update
delete a user hauser -delete
display all users hauser -display
System Operations
add a system to the cluster hasys -add
delete a system from the cluster hasys -delete
Modify a system attributes hasys -modify
list a system state hasys -state
Force a system to start hasys -force
Display the systems attributes hasys -display [-sys]
List all the systems in the cluster hasys -list
Change the load attribute of a system hasys -load
Display the value of a systems nodeid (/etc/llthosts) hasys -nodeid
Freeze a system (No offlining system, No groups onlining)
hasys -freeze [-persistent][-evacuate]
Note: main.cf must be in write mode
Unfreeze a system ( reenable groups and resource back online)
hasys -unfreeze [-persistent]
Note: main.cf must be in write mode
Dynamic Configuration
The VCS configuration must be in read/write mode in order to make changes. When in read/write mode the
configuration becomes stale, a .stale file is created in $VCS_CONF/conf/config. When the configuration is put
back into read only mode the .stale file is removed.
Change configuration to read/write mode haconf -makerw
Change configuration to read-only mode haconf -dump -makero
Check what mode cluster is running in
haclus -display |grep -i 'readonly'
0 = write mode
1 = read only mode
Check the configuration file
hacf -verify /etc/VRTS/conf/config
Note: you can point to any directory as long as it has main.cf and types.cf
convert a main.cf file into cluster commands hacf -cftocmd /etc/VRTS/conf/config -dest /tmp
convert a command file into a main.cf file
hacf -cmdtocf /tmp -dest /etc/VRTS/conf/config
Service Groups
add a service group haconf -makerw
hagrp -add groupw
hagrp -modify groupw SystemList sun1 1 sun2 2
hagrp -autoenable groupw -sys sun1
haconf -dump -makero
delete a service group haconf -makerw
hagrp -delete groupw
haconf -dump -makero
change a service group
haconf -makerw
hagrp -modify groupw SystemList sun1 1 sun2 2 sun3 3
haconf -dump -makero
Note: use the "hagrp -display" to list attributes
list the service groups hagrp -list
list the groups dependencies hagrp -dep
list the parameters of a group hagrp -display
display a service group's resource hagrp -resources
display the current state of the service group hagrp -state
clear a faulted non-persistent resource in a specific grp hagrp -clear [-sys]
Change the system list in a cluster
# remove the host
hagrp -modify grp_zlnrssd SystemList -delete
# add the new host (don't forget to state its position)
hagrp -modify grp_zlnrssd SystemList -add 1
# update the autostart list
hagrp -modify grp_zlnrssd AutoStartList
Service Group Operations
Start a service group and bring its resources online hagrp -online -sys
Stop a service group and takes its resources offline hagrp -offline -sys
Switch a service group from system to another hagrp -switch to
Enable all the resources in a group hagrp -enableresources
Disable all the resources in a group hagrp -disableresources
Freeze a service group (disable onlining and offlining)
hagrp -freeze [-persistent]
note: use the following to check "hagrp -display | grep TFrozen"
Unfreeze a service group (enable onlining and offlining)
hagrp -unfreeze [-persistent]
note: use the following to check "hagrp -display | grep TFrozen"
Enable a service group. Enabled groups can only be brought online
haconf -makerw
hagrp -enable [-sys]
haconf -dump -makero
Note to check run the following command "hagrp -display | grep Enabled"
Disable a service group. Stop from bringing online
haconf -makerw
hagrp -disable [-sys]
haconf -dump -makero
Note to check run the following command "hagrp -display | grep Enabled"
Flush a service group and enable corrective action. hagrp -flush -sys
Resources
add a resource haconf -makerw
hares -add appDG DiskGroup groupw
hares -modify appDG Enabled 1
hares -modify appDG DiskGroup appdg
hares -modify appDG StartVolumes 0
haconf -dump -makero
delete a resource haconf -makerw
hares -delete
haconf -dump -makero
change a resource
haconf -makerw
hares -modify appDG Enabled 1
haconf -dump -makero
Note: list parameters "hares -display"
change a resource attribute to be globally wide hares -global
change a resource attribute to be locally wide hares -local
list the parameters of a resource hares -display
list the resources hares -list
list the resource dependencies hares -dep
Resource Operations
Online a resource hares -online [-sys]
Offline a resource hares -offline [-sys]
display the state of a resource( offline, online, etc) hares -state
display the parameters of a resource hares -display
Offline a resource and propagate the command to its children hares -offprop -sys
Cause a resource agent to immediately monitor the resource hares -probe -sys
Clearing a resource (automatically initiates the onlining) hares -clear [-sys]
Resource Types
Add a resource type hatype -add
Remove a resource type hatype -delete
List all resource types hatype -list
Display a resource type hatype -display
List a partitcular resource type hatype -resources
Change a particular resource types attributes hatype -value
Resource Agents
add a agent pkgadd -d .
remove a agent pkgrm
change a agent n/a
list all ha agents haagent -list
Display agents run-time information i.e has it started, is it running ? haagent -display
Display agents faults haagent -display |grep Faults
Resource Agent Operations
Start an agent haagent -start[-sys]
Stop an agent haagent -stop[-sys]
LLT and GAB Commands | Port Membership | Daemons | Log Files | Dynamic Configuration | Users | Resources | Resource Agents | Service Groups | Clusters | Cluster Status | System Operations | Sevice Group Operations | Resource Operations | Agent Operations | Starting and Stopping
LLT and GRAB
VCS uses two components, LLT and GAB to share data over the private networks among systems.
These components provide the performance and reliability required by VCS.
LLT LLT (Low Latency Transport) provides fast, kernel-to-kernel comms and monitors network connections. The system admin configures the LLT by creating a configuration file (llttab) that describes the systems in the cluster and private network links among them. The LLT runs in layer 2 of the network stack
GAB GAB (Group membership and Atomic Broadcast) provides the global message order required to maintain a synchronised state among the systems, and monitors disk comms such as that required by the VCS heartbeat utility. The system admin configures GAB driver by creating a configuration file ( gabtab).
LLT and GAB files
/etc/llthosts
The file is a database, containing one entry per system, that links the LLT system ID with the hosts name. The file is identical on each server in the cluster.
/etc/llttab
The file contains information that is derived during installation and is used by the utility lltconfig.
/etc/gabtab
The file contains the information needed to configure the GAB driver. This file is used by the gabconfig utility.
/etc/VRTSvcs/conf/config/main.cf
The VCS configuration file. The file contains the information that defines the cluster and its systems.
Gabtab Entries
/sbin/gabdiskconf - i /dev/dsk/c1t2d0s2 -s 16 -S 1123
/sbin/gabdiskconf - i /dev/dsk/c1t2d0s2 -s 144 -S 1124
/sbin/gabdiskhb -a /dev/dsk/c1t2d0s2 -s 16 -p a -s 1123
/sbin/gabdiskhb -a /dev/dsk/c1t2d0s2 -s 144 -p h -s 1124
/sbin/gabconfig -c -n2
gabdiskconf
-i Initialises the disk region
-s Start Block
-S Signature
gabdiskhb (heartbeat disks)
-a Add a gab disk heartbeat resource
-s Start Block
-p Port
-S Signature
gabconfig
-c Configure the driver for use
-n Number of systems in the cluster.
LLT and GAB Commands
Verifying that links are active for LLT lltstat -n
verbose output of the lltstat command lltstat -nvv | more
open ports for LLT lltstat -p
display the values of LLT configuration directives lltstat -c
lists information about each configured LLT link lltstat -l
List all MAC addresses in the cluster lltconfig -a list
stop the LLT running lltconfig -U
start the LLT lltconfig -c
verify that GAB is operating
gabconfig -a
Note: port a indicates that GAB is communicating, port h indicates that VCS is started
stop GAB running gabconfig -U
start the GAB gabconfig -c -n
override the seed values in the gabtab file gabconfig -c -x
GAB Port Memberbership
List Membership
gabconfig -a
Unregister port f /opt/VRTS/bin/fsclustadm cfsdeinit
Port Function a gab driver
b I/O fencing (designed to guarantee data integrity)
d ODM (Oracle Disk Manager)
f CFS (Cluster File System)
h VCS (VERITAS Cluster Server: high availability daemon)
o VCSMM driver (kernel module needed for Oracle and VCS interface)
q QuickLog daemon
v CVM (Cluster Volume Manager)
w vxconfigd (module for cvm)
Cluster daemons
High Availability Daemon had
Companion Daemon hashadow
Resource Agent daemon
Web Console cluster managerment daemon CmdServer
Cluster Log Files
Log Directory /var/VRTSvcs/log
primary log file (engine log file) /var/VRTSvcs/log/engine_A.log
Starting and Stopping the cluster
"-stale" instructs the engine to treat the local config as stale
"-force" instructs the engine to treat a stale config as a valid one
hastart [-stale|-force]
Bring the cluster into running mode from a stale state using the configuration file from a particular server
hasys -force
stop the cluster on the local server but leave the application/s running, do not failover the application/s hastop -local
stop cluster on local server but evacuate (failover) the application/s to another node within the cluster hastop -local -evacuate
stop the cluster on all nodes but leave the application/s running
hastop -all -force
Cluster Status
display cluster summary hastatus -summary
continually monitor cluster hastatus
verify the cluster is operating hasys -display
Cluster Details
information about a cluster haclus -display
value for a specific cluster attribute haclus -value
modify a cluster attribute haclus -modify
Enable LinkMonitoring haclus -enable LinkMonitoring
Disable LinkMonitoring haclus -disable LinkMonitoring
Users
add a user hauser -add
modify a user hauser -update
delete a user hauser -delete
display all users hauser -display
System Operations
add a system to the cluster hasys -add
delete a system from the cluster hasys -delete
Modify a system attributes hasys -modify
list a system state hasys -state
Force a system to start hasys -force
Display the systems attributes hasys -display [-sys]
List all the systems in the cluster hasys -list
Change the load attribute of a system hasys -load
Display the value of a systems nodeid (/etc/llthosts) hasys -nodeid
Freeze a system (No offlining system, No groups onlining)
hasys -freeze [-persistent][-evacuate]
Note: main.cf must be in write mode
Unfreeze a system ( reenable groups and resource back online)
hasys -unfreeze [-persistent]
Note: main.cf must be in write mode
Dynamic Configuration
The VCS configuration must be in read/write mode in order to make changes. When in read/write mode the
configuration becomes stale, a .stale file is created in $VCS_CONF/conf/config. When the configuration is put
back into read only mode the .stale file is removed.
Change configuration to read/write mode haconf -makerw
Change configuration to read-only mode haconf -dump -makero
Check what mode cluster is running in
haclus -display |grep -i 'readonly'
0 = write mode
1 = read only mode
Check the configuration file
hacf -verify /etc/VRTS/conf/config
Note: you can point to any directory as long as it has main.cf and types.cf
convert a main.cf file into cluster commands hacf -cftocmd /etc/VRTS/conf/config -dest /tmp
convert a command file into a main.cf file
hacf -cmdtocf /tmp -dest /etc/VRTS/conf/config
Service Groups
add a service group haconf -makerw
hagrp -add groupw
hagrp -modify groupw SystemList sun1 1 sun2 2
hagrp -autoenable groupw -sys sun1
haconf -dump -makero
delete a service group haconf -makerw
hagrp -delete groupw
haconf -dump -makero
change a service group
haconf -makerw
hagrp -modify groupw SystemList sun1 1 sun2 2 sun3 3
haconf -dump -makero
Note: use the "hagrp -display
list the service groups hagrp -list
list the groups dependencies hagrp -dep
list the parameters of a group hagrp -display
display a service group's resource hagrp -resources
display the current state of the service group hagrp -state
clear a faulted non-persistent resource in a specific grp hagrp -clear
Change the system list in a cluster
# remove the host
hagrp -modify grp_zlnrssd SystemList -delete
# add the new host (don't forget to state its position)
hagrp -modify grp_zlnrssd SystemList -add
# update the autostart list
hagrp -modify grp_zlnrssd AutoStartList
Service Group Operations
Start a service group and bring its resources online hagrp -online
Stop a service group and takes its resources offline hagrp -offline
Switch a service group from system to another hagrp -switch
Enable all the resources in a group hagrp -enableresources
Disable all the resources in a group hagrp -disableresources
Freeze a service group (disable onlining and offlining)
hagrp -freeze
note: use the following to check "hagrp -display
Unfreeze a service group (enable onlining and offlining)
hagrp -unfreeze
note: use the following to check "hagrp -display
Enable a service group. Enabled groups can only be brought online
haconf -makerw
hagrp -enable
haconf -dump -makero
Note to check run the following command "hagrp -display | grep Enabled"
Disable a service group. Stop from bringing online
haconf -makerw
hagrp -disable
haconf -dump -makero
Note to check run the following command "hagrp -display | grep Enabled"
Flush a service group and enable corrective action. hagrp -flush
Resources
add a resource haconf -makerw
hares -add appDG DiskGroup groupw
hares -modify appDG Enabled 1
hares -modify appDG DiskGroup appdg
hares -modify appDG StartVolumes 0
haconf -dump -makero
delete a resource haconf -makerw
hares -delete
haconf -dump -makero
change a resource
haconf -makerw
hares -modify appDG Enabled 1
haconf -dump -makero
Note: list parameters "hares -display
change a resource attribute to be globally wide hares -global
change a resource attribute to be locally wide hares -local
list the parameters of a resource hares -display
list the resources hares -list
list the resource dependencies hares -dep
Resource Operations
Online a resource hares -online
Offline a resource hares -offline
display the state of a resource( offline, online, etc) hares -state
display the parameters of a resource hares -display
Offline a resource and propagate the command to its children hares -offprop
Cause a resource agent to immediately monitor the resource hares -probe
Clearing a resource (automatically initiates the onlining) hares -clear
Resource Types
Add a resource type hatype -add
Remove a resource type hatype -delete
List all resource types hatype -list
Display a resource type hatype -display
List a partitcular resource type hatype -resources
Change a particular resource types attributes hatype -value
Resource Agents
add a agent pkgadd -d .
remove a agent pkgrm
change a agent n/a
list all ha agents haagent -list
Display agents run-time information i.e has it started, is it running ? haagent -display
Display agents faults haagent -display |grep Faults
Resource Agent Operations
Start an agent haagent -start
Stop an agent haagent -stop
Subscribe to:
Posts (Atom)
