crash(ADM)
crash --
examine system images
Syntax
/usr/sbin/crash [-a analysisfile] [-b batchfile] [-d dumpfile] [-m moduledir] [-n namelist] [-w outputfile]
Description
The crash command is used to examine the system memory image
of a running or a crashed system by formatting and printing
control structures, tables, and other information.
The amount and type of information stored in a system memory dump can be
altered using the kernel parameter SYSDUMP_SELECTIVE.
Set this parameter to:
0-
to dump both kernel and user mapped memory.
1-
to dump only kernel mapped memory.
This is the default.
2-
to dump only a portion of kernel mapped memory.
If you change the value of SYSDUMP_SELECTIVE you must
run
idbuild(ADM)
to rebuild the kernel, and reboot the system.
When the dump is saved, you are prompted whether to save it in
compressed format -- compression reduces transmission time
if you are forwarding the dump to a remote site for analysis.
If the Encryption Utilities package is installed on your system,
you are also presented with a prompt asking if you want to encrypt the dump.
If you choose to encrypt the dump, you must supply an encryption key [see
crypt(C)
].
To decrypt the dump:
cat dumpfile | crypt key > newdumpfile
where dumpfile is the encrypted dump file, crypt key
is the encryption key that was specified when the dump was encrypted,
and newdumpfile is the resulting, unencrypted dump file.
If the PANICBOOT parameter in /etc/default/init is
set to ``YES'', the system automatically saves the memory dump
and reboots if a system panic occurs.
In this case, you are not prompted to compress or encrypt the dump.
Using crash to examine kernel memory statistics, history, and STREAMS performance
The kmdt and perf modules allow you to
examine kernel memory statistics and history, and streams performance
statistics, respectively.
These two modules are mutually exclusive.
Before obtaining the dump, configure either module into the kernel
as follows:
-
Execute the following command to configure either the kmdt module
or the perf module into your kernel.
Note that kmdt or perf must be the first argument to idtype,
and processor type is second (mp for multi-processor,
atup for uni-processor).
/etc/conf/bin/idtype {kmdt|perf} {mp|atup}
Here, the debug module (either kmdt or perf) is
required to be the first argument.
The second argument specifies the processor type (mp for
multi-processor, or atup for uni-processor).
-
Rebuild the kernel and reboot.
Configuring the kmdt module into the kernel makes these
additional functions available to crash (described in the
"Usage" section):
-
kmastats
-
kmahist
-
kmapools
Configuring the perf module into the kernel makes these
additional functions available to crash (described in the
"Usage" section):
-
strleaklist
-
strleakcnt
-
strperf
NOTE:
If you try to use a function which is not configured into the kernel
with one of the debug modules, you will get a message telling you
it is available only with a debug kernel.
To remove the debug module from the kernel, execute
idtype atup or idtype mp, rebuild the kernel,
and reboot.
Options
crash takes the following options:
-a analysisfile-
Use the -a option to have the crash utility
perform an analysis of the dump, and save the analysis in plain text
format to the directory specified by analysisfile.
The default analysisfile is /tmp/analysis.
-b batchfile-
Use the text file batchfile, which contains a list of
crash commands to be executed on the dump.
-d dumpfile-
Use dumpfile for the file containing the system memory image.
The default dumpfile is /dev/swap.
If you enter a value other than /dev/swap, it is assumed
that the system has crashed.
If a system dump is directed to a dump device other than the default swap
device, the dump will not be detected when the system boots up.
To enable the system to detect a dump that is not in the swap area when the
system boots, add the following entry to the file /etc/dumptab.
/dev/dump 0 -
-n namelist-
Use the text file namelist, which contains the symbol table
information needed for symbolic access to the system memory image
to be examined.
The default namelist is /stand/unix.
If a system image from another machine is to be
examined, the corresponding text file must be copied from that machine.
For active systems, crash uses system calls to the
running kernel to get symbol table information.
For that reason, only global symbols are available.
-m moduledir-
Use the specified directory, moduledir to look for modules.
When working on a dump, crash looks for the modules that were
loaded at the time of the dump, and adds those symbols
to its symbol table information.
By default, crash tries to find the modules in
the directories from which they were loaded.
The -m option specifies a directory where crash
should look for the modules instead.
If you use the -m option, crash only looks in the
specified directory.
You can only use a single instance of the -m
option on the command line, and
if you attempt to use the -m option more
than once, only the last one is valid; the others are ignored.
-w outputfile-
The output from a crash session is directed to outputfile.
The default outputfile is the standard output.
Input
When you execute the crash command, a session is initiated.
If you enter the ``?'' character at the command
prompt, crash provides
a help menu of the available commands.
Input during a crash session is of the form:
function [argument...]
where function is one of the crash functions described in
``Usage'', and argument is a list of qualifying data
that indicate which items of the system image are to be printed.
The default for process-related items is
the current process for a running system, or
the process that was running at the time of the
crash for a crashed system.
If the contents of a table are being dumped, the default is all active
table entries.
The following function options are available
to crash functions wherever they are semantically valid.
-e-
Display every entry in a table.
-f-
Display the full structure.
-n-
Display privilege names symbolically.
Default is to display in hexadecimal.
-p-
Interpret all address arguments in the command
line as physical addresses.
If they are not physical addresses, results are inconsistent.
-s process-
Specify a process slot other than the default.
-w file-
Redirect the command's output to a file.
The functions mode, defproc, and redirect
correspond to the function options -p, -s,
and -w.
The mode function may be used to set the address
translation mode to physical or
virtual for all subsequently entered functions;
defproc sets the value of the process slot argument
for subsequent functions; and
redirect redirects all subsequent output.
Output
Output from crash functions may be piped to another program
as follows:
function [argument . . .] ! shell_command
For example, mount ! grep rw
writes all mount table entries with an rw flag to the
standard output. The redirection option (-w) cannot be used with
this feature.
An argument can be either a symbol or a numeric argument.
Numeric arguments are assumed to be decimal unless
preceded by 0x or 0b prefixes for
hexadecimal or binary numbers.
Any argument that is not either a symbol or a number will be
reported as an error.
Each function, when executed, determines whether the
argument specified a valid slot or address.
Default bases on all arguments may be overridden.
The C conventions for designating the bases of numbers are recognized.
Aliases for functions may be any uniquely
identifiable initial substring of the function name.
Traditional aliases of one letter, such as p for
proc, remain valid.
Many functions accept different forms of entry for the same argument.
Requests for table information will accept a table entry number or
a range.
A range of slot numbers may be specified in the form a-b where
a and b are decimal numbers.
An expression consists of two operands
and an operator.
An operand may be an address, a symbol, or a number; the operator
may be +, -, *, /, &, or |.
An operand that is a number should be preceded by a radix prefix
if it is not a decimal
number (0 for octal, 0x for hexadecimal,
0b for binary).
The expression must be enclosed in parentheses.
Other functions accept any of these argument forms
that are meaningful.
Two abbreviated arguments to crash functions are
used throughout.
Both accept data entered in several forms.
They may be expanded into the following:
table_entry = address | slot | range
start_addr = address | symbol | expression
Usage
Following are the available functions in crash:
? [-w file]-
List available functions.
!command-
Escape to the shell and execute
command.
abuf [-w file] [-mode]-
Print audit buffer data in mode format.
mode is one of long (-l), short (-t), or byte
(-b).
The default mode for character and ASCII formats is byte;
the default mode for decimal, hexadecimal, and octal formats is long.
When mode is omitted, the previous value is used.
At the start of a crash session, the mode is long.
addstruct [-w file] structure_name-
adds a structure to the list shown
by size or size -f, to the structures and their field names
(its particularly useful for od).
For example, addstruct stat sys/stat.h
parses the declaration of struct stat in
/usr/include/sys/stat.h,
and creates a /usr/lib/crash/statoffs.so used thereafter (even
when you reinvoke crash).
If the header file turns out to need other header files included
before it can be parsed, then you can add those dependent header files.
If it does not parse correctly, you can edit /usr/lib/crash/addstruct
to correct its parsing.
as [-e] [-f] [-w file] [proc...]-
Print address space information on process segments.
base [-w file] number...-
Print number in binary, octal, decimal, and
hexadecimal.
A number in a radix other than decimal should be preceded by a prefix that
indicates its radix as follows: 0x, hexadecimal;
0, octal; and 0b, binary.
buffer [-w file] [-format] [-p] addr-
Alias: b
Print the contents of a buffer in the designated format.
The following format designations are recognized: -b,
byte: -c, character; -d, decimal; -x, hexadecimal;
-o, octal; and, -i, inode.
If no format is given, the previous format is used.
The default format at the beginning of a crash session is hexadecimal.
bufhdr [-f] [-w file] [[-p] addr...]-
Alias: buf
Print system buffer headers.
The -f option produces different output depending on
whether the buffer is local or remote.
callout [-w file] CG-
Print the callout table for a CPU group.
cg [-w file] [cgnum]-
Set the current CPU group to cgnum.
If cgnum is not specified then the current CPU
group is displayed.
cglocal [-w file] [-e | eng]-
Print the local information for the current CPU group, or for
the CPU specified by eng.
If -e is specified, information is printed
for all CPU groups.
class [-w file] [table_entry...]-
Print information about process scheduler classes.
deflwp [-w file] [-c]-
deflwp [-w file] [slot]-
Set the value of the lwp slot for the current process.
If slot is not specified, the current slot is displayed.
The default lwp slot for each process is 0.
If -c is specified, a reset is performed which gets context
for the current slot number.
defproc [-w file] [-c]-
defproc [-w file] [slot]-
Set the value of the default process slot argument.
The default process slot argument may be set to the
current slot number (-c) or
the slot number may be specified.
If no argument
is entered, the value of the previously set slot number is printed.
At the start of a crash session, the process slot is set to
the current process.
dis [-w file] [-a] start_addr [count]-
dis [-w file] [-a] [-c | [-p | -sproc] address] [count]-
Disassemble count instructions starting at start_addr.
The default count is 1.
The absolute option (-a) specifies a non-symbolic disassembly.
The -c option can be used in place of start_addr to
continue disassembly at the address at which a previous disassembly ended.
dispq [-l|-g] [-w file] [table_entry...]-
Print the dispatcher (scheduler) queues.
The -l option prints only the local dispatcher queues.
The -g option prints only the global dispatcher queue.
ds [-w file] virtual_address...-
Print the data symbol whose address is closest to, but not
greater than, the address entered.
eng [-w file] [eng_num] [-b]-
Set the current engine to eng_num.
If no eng_num is specified, then the
current engine number is displayed.
The -b option acts as a toggle to bind or unbind an engine.
file [-e] [-w file] [[-p] table_entry...]-
Alias: f
Print the file table.
filepriv [-n] [-w file] [-p table_entry...]-
Alias: fprv
Print the kernel privilege table.
fs [-w file] [[-p] table_entry...]-
Print the filesystem information table.
gdt [-e] [-w file] [slot [count]] table_entry...]-
Print the global descriptor table.
help [-w file] [function... | -e]-
Print a description of function, including syntax and aliases.
Use -e to print a help list for all functions.
hexmode [-w file] [ON | OFF]-
If hexmode is ON, commands expect hexadecimal input
and display hexadecimal output.
If hexmode is OFF, input and output are decimal.
htinode [-e] [-f] [-i] [-l] [inodeaddr...] [inodeno...]-
Alias: hti
Print the inode table for the
HTFS, EAFS, AFS, and S51K filesystems.
Using -e and -i together prints
inactive nodes. The -e option alone prints active nodes.
Use -l to display the free lists.
Specific inodes can be specified by inodeaddr (the
address in kernel memory), or by inodeno (inode number).
If inodeno is used, however, inodes might
be found in more than one filesystem. In this case, all of
the inodes that are found are printed.
idt [-e] [-w file ] [slot [count]]-
Print the interrupt descriptor table.
inode [-e] [-f] [-l] [-r] [slot [-last] | inodeaddr...]-
Alias: i
Print the inode table for the s5 filesystem. Use -r to
print all inactive inodes. Use -l to display the
free lists. Note that the s5 filesystem is not normally
configured for use on SCO OpenServer Release 6 systems. Use the
htinode command for
HTFS, EAFS, AFS, and S51K
filesystem inodes.
inpcb [-fnphx][-w file] address [address...]-
Print the Internet protocol control block (inpcb)
at the specified address(es).
The -n (next) and -p
options allow you to walk a chain of inpcb structures.
The -h option makes the -n and -p
options follow ``head'' (that is, normal) next
or previous links. This is the default behavior.
The -x option switches the command's behavior so that
-n and -p follow next or previous links in the hash
queue.
inpcbhash [-fshec] [-tu] [-w file]-
Print information about the Internet protocol control block hash
(inpcbhash) tables.
The -s option prints only statistics (default behavior), the
-h option prints hash entries,
the -c option causes the chains attached to an entry to be walked,
and the -e option will cause empty entries to be included.
The -t and -u options
display the TCP and UDP hash tables respectively.
If neither -t nor -u are specified, both are assumed.
kmahist [-[a|A] address] [-[l|L] lwp_address] [-[n|N] number]-
Display kernel memory allocation history.
This displays time, lwp, address, size, and the function doing the
allocation or free of the memory.
Options can be given in upper or lower case.
The option characters a and l can be
preceeded by``!'' to reverse the sense of the comparison.
The -a option argument address is the address of a
memory buffer.
The -l option argument lwp_address is the address of
the lwp (light weight process) you are interested in.
The -n option argument number specifies that only
number most recent history entries will be displayed.
the lwp (light weight process) you are interested in.
kmapools-
Display kernel memory pools statistics.
kmastats-
Display kernel memory allocator statistics: kernel metrics for
allocation and freeing of memory, failures, and the invocations table
(the number of allocations, frees and failures, by function).
lck [-w file] [filockaddr...]-
Print record locking information.
If file lock address arguments are given, the record lock list is printed.
If no argument is given, information on locks relative to inodes is printed.
ldt [-e] [-w file] [process [slot [count]]]-
Print the local descriptor table for the specified process.
The table is printed for the current process if process
is not specified.
linkblk [-e] [-w file] [[-p] table_entry...]-
Print the linkblk table.
map [-w file] mapname...-
Print the map structure of the given mapname.
mbuf [-flbmk][-w file] address [address...]-
Print the memory buffers (mbufs) at the specified address(es).
The -m option causes the
m_next
pointer to be followed
and each mbuf encountered to be printed.
The -k option causes the m_nextpkt
pointer to be
followed.
Using -k and -m together will perform both operations,
but first follows m_next
.
The -l option prints a long listing of all fields in the
m_hdr. The -b option includes details of the
M_dat
union.
mount [-e] [-w file] [[-p] table_entry...]-
Alias: m, vfs
Print information about mounted filesystems.
nm [-w file] symbol...-
Alias: symval, ts
Print value and type for the given symbol.
od [-p] [-w file] [-format] [-mode] [-s process] start_addr [count]-
Alias: rd
Print count values starting
at start_addr in one of the following
formats: character (-c),
decimal (-d), hexadecimal (-x),
octal (-o), ASCII (-a), or
hexadecimal/character (-h),
and one of the following
modes: long (-l), short (-t), or byte (-b).
The default mode for character and ASCII formats is byte;
the default mode for decimal, hexadecimal, and octal formats is long.
The format -h prints both hexadecimal and character
representations of the addresses dumped;
no mode needs to be specified.
When format or mode is omitted, the previous value is used.
At the start of a crash session, the format is
hexadecimal and the mode is long.
If no count is entered, 1 is assumed.
page [-e] [-w file] [[-p] table_entry...]-
Print information about pages.
panic-
Print the latest system notices, warnings, and panic messages
from the limited circular buffer kept in memory.
pcb [-e] [-w file] [process] [[-k | [[-u] [proc | #pio | idle] [lwp | eng]] [-i type=1]-
Print the process control block (TSS).
If no arguments are given, the active TSS
for the current process is printed.
If arguments are given, only one of
-k, -u, or -i can be specified.
plocal [-w file] [-e | eng]-
Print the processor local data for the current engine.
Use the -e option to print data for all engines, or use
eng to specify the engine number.
proc [-e] [-f [-n]] [-w file] [[-p] table_entry...#procid...]-
proc [-f [-n]] [-w file]-
Alias: p
Print the process table.
Process table information may be specified in two ways.
First, any mixture of table entries and process IDs may be entered.
Each process ID must be preceded by a ``#''.
The full option (-f) details most of the information in the
process table as well as the region for that process.
ptbl [-e] [-w file] [-sprocess] [[-p] addr [count]]-
Print information on page descriptor tables.
pty [-f] [-e] [-w file] [-s] [-h] [-l] [-t type]-
Print the pseudo ttys presently configured.
The -l, -h and -h options give
information about the STREAMS
modules ldterm, ptem and pckt, respectively.
qrun [-w file]-
Print the list of scheduled STREAMS queues.
queue [-e] [-s style] [-w file] [[-p] table_entry...]-
queue [-f] [-s style] [-w file] [[-p] table_entry...]-
Print STREAMS queues.
Up to three -s arguments can be specified: the first for the
write side, the second for the read side, and a third for read/write.
However, only one -s can be specified if other arguments are
to be used.
The -s option displays the symbolic STREAMS configuration.
quit-
Alias: q
Terminate the crash session.
redirect [-w file] [-c]-
redirect [-w file] [newfile]-
Used with a file name, redirects output
of a crash session to newfile.
If no argument is given, the file name to which output is being redirected
is printed.
Alternatively, the close option (-c) closes the previously set file
and redirects output to the standard output.
resmgr [-f [-w file]-
Print the resource manager database.
rtdptbl [-w file] [table_entry...]-
Print the real-time scheduler parameter table.
See
fp_dptbl(F).
rtproc [-w file]-
Print information about processes in the real-time scheduler class.
search [-p] [-w file] [-m mask] [-s process] pattern start_addr length-
Print the long words in memory that match pattern,
beginning at the start_addr for length long words.
The mask is ANDed (&) with each memory word and the result
compared against the pattern.
The mask defaults to 0xffffffff.
socket [-flb][-w file] [address [address...] ]-
Print out the socket associated with each socknode
in turn starting at the
socknodelist
anchor point in sockfs.
If one or more socket addresses are specified,
they are printed out in detail instead of as summaries.
The -b option prints a listing of the embedded
sockbuf structures.
The -l option prints a full listing of all fields except the
embedded sockbuf structures.
socknode [-f][-w file] [address [address...] ]-
Print details of each socknode in turn starting at
the
socknodelist
anchor point in sockfs
If one or more socknode addresses are specified,
they are printed out in detail instead of as summaries.
The -f option adds single-line details for the embedded
vnode and associated socket structure.
sinode [-e] [-f] [-w file] [-r] [-l] [[-p] table_entry...]-
Options
f
Include a single line description of the associated socket
w
redirect to file
Alias: si
Print the inode table for ufs or sfs filesystems.
Since the ufs/sfs incore inode contains the icommon
inode and the alternate inode, this function displays,
in addition to the icommon inode information,
all security data stored in the alternate inode.
This includes the Access Control List (ACL) count, the
extended ACL disk block pointer, and any inode resident
ACL entries.
size [-f] [-w file] [structure_name...]-
Print the size of the designated structure.
The (-x) option prints the size in hexadecimal.
If no argument is given,
a list of the structure names for which sizes are available is printed.
snode [-f] [-w file] [[-p] table_entry...]-
Print information about open special files.
stack [-e]-
stack [-t stackptr] [-w file] [process]-
Alias: s
Dump the stack.
If no arguments are entered, the kernel stack
for the current process is printed.
If -t is used, it expects a hexadecimal value for the stack pointer.
The interrupt stack and the stack for the current process are not
available on a running system.
stat [-w file]-
Print system statistics.
stream [-f] [-w file] [[-p] table_entry...]-
Print the STREAMS table.
strleaklist-
Print all streams in-use messages (a list of message blocks allocated
by functions in the kernel).
strleakcnt-
Print table of callers of streams in-use messages (adress, count, size,
and usage of the message blocks).
strperf-
Print streams performance statistics.
strstat [-w file]-
Print STREAMS statistics.
test [-w file] [debuglevel]-
Displays or changes the debug level.
trace [-w file] [-e] [-t] [process]-
Alias: t
Print stack trace.
tsdptbl [-w file] [table_entry...]-
Print the time-sharing scheduler parameter table.
See
ts_dptbl(F).
tslwp [-w file]-
Alias: tsproc
Print information about light-weight processes (LWPs)
in the time-sharing scheduler class.
tty [-e] [-f] [-l] [-w file] [-t type [[-p] table_entry...]|[-p] start addr]-
Valid types: console, sr, sx, sc.
Print the tty table.
If no arguments are given, the tty table for both tty types is printed.
If the -t option is used, the table for the single tty type
specified is printed.
If no argument follows the type
option, all entries in the table are printed.
A single tty entry may be specified using start_addr.
The -l option prints the line discipline information.
user [-e] [-w file] [process]-
Alias: u
Print the ublock for the designated process.
var [-w file]-
Alias: v
Print the tunable system parameters.
vfs [-e] [-w file] [[-p] table_entry...]-
Alias: mount, m
Print information about mounted filesystems.
vfssw [-w file] [[-p] table_entry...]-
Alias: fs
Print information about configured filesystem types.
vnode [-w file] [[-p] vnode_addr...]-
Print information about vnodes.
vtop [-w file] [-s process] start_addr...-
Print the physical address translation of the virtual address start_addr.
vxinode [-w file] [-e] [-f] [-l] [-r] [vx_inodeaddr...]-
Alias: vi, vinode
Print the VxFS filesystem (vxfs) inode table;
valid only for vxfs type filesystems.
-l prints the contents of all the icache lists (these hold the
vx_inodes) together with information for each inode indicating
if it is a free or attribute (or both) inode.
-r prints all the free vx_inodes.
vx_inodeaddr prints the vx_inode located at
address vx_inodeaddr.
Extensible functions
It is possible to add functions to crash without having access
to the source code.
This is useful for writing platform specific functions
or to obtain more information than is available in the default command set.
Adding new functions is accomplished with shared objects.
The shared object is created from a file consisting of the functions
that are to be added to the existing command set.
When crash is started, these
shared objects are read and the additional functions are incorporated
in the command set.
To create a shared object:
-
Create a file that contains the new commands.
This file should define the variable functab, which should
be an array type struct func.
This array points to the new commands that need to be added.
-
Compile the file with the same options as the kernel
build. For example, if crash is going to be run on top of a
uniprocessor kernel, then the file should be compiled with the
appropriate definitions for a uniprocessor kernel.
-
Use the -G option to the compiler to create the shared object.
crash reads the shared objects from directory
/usr/lib/crash and also from
files defined by environment variable CRASH_LIBS.
Extension example
An example of a file containing a new command (lbolt) is given below:
#include <stdio.h>
#include <sys/types.h>
#include <syms.h>
extern int get_lbolt();
/* function definition */
struct func {
char *name;
char *syntax;
int (*call)();
char *description;
};
struct func functab[] = {
"lbolt", " ", get_lbolt, "lbolt",
0,0,0,0
};
static struct syment *lbolt_sym = NULL;
extern struct syment *symsrch(char *);
extern FILE *fp;
get_lbolt(void)
{
time_t lbolt;
if (lbolt_sym == NULL && (lbolt_sym = symsrch("lbolt")) == NULL)
return 0;
readmem((void *)(lbolt_sym->n_value), 1, -1, &lbolt, sizeof(lbolt),
"lbolt");
fprintf(fp, "%lx\n",lbolt);
}
© 2005 The SCO Group, Inc. All rights reserved.
SCO OpenServer Release 6.0.0 -- 03 June 2005