http://www.slac.stanford.edu/BFROOT/www/Computing/Environment/Tools/Batch/exitcode.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html>
<head>
<title>Exit codes</title>
</head>
<body>

<center>
<h1>Job crashes and exit codes</h1></center>
<p>

<p>This webpage is a collection of information about job 
crashes and exit codes, gleaned from HyperNews and wherever 
else I could find it.

<p>If you have anything useful to add, or if any of the information 
is incorrect, <b>please feel free to edit the page!</b>

<h2>Quick diagnosis</h2>

<p>The overall impression I get from searching Hypernews is:

<ul>
<li> If you get core dumped, there was a problem in your code, and you 
should use the debugger.  
<li> If you do not get core dumped, 
then your exit code probably means you ran out of CPU time.
</ul>

<h2>Exit codes and kill-job signals</h2>

<p>The exit code from a batch job is a standard Unix termination status,
the same sort of number you get in a shell script from checking the "<code>$?</code>"
variable after executing a command.</p>

<p>
Typically, exit code 0 (zero) means successful completion.  Codes 1-127
are typically generated by your job itself calling
<code><a href="http://www.opengroup.org/onlinepubs/007908799/xsh/_exit.html">exit()</a></code>
with a non-zero value to terminate itself and indicate an error.
In BaBar we don't make very much use of this.  The most common such value you might 
see is 64, which is the value used by Framework to say that its event loop is
being stopped before all the requested data have been read, typically because time
ran out.
In recent BaBar releases you might also see 125, which we use as a code for a generic
"severe error"; the job log should contain a message stating what the error was.
</p>

<p>
Exit codes in the range 129-255 represent jobs terminated by Unix 
"<a href="http://www.opengroup.org/onlinepubs/007908799/xsh/signal.h.html">signals</a>".
Each type of signal has a number, and what's reported as the job exit code is the
signal number plus 128.
Signals can arise from within the process itself (as for SEGV, see below)
or be sent to the process by some external agent (such as the batch control system,
or your using the "<code>bkill</code>" command).
</p>

<p>
By way of example, then, exit code 64 means that the job deliberately terminated its 
execution by calling "<code>exit(64)</code>", 
exit code 137 means that the job received a signal 9,
and exit code 140 represents signal 12.
</p>

<p>
The specific meaning of the signal numbers is platform-dependent.  If
you are trying to figure out a problem that was seen on Linux, you
have to run the following commands on Linux.  We don't have Solaris or Mac OS batch
resources in BaBar at the moment, but if we did, you would have to match platforms
similarly when debugging.
</p>


<h3>terminationDecoder</h3>

<p>
BaBar provides a little program that will take your exit code 
and spit out an explanation.  
The program is called <code>terminationDecoder</code>.  Examples:

<pre>
[yakut] terminationDecoder 137
terminated by signal 9 (Killed)

[yakut] terminationDecoder 64
exited with code 64 (in Framework: stop requested, e.g., by CpuCheck)
</pre>
</p>


<h3>More details</h3>

<p>You can also look this up yourself;
if you know the signal number, then you can find out why the job was killed 
using the command "kill -l":

<pre>
[yakut] kill -l

HUP INT QUIT ILL TRAP ABRT BUS FPE KILL USR1 SEGV USR2 PIPE ALRM TERM STKFLT
CHLD CONT STOP TSTP TTIN TTOU URG XCPU XFSZ VTALRM PROF WINCH POLL PWR SYS
RTMIN RTMIN+1 RTMIN+2 RTMIN+3 RTMAX-3 RTMAX-2 RTMAX-1 RTMAX
</pre>

<p>So for example, if your job was killed by signal 6, then it got an 
"ABRT", which is short for ABORT.

<p>To find out what all the "<code>kill -l</code>" words mean, 
you can use the command:

<pre>
man 7 signal    
</pre>

<p>
(or, on Solaris, "<code>man -s 3HEAD signal</code>").
This will give you the man page for SIGNAL(7).  
Scroll down a bit and you will get a list of the kill-signal words 
with a short explanation.  Here is a sample:

<pre>
SIGHUP        1       Term    Hangup detected on controlling terminal
                              or death of controlling process
SIGINT        2       Term    Interrupt from keyboard
SIGQUIT       3       Core    Quit from keyboard
SIGILL        4       Core    Illegal Instruction
SIGABRT       6       Core    Abort signal from abort(3)
SIGFPE        8       Core    Floating point exception
SIGKILL       9       Term    Kill signal
SIGSEGV      11       Core    Invalid memory reference
SIGPIPE      13       Term    Broken pipe: write to pipe with no readers
SIGALRM      14       Term    Timer signal from alarm(2)
SIGTERM      15       Term    Termination signal
</pre>

<p>(Obviously, these are just the "<code>kill -l</code>" words, but with 
a "SIG" in front of them.)</p>

<p>
You may also find it useful to look at the file signal.h.  On 
a Linux machine, the location is:

<pre>
/usr/include/asm/signal.h
</pre>
</p>


<h2>Hypernews examples</h2>

<p>Here are some specific exit codes that came up 
in Hypernews.  Here I have recorded the HN responses.  
However, they might not be correct in all cases.  
(Maybe the exit codes can mean other things, too.)

<p><b>Exit code 9:</b> Ran out of CPU time.

<p><b>Exit code 64:</b> The framework ended the job nicely for you, most
likely because  
the job was running out of CPU time.  But it means you did not go through all  
the data requested.  The solution is to submit the job to a queue with more 
resources (bigger CPU time limit).

<p><b>Exit code 125:</b> An <code>ErrMsg(severe)</code> was reached in your job.

<p><b>Exit code 127:</b> Something wrong with the machine?

<p><b>Exit code 130:</b> The job ran out of CPU or swap time.  If swap 
time is the culprit, check for <a href="#memleak">memory leaks</a>.

<p><b>Exit code 131:</b> The job ran out of CPU or swap time.  If swap 
time is the culprit, check for <a href="#memleak">memory leaks</a>.

<p><b>Exit code 134:</b> The job is killed with an abort signal, and you 
probably got core dumped.  Often this is caused either by an <code>assert()</code>
or an <code>ErrMsg(fatal)</code> being hit in your job.
There may be a run-time bug in your code.  
<a href="http://www.slac.stanford.edu/BFROOT/www/doc/workbook/debug/debug.html">Use a debugger</a> 
like gdb or dbx to find out what's wrong.

<p><b>Exit code 137:</b> The job was killed because it exceeded the time limit.

<p><b>Exit code 139:</b> Segmentatation violation.  

<p><b>Exit code 140:</b> The job exceeded the "wall clock" time limit 
(as opposed to the CPU time limit).

<h2>HOWTO's guide to job-kill signals</h2>

The following is copied from <a href="http://www.slac.stanford.edu/BFROOT/dist/releases/newest/HOWTO/HOWTO-Basic-Debugging">HOWTO-Basic-Debugging</a>, 
which you should definitely consult to learn how to interpret, report, 
and deal with errors and crashes:
<br>
<br>

<DL>
<DT> SEGV 
<DD> A segmentation violation or segmentation fault typically 
means that something is trying to access memory that 
it shouldn't be accessing. One common example of this is 
trying to access memory through a NULL pointer, for example:

<pre>
sunprompt&gt; cat main.c
#include <iostream.h>
main()
{
  int* bunk(0);
  cout << *bunk << endl;
}
sunprompt&gt; CC main.c
sunprompt&gt; ./a.out
Segmentation fault (core dumped)
</pre>

<DT> ABRT
<DD> asserts are one common source of the "abort" signal, for example:

<pre>
sunprompt&gt; cat main.c
#include <assert.h>
main()
{
  int i=0;
  assert(i!=0);
}
sunprompt&gt; CC main.c
sunprompt&gt; ./a.out
Assertion failed: i!=0, file main.c, line 5
Abort (core dumped)
</pre>

Note that the actual assertion which was failed and the location is also printed.
An ABRT can also be generated from the BaBar <code>ErrMsg(fatal)</code> construct,
in which case your job log should contain a message explaining the error.

<DT> FPE
<DD> A "Floating Point Error" usually indicates a numerical 
problem such as a division by zero or an overflow. One example would be:

<pre>
osfprompt&gt; cat main.c
main()
{
  float a = 1.;
  float b = 0.;
  float c = a/b;
}
osfprompt&gt; g++ main.c
osfprompt&gt; ./a.out
Floating exception (core dumped)
</pre>

<DT>ILL

<DD> If you receive a signal like this ("Illegal Instruction"), means 
that, while running, your program has tried to execute a 
machine "instruction" which does not exist. This can happen 
for a variety of reasons, including:

<ul>
<li> a memory overwrite that happens to overwrite part of the
program stored in memory. This may result in the program
trying, for example, to execute data as if it is a
machine instruction.
<li> an attempt to take an executable compiled on one platform
for use on another, for example on an earlier version of the same chip.
<li> a truncated or corrupted executable is loaded for execution
<li> incomplete recompilation of source code, i.e. you changed one C++ class 
and didn't recompile all other code affected by that change.
</ul>

<DT> BUS
<DD> A "Bus Error" may come, for example, from accessing unaligned 
data (i.e. like trying to access a 4 byte integer with a 
pointer to the middle of it). What this means will vary 
from platform to platform. (I haven't come up with a good 
example of this one yet.)

<p>A "Bus Error" can also often indicate a memory overwrite, 
e.g. somebody wrote a number where a pointer is kept.  
<u>Often</u> caused by going past the end of an array and into 
the system pointers at the start of the next memory block.
</DL>

<h2>How do you know if you've exceeded your CPU time?</h2>

<p>To find out whether your job has exceeded the CPU time limit, you have 
to do 3 things:

<ol>
<li> Look at your log file to get the job's CPU time.
<li> Use the machine-dependent CPUF to convert the CPU time to SLAC time.  The 
formula is: SLAC time = CPU time * CPUF.
<li> Compare this to the time allowed by the queue in which the job was run.
</ol>

<p>Here is an example.

<p>First, look at the end of your log file:

<pre>
<hr>
Job &lt;VubRecoilUserApp VubXlnu.tcl SP-1237-BSemiExcl-Run5-R18b-1 MC&gt; was submitte
d from host &lt;yakut02&gt; by user &lt;penguin&gt;.
Job was executed on host(s) &lt;cob0313&gt;, in queue &lt;xlong&gt;, as user &lt;penguin&gt;.
&lt;/u/br/penguin&gt; was used as the home directory.
&lt;/u/br/penguin/vubrecoil/vub30/workdir&gt; was used as the working directory.
Started at Wed Feb  8 17:25:33 2006
Results reported at Wed Feb  8 19:27:28 2006

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
VubRecoilUserApp VubXlnu.tcl SP-1237-BSemiExcl-Run5-R18b-1 MC
------------------------------------------------------------

Exited with exit code 134.

Resource usage summary:

    CPU time   :   7058.71 sec.
    Max Memory :      2863 MB
    Max Swap   :      2968 MB

    Max Processes  :         3
    Max Threads    :         3
<hr>
</pre>

<p>The job was run on the machine cob0313.  

<pre>
&gt; bhosts -l cob0313
</pre>

<p>This tells you (among other things) that the CPUF 
for cob0313 is 7.65.

<p>The SLAC time for your job is thus:

<pre>
SLAC time = (CPU time) * CPUF = (7058.71 sec) * 7.65 = 53999.1 sec = 900 min
</pre>

<p>The next step to find out if this exceeds the CPU limit of the 
queue in which the job was run.  In this example, the job was 
the xlong queue:

<pre>
&gt; bqueues -l xlong
</pre>

<p>Among other things, this tells you the CPU limit for the queue:

<pre>
 CPULIMIT
 2900.0 min of slac
</pre>

<p>The job used only 900 minutes of SLAC time, less than the 2900 allowed 
by the xlong queue.  So the job did not exceed its CPU time limit.
It must have crashed for some other reason.

<h2>Memory Leaks</h2><a name="memleak">

<p>Jobs can also crash because of memory leaks --- things like dangling 
pointers or array overruns.  The following links may be helpful for tracking 
down memory leaks:
<ul>
<li><a href="http://www.slac.stanford.edu/BFROOT/www/Computing/Programming/QC/LeakCheck/index.html">Memory leaks webpage</a>
<li><a href="http://www.slac.stanford.edu/BFROOT/www/Computing/Environment/Tools/valgrind.html">Valgrind at BABAR</a>
</ul>

<br>
<hr>
<br>Author: <a href="mailto:smclac@physics.mcgill.ca">Sheila Mclachlin</a>
<br>Created: Feb 09, 2006.
<br>Last updated: Feb 13, 2006 by Gregory Dubois-Felsmann.

</body>
</html>
