logtalk_tester.1

.TH logtalk_tester 1 "December 29, 2024" "Logtalk 3.87.0" "Logtalk Documentation"

.SH NAME
logtalk_tester \- Logtalk script for automating running unit tests

.SH SYNOPSIS
.B logtalk_tester -v
.br
.B logtalk_tester -p \fIprolog\fR
[\fB-o \fIoutput\fR]
[\fB-m \fImode\fR\fR]
[\fB-f \fIformat\fR]
[\fB-d \fIresults\fR]
[\fB-t \fItimeout\fR]
[\fB-n \fIdriver\fR]
[\fB-s \fIprefix\fR]
[\fB-b \fItracker\fR]
[\fB-u \fIurl\fR]
[\fB-c \fIreport\fR]
[\fB-l \fIlevel\fR]
[\fB-e \fIexclude\fR]
[\fB-i \fIoptions\fR]
[\fB-g \fIgoal\fR]
[\fB-r \fIseed\fR]
[\fB-w\fR]
[\fB-- \fIarguments\fR]
.br
.B logtalk_tester -h

.SH DESCRIPTION
\fBlogtalk_tester\fR is a script for automating running Logtalk unit tests. This script runs unit tests found in the current directory and recursively in its sub-directories by scanning by default for \fItester.lgt\fR and \fItester.logtalk\fR source files. The script returns a non-zero exit status in case of failed tests or test set errors. When a \fItester.sh\fR file exists in the tests directory, the file is sourced before running the tests. The \fItester.sh\fR file is sourced with all the parameters passed to the script. The script prints by default a full report of test compilation errors and warnings, skipped test sets, broken test sets, crashed test sets, skipped tests, failed tests, test set stats, and test stats. Note that the existence of broken or crashed test sets may result in some inconsistency in test stats, notably in the number of passed and failed tests.

.SH OPTIONS
.TP
.BI \-v
Print script version.
.TP
.BI \-p
Backend Prolog compiler. Required. The following identifiers are used: B-Prolog - \fIb\fR, Ciao Prolog - \fIciao\fR, CxProlog - \fIcx\fR, ECLiPSe - \fIeclipse\fR, GNU Prolog - \fIgnu\fR, GNU Prolog (native code) - \fIgnunc\fR, JIProlog - \fIji\fR, XVM - \fIxvm\fR, SICStus Prolog - \fIsicstus\fR, SWI-Prolog - \fIswi\fR, SWI-Prolog with Logtalk pack - \fIswipack\fR, Tau Prolog - \fItau\fR, Trealla Prolog - \fItrealla\fR, XSB - \fIxsb\fR, YAP - \fIyap\fR.
.TP
.BI \-o
Output verbosity. Default is \fIverbose\fR. Valid values are \fIverbose\fR and \fIminimal\fR.
.TP
.BI \-m
Compilation mode. Default is \fInormal\fR. Valid values are \fIoptimal\fR, \fInormal\fR, \fIdebug\fR, and \fIall\fR. This option only sets the default compilation mode. Code compiled using an explicit mode is not affected.
.TP
.BI \-f
Format for writing the test results. Default is \fIdefault\fR. Valid values are \fIdefault\fR, \fItap\fR, \fIxunit\fR, and \fIxunit_net_v2\fR.
.TP
.BI \-d
Directory to store the test results. Default is the local \fI./logtalk_tester_logs\fR directory.
.TP
.BI \-t
Timeout in seconds for running each test set. Default is \fI0\fR (i.e., disabled).
.TP
.BI \-n
Name of the tests driver and sourced files (minus file name extensions). Default is \fItester\fR.
.TP
.BI \-s
Path prefix to suppress when printing test suite and test result paths. Default is the user home directory.
.TP
.BI \-b
Bug report server and optionally the bug report label(s). Valid values are \fIgithub\fR and \fIgitlab\fR. (no default). Use of this option requires the script to be called from a git repo directory or from one its sub-directories and the use of the \fI-s\fR and \fI-u\fR options. To set the label(s), use the syntax \fI-b server:labels\fR where \fIlabels\fR are a comma-separated list.
.TP
.BI \-u
Base URL for generating links to test files (no default). Must end with a slash and usually includes a git hash so that the link remains valid if the test files are later updated.
.TP
.BI \-c
Code coverage report. Default is \fInone\fR. Valid values are \fIxml\fR and \fInone\fR.
.TP
.BI \-l
Directory depth level to look for test sets. Level 1 means the current directory only. Default is to recursively look into all sub-directories.
.TP
.BI \-e
Regular expression for excluding directories (using the \fIfind\fR command support for POSIX-extended regular expressions).
.TP
.BI \-i
Integration script command-line options (no default).
.TP
.BI \-g
Initialization goal. Default is \fItrue\fR (no period). Called after loading the Logtalk compiler/runtime and before running the tests.
.TP
.BI \-r
Random generator starting seed (no default).
.TP
.BI \-w
Wipe test set default scratch directories (\fI./.lgt_tmp\fR and \fI./lgt_tmp\fR) before running it. This option should not be used when running parallel processes that use the same test sets.
.TP
.BI \--
Arguments to be passed to the tests (no default).
.TP
.B \-h
Provide help on script usage.

.SH "EXIT STATUS"
.TP
.B 0
All non-skipped tests pass (flaky test failures are ignored). There are no broken, timedout, or crashed test sets.
.TP
.B 1
Some (non-flaky) tests fail. But there are no broken, timedout, or crashed test sets.
.TP
.B 3
One or more test sets timedout while executing. But there are no broken or crashed test sets.
.TP
.B 5
One or more test sets are broken and thus could not be executed. But there are no crashed test sets.
.TP
.B 7
One or more test sets crashed the process running them.
.TP
.B 9
The sourced \fItester.sh\fR file returned a non-zero code.
.TP
.B 11
The \fIfind\fR command used internally to enumerate test sets returned a non-zero code (most likely due to a non-supported option).

.SH EXAMPLES
.TP
Run all tests in the current directory and sub-directories using the GNU Prolog backend, with results in TAP format, code coverage reports in XML format, and passing arguments \fIfoo\fR, \fIbar\fR, and \fIbaz\fR to the tests:
\fBlogtalk_tester -p gnu -tap -c xml -- foo bar baz\fR
.PP
.TP
Run all tests in the current directory only using the ECLiPSe backend, with a timeout of 4 minutes per test set:
\fBlogtalk_tester -p eclipse -l 1 -t 240\fR
.PP
.TP
Run all tests in the current directory and sub-directories using the SICStus Prolog backend and a random generator starting seed:
\fBlogtalk_tester -p sicstus -r "seed(3172,9814,20125)"\fR
.PP
.TP
Run all tests in the current directory and sub-directories using the SWI-Prolog backend and looking for \fItester_debug\fR driver and sourced files:
\fBlogtalk_tester -p swi -n tester_debug\fR
.PP
.TP
Run all tests in the current directory and sub-directories using the Tau Prolog backend, caching the compilation of the testing tool for better performance, but wiping the test set default scratch directories before running it:
\fBlogtalk_tester -p tau -g "set_logtalk_flag(clean,off)" -w\fR
.PP
.TP
Change the default Logtalk compiler scratch directory for compiling the tests:
\fBlogtalk_tester -p sicstus -g "set_logtalk_flag(scratch_directory,'$LOGTALKUSER/scratch')"\fR
.PP
.TP
Allow parallel Logtalk processes to run the same tests concurrently:
\fBlogtalk_tester -p xvm "-i $LOGTALKUSER/library/parallel_logtalk_processes_setup.pl"\fR
.PP
.TP
Run the Logtalk distribution tests using the Trealla Prolog backend, passing a base URL that can be used for creating links to the test files:
\fBlogtalk_tester -p trealla -f xunit_net_v2 -s "$LOGTALKUSER/" -u https://github.com/LogtalkDotOrg/logtalk3/tree/55aa900775befa135e0d5b48ea63098df8b97f5c/\fR
.PP
.TP
Run the Logtalk distribution tests using the GNU Prolog backend, passing a base URL to automatically create bug reports:
\fBlogtalk_tester -p gnu -b github:bug,auto -s "$HOME/logtalk3/" -u https://github.com/LogtalkDotOrg/logtalk3/tree/$(git rev-parse HEAD)/\fR
.PP
.TP
Run all tests in the current directory and sub-directories using the YAP backend with its disable signal handling option:
\fBlogtalk_tester -p yap -i "-nosignals"\fR
.PP
.TP
Run all Prolog standards compliance tests with the exception of the \fIcatch_3\fR and \fIcut_0\fR test sets:
\fBlogtalk_tester -p swi -e ".*/(catch_3|cut_0)/.*"\fR

.SH IMPLEMENTATION NOTES
When the script detects either a \fItimeout\fR or a \fIgtimeout\fR command (provided by the GNU \fIcoreutils\fR package), it will use it to run each test set if the timeout option is set to a value greater than zero. This package can be easily installed in POSIX operating-systems (including Linux and macOS) if not provided by default. On Windows operating-systems, the native timeout command is not usable for this purpose. But the Git for Windows installer includes the GNU \fIcoreutils\fR package, which can be used from the bundled Bash shell.
.TP
Installation of the GNU \fIsed\fR command, when not available by default, is strongly recommended. The Git for Windows installer includes this command, which can be used from the bundled Bash shell.
.TP
The version of the \fIfind\fR command provided by some operating-systems doesn't support (extended) regular-expressions, thus preventing using the \fI-e\fR option.
.TP
The script kills all created child processes on exit using the \fIpkill\fR command.
.TP
The \fIgnunc\fR backend requires creating a GNU Prolog executable that includes Logtalk named \fIgplgtnc\fR and making it available in the system \fIPATH\fR. This executable can be created using the embedding script provided in the Logtalk distribution. For example:
\fBgprolog_embed_logtalk.sh -c -d ~/collect -s none -- --new-top-level && mv ~/collect/logtalk ~/collect/gplgtnc\fR

.SH ENVIRONMENT
.TP
.B LOGTALKHOME
Specifies the location of the Logtalk installation.
.TP
.B LOGTALKUSER
Specifies the location of the Logtalk user directory.

.SH FILES
.TP
.BI $LOGTALKUSER/library/tester_versions.lgt
Logtalk source file used for printing the Logtalk and backend Prolog compiler versions.
.TP
.BI $LOGTALKUSER/library/parallel_logtalk_processes_setup.pl
Logtalk source file used for parallel Logtalk processes with selected backend Prolog compilers.
.TP
.BI $LOGTALKUSER/tools/lgtunit/automation_report.lgt
Logtalk source file that intercepts unit test execution messages and generates report files that are parsed by this script.

.SH "SEE ALSO"
The Logtalk website at https://logtalk.org/
.PP
The Logtalk User and Reference Manuals at \fB$LOGTALKHOME/manuals/index.html\fR

.SH AUTHOR
Paulo Moura <pmoura@logtalk.org>

.SH COPYRIGHT
Copyright (c) 1998-2024 Paulo Moura.