README.md 8.62 KB
Newer Older
1 2

[![License: LGPL v3](https://img.shields.io/badge/License-LGPL%20v3-blue.svg)](https://www.gnu.org/licenses/lgpl-3.0)
3 4
[![pipeline status](https://gitlab.bioinfo-diag.fr/vidjil/should/badges/dev/pipeline.svg)](https://gitlab.bioinfo-diag.fr/vidjil/should/commits/dev)
[![coverage report](https://gitlab.bioinfo-diag.fr/vidjil/should/badges/dev/coverage.svg)](https://gitlab.bioinfo-diag.fr/vidjil/should/commits/dev)
5

6
### `should` -- Test command-line applications through `.should` files
7 8


Mathieu Giraud's avatar
Mathieu Giraud committed
9
`should` is a single-file program to test command-line applications on Unix-like systems.
10 11
It checks the standard output, or possibly another file, and parses for exact or regular expressions,
possibly while counting them and checking their number of occurrences.
Mathieu Giraud's avatar
Mathieu Giraud committed
12 13
It also parses and tests JSON data.
`should` outputs reports in [`.tap` format](https://testanything.org/tap-specification.html)
14
and in JUnit-like XML.
15

Mathieu Giraud's avatar
Mathieu Giraud committed
16
`should` is written in Python with no external dependencies except from Python >= 3.4
Mathieu Giraud's avatar
Mathieu Giraud committed
17
and is intended to work on any command-line application
18 19 20
-- should your application outputs something, you can test it!


21 22
### Download

23
The archive [should-3.0.0.tar.gz](https://gitlab.bioinfo-diag.fr/vidjil/should/-/archive/3.0.0/should-3.0.0.tar.gz)
24
also includes this documentation with demo files.
25
In your projetcts, you only need the [should.py](https://gitlab.bioinfo-diag.fr/vidjil/should/raw/master/src/should.py) file.
26

27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73

### Basic usage


The [demo/hello.should](demo/hello.should) example covers the basic functionality :

```shell
# A .should file launches any command it encounters.
# Every line starting with a `#` is a comment.

echo "hello, world"

# Lines containing a `:` are test lines.
# The `test expression` is what is found at the right of the `:`.
# It should be found in the stdout, at least one time.

:world
:lo, wo


# What is at the left of the `:` are modifiers.
# One can specify the exact number of times the test expression has to appear.

1:hello
0:Bye-bye


# Lines beginning by `$` give a name to the following tests

$ Check space before world
1:, wo

$ Check misspellings
0:wrld
0:helo


# The `r` modifier triggers Python regular expressions

$ Two o´s
r: o.*o

$ A more flexible test
r: [Hh]ello,\s+world
```


74
The test is then run by calling `should` on the `.should` file:
75 76

```shell
77
> ./should demo/hello.should
78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101
$LAUNCHER=

demo/hello.should
**echo "hello, world"**
  stdout --> 1 lines
  stderr --> 0 lines

ok
ok
ok
ok
ok - Check space before world
ok - Check misspellings
ok - Check misspellings
ok - Two o´s
ok - A more flexible test
ok - Exit code is 0
==> ok - ok:10 total:10 tests

demo/hello.should
==> ok - ok:10 total:10 tests
```


102
`should` can be run on several `.should` files at once. In this case, it furthers shows statistics
103 104 105
on all tests. Here 39 out of 40 tests passed from 8 `.should` files.

```shell
106
> ./should demo/*.should
107 108 109 110 111 112 113 114 115 116
(...)

Summary ==> ok - ok:8 total:8 files
Summary ==> ok - ok:39 TODO:1 total:40 tests

```


### Documentation

Mathieu Giraud's avatar
Mathieu Giraud committed
117
The documentation is completed by the files in [demo/](demo/).
118 119 120

The [demo/cal.should](demo/cal.should) example shows
matching on several lines (`l`), counting inside lines (`w`),
121 122
ignoring whitespace differences (`b`), expecting a test to fail (`f`)
or allowing a test to fail (`a`),
Mathieu Giraud's avatar
Mathieu Giraud committed
123
requiring less than or more than a given number of expressions (`<`/`>`).
124

125
[demo/commands.should](demo/commands.should) shows that several commands can be used into a same `.should` file. Tests are flushed after each set of commands.
Mathieu Giraud's avatar
Mathieu Giraud committed
126

127
[demo/exit-codes.should](demo/exit-codes.should) shows how to require a particular exit code with `!EXIT_CODE`.
128

129 130 131
[demo/unicode.should](demo/unicode.should) shows that unicode characters can be put
in test comment and strings, provided that a UTF-8 locale has been set up.

132 133 134 135
[demo/variables.should](demo/variables.should) shows how to define and use variables.

[demo/launcher.should](demo/launcher.should) shows how to define a "launcher" that is appended before every command,
for example to launch tools like `valgrind` on a test set.
136
[demo/extra.should](demo/extra.should) show how to add an argument to every tested command.
137

Mathieu Giraud's avatar
Mathieu Giraud committed
138
[demo/json.should](demo/json.should) shows how to test JSON data (keys and values).
139 140 141 142

**Options and modifiers**

```shell
Mathieu Giraud's avatar
Mathieu Giraud committed
143 144 145 146 147 148 149
usage: should [--cd PATH] [--cd-same] [--launcher CMD] [--extra ARG]
              [--mod MODIFIERS] [--var NAME=value] [--timeout TIMEOUT]
              [--shuffle] [--no-a] [--no-f] [--only-a] [--only-f]
              [--retry] [--retry-warned]
              [--log] [--tap] [--xml]
              [-v] [-q] [--fail-a]
              [-h] [--version]
150
              should-file [should-file ...]
151 152 153 154

Test command-line applications through .should files

positional arguments:
Mathieu Giraud's avatar
Mathieu Giraud committed
155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170
  should-file        input files (.should)

running tests (can also be set per test in !OPTIONS):
  --cd PATH          directory from which to run the test commands
  --cd-same          run the test commands from the same directory as the .should files
  --launcher CMD     launcher preceding each command (or replacing $LAUNCHER)
  --extra ARG        extra argument after the first word of each command (or replacing $EXTRA)
  --mod MODIFIERS    global modifiers (uppercase letters cancel previous modifiers)
                       f/F consider that the test should fail
                       a/A consider that the test is allowed to fail
                       r/R consider as a regular expression
                       w/W count all occurrences, even on a same line
                       i/I ignore case changes
                       b/B ignore whitespace differences as soon as there is at least one space. Implies 'r'
                       l/L search on all the output rather than on every line
                       z/Z keep leading and trailing spaces
Mathieu Giraud's avatar
Mathieu Giraud committed
171
                       j/J interpret json data. Implies 'lw'
Mathieu Giraud's avatar
Mathieu Giraud committed
172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194
                       >   requires that the expression occurs strictly more than the given number
                       <   requires that the expression occurs strictly less than the given number
  --var NAME=value   variable definition (then use $NAME in .should files)
  --timeout TIMEOUT  Delay (in seconds) after which the task is stopped (default: 120)

selecting tests to be run:
  --shuffle          shuffle the tests
  --no-a             do not launch 'a' tests
  --no-f             do not launch 'f' tests
  --only-a           launches only 'a' tests
  --only-f           launches only 'f' tests
  --retry            launches only the last failed tests
  --retry-warned     launches only the last failed or warned tests

controlling output:
  --log              stores the output into .log files
  --tap              outputs .tap files
  --xml              outputs JUnit-like XML into should.xml
  -v, --verbose      increase verbosity
  -q, --quiet        verbosity to zero
  --fail-a           fail on passing 'a' tests
  -h, --help         show this help message and exit
  --version          show program version number and exit
Mathieu Giraud's avatar
Mathieu Giraud committed
195
```
196

Mathieu Giraud's avatar
Mathieu Giraud committed
197 198 199 200 201 202 203 204 205 206 207 208
**Configuration files.**
Options can be set through a `should.cfg` file.
For example, a `should.cfg`.with the following lines is equivalent to giving `--xml --mod r` options to every `should` call:
```
--xml

--mod
r
```

Other configuration files can be provided when calling should with `@my.cfg`, see [demo/variables.should](demo/variables.should).

Mathieu Giraud's avatar
Mathieu Giraud committed
209
**Output.**
210
By default, `should` only writes to the standard output.
211 212 213
The `--log`, `--tap` and `--xml` options enable to store the actual output of the tested commands
as well as [`.tap` files](https://testanything.org/tap-specification.html)
and a JUnit `should.xml` file.
Mathieu Giraud's avatar
Mathieu Giraud committed
214

215
**Exit code.**
Mathieu Giraud's avatar
Mathieu Giraud committed
216
`should.py` returns `0` when all the tests passed (or have been skipped, or marked as `failed-with-ALLOW` with `a`).
217 218
As soon as one regular test fails, or as soon as a test marked with `TODO` pass,
`should.py` returns `1`.
219 220 221 222


### Alternatives

223 224 225 226
* Aruba: Testing with Cucumber, RSpec, Minitest -- https://github.com/cucumber/aruba
* Cram: It's test time -- https://bitheap.org/cram/
* Bats: Bash Automated Testing System -- https://github.com/sstephenson/bats
* Expect -- http://core.tcl.tk/expect
227
* Tush: Literate testing for command-line programs -- https://github.com/darius/tush
228 229 230 231 232 233

See also https://stackoverflow.com/questions/353198/best-way-to-test-command-line-tools.


### History

234
`should` is a refactor of an earlier shell script that was developed and heavily used since 2013
235 236
to test two bioinformatics programs,
[CRAC](http://crac.gforge.inria.fr/) (mapping of RNA sequences, discovery of transcriptomic and genomic variants),
237 238
[Vidjil-algo](http://www.vidjil.org/) (clustering and analysis of lymphocyte clusters from their DNA sequences,
contains [700+ `should` tests](https://gitlab.inria.fr/vidjil/vidjil/tree/dev/algo/tests/should-get-tests) in about 100 files).