Commit 001846a3 authored by Mathieu Giraud's avatar Mathieu Giraud

README.md: draft welcome page and documentation

parent 1803994c
Pipeline #709 failed with stages
in 4 seconds
[![License: LGPL v3](https://img.shields.io/badge/License-LGPL%20v3-blue.svg)](https://www.gnu.org/licenses/lgpl-3.0)
### `should.py` -- Test command-line applications through `.should` files
`should.py` is a single-file program to test command-line applications.
It checks the standard output, or possibly another file, and parses for exact or regular expressions,
possibly while counting them and checking their number of occurrences.
It outputs reports in [`.tap` format](https://testanything.org/tap-specification.html)
and (soon, #xxxx) in JUnit XML.
`should.py` is written in Python 3 but is intended to work on any command-line application
-- should your application outputs something, you can test it!
### Basic usage
The [demo/hello.should](demo/hello.should) example covers the basic functionality :
```shell
# A .should file launches any command it encounters.
# Every line starting with a `#` is a comment.
echo "hello, world"
# Lines containing a `:` are test lines.
# The `test expression` is what is found at the right of the `:`.
# It should be found in the stdout, at least one time.
:world
:lo, wo
# What is at the left of the `:` are modifiers.
# One can specify the exact number of times the test expression has to appear.
1:hello
0:Bye-bye
# Lines beginning by `$` give a name to the following tests
$ Check space before world
1:, wo
$ Check misspellings
0:wrld
0:helo
# The `r` modifier triggers Python regular expressions
$ Two o´s
r: o.*o
$ A more flexible test
r: [Hh]ello,\s+world
```
The test is then run by calling `should.py` on the `.should` file:
```shell
> python3 src/should.py demo/hello.should
$LAUNCHER=
demo/hello.should
**echo "hello, world"**
stdout --> 1 lines
stderr --> 0 lines
ok
ok
ok
ok
ok - Check space before world
ok - Check misspellings
ok - Check misspellings
ok - Two o´s
ok - A more flexible test
ok - Exit code is 0
==> ok - ok:10 total:10 tests
demo/hello.should
==> ok - ok:10 total:10 tests
```
`should.py` can be run on several `.should` files at once. In this case, it furthers show statistics
on all tests. Here 39 out of 40 tests passed from 8 `.should` files.
```shell
> python3 src/should.py demo/*.should
(...)
Summary ==> ok - ok:8 total:8 files
Summary ==> ok - ok:39 TODO:1 total:40 tests
```
### Documentation
The files in [demo/](demo/) showcase testing modifiers:
parsing exact or regular expression (`r`), matching on several lines (`l`), counting inside lines (`w`),
ignoring whitespace differences (`b`), expecting a test to fail (`f`).
[demo/variables.should](demo/variables.should) shows how to define and use variables.
[demo/launcher.should](demo/launcher.should) shows how to define a "launcher" that is appended before every command,
for example to launch tools like `valgrind` on a test set.
**Options and modifiers**
```shell
python3 should.py -h
usage: should.py [-h] [--cd PATH] [--cd-same] [--launcher CMD]
[--mod MODIFIERS] [--var NAME=value] [--log] [--tap] [-v]
[-q]
should-file [should-file ...]
Test command-line applications through .should files
positional arguments:
should-file input files (.should)
optional arguments:
-h, --help show this help message and exit
--cd PATH directory from which to run the test commands
--cd-same run the test commands from the same directory as the .should files
--launcher CMD launcher preceding each command (or replacing $LAUNCHER)
--mod MODIFIERS global modifiers (uppercase letters cancel previous modifiers)
f/F consider that the test should fail
r/R consider as a regular expression
w/W count all occurrences, even on a same line
b/B ignore whitespace differences as soon as there is at least one space. Implies 'r'
l/L search on all the output rather than on every line
z/Z keep leading and trailing spaces
--var NAME=value variable definition (then use $NAME in .should files)
output options:
--log stores the output into .log files
--tap outputs .tap files
-v, --verbose increase verbosity
-q, --quiet verbosity to zero
```
**Exit code.**
`should.py` returns `0` when all the tests passed (or have been skipped, or marked as TODO with `f`). A
As soon as one non-TODO test fails, it returns `1`.
### Alternatives
* https://github.com/cucumber/aruba
* https://bitheap.org/cram/
* https://github.com/sstephenson/bats
See also https://stackoverflow.com/questions/353198/best-way-to-test-command-line-tools.
### History
`should.py` is a refactor of an earlier shell script that was developed and heavily used since 2014
to test [Vidjil-algo](https://gitlab.inria.fr/vidjil/vidjil/tree/dev/algo/tests/should-get-tests),
a bioinformatics program describing lymphocyte clusters from their DNA sequences.
  • Que se passe-t-il si la commande à lancer contient un : ?

    Par exemple echo "He said: “Who are you?”" (on peut imaginer des cas plus réalistes avec un awk où les champs sont séparés par des :, ou un scp, ou…)

    By Mikaël Salson on 2018-04-18T10:03:46 (imported from GitLab project)

  • We use in this case !LAUNCH:. See fbf2c30d.

    By Mathieu Giraud on 2018-04-18T10:08:31 (imported from GitLab project)

  • en fait ça date même de crac ;)

    By Mikaël Salson on 2018-04-18T10:10:00 (imported from GitLab project)

  • Yes, I was thinking to Alecsia but I forgot CRAC. Please update as needed :-)

    By Mathieu Giraud on 2018-04-18T10:12:16 (imported from GitLab project)

  • 659727f8

    By Mathieu Giraud on 2018-06-13T09:56:44 (imported from GitLab project)

Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment