Skip to content

Commit 8c5d7ac

Browse files
iHiDiHiD
committed
Start adding test runner docs
1 parent 5fe420a commit 8c5d7ac

File tree

9 files changed

+234
-9
lines changed

9 files changed

+234
-9
lines changed

‎README.md

+7
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,10 @@
33
# Exercism Docs
44

55
These are the official docs for Exercism. We welcome all and any contributions.
6+
7+
## How this repo is organised
8+
9+
There are three top-level repositories:
10+
- **anatomy:** How Exercism fits together with technical specs for maintainers.
11+
- **product:** The product philosophy and decisions behind Exercism.
12+
- **user-facing:** User facing docs for FAQs, help, etc

‎TODO.md

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# TODO
2+
3+
These are things that should be done before this repo becomes public:

‎maintenance/.gitkeep ‎anatomy/.gitkeep

File renamed without changes.

‎anatomy/README.md

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# The Anatomy of Exercism
2+
3+
The documents in this section explore the different areas of Exercism, how they are built, and the technical specifications for each area.
4+
5+
These documents are written for maintainers and contributors to Exercism, rather than end-users, and so may presume more innate knowledge of Exercism.
6+

‎anatomy/test-runners/README.md

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Test Runners
2+
3+
A Test Runner is a piece of software that runs a an exercise's tests against a student's code.
4+
5+
Test runners give us two advantages
6+
1. They enable in-browser coding, where a student can iterate on a solution seeing whether the tests pass or fail.
7+
2. They allow us to provide information to mentors on whether a student's code "works" or not.
8+
9+
Each language has it's own test runner, written in that language.
10+
The website acts as the orchestrator between the test runners and student's submissions.
11+
12+
Each test runner lives in the Exercism GitHub organsation in a repository named `$LANG-test-runner` (e.g. `ruby-test-runner`).
13+
You can explore the different test runners [here](https://github.com/exercism?q=-test-runner).
14+
15+
If you would like to get involved in helping with an existing Test Runner, please open an issue in its repository asking if there is somewhere you can help.
16+
If you would like to create a Test Runner for a language that currently does not have one, please follow the [creating a Test Runner](creating-from-scratch.md) instructions.
17+
18+
This directory contains the following information:
19+
- **`creating-from-scratch.md:`** Information on creating a Test Runner from scratch.
20+
- **`interface.md:`** The Test Runner interface.
21+
- **`docker.md:`** How to build a Docker image with Docker for local testing and deployment.
22+
23+
24+

‎anatomy/test-runners/creating-from-scratch.md

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Creating an Test Runner
2+
3+
Firstly, thank you for your interest in creating a Test Runner!
4+
5+
These are the steps to get going:
6+
7+
1. Check [our repository list for an existing `...-test-runner`](https://github.com/exercism?q=-test-runner) to ensure that one doesn't already exist.
8+
2. Scan the [contents of this directory](./) to ensure you are comfortable with the idea of creating an Test Runner.
9+
3. Open an issue at [exercism/exercism](https://github.com/exercism/exercism/issues/new) introducing yourself and telling us which language you'd like to create a Test Runner for.
10+
11+
We have an incredibly friendly and supportive community who will be happy to help you as you work through this! Just create new issues in this repository as needed :slightly-smiling-face:

‎anatomy/test-runners/docker.md

+32
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Docker
2+
3+
Our Test Runners are deployed as Docker images.
4+
5+
Each Test Runner requires a Dockerfile.
6+
This file specifies how the machine is built.
7+
It should live at the root directory of your repository and should be called `Dockerfile`.
8+
9+
The Dockerfile should create the minimal image needed for the Test Runner to run the tests.
10+
11+
It should produce an image with as a small a size as possible.
12+
The minimal-sized image is often based on Alpine Linux.
13+
Applying the official [Dockerfile best practices](https://docs.docker.com/develop/develop-images/Dockerfile_best-practices/) can help to create a minimal image.
14+
15+
## Executing the Docker container
16+
17+
When we run the Docker container we execute a `run.sh` script.
18+
To ensure this works properly the following rules must be following:
19+
20+
- The working directory should be `/opt/test-runner`.
21+
- There should be a `/opt/test-runner/bin/run.sh` script that can be called with 3 parameters:
22+
the `exercise slug`, the path to the `solution folder`, and the path to the `output folder`.
23+
For more information see [The Interface](./interface.md).
24+
25+
## Local development
26+
27+
To allow for local development we have produced an exectuable called [tooling-webserver](https://github.com/exercism/tooling-webserver/blob/master/README.md#installation-docker).
28+
Please follow its installation instructions.
29+
30+
## Creating/editing a Dockerfile
31+
32+
All changes to Dockerfiles require a PR review from the @exercism/ops team.

‎anatomy/test-runners/interface.md

+151
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
# The Interface
2+
3+
Test runners have the single responsibility of taking a solution, running all tests and returning a standardised output.
4+
All interactions with the Exercism website are handled automatically and are not part of this specification.
5+
6+
## Execution
7+
8+
- A test runner should provide an executable script. You can find more information in the [docker.md](./docker.md) file.
9+
- The script will receive three parameters:
10+
- The slug of the exercise (e.g. `two-fer`).
11+
- A path to an input directory (with a trailing slash) containing the submitted solution file(s) and any other exercise file(s). This directory should be considered as read-only. It's technically possible to write into it but it's better to use `/tmp` for temporary files (e.g. for compiling sources).
12+
- A path to an output directory (with a trailing slash). This directory is writable.
13+
- The script must write a `results.json` file to the output directory.
14+
- The runner must exit with an exit code of 0 if it has run successfully, regardless of the status of the tests.
15+
16+
## Output format
17+
18+
The `results.json` file should be structured as followed:
19+
20+
```json
21+
{
22+
"status": "fail",
23+
"message": null,
24+
"tests": [
25+
{
26+
"name": "Test that the thing works",
27+
"status": "fail",
28+
"message": "Expected 42 but got 123123",
29+
"output": "Debugging information output by the user",
30+
"cmd": "assert_equal 42, answerToTheUltimateQuestion()",
31+
}
32+
]
33+
}
34+
```
35+
36+
### Top level
37+
38+
#### Status
39+
40+
> key: `status`
41+
42+
The following overall statuses are valid:
43+
- `pass`: All tests passed
44+
- `fail`: At least one test failed
45+
- `error`: To be used when the tests did not run correctly (e.g. a compile error, a syntax error)
46+
47+
#### Message
48+
49+
> key: `message`
50+
51+
Where the status is `error` (the tests fail to execute cleanly), the top level `message` key should be provided. It should provide the occurring error to the user. As it is the only piece of information a user will receive on how to debug their issue, it must be as clear as possible. For example, in Ruby, in the case of a syntax error, we provide the error and stack trace. In compiled languages, the compilation error should be provided. The top level `message` value is not limited in length.
52+
53+
When the status is not `error`, either set the value to `null` or omit the key entirely.
54+
55+
### Per-test
56+
57+
#### Name
58+
59+
> key: `name`
60+
61+
This is the name of the test in a human-readable format.
62+
63+
#### Command
64+
65+
> key: `cmd`
66+
67+
This is the body of the command that is being tests. It should have any `skip` annotations removed. For example, the following Ruby test:
68+
```ruby
69+
def test_duplicate_items_uniqs_list
70+
skip
71+
cart = ShoppingCart.new
72+
cart.add(:STARIC)
73+
cart.add(:MEDNEW)
74+
cart.add(:MEDNEW)
75+
assert_equal 'Newspaper, Rice', cart.items_list
76+
end
77+
```
78+
79+
... should return a cmd value of:
80+
```ruby
81+
"cart = ShoppingCart.new
82+
cart.add(:STARIC)
83+
cart.add(:MEDNEW)
84+
cart.add(:MEDNEW)
85+
assert_equal 'Newspaper, Rice', cart.items_list"
86+
```
87+
88+
(with linebreaks replaced by `\n` to make the JSON valid).
89+
90+
#### Status
91+
92+
> key: `status`
93+
94+
The following per-test statuses are valid:
95+
- `pass`: The test passed
96+
- `fail`: The test failed
97+
- `error`: The test errored
98+
99+
#### Message
100+
101+
> key: `message`
102+
103+
The per-test `message` key is used to return the results of a failed test. It should be as human-readable as possible. Whatever is written here will be displayed to the student when their test fails. If there is no error message, either set the value to `null` or omit the key entirely. It is also permissible to output test suite output here. The `message` value is not limited in length.
104+
105+
#### Output
106+
107+
> key: `output`
108+
109+
The per-test `output` key should be used to store and output anything that a user deliberately outputs for a test.
110+
111+
- It should be attached to all test results that produce user output.
112+
- Only content outputted by a user manually should show - not automatic output by the test-runner.
113+
- You may either capture content that is output through normal means (e.g. `puts` in Ruby, `print` in Python or `Debug.WriteLine` in C#), or you may provide a method that the user may use (e.g. the Ruby Test Runner provides a user with a globally available `debug` method that they can use, which has the same characteristics as the standard `puts` method).
114+
- The output **must** be limited to 500 chars. Either truncating with a message of "Output was truncated. Please limit to 500 chars" or returning an error in this situation are acceptible.
115+
116+
### UI/UX concerns
117+
118+
#### On test failure
119+
120+
When a student's solution fails a test, it should display something like:
121+
122+
```text
123+
Test Code:
124+
<cmd>
125+
126+
Test Result:
127+
<message>
128+
```
129+
130+
#### On test success
131+
132+
When the solution passes a test, it should display something like:
133+
134+
```text
135+
Test Code:
136+
<cmd>
137+
```
138+
139+
### How to add metadata for your language's test suite
140+
141+
All roads lead to Rome and there is no prescribed pattern to arrive at this. There are several approaches taken so far:
142+
143+
- Auxillary JSON files compiled manually, merged with test results during the test run-time.
144+
- Automated static analysis of the test suite, merged with test results during the test run-time.
145+
- This may be accomplished by AST analysis or text-parsing
146+
147+
## Debugging
148+
149+
The contents of `stdout` and `stderr` from each run will be stored in files that can be viewed later.
150+
151+
You may write an `results.out` file to the output directory, which contains debugging information you want to later view.

‎product/test-runners.md

-9
This file was deleted.

0 commit comments

Comments
 (0)