Merge pull request #215 from jpthiele/local_testing_R_Julia

Add local testing for R and Julia

content/code/R/example.R

@@ -0,0 +1,15 @@
+if (!require(testthat)) install.packages(testthat)
+add <- function(a, b) {
+    return(a + b)
+}
+
+test_that("Adding integers works", {
+    res <- add(2, 3)
+
+    # Test that the result has the correct value
+    expect_identical(res, 5)
+
+    # Test that the result is numeric
+    expect_true(is.numeric(res))
+})
+

content/code/julia/example.jl

@@ -0,0 +1,8 @@
+function myadd(a,b) 
+    return a+b
+end
+
+using Test
+@testset "myadd" begin
+    @test myadd(2,3) == 5
+end

content/code/python/example.py

@@ -0,0 +1,6 @@
+def add(a, b):
+    return a + b
+
+def test_add():
+    assert add(2, 3) == 5
+    assert add('space', 'ship') == 'spaceship'

content/guide.md

@@ -3,7 +3,7 @@
 ## Detailed schedule
 
 - 9:00-9:15 [Motivation](https://coderefinery.github.io/testing/motivation/)
-- 9:15-9:40 [Testing locally](https://coderefinery.github.io/testing/pytest/)
+- 9:15-9:40 [Testing locally](https://coderefinery.github.io/testing/locally/)
     - explain the exercise: 5 min
     - **20 min exercise**
 - 9:40-10:00 [Automated testing](https://coderefinery.github.io/testing/continuous-integration/)

content/index.rst

@@ -39,7 +39,7 @@ see how automated testing works and practice designing and writing tests.
    :delim: ;
 
    15 min ; :doc:`motivation`
-   25 min ; :doc:`pytest`
+   25 min ; :doc:`locally`
    30 min ; :doc:`continuous-integration`
    30 min ; :doc:`test-design`
    5 min ; :doc:`conclusions`
@@ -50,7 +50,7 @@ see how automated testing works and practice designing and writing tests.
    :caption: The lesson
 
    motivation
-   pytest
+   locally
    continuous-integration
    test-design
    conclusions

content/locally.md

@@ -0,0 +1,299 @@
+# Testing locally
+
+```{questions}
+- How hard is it to set up a test suite for a first unit test?
+```
+
+
+## Exercise 
+
+
+In this exercise we will make a simple function and use
+one of the language specific test frameworks to test it.
+
+* This is easy to use by almost any project and doesn't rely on any
+  other servers or services.  
+* The downside is that you have to remember to run it yourself.
+
+```````{exercise} Local-1: Create a minimal example (15 min)
+In this exercise, we will create a minimal example using
+the [pytest](http://doc.pytest.org), run the test, and show what
+happens when a test breaks.
+
+
+1. Create a new directory and change into it:
+   ```console
+   $ mkdir local-testing-example
+   $ cd local-testing-example
+   ```
+
+2. Create an example file and paste the following code into it
+`````{tabs}
+  ````{group-tab} Python
+    Create `example.py` with content
+
+    ```{literalinclude} code/python/example.py
+    :language: python
+    ``` 
+    This code contains one genuine function and a test function.
+   `pytest` finds any functions beginning with `test_` and treats them
+   as tests.
+  ````
+
+  ````{group-tab} R
+    Create `example.R` with content 
+    ```{literalinclude} code/R/example.R
+    :language: R
+    ```
+    A test with `testthat` is created by calling 
+    `test_that()` with a test name and code as arguments.
+  ````
+
+  ````{group-tab} Julia
+    Create `example.jl` with content 
+    ```{literalinclude} code/julia/example.jl
+    :language: Julia
+    ```
+    The package `Test.jl` handles all testing. 
+    A test(set) is added with `@testset` 
+    and a test itself with `@test`.
+  ````
+
+`````
+
+3. Run the test
+`````{tabs}
+  ````{group-tab} Python
+   ```console
+   $ pytest -v example.py
+
+   ============================================================ test session starts =================================
+   platform linux -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /home/user/pytest-example/venv/bin/python3
+   cachedir: .pytest_cache
+   rootdir: /home/user/pytest-example, inifile:
+   collected 1 item
+
+   example.py::test_add PASSED
+
+   ========================================================= 1 passed in 0.01 seconds ===============================
+   ```
+   Yay! The test passed!
+
+   Hint for participants trying this inside Spyder or IPython: try `!pytest -v example.py`.
+  ````
+  ````{group-tab} R
+    ```console
+    $ Rscript example.R 
+    
+    Loading required package: testthat
+    Test passed 🎉 
+    ```
+    Yay! The test passed! 
+
+    Note that the emoji is random and might be different for you.
+  ````
+  ````{group-tab} Julia
+    ```console
+    $ julia example.jl 
+    
+    Test Summary: | Pass  Total  Time
+    myadd         |    1      1  0.0s
+    ```
+    Yay! The test passed! 
+  ````
+`````
+4. Let us break the test!
+
+Introduce a code change which breaks the code and check
+whether out test detects the change:
+`````{tabs}
+  ````{group-tab} Python
+   ```console
+   $ pytest -v example.py
+
+   ============================================================ test session starts =================================
+   platform linux -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /home/user/pytest-example/venv/bin/python3
+   cachedir: .pytest_cache
+   rootdir: /home/user/pytest-example, inifile:
+   collected 1 item
+
+   example.py::test_add FAILED
+
+   ================================================================= FAILURES =======================================
+   _________________________________________________________________ test_add _______________________________________
+
+       def test_add():
+   >       assert add(2, 3) == 5
+   E       assert -1 == 5
+   E         --1
+   E         +5
+
+   example.py:6: AssertionError
+   ========================================================= 1 failed in 0.05 seconds ==============
+   ```
+   Notice how pytest is smart and includes context: lines that failed,
+   values of the relevant variables.
+  ````
+  ````{group-tab} R
+    ```console
+    $ Rscript example.R 
+
+    ── Failure: Adding integers works ──────────────────────────────
+    `res` not identical to 5.
+    1/1 mismatches
+    [1] -1 - 5 == -6
+
+    Error: Test failed
+    Execution halted
+    ```
+    `testthat` tells us exactly which test failed and how
+    but does not include more context.
+  ````
+  ````{group-tab} Julia
+    ```console
+    $ julia example.jl
+
+    myadd: Test Failed at /home/user/local-testing-example/example.jl:7
+    Expression: myadd(2, 3) == 5
+    Evaluated: -1 == 5
+
+    Stacktrace:
+    [1] macro expansion
+    @ ~/opt/julia-1.10.0/share/julia/stdlib/v1.10/Test/src/Test.jl:672 [inlined]
+    [2] macro expansion
+    @ ~/local-testing-example/example.jl:7 [inlined]
+    [3] macro expansion
+    @ ~/opt/julia-1.10.0/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
+    [4] top-level scope
+    @ ~/local-testing-example/example.jl:7
+    Test Summary: | Fail  Total  Time
+    myadd         |    1      1  0.6s
+    ERROR: LoadError: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.
+    in expression starting at /home/user/local-testing-example/example.jl:6
+    ```
+    Notice how `Test.jl` is smart and includes context: 
+    Lines that failed, evaluated and expected results.
+  ````
+`````
+```````
+
+```````{challenge} (optional) Local-2: Create a test that considers numerical tolerance (10 min)
+Let's see an example where the test has to be more clever in order to
+avoid false negative.
+
+In the above exercise we have compared integers.  In this optional exercise we
+want to learn how to compare floating point numbers since they are more tricky
+(see also ["What Every Programmer Should Know About Floating-Point Arithmetic"](https://floating-point-gui.de/)).
+
+The following test will fail and this might be surprising. Try it out:
+`````{tabs}
+  ````{group-tab} Python
+   ```python
+   def add(a, b):
+       return a + b
+
+   def test_add():
+       assert add(0.1, 0.2) == 0.3
+   ```
+  ````
+  ````{group-tab} R
+   ```R
+   add <- function(a, b){
+       return a + b
+   }
+
+   test_that("Adding floats works", { 
+       expect_identical(add(0.1, 0.2),0.3)
+   })
+   ```
+  ````
+  ````{group-tab} Julia
+   ```Julia
+   function myadd(a,b)
+       return a + b
+    end
+
+   using Test
+   @testset "myadd" begin
+       @test myadd(0.1, 0.2) == 0.3
+   end
+   ```
+  ````
+`````
+
+Your goal: find a more robust way to test this addition.
+```````
+
+```````{solution} Solution: Local-2
+
+`````{tabs}
+  ````{group-tab} Python
+    One solution is to use
+    [pytest.approx](https://docs.pytest.org/en/4.6.x/reference.html#pytest-approx):
+    ```python
+    from pytest import approx
+
+    def add(a, b):
+        return a + b
+
+    def test_add():
+        assert add(0.1, 0.2) == approx(0.3)
+    ```
+
+    But maybe you didn't know about
+    [pytest.approx](https://docs.pytest.org/en/4.6.x/reference.html#pytest-approx):
+    and did this instead:
+    ```python
+    def test_add():
+        result = add(0.1, 0.2)
+        assert abs(result - 0.3) < 1.0e-7
+    ```
+    This is OK but the `1.0e-7` can be a bit arbitrary.
+  ````
+  ````{group-tab} R
+    One solution is to use
+    [expect_equal](https://testthat.r-lib.org/reference/equality-expectations.html) which allows for roundoff errors:
+    ```R
+    test_that("Adding floats works with equal", {
+        res <- add(0.1, 0.2)
+        expect_equal(res,0.3)
+        expect_true(is.numeric(res))
+    })
+    ```
+
+    But maybe you didn't know about it and used the 'less than' comparison of [expect_lt](https://testthat.r-lib.org/reference/comparison-expectations.html) instead:
+    ```R
+    test_that("Adding floats works with lt", {
+        res <- add(0.1, 0.2)
+        expect_lt(abs(res-0.3),1.0e-7)
+        expect_true(is.numeric(res))
+    })
+    ```
+    This is OK but the `1.0e-7` can be a bit arbitrary.
+  ````
+  ````{group-tab} Julia
+    One solution is to use `\approx`:
+    ```Julia
+    @testset "Add floats with approx" begin
+        @test myadd(0.1,0.2) ≈ 0.3
+        #Variant with specifying a tolerance
+        @test myadd(0.1,0.2) ≈ 0.3 atol=1.0e-7
+    end
+    ```
+
+    But maybe you didn't know about `\approx`
+    and did this instead:
+    ```python
+        @test abs(myadd(0.1,0.2)-0.3) < 1.0e-7
+    ```
+    This is OK but the `1.0e-7` can be a bit arbitrary.
+  ````
+````` 
+````````
+
+---
+
+```{keypoints}
+- Each test framework has its way of collecting and running all test functions, e.g. functions beginning with `test_` for `pytest`.
+- Python, Julia and C/C++ have better tooling for automated tests than Fortran and you can use those also for Fortran projects (via `iso_c_binding`).
+```