An REPL oriented testing framework for clojure and clojurescript

tst is a testing library for clojure(script) that is designed to be used from the REPL. In this article, I'm going to describe how to use it.

The base API consists of two entities, the macro testing and the function run-test.

Defining a test suite

testing macro is used to define the test suite tree. The leaf nodes in this tree will contain the testing logic and the inner nodes will be used to group multiple test suites. An invocation of the testing macro returns a test suite as a value:

(def suite1 (testing :test []
                    {:result :OK
                     :message "A simple test suite"))

The first argument to the macro is the name of the test suite (:test in this case). How the rest of the arguments are interpreted depends on whether the test suite is a leaf node or an inner node.

Leaf Nodes

Leaf nodes must have a vector after the name. The above example is a leaf node. As you can see, it looks a lot like a function call. You are supposed to put all the testing logic inside these "functions". It must return a clojure map with :result as a key. It can throw an error too. The empty vector isn't there just for marking the leaf nodes. I'll describe how to use them a little later.

Inner Nodes

Inner nodes must contain nested testing macro calls following the name:

(def suite2 (testing :a
                     (testing :b []
                              {:result :OK})
                     (testing :c []
                              {:result  :ERR
                               :message "failing test"})
                     (testing :d []
                              (throw (new Error))
                              {:result  :OK
                               :message "ignored"}) ))

Manipulating test suites

You can combine multiple test suites into a single suite using combine-tests function:

(def combined (combine-tests [suite1 suite2]))

A common pattern is to define parts of test suite in different modules and combining them into one big suite in the main testing module.

You can also pull out parts of the test suite as if it were a regular clojure map (it is a regular clojure map):

(def suite3 (get combined :test))
(def suite4 (get-in combined [:a :c]))

This can be used to run a subset of the full test suite.

Running test suites

run-test function can be used to execute a test suite. It returns a result tree whose structure matches that of the given test suite.

(run-test combined)
{:test {:result :OK,
        :message "A simple test suite"},
 :a {:b {:result :OK},
     :c {:result :ERR,
         :message "failing test"},
     :d {:result :EXCEPTION,
         :exception# error}}}

(run-test (get-in combined [:a :c]))
{:result :ERR, :message "failing test"}

Test state

Consider the following test unit:

(testing :suite []
         (let [a (get_a)
               b (get_b a)]
           (if (valid_value? b)
             {:result :OK}
             {:result :ERR
              :cause  "Invalid result"
              :a      a
              :b      b})))

What if get_b throws an exception. You'll get the following result when you execute this test:

{:result    :EXCEPTION
 :exception# error}

The exception info will contain the stack trace, but not the value of local variable a that caused get_b to throw an exception.

You can resolve this dilemma by using state variables. The state variables for a test unit are declared in the argument vector of the "function" for that test unit:

(testing :suite [val_a]
         (let [a (get_a)
               _ (reset! val_a a)
               b (get_b a)]
           (if (valid_value? b)
             {:result :OK}
             {:result :ERR
              :cause  "Invalid result"
              :a      a
              :b      b})))

The execution context of the test unit will have an atom for each state variable that is declared. You can store intermediate state of the test unit inside these atoms, and this state will be available in the test result:

{:suite {:result :EXCEPTION,
         :exception# error,
         :state {val_a 1}}}

Navigating the test result tree

You can use the get-failed function to filter tests with (not= status :OK):

(get-failed (run-test combined))
{:a {:c {:result :ERR, :message "failing test"},
     :d {:result :EXCEPTION,
         :exception# error}}}

You can use functions flatten-result and treefy-result if your need more granularity:

(->> (run-test combined)
     flatten-result
     (filter my-result-filter)
     treefy-result)

flatten-result turns a result tree into a sequence of results and treefy-result turns a sequence of results back into a tree:

(def failed (flatten-result (get-failed (run-test combined))))

; number of failed results
(count failed)

; first failed test:
(first failed)

; stats about test results
(frequencies (map :result (flatten-result (run-test combined))))
{:OK 2, :ERR 1, :EXCEPTION 1}