martian

Oliver Hine

HTTP in Clojure


(defn create-pet []
  (http/post "https://api.com/create-pet/dog" {:as :json}))

Route params


(defn create-pet [species]
  (http/post (format "https://api.com/create-pet/%s" species)
             {:as :json}))

query params


(defn create-pet [species age]
  (http/post (format "https://api.com/create-pet/%s" species)
             {:as :json
              :query-params {:age age}))

body params


(defn create-pet [species name age]
  (http/post (format "https://api.com/create-pet/%s" species)
             {:as :json
              :query-params {:age age}
              :body (json/encode {:name name))
              :headers {"Content-Type" "application/json"}}))

metrics, environments, authentication...


(defn create-pet [metrics host port creds species name age]
  (timing metrics "create-pet"
    (http/post (format "https://%s:%s/create-pet/%s" host port species)
               {:as :json
                :query-params {:age age}
                :body (json/encode {:name name))
                :headers {"Content-Type" "application/json"
                          "Authorization" (str "Token" creds)}})))

Signal to noise ratio

(defn create-pet [metrics host port creds species name age]
  (timing metrics "create-pet"
    (http/post (format "https://%s:%s/create-pet/%s" host port species)
               {:as :json
                :query-params {:name name}
                :body (json/encode {:age age))
                :headers {"Content-Type" "application/json"
                          "Authorization" (str "Token" creds)}})))

If we look at that last request and pick out the bits we care about, the bits that drive functionality, we find there are only four. Functionality is what we are trying to achieve, that adds value to our software, and are domain level things that make sense to us. Everything else is how we do it, and it doesn't really add any value to what we are trying to achieve and is the sort of thing that is better left to computers. For example, encoding - I don't really care how my computer talks to another computer, I want them to sort that out between them. I'm trying to solve problems in the business domain and that's hard enough without having to make decisions all the time on things like encoding. The signal to noise ratio is very low here, and that's something we can improve.

The Server


(defn create-pet [request]
  (let [species (get-in request [:route-params :species])
        age (Integer/valueOf (get-in request [:query-params :age]))
        name (-> request
                 :body
                 (json/decode keyword)
                 :name)]
    (create-pet species name age)))

That's not a dog

Signatures


              (fn create-pet [species name age])

This is a dog

Data describing data


{:route-params {:species Species}
 :query-params {:age Age}
 :body-params  {:name Name}

API descriptions


(defhandler create-pet
  {:parameters {:path-params  {:species Species}
                :query-params {:age Age}
                :body-params  {:name Name}}
   :responses {201 {:body {:id Id}}}}
   ...)

oliyh/pedestal-api

Moves like Swagger

life on mars

Make a Martian


(def m (martian-http/bootstrap-swagger
        "https://pedestal-api.herokuapp.com/swagger.json"))

Contact Martian


(martian/response-for m :create-pet {:name "Charlie"
                                     :species "Dog"
                                     :age 3})

What does it mean?

Cleaner code
Specification always up-to-date
Easier refactoring
Explicit knowledge

It may seem like a small improvement to the code, but the implications are bigger than replacing five lines of code with one line. Your code is cleaner not just in terms of line count but in terms of concepts. You're no longer being asked to remember what HTTP status codes mean, or whether a parameter might need url encoding. Your mind is freed to concentrate on what it really means to create a pet and all the deep philosophical ramifications that has. It is also, without being obvious, DRY - don't repeat yourself. The description of the API is generated from the API itself so it's always guaranteed to be up to date. By writing your own code that says, for example, that species is a route parameter, you're unconciously repeating the code on the server with the usual caveat to repetition - one version will end up being wrong, and it's going to be the client side in this case. For this reason small refactorings to the API can be made without your code having to change at all. If a POST is changed to a PUT, or a url is changed, or a query parameter becomes a route parameter you won't have to change your code because you haven't specified those things on your side at all - you're letting the server describe all those things to you. We no longer have implicit knowledge about the API buried in our code, we're using the explicit description from the golden source - the remote API itself.

Interceptors

I'll just speak for a few minutes now on a pattern I used to implement martian. Anyone familiar with pedestal or even re-frame will have come across the concept of interceptors before. Essentially they behave like frames on a call stack - code that is executed before the next call down the stack, and code that is executed when the result bubbles back up. However they are described by data and they can be manipulated like data. In order to do something like time a request we would want to add a frame to the call stack just before the operation we want to time. When your call stack is data this just becomes a simple conj operation! Interceptors are like a reified call stack, and this is a very powerful concept. Martian uses interceptors to perform each of the steps it takes between invocation and returning the response. These steps shown here include conforming the HTTP request map with the method, url etc, choosing the most efficient way to serialise data on the way down on the left hand side, making the http call at the bottom, waiting for the response, then up the right hand side we are deserialising it and returning it to the call site. This is martian's default interceptor stack.

Security is everything

Authentication interceptor


(def authentication
  {:name ::authentication
   :enter (fn [ctx]
            (assoc-in ctx [:request :headers "Authorization"]
                      "Token: 12456abc"))})

Update the call stack


(def m (martian-http/bootstrap-swagger
        "https://pedestal-api.herokuapp.com/swagger.json"
        {:interceptors (concat martian/default-interceptors
                               [authentication
                                martian-http/encode-body
                                (martian-http/coerce-response)
                                martian-http/perform-request])}))

Timing is everything

Timing interceptor


(def timing
  {:name ::timing
   :enter (fn [ctx] (assoc ctx ::start (t/now)))
   :leave (fn [ctx] (assoc ctx ::duration
                           (t/minus (t/now) (::start ctx))))})

Update the call stack (again!)


(def m (martian-http/bootstrap-swagger
        "https://pedestal-api.herokuapp.com/swagger.json"
        {:interceptors (concat martian/default-interceptors
                               [authentication
                                martian-http/encode-body
                                (martian-http/coerce-response)
                                timing
                                martian-http/perform-request])}))

Interceptors as an API

Better than multimethods
Better than binding dynamic vars
Better than arbitrary options maps

Interceptors actually provide a brilliant way of allowing the developer to change the way the library works. They can have full control of what happens and when it happens - the martian library is a set of composable interceptors rather than a rigid framework. Although interceptors were developed by the pedestal team as a bit of an extension to the ring specification, they provide generic behaviour and could be used in any library to provide all the benefits I've talked about. Traditionally you might choose multimethods, dynamic vars or an options map to let the developer change behaviour. Inteceptors are better than using multimethods because multimethods only allow you to provide implementations for particular dispatch values. You can't choose how the dispatch values are chosen, you can't reorder anything, you can't add new behaviour, only adapt existing behaviour. They're better than binding dynamic vars because they rely on state and makes it hard to reason about how a system works when state can change at any time, and also have many of the same drawbacks as multimethods. Dynamic vars also cannot cross thread boundaries so using httpkit with it's asynchronous response callback wouldn't work properly. They're also better than supporting arbitrary options maps, which are often poorly documented and still don't let you add or extend behaviour beyond what the author originally allows. I humbly submit that exposing the interceptor stack is the best way of making a library extendable, reusable and ultimately more useful to the developer whilst reducing the demand on the author to maintain and add features to it. I certainly plan to follow the same pattern for future libraries.

Testing

Always test the right thing

Stub servers

Drift away from real life
Manually written cases
Slow to run

Mocking

martian-test

Uses production definition - always correct
Generative - complete
Fast
Readable error messages

martian-test example


(def m (-> (martian/bootstrap "https://petstore.com" api-definition)
           (martian-test/respond-with :success)))

(martian/response-for m :create-pet {:name "Charlie"})
;; => ExceptionInfo Value cannot be coerced to match schema:
;;                  {:species missing-required-key}

(martian/response-for m :create-pet {:name "Charlie"
                                     :species "Dog"
                                     :age 3})
;; => {:status 201, :body {:id -3}}

When we bootstrap martian, we do it the same way as we do in production, so it has exactly the same definitions. This is very important. Then we tell it to always respond with success, at the top. This respond-with behaviour removes the interceptor which makes the actual http request and replaces it with one that simply generates a response. On the next line you see when we make a bad call, like forgetting to provide the species of our pet for example, we get an exception which tells us this clearly. We haven't had to call the server, we've just validated the request against the schema that the server has provided. Below that, when we make a good call, it succeeds and we get a generated response, and this response is generated from the responses that the server has told us it can generate without having to go near the server itself. This means our tests can prove that we are always passing good data in to our API calls, which is a really great thing to be able to test. If someone changes an API in a way that's incompatible with the way you use it, you'll get a failing test without hammering a real system with your test data. This shows the behaviour of the martian functions, but you can imagine your own code wrapped around these interacting with the data passing into and out of the remote API, and how this test behaviour can help. When you've built up your test suite you now have the ability to do something pretty special. If someone changes an API that you use in a way that will break your code, your tests will fail and you will know exactly where it broke. If they change their API in a way that won't break your code, you'll be confident that you're still compatible just by running your test suite. That's something I've never been able to do before.

Response schemas

(defhandler create-pet
  {:parameters {:path-params  {:species Species}
                :query-params {:age Age}
                :body-params  {:name Name}}
   :responses {201 {:body {:id Id}}
               401 {:body {:message s/Str}}
               402 {:body {:amount s/Int}}
               403 {:body {:message s/Str}}
               ...}}
   ...)

http.cat

Generating responses


(def create-pet-responses
  (martian-test/response-generator m :create-pet))

(generate create-pet-responses)
;; => {:status 200, :body {:id -1472372}}

You can also write truly generative tests using test.check. Creating a generator is easy - you just ask martian for one for the endpoint you want to test. They use schema generators underneath so behave in the same way. Already we're thinking more - what does it mean that an id could be negative? The age of a pet is optional, what happens when I load a pet that doesn't have an age? martian-test comes with all the plumbing you need to create a martian which will return generated responses and let you make generic assertions. You can for example assert that whenever a pet is successfully created, you always notify the user that their action was successful, or if there was a failure that something went wrong. Your integration tests can be much more robust. There are examples on how to do this on github.

Assumptions

Assumptions in testing can be quite a dangerous thing. You might have tested that you can leap, assuming you're standing on a high friction surface. Assumptions about a remote API can easily creep into your codebase without you knowing: these assumptions are implicit and get lost as data propagates through your system. By explicitly describing the remote API at the edge of your system and using generative testing you can avoid a lot of pitfalls and get clearer errors when integrating. Generative tests force you to consider all the permutations that can exist and don't let you make these assumptions. Complexity significantly increases when you start interacting with remote services written by other people, and just because the service behaved as you expected it to for a narrow range of inputs you never know how it might behave under duress.

No Swagger? No problem


(martian/bootstrap "https://api.org"
                   [{:route-name :create-pet
                     :path-parts ["/pets/" :species]
                     :method :put
                     :path-schema {:species s/Str}
                     :body-schema {:name                 s/Str
                                   (s/optional-key :age) s/Int}}]
                   {:produces ["application/json"]
                    :consumes ["application/json"]})

Any implementation

http-kit/http-kit

dakrone/clj-http

r0man/cljs-http

your-face/here

The actual HTTP client you use underneath martian is up to you. The original driver behind this was using it in Clojurescript as well as in Clojure, and because everything is implemented as an interceptor this turned out to be really easy to do. You can even retain the behaviour of these libraries through martian - the asynchronous httpkit solution for example handles the leave phases of all the interceptors inside the callback which gets called when the response comes back, which is normally the place where your thread bindings would get lost and you've lost a lot of context. In martian, we pick up the entire context containing the interceptor queue and stack and put it into the callback - no context lost at all.

Alternatives

swagger

finagle

TL;DR

Separate your domain from implementation
Describe data for great good
Interceptor all the things

Thanks

oliyh/martian