Have you ever wanted to create your very own client for a REST API? That's what this book is all about.

There's quite a pair of acronyms right there. Let's break it down a bit:

• REST, or REpresentational State Transfer, basically describes a specific way that one can use web requests to get information from the remote system and subsequently do things with that data, still on the remote system.
• An API, or Application Programming Interface is a collection of functions, procedures, or system calls that one uses to make their program work with a piece of software that is not itself.

Bringing it all together, a REST API is a remote system that allows us to use that system for our programs as if it was a part of our program.

The best layman analogy that I have for this is my dog, Jake:

• When I look at Jake and say "sit," he ignores me, but in a perfect world, he would sit.
• When I look at Jake and say "good boy," he wags his tail.
• When I look at Jake and say "potty," he runs for the door.

The API for the K9-Jake Supercomputer is "sit," "good boy," and "potty." If he were hooked up to a web server (please, don't form a mental image of this, nothing good can come from it), there would be an equivalent REST API.

A client is a package that knows specifically how to talk to the REST API for which it is written, generally making it easier to interact with that API in your own programs.

Hi there, I'm Dennis, and I'm a Professional Services Engineer at Engine Yard. Really, that's just a fancy way to say that I do the "whatever it takes" on behalf of customers that have very custom needs.

I've been creating software and API consumer libraries for a pretty long time, and the latter of those ideas is something that I inexplicably enjoy quite a lot.

When I'm not doing that sort of thing, I can sometimes be found drawing, trying to get better at basement lutherie, making music on computerboxes, or convincing my dog that it's totally okay if he wants to leave me a few inches on the bed.

Hi there, you're you. As I see it, there are a few possibilities as to who you are:

• You might be a friend or colleague that I've asked to read over this book to help me figure out how to most effectively communicate the ideas that I'm presenting
• You might be a relatively new software developer that needs to figure out how to consume an API for which there is not an existing client
• You might be a seasoned software developer that is looking for a different approach to API client design
• You might even be a person that hasn't touched a computer before, but wants to learn about the way things like the Twitter and Facebook mobile apps communicate with their respective services

Regardless of who you are, I very much hope that you enjoy the ride.

Every REST API that I've used, regardless of how well documented they may be, has required me to do a bit of journeying and mapping along the way to figure out how things really work. As a reflection of this, the API provider will often publish clients for the popular languages of the day, but that isn't always the case.

For example, the Engine Yard API is relatively well-documented, and Engine Yard does provide a Ruby client, but what if I want to consume this API from a Python program?

That is the example that we'll use for the purposes of this illustration: implementing a Python client for the Engine Yard API. That said, the techniques used here should be fairly easily adapated to nearly any language or API (web based or otherwise).

Effectively, this book is a quasi-real-time illustration of the designs and processes that I use for charting the seas of web APIs. When a new term or acronym appears, I'll do my best to break it down for folks that might not be familiar, but there's a decent chance that I'll gloss over one or two along the way due to their ubiquity in the field.

Also, there will be quite a lot of Python code in this book. When you've finished, if you've diligiently typed out all of the examples into the correct files, you'll actually have a working client for the API that I'm mapping. If you'd like to forego that process, you can find the finished client on Github, but I promise that you'll learn more the other way.

In the beginning those code examples will be entire files, but as we progress and those files become large, snarling behemoths, I'll start showing only the parts that change. Don't worry, though, I'll let you know exactly where that happens.

Every now and then, I'll drop in a list of questions that won't have answers. You don't strictly have to do these exercises, but I'd strongly urge you to do so, because learning is learning, and learning is awesome.

These people greatly helped me along the way by reading the various iterations to make content suggestions, point out things that I've forgotten, increase readability and comprehensibility, or even just finding typos.

These are good people. You should find them and listen to the things that they have to say as well.

• Adam Delong
• Carlos Cano
• Daniel Valfre
• Matt Jones
• Radamanthus Batnag

As seems to be the case with all books like this one, there are a few things that should probably be stated up front:

• As mentioned above, I'm currently employed by Engine Yard. While one might argue that I've chosen the Engine Yard API as the example for this book to be a sign of bias, the reality is that I've chosen it because it's a fairly complicated (to the point of being difficult) API to consume, and there are some hidden gems in it that can help to illustrate how one might handle situations in which they're stuck.
• This very much is not the only way that one could develop an API client. This is, however, the method that I prefer, so I'm rolling with it.
• Python is being used as the implementation language in this book, but this is just an example of a language that is not officially supported by the API provider. As it were, there are partial implementations of the client that is being developed in this book in the following languages: Go, Javascript, ooc, Ruby, and Rust
• I don't actually know Python, so this will be a learning experience for me as well. I'm using Python 3.6.4 locally to develop this client as we go, but it's important to note that this should probably not be considered an example of idiomatic Python code, let alone high-quality Python code. I honestly don't know Python well enough to even be able to make that previous statement.

This chapter is all about doing the very first things that I do when staring this sort of project. It's not really about installing Python, setting up a new project space, or anything like that. More so, it's about what happens after all of those things.

The first step in the process, at least for me, is to take a look at whatever documentation may exist for the API that I'm trying to map to see if I can find some basic hints for its usage.

In the case of the Engine Yard API, there's actually quite a lot of information that we can get from the API overview:

• It's an authenticated HTTP API
• The authentication token can be passed in either as part of the query or as the X-EY-TOKEN request header
• From this page, we know that the GET, POST, PUT, and PATCH HTTP verbs are used
• We know that the API can both accept and serve the application/json MIME type
• From the sidebar, we can see that there are a good number of endpoints off of the API root

Taking a look around at some of those endpoints, we learn even more about this specific API:

• The API is versioned
• The desired API version is specified as part of the Accept request header: application/vnd.engineyard.v3+json
• The DELETE HTTP verb is also used in this API

Now that we have some information about the API that we're mapping, we can start designing our client. Here's what we know:

• We need to be able to handle HTTP interactions with a JSON-centric REST API
• That API is authenticated
• That API is versioned
• That API uses an uncommon version specification strategy
• We need to be able to handle multiple endpoints on that API

We'll break those down a bit further in a few minues, but before we do that, we should think a bit about the overall design of our client.

There are a few schools of thought as to what an API client should look like. At the extremes of that group are basically "a library full of functions" and "a web ORM."

I prefer something that's a bit between the two.

For nearly every API client that I've written, I've gone with a design that looks a lot like the Data Mapper pattern overlaid on top of a Repository. I'm not incredibly familiar with libraries that work this way outside of the Ruby community. From that community, though, the best examples are probably Ruby Object Mapper and Hanami Model.

What we're going to do is to break this API client project into two main phases:

1. First, we're going to create a HTTP driver that interfaces with the REST API. This driver is basically our doorway to all communication with the upstream API. It takes care of authentication, request and response handling, and is really the core of the whole deal.
2. The second phase is the modeling of the upstream API endpoints and the data structures specific to them.

Before we move forward, the hardest part of the whole project needs to be done: we need to name the project. In keeping with the naming for my other EY API client implementations, I've called this project maury, but you can call it anything that you'd like.

With that, let's be a bit proactive and determine the external dependencies that we want to use for the project.

It might seem a bit like putting the cart before the horse, but I'm not very familiar with Python at this point. It makes sense to do a bit of research to find out if it has support for the things that we already know our client needs to do.

Technically, all of the things that we need in order to implement the core driver is provided in the Python standard library. After a bit of comparitive research, though, it seems that the non-standard options provided by the community might well be easier to use (and are often suggested over the standard library).

So, let's take a look at the options.

Interestingly, the documentation for Python's standard library http.client package directs us to check out a community-provided library instead for a higher-level interface: requests.

After looking at how both of these options are used, I'm totally going the easy route by pulling in the requests package.

In my experience, we're going to end up building some URLs in this implementation. After a bit of trial and error with Python's built-in mechanism for this, I left with a bad taste in my mouth and went in search for another option.

More googling and a bit of playing around led me to the furl package.

While researching the above concepts, I started thinking about testing.

We're going to use nose to make writing and running our tests easier. Oh, yeah, we're testing. I'd hate for Bryan Liles to come after me for showing you how to do this stuff without showing you how I TATFT.

So, now that we have our external dependencies, let's add them to our setup.py:

from setuptools import setup

install_requires=[
    'furl',
    'requests',
    ]

tests_require = [
    'mock',
    'nose',
    ]

setup(
        name = 'maury',
        version = '0.1.0',
        description = 'A experimental client for the Engine Yard API',
        license = 'MIT',
        packages = ['maury'],
        install_requires = install_requires,
        tests_require = tests_require,
        test_suite = "nose.collector",
        zip_safe = False)

We're now entering Phase 1 of the project: implementing the core HTTP driver for the client.

To that end, let's take the information that we gathered in the previous chapter and generate the requirements for the driver.

We already have a name for the project. Now that the hardest part of any project is over with, we should identify the base requirements for our core client driver:

• It must speak HTTP
• It must accept an authentication token
• For the sake of flexibility, it should also accept the base URL for the API
• It must be able to use the HTTP verbs in question: GET, POST, PUT, PATCH, and DELETE
• It must be able to set the X-EY-TOKEN, Content-Type, and Accept headers
• For convenience, it should treat all requests as relative, so it must be able to build URLs for each request

That doesn't really sound like a lot, but the driver that we implement through the rest of this chapter is all we really need to work with the API. Don't worry, we're not stopping there at all. We have to crawl before we can walk, though.

So, let's get started crawling.

I like to start my implementation with the first, least-destructive HTTP verb in mind: GET.

So, let's get started developing our Client class. Why am I using a class here? Aside from the general when in Rome ... rule (though I'm told that it's almost bad form to do OOP in this OO language, by somebody who's probably trollin'), I have some plans down the line for how this client will be used. Also, our requirements flat out state that we need to keep track of some data: the base URL for the API and an authentication token.

Let's start there! In maury/client.py, we'll put the following:

class Client(object):
    """A base driver that talks to the Engine Yard API"""

    def __init__(self, base_url = 'https://api.engineyard.com', token = None):
        """Instantiate a new Client instance
        
        Keyword arguments:
        base_url -- the base URL of the API (default: 'https://api.engineyard.com')
        token -- the API authentication token (default: None)
        """

        self.__base_url = base_url
        self.__token = token

There we go. We now have a Client class that accepts a base URL and a token. Our work is done!

Dangit. Okay. More work it is.

Our requirements say that we need to be able to construct URLs based off of a relative endpoint path. So, let's do that in maury/client.py, too:

from furl import furl

class Client(object):
    """A base driver that talks to the Engine Yard API"""

    def __init__(self, base_url = 'https://api.engineyard.com', token = None):
        """Instantiate a new Client instance
        
        Keyword arguments:
        base_url -- the base URL of the API (default: 'https://api.engineyard.com')
        token -- the API authentication token (default: None)
        """

        self.__base_url = base_url
        self.__token = token

    def __construct_request_url(self, path):
        """Construct a URL for an API endpoint.
        
        Given a relative endpoint path, construct a fully-qualified API URL.
        """

        # Get a URL object that we can edit
        u = furl(self.__base_url)

        # Set the path to the endpoint in question
        u.path = path

        # Return the modified URL
        return u.url

Well, that was easy. That's two requirements down ... let's go for a third.

We need to be able to speak HTTP! Back to maury/client.py:

from furl import furl
import requests

class Client(object):
    """A base driver that talks to the Engine Yard API"""

    def __init__(self, base_url = 'https://api.engineyard.com', token = None):
        """Instantiate a new Client instance
        
        Keyword arguments:
        base_url -- the base URL of the API (default: 'https://api.engineyard.com')
        token -- the API authentication token (default: None)
        """

        self.__base_url = base_url
        self.__token = token

    def __construct_request_url(self, path):
        """Construct a URL for an API endpoint.
        
        Given a relative endpoint path, construct a fully-qualified API URL.
        """

        # Get a URL object that we can edit
        u = furl(self.__base_url)

        # Set the path to the endpoint in question
        u.path = path

        # Return the modified URL
        return u.url

It's almost cheating, but simply importing the requests package means that we can speak HTTP in this module. Moving on ...

This might be the most complicated requirement that we've tackled so far, and it's not the lowest-hanging fruit on the list of remaining requirements, but it makes sense to do this next because of reasons.

So, let's implement the first public method in our Client class: get:

from furl import furl
import requests

class Client(object):
    """A base driver that talks to the Engine Yard API"""

    def __init__(self, base_url = 'https://api.engineyard.com', token = None):
        """Instantiate a new Client instance
        
        Keyword arguments:
        base_url -- the base URL of the API (default: 'https://api.engineyard.com')
        token -- the API authentication token (default: None)
        """

        self.__base_url = base_url
        self.__token = token

    def __construct_request_url(self, path):
        """Construct a URL for an API endpoint.
        
        Given a relative endpoint path, construct a fully-qualified API URL.
        """

        # Get a URL object that we can edit
        u = furl(self.__base_url)

        # Set the path to the endpoint in question
        u.path = path

        # Return the modified URL
        return u.url

    def get(self, path, params = None):
        """Perform an HTTP GET on the API.

        Given an endpoint path and a dictionary of parameters, send the request
        to the aPI and return the result.

        Positional arguments:
        path -- the path of the API endpoint to GET

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        """

        response = requests.get(self.__construct_request_url(path),
                params = params)

        return response.text

That's fine and good, and it technically fulfills the implements HTTP GET requirement, but there are a few problems with this implementation:

• It won't work: this is an authenticated API, but we're not handling authentication
• It might not work: we haven't specified an API version
• It might not work: we are assuming that the API is always in top notch operational condition and that we are always sending a valid request ... there is no error handling at all

Let's fix those in that order.

So, in order to provide our authentication token to the API, we either have to pass it in as part of the query string, or we have to set the X-EY-TOKEN header. Between the two of these, the header option is more secure and just plain less messy, so let's do that.

}}

from furl import furl
import requests

class Client(object):
    """A base driver that talks to the Engine Yard API"""

    def __init__(self, base_url = 'https://api.engineyard.com', token = None):
        """Instantiate a new Client instance
        
        Keyword arguments:
        base_url -- the base URL of the API (default: 'https://api.engineyard.com')
        token -- the API authentication token (default: None)
        """

        self.__base_url = base_url
        self.__token = token

    def __construct_request_url(self, path):
        """Construct a URL for an API endpoint.
        
        Given a relative endpoint path, construct a fully-qualified API URL.
        """

        # Get a URL object that we can edit
        u = furl(self.__base_url)

        # Set the path to the endpoint in question
        u.path = path

        # Return the modified URL
        return u.url

    def get(self, path, params = None):
        """Perform an HTTP GET on the API.

        Given an endpoint path and a dictionary of parameters, send the request
        to the aPI and return the result.

        Positional arguments:
        path -- the path of the API endpoint to GET

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        """

        response = requests.get(self.__construct_request_url(path),
                params = params,
                headers = {'X-EY-TOKEN' : self.__token})

        return response.text

That's taken care of. Next up is to specify the API version.

In order to specify the version of the API that we want to use, we have to pass it in as part of the Accept header.

from furl import furl
import requests

class Client(object):
    """A base driver that talks to the Engine Yard API"""

    def __init__(self, base_url = 'https://api.engineyard.com', token = None):
        """Instantiate a new Client instance
        
        Keyword arguments:
        base_url -- the base URL of the API (default: 'https://api.engineyard.com')
        token -- the API authentication token (default: None)
        """

        self.__base_url = base_url
        self.__token = token

    def __construct_request_url(self, path):
        """Construct a URL for an API endpoint.
        
        Given a relative endpoint path, construct a fully-qualified API URL.
        """

        # Get a URL object that we can edit
        u = furl(self.__base_url)

        # Set the path to the endpoint in question
        u.path = path

        # Return the modified URL
        return u.url

    def get(self, path, params = None):
        """Perform an HTTP GET on the API.

        Given an endpoint path and a dictionary of parameters, send the request
        to the aPI and return the result.

        Positional arguments:
        path -- the path of the API endpoint to GET

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        """

        response = requests.get(self.__construct_request_url(path),
                params = params,
                headers = {
                    'X-EY-TOKEN' : self.__token,
                    'accept' : 'application/vnd.engineyard.v3+json',
                    })

        return response.text

We specified the heck out of that API version, I reckon. One more to go!

This one might seem a bit weird if you're actually familiar with Python, but since I'm not particularly, I'm going to do the thing that makes the most sense to me, heavily influenced by other languages that I know better.

You see, the thing here is that I don't actually know much about how to handle exceptions in Python, and I actually don't like using exceptions for error handling. Some of my favorite languages use some form of multiple return for error handling. While I could do that with a list or a tuple, there's a technique that I prefer: let's make a Result class in maury/result.py:

class Result(object):
    """The result of an operation.

    A result has two parts: a body, and an error.
    If the result contains an error, things are not ok.
    If the result contains no error, things are ok.
    """

    def __init__(self, body, error):
        """Set up a new Result.

        Positional arguments:
        body -- the content to pass along if things are ok
        error -- the content to pass along if things are not ok
        """

        self.__body = body
        self.__error = error

    @property
    def ok(self):
        """Are things ok?

        If the result has an error, this is false. Otherwise, true.
        """

        return self.__error == None

    @property
    def body(self):
        """The positive result content"""

        if self.ok:
            return self.__body

        return None

    @property
    def error(self):
        """The negative result content"""
        return self.__error

Now that we have a way to express both positive and negative results, let's tie it in and use it in our Client:

from furl import furl
import requests
from .result import Result

class Client(object):
    """A base driver that talks to the Engine Yard API"""

    def __init__(self, base_url = 'https://api.engineyard.com', token = None):
        """Instantiate a new Client instance
        
        Keyword arguments:
        base_url -- the base URL of the API (default: 'https://api.engineyard.com')
        token -- the API authentication token (default: None)
        """

        self.__base_url = base_url
        self.__token = token

    def __construct_request_url(self, path):
        """Construct a URL for an API endpoint.
        
        Given a relative endpoint path, construct a fully-qualified API URL.
        """

        # Get a URL object that we can edit
        u = furl(self.__base_url)

        # Set the path to the endpoint in question
        u.path = path

        # Return the modified URL
        return u.url

    def get(self, path, params = None):
        """Perform an HTTP GET on the API.

        Given an endpoint path and a dictionary of parameters, send the request
        to the aPI and return the result.

        Positional arguments:
        path -- the path of the API endpoint to GET

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        """

        response = requests.get(self.__construct_request_url(path),
                params = params,
                headers = {
                    'X-EY-TOKEN' : self.__token,
                    'accept' : 'application/vnd.engineyard.v3+json',
                    })

        if response.ok:
            return Result(response.text, None)

        return Result(
                None,
                "The API returned the following status: %d" % response.status_code
                )

There we go. This feels more like something that will actually work. I should probably prove that with some tests ...

Usually, I do most all of my development in a test-driven (or test-first) manner, but when I'm learning a new language, I prefer to get used to the language before I try to get used to its testing mechanisms. At any rate, let's write our first test.

Since we're testing the client, we should probably figure out a way to mock out the actual HTTP requests. Otherwise, we're going to have to be online to run our tests, which is kind of a drag. It turns out, though, that requests-mock is a thing, so let's pull that into our test requirements in setup.py:

from setuptools import setup

install_requires=[
    'furl',
    'requests',
    ]

tests_require = [
    'mock',
    'nose',
    'requests-mock',
    ]

setup(
        name = 'maury',
        version = '0.1.0',
        description = 'A experimental client for the Engine Yard API',
        license = 'MIT',
        packages = ['maury'],
        install_requires = install_requires,
        tests_require = tests_require,
        test_suite = "nose.collector",
        zip_safe = False)

So far, so good. Now let's try actually writing a test or two in tests/test_client.py:

from unittest import TestCase
import requests_mock

from maury.client import Client

class TestClient(TestCase):
    @requests_mock.Mocker()
    def test_get(self, m):
        # The happy path
        m.get('https://api.engineyard.com/sausages', text='gold')

        c = Client(token = 'faketoken')
        result = c.get('sausages')
        self.assertTrue(result.ok)
        self.assertEqual(result.body, 'gold')

        # The happy path with params
        m.get('https://api.engineyard.com/sausages?color=gold', text='yep')

        result = c.get('sausages', params = {'color' : 'gold'})
        self.assertTrue(result.ok)
        self.assertEqual(result.body, 'yep')

        # A wild API error appears!
        m.get(
                'https://api.engineyard.com/ed209',
                status_code = 500,
                text = 'Drop your weapon. You have 20 seconds to comply.')

        result = c.get('ed209')
        self.assertFalse(result.ok)
        self.assertFalse(result.error == None)

        # PEBCAK
        m.get(
                'https://api.engineyard.com/404',
                status_code = 404,
                text = 'You are now staring into the void. It is staring back.')

        result = c.get('404')
        self.assertFalse(result.ok)
        self.assertFalse(result.error == None)

When we run our tests with python setup.py test, we get (along with a ton of noise) the following output:

test_get (maury.tests.test_client.TestResult) ... ok

That sounds like a winner in my book. There are still a few things we should straighten out before we move on, though. First thing being first ... textual responses are fine and all, but we're really more interested in JSON responses.

The requests package is nice enough to automagically convert JSON responses for us, so let's change the tests for that.

from unittest import TestCase
import requests_mock

from maury.client import Client

class TestClient(TestCase):
    @requests_mock.Mocker()
    def test_get(self, m):
        # The happy path
        m.get('https://api.engineyard.com/sausages', text='{"sausaged":"gold"}')

        c = Client(token = 'faketoken')
        result = c.get('sausages')
        self.assertTrue(result.ok)
        self.assertEqual(result.body, {'sausages' : 'gold'})

        # The happy path with params
        m.get(
                'https://api.engineyard.com/sausages?color=gold',
                text='{"sausages":"yep"}')

        result = c.get('sausages', params = {'color' : 'gold'})
        self.assertTrue(result.ok)
        self.assertEqual(result.body, {'sausages' : 'yep'})

        # A wild API error appears!
        m.get(
                'https://api.engineyard.com/ed209',
                status_code = 500,
                text = 'Drop your weapon. You have 20 seconds to comply.')

        result = c.get('ed209')
        self.assertFalse(result.ok)
        self.assertFalse(result.error == None)

        # PEBCAK
        m.get(
                'https://api.engineyard.com/404',
                status_code = 404,
                text = 'You are now staring into the void. It is staring back.')

        result = c.get('404')
        self.assertFalse(result.ok)
        self.assertFalse(result.error == None)

If we run the tests right now, we get failures. That's because we changed the test specification, but we haven't changed the code yet. That's awesome, because that's the sort of test-driven thing that allows us to actually change the code intentionally (rather than otherwise). So, let's intentionally change maury/client.py so we get JSON in our results instead of raw text:

from furl import furl
import requests
from .result import Result

class Client(object):
    """A base driver that talks to the Engine Yard API"""

    def __init__(self, base_url = 'https://api.engineyard.com', token = None):
        """Instantiate a new Client instance
        
        Keyword arguments:
        base_url -- the base URL of the API (default: 'https://api.engineyard.com')
        token -- the API authentication token (default: None)
        """

        self.__base_url = base_url
        self.__token = token

    def __construct_request_url(self, path):
        """Construct a URL for an API endpoint.
        
        Given a relative endpoint path, construct a fully-qualified API URL.
        """

        # Get a URL object that we can edit
        u = furl(self.__base_url)

        # Set the path to the endpoint in question
        u.path = path

        # Return the modified URL
        return u.url

    def get(self, path, params = None):
        """Perform an HTTP GET on the API.

        Given an endpoint path and a dictionary of parameters, send the request
        to the aPI and return the result.

        Positional arguments:
        path -- the path of the API endpoint to GET

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        """

        response = requests.get(self.__construct_request_url(path),
                params = params,
                headers = {
                    'X-EY-TOKEN' : self.__token,
                    'accept' : 'application/vnd.engineyard.v3+json',
                    'content-type' : 'application/json',
                    })

        if response.ok:
            return Result(response.json(), None)

        return Result(
                None,
                "The API returned the following status: %d" % response.status_code
                )

Now that we've updated the client, the tests pass again. Sometimes, I do love developering. As you can see, we've also specified that we would like JSON back in our responses via the headers for the request.

There's one last bit of business to take care of before we consider this iteration complete.

The Result class looks good. It gives us a clean way to communicate back to the code that's using our Client. But does it? We should illustrate that with a test in tests/test_result.py:

from unittest import TestCase

from maury.result import Result

class TestResult(TestCase):
    def test_ok(self):
        good = Result('yay', None)
        bad = Result(None, 'uh-oh')

        self.assertTrue(good.ok)
        self.assertFalse(bad.ok)

    def test_body(self):
        body = "head and shoulders, knees and toes"
        result = Result(body, None)

        self.assertEqual(result.body, body)

        result = Result(body, 'Onoes!')

        self.assertEqual(result.body, None)

    def test_error(self):
        body = 'eyes and ears and mouth and nose'
        error = "I've made a terrible mistake"
        result = Result(body, error)

        self.assertEqual(result.error, error)

That does it. If all our client ever has to do is provide the ability to make GET requests against the API, we're done!

Of course, our requirements state that we have to also be able to make POST requests to the API. So, we have more work cut out for us. Luckily, the handling of any given verb is quite a lot like the handling of any other given verb.

Yeah, not really. More than anything, I wanted to take this opportunity to let you know that from this point on, listing the entirety of the referenced files would become unwieldy rather quickly.

So, from now on, I'll just show the changes in the code examples instead of the entire file, where possible.

Now that we're a little more familiar with the language and its unittest framework, let's change gears a bit. We're going to start our POST feature with a new test in maury/tests/test_client.py:

    @requests_mock.Mocker()
    def test_post(self, m):
        # The happy path
        m.post('https://api.engineyard.com/sausages', text='{"sausages":"gold"}')

        c = Client(token = 'faketoken')
        result = c.post('sausages')
        self.assertTrue(result.ok)
        self.assertEqual(result.body, {'sausages' : 'gold'})

        # The happy path with params
        m.post(
                'https://api.engineyard.com/sausages?color=gold',
                text='{"sausages":"yep"}')

        result = c.post('sausages', params = {'color' : 'gold'})
        self.assertTrue(result.ok)
        self.assertEqual(result.body, {'sausages' : 'yep'})

        # A wild API error appears!
        m.post(
                'https://api.engineyard.com/ed209',
                status_code = 500,
                text = 'Drop your weapon. You have 20 seconds to comply.')

        result = c.post('ed209')
        self.assertFalse(result.ok)
        self.assertFalse(result.error == None)

        # PEBCAK
        m.post(
                'https://api.engineyard.com/404',
                status_code = 404,
                text = 'You are now staring into the void. It is staring back.')

        result = c.post('404')
        self.assertFalse(result.ok)
        self.assertFalse(result.error == None)

As you can see, our post test is almost identical to our get test. That's because, as mentioned above, all of the verbs are handled in more or less the same way. The big difference here is that post involves not just a path and a params dict, but also a dict of data to be POSTed to the endpoint.

After running our tests, we see that our test_post test yields an error. That's because we don't have a post method in our client yet. Let's do that.

Since they test the same (aside from the extra argument), it stands to reason that get and post should have rather similar implementations. Let's do a quick copypasta in maury/client.py and see how that works out:

    def post(self, path, params = None, data = None):
        """Perform an HTTP POST on the API.
        
        Given an endpoint path, a dictionary of parameters, and a dictionary of
        POST data, send the request to the API and return the result.

        Positional arguments:
        path -- the path of the API endpoint you wish to POST

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        data -- a dictionary of POST data (default: None)
        """

        response = requests.post(self.__construct_request_url(path),
                params = params,
                json = data,
                headers = {
                    'X-EY-TOKEN' : self.__token,
                    'accept' : 'application/vnd.engineyard.v3+json',
                    'content-type' : 'application/json',
                    })

        if response.ok:
            return Result(response.json(), None)

        return Result(
                None,
                "The API returned the following status: %d" % response.status_code
                )

Running our tests now yields a success:

test_post (maury.tests.test_client.TestResult) ... ok

So, one could argue that we're done at this point, but there's something that's bugging me a bit about this implementation. Take a look at the get and post methods. Notice how similar they are? We should probably take this opportunity to go ahead and refactor those similarities away.

For those not used to the term or the practice, refactoring is basically the act of rearranging the code within a program to increase the simplicity of the system without altering its behavior. That's not really the proper definition of the term, but that is the way that I think about it.

There are a several interpretations one could use for "simplicity" in this context. I think about two things:

• How easy is it to figure out how the module works?
• How smelly is the code?

Our client module is fairly small, and it's not very difficult to figure out how it works for the moment. However, it is slightly smelly due to the high degree of code duplication between the get and post methods. In addition to this, each of those high-duplication methods also have variant execution paths depending on the API response.

We can't totally remove the duplicated code and those variant conditional execution paths, but we can minimize those smells by method extraction.

Before we start, let's set some ground rules that we will use every time we refactor anything going forward:

• We MAY change the module that we're refactoring
• We MAY create new methods within the module
• We MAY create new modules and hand work off to them
• We MAY NOT alter our tests (otherwise, we're redesinging, not refactoring)
• We MUST have a passing test suite after every change (otherwise, we have broken our module)

So, one of the things that makes these two methods so similar, aside from all verbs being very similar in the first place, is that they both interpret the API response identically. That being the case, we can construct a common method to use for response processing in maury/client.py:

    def __process_response(self, response):
        """Process an API response into a Result."""

        if response.ok:
            return Result(response.json(), None)

        return Result(
                None,
                "The API returned the following status: %d" % response.status_code
                )

    def get(self, path, params = None):
        """Perform an HTTP GET on the API.

        Given an endpoint path and a dictionary of parameters, send the request
        to the aPI and return the result.

        Positional arguments:
        path -- the path of the API endpoint to GET

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        """

        response = requests.get(self.__construct_request_url(path),
                params = params,
                headers = {
                    'X-EY-TOKEN' : self.__token,
                    'accept' : 'application/vnd.engineyard.v3+json',
                    'content-type' : 'application/json',
                    })

        return self.__process_response(response)

    def post(self, path, params = None, data = None):
        """Perform an HTTP POST on the API.
        
        Given an endpoint path, a dictionary of parameters, and a dictionary of
        POST data, send the request to the API and return the result.

        Positional arguments:
        path -- the path of the API endpoint you wish to POST

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        data -- a dictionary of POST data (default: None)
        """

        response = requests.post(self.__construct_request_url(path),
                params = params,
                json = data,
                headers = {
                    'X-EY-TOKEN' : self.__token,
                    'accept' : 'application/vnd.engineyard.v3+json',
                    'content-type' : 'application/json',
                    })

        return self.__process_response(response)

That's a little better, and thanks to our tests, we can see that the behavior has not changed. Yay TDD! Still, I see at least one more thing that I don't like: those repeated headers seem like a perfect aspect to reconsider.

Now, there are a lot of ways that we could switch up the request headers dictionary. I usually go for a private method for things like this, but I'm also not familiar enough with Python to know how much of an impact on resource usage and performance constantly generating new dicts will have. That being the case, let's jump into maury/client.py and see if we can find another way:

    def __init__(self, base_url = 'https://api.engineyard.com', token = None):
        """Instantiate a new Client instance
        
        Keyword arguments:
        base_url -- the base URL of the API (default: 'https://api.engineyard.com')
        token -- the API authentication token (default: None)
        """

        self.__base_url = base_url
        self.__headers = {
                'X-EY-Token' : token,
                'accept' : 'application/vnd.engineyard.com.v3+json',
                'content-type' : 'application/json'
                }
    
    def get(self, path, params = None):
        """Perform an HTTP GET on the API.

        Given an endpoint path and a dictionary of parameters, send the request
        to the aPI and return the result.

        Positional arguments:
        path -- the path of the API endpoint to GET

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        """

        response = requests.get(self.__construct_request_url(path),
                params = params,
                headers = self.__headers)

        return self.__process_response(response)

    def post(self, path, params = None, data = None):
        """Perform an HTTP POST on the API.
        
        Given an endpoint path, a dictionary of parameters, and a dictionary of
        POST data, send the request to the API and return the result.

        Positional arguments:
        path -- the path of the API endpoint you wish to POST

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        data -- a dictionary of POST data (default: None)
        """

        response = requests.post(self.__construct_request_url(path),
                params = params,
                json = data,
                headers = self.__headers)

        return self.__process_response(response)

What we did there was to store the headers dictionary directly in the client object as __headers. Also, since we don't use the token for anything else, we are no longer storing the token at all. Also, the tests still pass, so it looks like we're still good.

Can we go further? Sure, we can, but at this point, we probably shouldn't.

So far, all verb implementations have involved making an API request, then processing the API response. We have implemented less than half of the verbs that we have to implement, though.

That being the case, let's follow the advice that I'd imagine Sandy Metz would give right now: let's just keep working on the requirements until we're sure that we can safely refactor further.

While it's not strictly the case, PUT and PATCH are fairly often used interchangeably on REST APIs. In the previous section, we implemented the POST verb, which is typically used for creating a brand new entity on the remote service. In contrast, PUT and PATCH are most typically used to update an entity that already exists. Considering that they are often used synonymously, we're going to go ahead and implement these at the same time.

Another fortunate bit is that put and patch have the same signature as post. That being the case, let's go ahead and copypasta the post test in maury/tests/test_client.py and modify it for our new methods:

    @requests_mock.Mocker()
    def test_put(self, m):
        # The happy path
        m.put('https://api.engineyard.com/sausages', text='{"sausages":"gold"}')

        c = Client(token = 'faketoken')
        result = c.put('sausages')
        self.assertTrue(result.ok)
        self.assertEqual(result.body, {'sausages' : 'gold'})

        # The happy path with params
        m.put(
                'https://api.engineyard.com/sausages?color=gold',
                text='{"sausages":"yep"}')

        result = c.put('sausages', params = {'color' : 'gold'})
        self.assertTrue(result.ok)
        self.assertEqual(result.body, {'sausages' : 'yep'})

        # A wild API error appears!
        m.put(
                'https://api.engineyard.com/ed209',
                status_code = 500,
                text = 'Drop your weapon. You have 20 seconds to comply.')

        result = c.put('ed209')
        self.assertFalse(result.ok)
        self.assertFalse(result.error == None)

        # PEBCAK
        m.put(
                'https://api.engineyard.com/404',
                status_code = 404,
                text = 'You are now staring into the void. It is staring back.')

        result = c.put('404')
        self.assertFalse(result.ok)
        self.assertFalse(result.error == None)

    @requests_mock.Mocker()
    def test_patch(self, m):
        # The happy path
        m.patch('https://api.engineyard.com/sausages', text='{"sausages":"gold"}')

        c = Client(token = 'faketoken')
        result = c.patch('sausages')
        self.assertTrue(result.ok)
        self.assertEqual(result.body, {'sausages' : 'gold'})

        # The happy path with params
        m.patch(
                'https://api.engineyard.com/sausages?color=gold',
                text='{"sausages":"yep"}')

        result = c.patch('sausages', params = {'color' : 'gold'})
        self.assertTrue(result.ok)
        self.assertEqual(result.body, {'sausages' : 'yep'})

        # A wild API error appears!
        m.patch(
                'https://api.engineyard.com/ed209',
                status_code = 500,
                text = 'Drop your weapon. You have 20 seconds to comply.')

        result = c.patch('ed209')
        self.assertFalse(result.ok)
        self.assertFalse(result.error == None)

        # PEBCAK
        m.patch(
                'https://api.engineyard.com/404',
                status_code = 404,
                text = 'You are now staring into the void. It is staring back.')

        result = c.patch('404')
        self.assertFalse(result.ok)
        self.assertFalse(result.error == None)

As expected, these tests fail, as we don't yet have either the put or patch mehods in our client.

So, let's crack open maury/client.py again, copypasta the post method, and modify it to fit our new methods:

    def put(self, path, params = None, data = None):
        """Perform an HTTP PUT on the API.
        
        Given an endpoint path, a dictionary of parameters, and a dictionary of
        PUT data, send the request to the API and return the result.

        Positional arguments:
        path -- the path of the API endpoint you wish to PUT

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        data -- a dictionary of PUT data (default: None)
        """

        r = requests.put(self.__construct_request_url(path),
                params = params,
                json = data,
                headers = self.__headers)

        return self.__process_response(r)

    def patch(self, path, params = None, data = None):
        """Perform an HTTP PATCH on the API.
        
        Given an endpoint path, a dictionary of parameters, and a dictionary of
        PATCH data, send the request to the API and return the result.

        Positional arguments:
        path -- the path of the API endpoint you wish to PATCH

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        data -- a dictionary of PATCH data (default: None)
        """

        r = requests.patch(self.__construct_request_url(path),
                params = params,
                json = data,
                headers = self.__headers)

        return self.__process_response(r)

With that, our tests pass, and we've now implemented 80% of the HTTP verbs that we need to meet our requirements. That means it's time to take another look and decide if we should refactor further.

It might seem obvious that since we've already made a common method to handle the processing of an API response that it would also make sense to break out a method that performs the requests in question.

That is a fine observation, and it is indeed what I would usually do myself. I'm not going to do so here, but let's explore it a bit.

In languages with which I have a higher degree of familiarity, I do HTTP requests the "hard" way. That is, so far, we've been doing things the easy way with methods like request.get(). What I'll usually do is to wrap the HTTP module's generic method is to send a request to the server, specifying the verb along the way. Then, my verb-related methods just dispatch to that wrapper.

In Python, it would look something like this (docstrings shortened for brevity):

    def __make_request(self, verb, path, params = None, data = None):
        """Send an HTTP request to the server."""

        # send the request to the server, return the processed response

    def get(self, path, params = None):
        """Perform an HTTP GET on the API."""

        return self.__make_request('GET', path, params = params)

    def post(self, path, params = None, data = None):
        """Perform an HTTP POST on the API."""

        return self.__make_request('POST', path, params = params, data = data)

    def put(self, path, params = None, data = None):
        """Perform an HTTP PUT on the API."""

        return self.__make_request('PUT', path, params = params, data = data)

    def patch(self, path, params = None, data = None):
        """Perform an HTTP PATCH on the API."""

        return self.__make_request('PATCH', path, params = params, data = data)

Granted, if we run our test suite right now, we get just a ton of errors, because we changed the verbs. The requests package, contrary to my initial thoughts, will allow us to do things the hard way pretty easily. So, let's fill in that __make_request method:

    def __make_request(self, verb, path, params = None, data = None):
        """Send an HTTP request to the server."""

        response = requests.request(
                # Uppercase the verb, just in case
                verb.upper(),
                self._construct_request_url(path),
                params = params,
                json = data,
                headers = self.__headers,
                allow_redirects = True
                )

        return self.__process_response(response)

Okay, that's actually not so bad. The only reason that it leaves a bad taste in my mouth at all is that finding requests.request() was a bit difficult. It is mentioned in the docs (under the Advanced Usage section), but that's not how I found it. I actually ended up digging through the source until I figured out how requests.get() works, then working backwards.

As it were, that's also where I got the allow_redirects argument. All of the verb-related methods in the requests package set this argument to True by default before dispatching to the request() method, so it seems a good idea for us to do so as well. Another fun tidbit: our driver's verb implementations now work almost identically to the way that the verbs in the requests package do.

I rather like this design. Our public verb methods don't do any heavy lifting. Instead, they dispatch to private (not really, but by Python social convention) methods that do all of the heavy lifting.

Our driver is very nearly complete. We only have one verb left to go, so let's get to it!

As it would happen, delete has the same signature and expectations as get, so let's go ahead and copypasta test_get and modify it to test_delete in maury/tests/test_client.py:

    @requests_mock.Mocker()
    def test_delete(self, m):
        # The happy path
        m.delete('https://api.engineyard.com/sausages', text='{"sausages":"gold"}')

        c = Client(token = 'faketoken')
        result = c.delete('sausages')
        self.assertTrue(result.ok)
        self.assertEqual(result.body, {'sausages' : 'gold'})

        # The happy path with params
        m.delete(
                'https://api.engineyard.com/sausages?color=gold',
                text='{"sausages":"yep"}')

        result = c.delete('sausages', params = {'color' : 'gold'})
        self.assertTrue(result.ok)
        self.assertEqual(result.body, {'sausages' : 'yep'})

        # A wild API error appears!
        m.delete(
                'https://api.engineyard.com/ed209',
                status_code = 500,
                text = 'Drop your weapon. You have 20 seconds to comply.')

        result = c.delete('ed209')
        self.assertFalse(result.ok)
        self.assertFalse(result.error == None)

        # PEBCAK
        m.delete(
                'https://api.engineyard.com/404',
                status_code = 404,
                text = 'You are now staring into the void. It is staring back.')

        result = c.delete('404')
        self.assertFalse(result.ok)
        self.assertFalse(result.error == None)

Okay, we have a failing test now, so we're going to do the obvious.

Since the signature is the same as get, we're going to copypasta the get method definition and modify it to fit our new delete method in maury/client.py:

    def delete(self, path, params = None):
        """Perform an HTTP DELETE on the API.
        
        Given an endpoint path and a dictionary of parameters, send the request
        to the API and return the result.

        Positional arguments:
        path -- the path of the API endpoint you wish to DELETE

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        """

        return self.__make_request('DELETE', path, params = params)

There we have it. Now that all of the defined requirements are met and all of our tests pass, our client driver is technically complete.

For the sake of being able to see everything all at once, here is the full source for the complete driver:

from furl import furl
import requests
from .result import Result

class Client(object):
    """A base driver that talks to the Engine Yard API"""

    def __init__(self, base_url = 'https://api.engineyard.com', token = None):
        """Instantiate a new Client instance
        
        Keyword arguments:
        base_url -- the base URL of the API (default: 'https://api.engineyard.com')
        token -- the API authentication token (default: None)
        """

        self.__base_url = base_url
        self.__headers = {
                'X-EY-Token' : token,
                'accept' : 'application/vnd.engineyard.com.v3+json',
                'content-type' : 'application/json'
                }

    def __construct_request_url(self, path):
        """Construct a URL for an API endpoint.
        
        Given a relative endpoint path, construct a fully-qualified API URL.
        """

        # Get a URL object that we can edit
        u = furl(self.__base_url)

        # Set the path to the endpoint in question
        u.path = path

        # Return the modified URL
        return u.url

    def __process_response(self, response):
        """Process an API response into a Result."""

        if response.ok:
            return Result(response.json(), None)

        return Result(
                None,
                "The API returned the following status: %d" % response.status_code
                )

    def __make_request(self, verb, path, params = None, data = None):
        """Send an HTTP request to the server.

        Given an HTTP verb, an endpoint path, a dictionary of parameters,
        and a dictionary of body data, send the request to the API and
        return the result.

        Positional arguments:
        verb -- the HTTP verb to use for the request
        path -- the path of the API endpoint you wish to address

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        data -- a dictionary of POST data (default: None)
        """

        response = requests.request(
                # Uppercase the verb, just in case
                verb.upper(),
                self.__construct_request_url(path),
                params = params,
                json = data,
                headers = self.__headers,
                allow_redirects = True
                )

        return self.__process_response(response)

    def get(self, path, params = None):
        """Perform an HTTP GET on the API.
        
        Given an endpoint path and a dictionary of parameters, send the request
        to the API and return the result.

        Positional arguments:
        path -- the path of the API endpoint you wish to GET

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        """

        return self.__make_request('GET', path, params = params)

    def post(self, path, params = None, data = None):
        """Perform an HTTP POST on the API.
        
        Given an endpoint path, a dictionary of parameters, and a dictionary of
        POST data, send the request to the API and return the result.

        Positional arguments:
        path -- the path of the API endpoint you wish to POST

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        data -- a dictionary of POST data (default: None)
        """

        return self.__make_request('POST', path, params = params, data = data)

    def put(self, path, params = None, data = None):
        """Perform an HTTP PUT on the API.
        
        Given an endpoint path, a dictionary of parameters, and a dictionary of
        PUT data, send the request to the API and return the result.

        Positional arguments:
        path -- the path of the API endpoint you wish to PUT

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        data -- a dictionary of PUT data (default: None)
        """

        return self.__make_request('PUT', path, params = params, data = data)

    def patch(self, path, params = None, data = None):
        """Perform an HTTP PATCH on the API.
        
        Given an endpoint path, a dictionary of parameters, and a dictionary of
        PATCH data, send the request to the API and return the result.

        Positional arguments:
        path -- the path of the API endpoint you wish to PATCH

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        data -- a dictionary of PATCH data (default: None)
        """

        return self.__make_request('PATCH', path, params = params, data = data)

    def delete(self, path, params = None):
        """Perform an HTTP DELETE on the API.
        
        Given an endpoint path and a dictionary of parameters, send the request
        to the API and return the result.

        Positional arguments:
        path -- the path of the API endpoint you wish to DELETE

        Keyword arguments:
        params -- a dictionary of query params (default: None)
        """

        return self.__make_request('DELETE', path, params = params)

Keep in mind, I said that it's technically complete, in as much as it does everything that we need for it to do. That doesn't necessarily mean that we're done, though.

How usable is the maury package right now? How many layers of documentation and submodules does one have to dig through to find out how to create a Client?

While it's not strictly necessary, I like to provide an easy entrypoint at the top-level of the package for things that get used a lot. That way, a developer using my library can just import the top level package, ask it for a client, and get rolling.

First thing first, let's add a test for the function that we want to create in maury/tests/test_new_client.py:

from unittest import TestCase

import maury

class TestMaury(TestCase):
    def test_new_client(self):
        s = maury.new_client(token = 'token12345')
        self.assertIsInstance(s, maury.Client)

That's a pretty simple test (almost too simple, really), but it will do for our purposes right now. What's more, it fails if we try to run our tests right now, so we're definitely on the right track! Let's go ahead and implement this function in maury/__init__.py:

from .client import Client

def new_client(base_url = 'https://api.engineyard.com', token = None):
    """Create a new client driver instance.
    
    Keyword arguments:
    base_url -- the base URL of the API (default: 'https://api.engineyard.com')
    token -- the API authentication token (default: None)
    """

    return Client(base_url = base_url, token = token)

Now we have a somewhat friendly and complete driver for the API that we're mapping. That's it for Phase 1, and now we're ready to move on to Phase 2.

Feel free to take a break. Phase 2 is, if you can imagine it, a lot more reliant on code snippets than Phase 1 is, and it's also a rather tedious and repetitive affair.

Also, just to be clear, you don't have to go any further than you already have. If you simply want to provide a lowish-level client that knows how to talk to the API that you're mapping so others can have a handy building block on which to build their API interactions, you're done. I like to go a little further than that, though, so I'll go ahead and trudge forward.

Remember that I said that I'd ask you questions on occasion? Here are a few that come to mind:

• If our driver works so much like the requests package, why not just use that package directly?
• Why do we keep refactoring such a small module?
• Off the top of your head, what would be different about this driver if we were mapping the Twilio API rather than the Engine Yard API?
• How about the Mastodon API?

Welcome to Phase 2! From here on out, we'll be building out the implementations for the various endpoints that are available, as per the API docs.

First, though, we need to do a bit of thinking about how these implementations will look. Also, for reasons that will become obvious fairly quickly, we need to make a decision about mocking.

So, let's dig into overall endpoint design.

When we did the initial documentation investigation to develop our driver, we gave just a very quick glance at the documentation for the individual endpoints. Now we're going to dig in a bit further to find traits that are common to all of those endpoints.

Looking only at the lists at the beginning of each endpoint page, we can get the following information:

• With few exceptions, all endpoints have a way to get a list of entities
• With few exceptions, all endpoints have a way to get a single entity
• Only a few of the endpoints have meaningful write operations
• A few endpoints have RPC-like operations that can be performed
• Most endpoints can be used as a sub-endpoint of another endpoint (ie "get all accounts for a user")

Looking a bit further on each of these pages, we can glean even more information about those above concepts:

• When listing entities from an endpoint, the returned object is a root node with a list, and the name of that node is the entity type of items in the list
• When getting a single entity from an endpoint, the returned object is a root node with an embedded object, and the name of that node is the entity type of that embedded object
• When querying for either multiple entities or a single entity, most endpoints support passing in a list of query params to filter the result

As they all have slightly different requiremnts, it's a little more difficult to do much design for these endpoints as a whole. Still, we can lay out some basic design elements for all of them:

• An endpoint is wholly contained within a maury submodule
• An endpoint contains an Entity class
• An endpoint contains one or more module-level functions
• These module-level functions all take a Client instance as their first argument

Let's break that down a bit further.

Every formal computer science class that I've taken has emphasized the importance of modular code. The reasoning that's always used is that modular code is code that is easy to re-use. I'm not going to deny that, but I honestly think there's a more important quality to modular code:

A highly modular system contains many clear seams.

What do I mean by "seam?" Much like the seams in your clothing that join two pieces of fabric, a seam in a software system is the methods, references, and so on that allow two pieces of the system to work together. There are a few reasons that seams are handy, but here are the ones that I care about the most, and they're related to each other:

• A seam is a clear, concrete separation of concerns
• A seam gives us a concrete border along which we can mock the code that is not directly under test

In object-oriented design, particularly in terms of [SOLID](https://en.wikipedia.org/wiki/SOLID_(object-oriented_design), one of the big goals is to limit the number of responsibilities given to a single object. Heck, the Single Responsibility Principle is the "S" in SOLID.

That's fine and good for individual classes, but I like to apply SRP as much as is possible to the module that contains those classes. For example, you'll see in the following chapters that the "accounts" module only does account stuff, the "environments" module only does environment stuff, so on.

Sometimes, it's incredibly inconvenient to actually run all of the code that's invovled with generating a result in our tests. For example, in our driver implementation, you saw that we used requests-mock to fake out the raw HTTP calls that were made.

Generally speaking, there are two big important rules for mocking:

• You cannot mock the code under test
• You can only mock documented interfaces for which you can reasonably replicate behavior

This is usually rolled up neatly as "mock along the seam." In the case of our driver, we don't "own" the upstream API, so that is a natural seam. That being the case, we mocked specific known behaviors of the upstream API.

Generally speaking, it's perfectly fine to mock code along any seam in your system. We'll get deeper into this in the following chapters, and it's a technique that I very much hope carries over into all of your projects if you don't already work this way.

In both the Data Mapper and the Repository patterns, the actual data object that is returned from the data store is typically referred to as an "entity." In the case of Repository, it's usually something like a Python dictionary, and in the case of Data Mapper, it's typically an object with reader methods for the properties of the object.

Take a look back at the Engine Yard API documentation: more or less every request returns a JSON object that describes one or more instance of the requested resource. Those are entities in the broad sense.

I mentioned that the API client design that I like is somewhat like the Data Mapper pattern overlayed on top of the Repository pattern. That being the case, here's the general design that we'll use for all Entity classes:

• An Entity has a reader for each important property
• An Entity has a general reader that can access any property
• An Entity should be immutable

According to the Accounts documentation, there are four basic uses for the endpoint:

• List all accounts (GET a collection of accounts visible to the current user)
• List all all accounts for a given user (GET a collection of accounts that are visible to the current user and associated with the given user)
• Show an account (GET a single account)
• Update an account's emergency contact (PUT new data for the account)

Two of these are problematic right off the top. For one, it appears that the docs don't actually tell us anything about the Users endpoint. Also, while the last use is described only in terms of updating the emergency contact, the parameter list for the use case implies that we can also update the account name.

Before we actually implement any of those, though, let's go ahead and define the Entity for this module ...

This is the first entity that we're defining, so it's going to be a bit more work than the entities for pretty much all other endpoints. Let's review the general requirements for an Entity:

• It must have a general reader for any property
• It must have a specific reader for all important properties
• It should be immutable

Why would we want the Entity to be immutable (or as close as we can get to it in Python)? In the end, I can bring up several arguments both for and against the notion of immutable data. The reality here is that after working for a long time with both sorts of systems, I've had a much better experience with those that treat data as data.

I don't have a lot of incredibly strong opinions about developering in general, but the idea that data should be data and nothing more is definitely a member of that club.

So, we want to be able to read any given property (by key) from the entity. Let's start with a test and work backwards from there. Since we're effectively just fetching a value from a dictionary, let's call the method fetch and add our test to maury/tests/test_accounts.py:

from unittest import TestCase

import maury.accounts as accounts

class TestAccounts(TestCase):
    def test_entity_fetch(self):
        data = {
                'id' : 'someaccount'
                }

        e = accounts.Entity(data = data)

        # When the property is known, we return the associated value
        self.assertEqual(e.fetch('id'), data['id'])

        # When the property is unknown, we return None
        self.assertEqual(e.fetch('unknown_key'), None)

That seems to cover the basic rules around fetch: if the key is in the Entity's data dictionary, we return the value for that key. Otherwise, we return None. Granted, if we run that test, we get an error about the class not existing. Let's fix that in maury/accounts.py:

class Entity(object):
    """A data structure model for an Account"""

    pass

That's enough to run our tests and ... get a different error. Such is the nature of the test-driven development beast. Now it's complaining that we can't actually instantiate an Entity the way that we're trying to, so let's fix that:

class Entity(object):
    """A data structure model for an Account"""

    def __init__(self, data = {}):
        """Instantiate an Entity with some data"""

        self.__data = data

        # Ensure that __data is a dict
        if type(self.__data) is not dict:
            self.__data = {}

We're getting closer. Now we get yet another error, and it's because we haven't a fetch method for our Entity objects. Back to maury/accounts.py:

class Entity(object):
    """A data structure model for an Account"""

    def __init__(self, data = {}):
        """Instantiate an Entity with some data"""

        self.__data = data

        # Ensure that __data is a dict
        if type(self.__data) is not dict:
            self.__data = {}

    def fetch(self, key):
        """Get a data property from the Entity
        
        Positional Arguments:
        key -- the name of the data property we wish to access
        """

        pass

Now we're getting somewhere. We're no longer getting Python errors when we run our accounts test ... now we're getting an actual test failure:

FAIL: test_entity_fetch (maury.tests.test_accounts.TestAccounts)

Looking a bit further, we see that our primary assertion is not being provided by our fetch implementation. That is, right now, fetch literally doesn't return anything (it returns the default Python return, None). So, let's make it return the expected information:

    def fetch(self, key):
        """Get a data property from the Entity
        
        Positional Arguments:
        key -- the name of the data property we wish to access
        """

        return self.__data[key]

Alright, our first assertion passes ... which leads us to another Python error in our tests:

KeyError: unknown key

That indicates that our second scenario is failing (though we could arguably be testing that in a way to produce a failure rather than an error). So, what we want is for fetch to return the actual requested data if the key is known, but to return a default None value if the key is not known. Since it's guaranteed that an Entity's internal __data is a dict, we can use a handy method for that:

    def fetch(self, key):
        """Get a data property from the Entity
        
        Positional Arguments:
        key -- the name of the data property we wish to access
        """

        return self.__data.get(key, None)

Our tests are now passing, and that's awesome. We've implemented a general reader for any data property of an account entity.

We also require that the Entity have a specific reader for all "important" data properties to make things easier for the end-users of our client. We're nice folks like that. So, let's head back over to the Accounts docs and figure out what some of the important properties are.

At least for now, I've narrowed it down to basically the following:

• id -- the identifier used on the API to reference the account
• name -- the name used on the API to reference the account
• emergency_contact -- the emergency contact for the account

Let's start by adding a test to maury/tests/test_accounts.py:

    def test_entity_id(self):
        data = {
                'id' : 'someaccount'
                }

        good = accounts.Entity(data = data)
        bad = accounts.Entity(data = {})

        # An entity with an id returns that id
        self.assertEqual(good.id, data['id'])

        # An entity without an id returns None
        self.assertEqual(bad.id, None)

This test is actually quite a lot like our fetch test, as do the initial results. That being the case, let's go ahead and implement the id method by using the fetch method:

    @property
    def id(self):
        """Get the account's ID"""

        return self.fetch('id')

Boom. Our tests pass, and we can move on to the next reader.

The other two specific readers are just riffs on the id reader. Let's go ahead and add the tests for those to maury/tests/test_accounts.py:

    def test_entity_name(self):
        data = {
                'name' : 'someaccount'
                }

        good = accounts.Entity(data = data)
        bad = accounts.Entity(data = {})

        # An entity with a name returns that name
        self.assertEqual(good.name, data['name'])

        # An entity without a name returns None
        self.assertEqual(bad.name, None)

    def test_entity_emergency_contact(self):
        data = {
                'emergency_contact' : '911'
                }

        good = accounts.Entity(data = data)
        bad = accounts.Entity(data = {})

        # An entity with an emergency contact returns that contact
        self.assertEqual(good.emergency_contact, data['emergency_contact'])

        # An entity without an emergency contact returns None
        self.assertEqual(bad.emergency_contact, None)

We already know that these tests fail, and we already know why ... there's not an implementation for these properties. Let's go ahead and add them to maury/accounts.py:

    @property
    def name(self):
        """Get the account's name"""

        return self.fetch('name')

    @property
    def emergency_contact(self):
        """Get the account's emergency contact"""

        return self.fetch('emergency_contact')

Boom. Our tests pass, and all of our (currently required) specific readers are implemented.

The last requirement that we have is that entities should be immutable. If I understand everything properly, this is unfortunately not technically possible for third-party types, like our Entity class.

To that end, we're using Python's social conventions to get as close as we can here (without dropping down to C to implement our data structure). That is, an Entity's data is stored in its __data member, and the social convention in Python is to not directly access methods and members that begin with __.

I'd personally prefer a stronger guarantee here. That said, this tradeoff also happens in just about all of the existing maury implementations (the notable exception being maury-rust).

So, while it's technically possible for somebody using our client to modify the makeup of a given account Entity, it would be bad form for them to do so.