Practical e2e testing

December 15, 2021

by Afanasy Barbarov

Situation: 3 microservices communicating to each other in a docker environment.

Task: Configure sample pipeline to run e2e tests on build

E2E

One key responsibility of Site Reliability Engineers is to quantify confidence in the systems they maintain. This is achievable with end-to-end (E2E) testing (but not limited to). End-to-end testing is a methodology used to test whether the flow of an application is performing as designed from start to finish. E2E tests are on the top of the testing pyramid. Googlers say, that they should split by 70%/20%/10% - for unit, integration, and E2E respectively. The numbers can vary, but the shape should look like a pyramid. I'd say, there should be linters and static analyzers at the very bottom since code style is always a point of debate within a team, and leaving that to the machine helps a lot and saves time and psychical energy.

The purpose of end-to-end tests is to identify system dependencies and that the integrated components of an application function as expected. The entire application is tested in a real-world scenario such as communicating with the database, network, hardware, and other applications. E2E tests simulate real-user behavior and thus cover all cases with main functionality in the application or service.

An example of E2E test of an email application might look like:

Logging in to the application
Opening inbox
Composing and sending an email
Logging out of the application

Preparation for E2E test:

Analyze requirements. Focus on critical things. Don't write tests just to cover every possible step.
Set up a staging environment in alignment with all the requirements
List down how every service needs to respond
List down testing methods required to test these responses.
Design test cases
Run tests, study, and save the result

Ideal setup (from my point of view)

Let's assume, that there are 10 microservices (each in its own Docker registry), communicating to each other in the real world. An ideal setup (for me) would be to have E2E running on each merge to the main branch (per microservice), and additionally, running them every night. Each test run consists of

Building up the latest version and pushing to the registry (only on push to the main branch).
Spinning up all microservices (by pulling the latest version from the registry and running up with docker compose)in the team's staging environment.
All end-to-end tests are then run against this staging environment.
A Slack/email notification is sent to the team channel with a summary report.

Common pitfalls

Finding the root cause for a failing E2E test is painful and takes time.
They keep the state (as you are running them on a copy of a real system). Thus each test needs a clean setup.
Consider running tests in parallel to speed up the process. From the other side, if the test setup allows parallel run, consider that each run may result in different output. Again, a clean setup will help but will increase the total time of the test run.
E2E tests increase the product delivery time (by adding complexity). So choose wisely what to test and consider the team effort.

Best practices

It is important to follow the practices outlined below - to ensure smooth testing and feasible cost management

Avoid calling third-party services. You test your own system, no need to test the others. Use contract testing tools, e.g. [Pact]https://pact.io/ to mimic third-party behavior.
Focus on "happy paths". End-to-end tests tend to run longer than all other types of tests, so test erroneous scenarios only if it's needed for the business. Don't use the E2E test to validate e.g. the login form, there should be unit tests for that.
Focus on features whose failure will cause maximum issues. Examples of such features could be authentication or payment service. Start with these features and design more elaborate test cases to verify them.
Optimize setup and teardown mechanisms. A common approach is to run tests for a dockerized infrastructure.

Lights, Camera, Action!

Let's create a new Github repository, called practical-e2e, and create 3 microservices for it - authentication, products, and payment.

The code for the Authentication service:

package main

import (
   "encoding/json"
   "io/ioutil"
   "log"
   "net/http"
)

type AuthRequest struct {
   Username string `json:"username"`
   Password string `json:"password"`
}

type TokenRequest struct {
   Token string `json:"token"`
}

type Response struct {
   Message string `json:"message"`
}

func authenticate(w http.ResponseWriter, r *http.Request) {
   body, err := ioutil.ReadAll(r.Body)
   if err != nil {
   	panic(err)
   }

   var credentials AuthRequest
   json.Unmarshal(body, &credentials)

   w.Header().Set("Content-Type", "application/json")
   if credentials.Username == "username"
     && credentials.Password == "password" {
   	w.WriteHeader(http.StatusOK)
   	json.NewEncoder(w).Encode(Response{
   		Message: "TOKEN: SECRET AUTH TOKEN",
   	})
   } else {
   	w.WriteHeader(http.StatusUnprocessableEntity)
   }
}

func validate(w http.ResponseWriter, r *http.Request) {
   body, err := ioutil.ReadAll(r.Body)
   if err != nil {
   	panic(err)
   }

   var credentials TokenRequest
   err = json.Unmarshal(body, &credentials)

   w.Header().Set("Content-Type", "application/json")
   if credentials.Token == "TOKEN: SECRET AUTH TOKEN" {
   	w.WriteHeader(http.StatusOK)
   	json.NewEncoder(w).Encode(Response{
   		Message: "VALID TOKEN",
   	})
   } else {
   	w.WriteHeader(http.StatusUnauthorized)
   }
}

func main() {
   http.HandleFunc("/authenticate", authenticate)
   http.HandleFunc("/validate", validate)

   log.Println("Listening Authentication service on port :8081")
   log.Fatal(http.ListenAndServe(":8081", nil))
}

Product service:

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
)

type Product struct {
	Id   int64  `json:"id"`
	Name string `json:"name"`
}

type Response struct {
	Message string `json:"message"`
}

var products = []Product{
	{Id: 1, Name: "Product 1"},
	{Id: 2, Name: "Product 2"},
	{Id: 3, Name: "Product 3"},
}

func list(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)

	json.NewEncoder(w).Encode(products)
}

func order(w http.ResponseWriter, r *http.Request) {
	token := r.Header.Get("Authorization")
	authReq, _ := json.Marshal(map[string]string{
		"token": token,
	})

	authRes, _ := http.Post("http://localhost:8081/validate",
	 "application/json", bytes.NewBuffer(authReq))
	defer authRes.Body.Close()

	if authRes.StatusCode == http.StatusUnauthorized {
		w.WriteHeader(http.StatusForbidden)
		json.NewEncoder(w).Encode(Response{
			Message: fmt.Sprintf("Cannot order product:
			 access denied"),
		})
		return
	}

	body, _ := ioutil.ReadAll(r.Body)

	var product Product
	json.Unmarshal(body, &product)

	w.Header().Set("Content-Type", "application/json")
	for _, p := range products {
		if p.Id == product.Id {
			if pay(p, token) {
				w.WriteHeader(http.StatusOK)
				json.NewEncoder(w).Encode(Response{
					Message: fmt.Sprintf("Successfully ordered
						product %d", product.Id),
				})
			} else {
				w.WriteHeader(http.StatusUnprocessableEntity)
				json.NewEncoder(w).Encode(Response{
					Message: fmt.Sprintf("Cannot order product
					 %d: payment failed", product.Id),
				})
			}
			return
		}
	}

	w.WriteHeader(http.StatusUnprocessableEntity)
	json.NewEncoder(w).Encode(Response{
		Message: fmt.Sprintf("Product not found %d", product.Id),
	})
}

func pay(product Product, token string) bool {
	payReq, _ := json.Marshal(product)
	req, _ := http.NewRequest("POST",
		"http://localhost:8082/pay", bytes.NewBuffer(payReq))
	req.Header.Set("Authorization", token)
	req.Header.Set("Content-Type", "application/json")

	client := &http.Client{}
	resp, _ := client.Do(req)
	defer resp.Body.Close()

	return resp.StatusCode == http.StatusOK
}

func main() {
	http.HandleFunc("/list", list)
	http.HandleFunc("/order", order)

	log.Println("Listening Product service on port :8083")
	log.Fatal(http.ListenAndServe(":8083", nil))
}

And the payment service:

package main

import (
	"bytes"
	"encoding/json"
	"io/ioutil"
	"log"
	"net/http"
)

type Product struct {
	Id   int64  `json:"id"`
	Name string `json:"name"`
}

func pay(w http.ResponseWriter, r *http.Request) {
	authReq, _ := json.Marshal(map[string]string{
		"token": r.Header.Get("Authorization"),
	})

	authRes, _ := http.Post("http://localhost:8081/validate",
	 "application/json", bytes.NewBuffer(authReq))
	defer authRes.Body.Close()

	if authRes.StatusCode == http.StatusUnauthorized {
		w.WriteHeader(http.StatusForbidden)
		return
	}

	body, _ := ioutil.ReadAll(r.Body)

	var product Product
	json.Unmarshal(body, &product)
	body, _ = ioutil.ReadAll(r.Body)

	w.WriteHeader(http.StatusOK)
}

func main() {
	http.HandleFunc("/pay", pay)

	log.Println("Listening Payment service on port :8082")
	log.Fatal(http.ListenAndServe(":8082", nil))
}

A successful e2e test will be as follows:

Get the list of products: GET 8083/list
Authenticate and get token: GET 8081/authenticate
Order product with ID=1 with Authorization header provided
Check that the status code is 200

A failing e2e test will be

Get the list of products: GET 8082/list
Try to order a product with ID=3 without Authorization header
Check that the status code is 422

Let's set up a Github action that triggers these tests. First, let's create Dockerfiles for each microservice. This is for the auth service:

# syntax=docker/dockerfile:1

##
## Build
##
FROM golang:1.17-buster AS build

WORKDIR /app

COPY go.mod ./
#COPY go.sum ./
RUN go mod download

COPY *.go ./

RUN go build -o /auth

##
## Deploy
##
FROM gcr.io/distroless/base-debian10

WORKDIR /

COPY --from=build /auth /auth

EXPOSE 8081

USER nonroot:nonroot

ENTRYPOINT ["/auth"]

And they are pretty similar for the other two microservices (only the name changes). Also, we need to adjust the code a bit, in order for each microservice to be discoverable by others. To do so, replace "http://localhost" with either "http://auth" or "http://payment" where needed. And here is a docker-compose file to join all together:

version: '3'

services:
  auth:
    image: auth
    build:
      context: auth
    restart: always
    ports:
      - "8081:8081"
  payment:
    image: payment
    build:
      context: payment
    restart: always
    ports:
      - "8082:8082"
  product:
    image: product
    build:
      context: product
    restart: always
    ports:
      - "8083:8083"

So, everything is set up, time to write our e2e tests! You can write e2e tests in the language of your choice, or, if you have time, do it manually first, to check at least that everything is properly configured. Usually, it's something high-level, like Cypress or similar, which spins up a browser and behaves like a real user. We will just imitate such a test with simple curl commands and just check the output from Github Actions.

name: Test
on:
  pull_request:
  push:
    branches:
     - main

jobs:
  e2e:
    name: Run e2e tests
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Build and run containers
        run: docker-compose up -d
      - name: Fake successful scenario
        run:  |
          curl --request POST --url http://localhost:8081
          /authenticate \
          --header 'content-type: application/json' \
          --data '{"username": "username", "password": "password"}'
          curl --request GET --url http://localhost:8083/list \
          --header 'content-type: application/json'
          curl --request POST --url http://localhost:8083/order \
          --header 'content-type: application/json'\
          --header 'Authorization: TOKEN: SECRET \
          AUTH TOKEN' --data '{"id": 1 }'
          echo "PASS OK"
      - name: Fake erroneous scenario
        run:  |
          curl --request GET --url http://localhost:8083/list \
          --header 'content-type: application/json'
          curl --request POST --url http://localhost:8083/order \
          --header 'content-type: application/json' \
          --header 'Authorization: FAKE_TOKEN' --data '{"id": 1'}
          echo "FAIL OK"

And lookig at the Github Actions output, we'll see that the test scenarios gives us expected results.

  curl --request POST --url http://localhost:8081/
  authenticate \
  --header 'content-type: application/json' \
  --data '{"username": "username", "password": "password"}'
  curl --request POST --url http://localhost:8081/
  authenticate \
  --header 'content-type: application/json' \
  --data '{"username": "username", "password": "password"}'
  curl --request GET --url http://localhost:8083/list \
  --header 'content-type: application/json'
  curl --request POST --url http://localhost:8083/order \
  --header 'content-type: application/json' \
  --header 'Authorization: TOKEN: SECRET AUTH TOKEN' \
  --data '{"id": 1 }'
  echo "PASS OK"
  shell: /usr/bin/bash -e {0}
  % Total    % Received % Xferd  Average Speed Time Time Time Current
                                 Dload  Upload Total Spent Left Speed

  0    0   0    0   0    0    0    0 --:--:-- --:--:-- --:--:--     0
100  87 100  39  100  48 19500 24000 --:--:-- --:--:-- --:--:-- 43500
{"message":"TOKEN: SECRET AUTH TOKEN"}
  % Total    % Received % Xferd  Average Speed Time  Time Time Current
                                 Dload  Upload Total Spent Left Speed

  0  0  0    0  0 0      0      0 --:--:-- --:--:-- --:--:--     0
100  86 100  86 0 0  28666      0 --:--:-- --:--:-- --:--:-- 28666
[{"id":1,"name":"Product 1"},{"id":2,"name":"Product 2"},
{"id":3,"name":"Product 3"}]
  % Total  % Received % Xferd  Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0  0 0   0  0   0      0      0 --:--:-- --:--:-- --:--:--     0
100 55 100 45 100  10   5625   1250 --:--:-- --:--:-- --:--:--  6875
{"message":"Successfully ordered product 1"}
PASS OK
0s
Run curl --request GET --url http://localhost:8083/list\
--header 'content-type: application/json'
  % Total  % Received % Xferd Average Speed Time  Time  Time  Current
                             Dload  Upload Total Spent  Left  Speed

  0 0  0    0   0 0      0  0 --:--:-- --:--:-- --:--:--     0
100 86 100  86  0 0  43000  0 --:--:-- --:--:-- --:--:-- 86000
[{"id":1,"name":"Product 1"},{"id":2,"name":"Product 2"},
{"id":3,"name":"Product 3"}]
  % Total  % Received % Xferd  Average Speed Time Time  Time  Current
                              Dload  Upload  Total Spent Left  Speed

  0  0  0   0   0    0    0      0 --:--:-- --:--:-- --:--:--     0
100 59  100 50  100  9  16666 3000 --:--:-- --:--:-- --:--:-- 19666
{"message":"Cannot order product: access denied"}
FAIL OK

If you are looking for the code only, here it is - https://github.com/abarbarov/practical-e2e

Hometask

Change the code, so the test breaks only if something in payment fails. Hint: you can hardcode some corner cases especially for that.

Results:

This is a brief introduction to the test setup. There might be a special QA department in an organization, that automates testing and they know a lot more about it. This post is not to steal their job, but to provide general concepts and show a simple setup to glue all together. Don't stop and have fun!

That's all, folks!