Jinhua Blog

burl: A simple but flexible HTTP/3 testing framework based on bash and curl

 nginx  2006  10 mins

burl◎ Ancient One

TL;DR

I’ve created a new HTTP/3 testing framework based on bash and curl:

https://github.com/kingluo/burl

Chinese blog version:

https://zhuanlan.zhihu.com/p/675741855

Background

A few months ago, when I ported the QUIC patches from the nginx mainline to APISIX and tried to test it, I found that test::nginx didn’t work very well. It uses the wrong listen directive parameter “http3” instead of “quic” (probably due to version differences).

So I was wondering if I could design a simple testing framework because the latest curl already fully supports HTTP/3.

Many testing frameworks prefer DSLs (Domain Specific Languages) such as test::nginx and Hurl because it is more user-friendly. And it looks more data-driven. However, this is not good for developers looking to extend its functionality as they would need to learn the language behind it , take the time to understand the core and extension APIs by reading the code base of this testing framework. In other words, DSL and the language behind it create a knowledge gap for developers because it is not intuitive and cannot map the functions corresponding to the DSL to the language itself. Especially when the framework itself lacks capabilities and needs to be expanded, this gap will bring significant learning costs.

Take test::nignx as an example, which is implemented in Perl. When I need to verify the response body, decrypt it by pre-shared key, The built-in function doesn’t help as it only supports regular expression matching. So I need to write a bunch of perl code to do this. However, I’m not familiar with Perl, so this isn’t an easy task. As we all know, Perl is not a mainstream language and has a steep learning curve. At that time, I was thinking, how great it would be if I could write code in Python.

The expressive capabilities of test::nginx are limited, even though it already supports sending requests using curl, because it is not a scripting system (Of course, you can extend it using Perl modules, but due to the limitations of the framework design, this is a difficult task for most people).

  1. It assumes your input is a text string, you cannot use other formats such as protobuf, and you cannot compress or encrypt text.
  2. The only output post-processing supported is regular matching, so you cannot parse and verify other forms of content, such as decryption.
  3. You cannot describe complex test procedures, such as sending three requests to trigger the request quota and verify the rate-limiting results.
  4. You cannot make other types of requests such as GRPC, or SOAP.

In my practice, when I have such needs that exceed test::nginx’s capabilities, I have to delegate them to the lua part, which is very troublesome.

For example, when I need to parse and verify the correctness of the CAS Auth plugin, it involves a lot of client-side CAS protocol details, so you have to implement the entire logic in the lua part, which increases the difficulty of maintaining this mix. As we all know, neither Perl nor Lua are popular languages. ;-(

You can see the test case below involves additional lua lib development to test CAS. And, to accommodate test::nginx’s lack of response parsing, you have to do the conversion in the lua environment. Quite annoying, at least to me, because I’ve written a lot of test cases like this over the years.

=== TEST 2: login and logout ok
--- config
    location /t {
        content_by_lua_block {
            local http = require "resty.http"
            local httpc = http.new()
            -- Test-specific CAS lua lib
            local kc = require "lib.keycloak_cas"

            local path = "/uri"
            local uri = "http://127.0.0.1:" .. ngx.var.server_port
            local username = "test"
            local password = "test"

            local res, err, cas_cookie, keycloak_cookie = kc.login_keycloak(uri .. path, username, password)
            if err or res.headers['Location'] ~= path then
                ngx.log(ngx.ERR, err)
                ngx.exit(500)
            end
            res, err = httpc:request_uri(uri .. res.headers['Location'], {
                method = "GET",
                headers = {
                    ["Cookie"] = cas_cookie
                }
            })
            assert(res.status == 200)
            ngx.say(res.body)

            res, err = kc.logout_keycloak(uri .. "/logout", cas_cookie, keycloak_cookie)
            assert(res.status == 200)
        }
    }
--- response_body_like
uri: /uri
cookie: .*
host: 127.0.0.1:1984
user-agent: .*
x-real-ip: 127.0.0.1

Similarly, Hurl also uses a DSL to describe tests. Its backend language is Rust. So in terms of scalability, it seems to be worse than test::nginx because it needs to compile the code. However, Rust’s ecosystem is much better than Perl’s. So you can write code easily. But anyway, you need to write the code in a different language than the DSL.

Is there a simple testing framework that can satisfy the following requirements?

  1. No DSL, just simple shell scripts
  2. Easy to extend and programming language agnostic
  3. Can be used to initialize and test any server, not limited to nginx, such as envoy

Bash scripting

Shell is the most important tool under the terminal in our daily work. Every administrator and developer needs to enter commands to complete their work. The most interesting thing is that the shell is also a programming platform, that is, scripts. When we talk about scripts, we care about two aspects: the power of the shell language itself and the available commands, which act like a standard library for a high-level programming language.

In the ancient times of computer history, the functions of the shell were very weak and the commands were very limited, so Perl was born. But now, things have changed. Many advanced shells have emerged, such as bash and zsh, which provide many excellent language elements for expressing complex logic. There are also a growing number of convenient and elegant commands, such as jq, which parses JSON with compact expressions. In Linux and Mac, the default shell is bash, so it’s right under our noses, so why not use it to complete our testing tasks? Why bother with complicated programming to get the job done, right?

Shell commands look very natural and straightforward. The command line is made up of words, just like human language, so why use a DSL? Furthermore, if a simple command line doesn’t meet your requirements, you can refactor it using variables and flow control statements, just like any mainstream programming language. Note that scripting is optional, i.e. you won’t encounter any complexity as a simple command line does everything, so everything works like a DSL, but better.

For example, if I need to send a GET and check for 404, two simple commands are enough:

1
2
3
4
5

# send request
REQ /anything/foobar

# validate the response headers
HEADER -x "HTTP/3 404"

But I need to check if Prometheus is logging 404 events as the correct number, how can I do this? Parses Prometheus metric messages and checks for counter changes in a shell function. Everything still looks simple, right? That’s the beauty of scripting.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

count_404() {
    curl http://127.0.0.1:9091/apisix/prometheus/metrics 2>&1 | \
        grep -F 'apisix_http_status{code="404",route="1",matched_uri="/anything/*"' | \
        awk '{print $2}'
}

# send the first request
REQ /anything/foobar

# get current counter
cnt1=`count_404`

# send the second request
REQ /anything/foobar

cnt2=`count_404`

# check if the counter is increased
((cnt2 == cnt1 + 1))

You can use flow control statements to express any business logic.

For example, to test whether the request limiting plugin is working, you can use a loop.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# consume the quota
for ((i=0;i<2;i++)); do
    # send request
    REQ /httpbin/get -X GET --http3-only

    # validate the response headers
    HEADER -ix "HTTP/3 200"
done

# no quota
REQ /httpbin/get -X GET --http3-only
HEADER -x "HTTP/3 503"
HEADER -ix "x-ratelimit-remaining: 0"

burl: bash + curl

burl is a simple but flexible HTTP/3 testing framework based on bash and curl.

Design

  1. The test file contains one or more test cases, and an optional initial part of the file header, such as configuring nginx.conf and starting nginx via template rendering.
  2. Each test case consists of three parts:
    1. Construct and send the request, and save the response header and response body to files for subsequent steps.
    2. Verify the response headers, for example using “grep”.
    3. Parse and validate the response body, for example with the “jq” expression.
  3. Easily extensible, you can validate responses (steps ii and iii) using any command or other advanced script, such as Python.
  4. Failure of any command will stop the testing process (enabled via the “set -euo pipefail” bash option).
  5. The test process is echoed by default (enabled via the “set -x” bash option).

Synopsis

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

#!/usr/bin/env burl

# Optional initialization here...
# Before all test cases are executed.
# For example, render nginx.conf and start nginx.
SET NGX_CONF_HTTP <<EOF
upstream test_backend {
    server $(dig +short nghttp2.org):443;

    keepalive 320;
    keepalive_requests 1000;
    keepalive_timeout 60s;
}
EOF

SET NGX_CONF <<'EOF'
location / {
    add_header Alt-Svc 'h3=":443"; ma=86400';
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host "nghttp2.org";
    proxy_pass https://test_backend;
}
EOF

START_NGX

TEST 1: test case

# Send request
# REQ is a curl wrapper so you can apply any curl options to suit your needs.
# Check https://curl.se/docs/manpage.html for details.
REQ /httpbin/anything --http3 -d foo=bar -d hello=world

# Validate the response headers
# HEADER is a grep wrapper so you can apply any grep options and regular expressions to suit your needs.
HEADER -x "HTTP/3 200"

# Validate the response body, e.g. JSON body
# JQ is a jq wrapper so you can apply any jq options and jq expression to suit your needs.
JQ '.method=="POST"'
JQ '.form=={"foo":"bar","hello":"world"}'

TEST 2: another test case
# ...

# More test cases...

Examples

APISIX

  1. Test MTLS whitelist
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

TEST 2: route-level mtls, skip mtls

ADMIN put /ssls/1 -d '{
    "cert": "'"$(<${BURL_ROOT}/examples/certs/server.crt)"'",
    "key": "'"$(<${BURL_ROOT}/examples/certs/server.key)"'",
    "snis": [
        "localhost"
    ],
    "client": {
        "ca": "'"$(<${BURL_ROOT}/examples/certs/ca.crt)"'",
        "depth": 10,
        "skip_mtls_uri_regex": [
            "/httpbin/get"
        ]
    }
}'

sleep 1

REQ /httpbin/get --http3-only

# validate the response headers
HEADER -x "HTTP/3 200"

# validate the response body, e.g. JSON body
JQ '.headers["X-Forwarded-Host"] == "localhost"'
  1. Test HTTP/3 Alt-Svc
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

ADMIN put /ssls/1 -d '{
    "cert": "'"$(<${BURL_ROOT}/examples/certs/server.crt)"'",
    "key": "'"$(<${BURL_ROOT}/examples/certs/server.key)"'",
    "snis": [
        "localhost"
    ]
}'

ADMIN put /routes/1 -s -d '{
    "uri": "/httpbin/*",
    "upstream": {
        "scheme": "https",
        "type": "roundrobin",
        "nodes": {
            "nghttp2.org": 1
        }
    }
}'



TEST 1: check if alt-svc works

altsvc_cache=$(mktemp)
GC "rm -f ${altsvc_cache}"

REQ /httpbin/get -k --alt-svc ${altsvc_cache}
HEADER -x "HTTP/1.1 200 OK"

REQ /httpbin/get -k --alt-svc ${altsvc_cache}
HEADER -x "HTTP/3 200"

SOAP

Send a SOAP request to the web service and verify the response.

Construct JSON input using jo and validate JSON output using jq expressions.

Powered by Python zeep.

1
2
3
4
5

TEST 1: test a simple Web Service: Add two numbers: 1+2==3

SOAP_REQ \
    'https://ecs.syr.edu/faculty/fawcett/Handouts/cse775/code/calcWebService/Calc.asmx?WSDL' \
    Add `jo a=1 b=2` '.==3'

XML

Powered by xmltodict.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

TEST 2: GET XML

# send request
REQ /httpbin/xml

# validate the response headers
HEADER -x "HTTP/1.1 200 OK"
HEADER -x "Content-Type: application/xml"

# validate the response XML body
XML '.slideshow["@author"]=="Yours Truly"'

Conclusion

I think the HTTP testing framework is not difficult and does not require the use of high-level programming languages to implement. DSL is effectively a meaningless gap between framework implementation and representation because we need to incur translation costs, especially for compiled programming languages, and For testers, learning it is a burden. So why not unify them and script them? With the help of bash (the standard shell used daily in almost all operating systems) and curl, we can handle all our testing efforts easily.

Welcome to discuss and contribute to burl:

https://github.com/kingluo/burl

See Also: