Jinhua Blog

coco: a simple stackless, single-threaded, and header-only C++20 coroutine library

luajit.io@gmail.com (jinhualuo) — Fri, 02 Jan 2026 00:00:00 +0000

TL;DR

coco is a simple stackless, single-threaded, and header-only C++20 coroutine library that leverages native C++20 coroutines for async/await programming with Go-like channels and waitgroups.

https://github.com/kingluo/coco

Background

Don’t communicate through shared memory, share memory through communication.

I have always been impressed by Golang’s CSP programming, which is very expressive and couples different business logic together through channels.

Initially, I implemented coco using macros to make coroutines look like Go, but use a switch-based state machine behind the scenes. However, with the advent of C++20 coroutines, I realized we could leverage native language support for even better performance and cleaner syntax.

Design

Uses C++20 coroutines for native async/await support.
Header-only library with no external dependencies.
Stackless coroutines with async/await implementation.
Channel and waitgroup primitives like Go.
Single-threaded, no locks required.
Minimal performance overhead.
Simple FIFO scheduler for managing multiple coroutines.

Synopsis

Defining, Resuming and Running Coroutines:

// Define a coroutine function - must return co_t and use co_await/co_yield/co_return
co_t my_coroutine(int id) {
    std::cout << "Coroutine " << id << " started" << std::endl;
    co_yield resched;  // Yield control and reschedule for later execution
    std::cout << "Coroutine " << id << " resumed" << std::endl;
    co_return;
}

int main() {
    // Create coroutine instance
    auto coro = my_coroutine(1);

    // Resume the coroutine - schedules it for execution
    coro.resume();

    // Run the scheduler to execute all scheduled coroutines
    scheduler_t::instance().run();

    return 0;
}

Joining Coroutines:

co_t task1() {
    std::cout << "Task 1 started" << std::endl;
    co_yield resched;
    std::cout << "Task 1 finished" << std::endl;
    co_return;
}

co_t task2() {
    std::cout << "Task 2 started" << std::endl;
    co_yield resched;
    std::cout << "Task 2 finished" << std::endl;
    co_return;
}

co_t main_task() {
    // Start task1 and wait for it to complete using join()
    co_await go([]() { return task1(); }).join();

    // After task1 completes, start task2
    co_await go([]() { return task2(); }).join();

    std::cout << "Both tasks completed sequentially" << std::endl;
    co_return;
}

int main() {
    auto task = main_task();
    task.resume();
    scheduler_t::instance().run();
    return 0;
}

Channels:

co_t example() {
    chan_t<int> ch(1);  // Buffered channel with capacity 1

    // Producer coroutine
    auto producer = [&ch]() -> co_t {
        for (int i = 0; i < 3; i++) {
            std::cout << "Sending: " << i << std::endl;
            bool ok = co_await ch.write(i);
            if (!ok) {
                std::cout << "Channel closed, stopping producer" << std::endl;
                break;
            }
        }
        ch.close();
        std::cout << "Producer finished" << std::endl;
        co_return;
    };

    // Consumer coroutine
    auto consumer = [&ch](const std::string& name) -> co_t {
        while (true) {
            auto result = co_await ch.read();
            if (result.has_value()) {
                std::cout << name << " received: " << result.value() << std::endl;
                // Yield and reschedule to allow fair distribution among consumers
                co_yield resched;
            } else {
                std::cout << name << " channel closed" << std::endl;
                break;
            }
        }
        co_return;
    };

    // Start producer and consumers
    auto prod = producer();
    auto cons1 = consumer("Consumer1");
    auto cons2 = consumer("Consumer2");

    // Schedule coroutines for execution
    prod.resume();
    cons1.resume();
    cons2.resume();

    // Run scheduler to execute all scheduled coroutines
    scheduler_t::instance().run();

    co_return;
}

Waitgroups:

co_t worker(int id, wg_t& wg) {
    std::cout << "Worker " << id << " starting" << std::endl;
    co_yield resched;  // Yield control, automatically rescheduled
    std::cout << "Worker " << id << " finished" << std::endl;
    wg.done();
    co_return;
}

co_t example() {
    wg_t wg;

    // Add workers to waitgroup
    wg.add(3);

    // Start workers using go()
    auto w1 = go([&wg](){ return worker(1, wg); });
    auto w2 = go([&wg](){ return worker(2, wg); });
    auto w3 = go([&wg](){ return worker(3, wg); });

    // Wait for all workers to complete
    co_await wg.wait();
    std::cout << "All workers completed!" << std::endl;

    co_return;
}

int main() {
    auto task = example();
    task.resume();
    scheduler_t::instance().run();
    return 0;
}

Example: A Simple Webserver Based on io_uring

In “Lord of the io_uring”, there is an example that implements a simple webserver. You can see that the callback style makes the code difficult to read and maintain.

void server_loop(int server_socket) {
    struct io_uring_cqe * cqe;
    struct sockaddr_in client_addr;
    socklen_t client_addr_len = sizeof(client_addr);
    add_accept_request(server_socket, & client_addr, & client_addr_len);
    while (1) {
        int ret = io_uring_wait_cqe( & ring, & cqe);
        struct request * req = (struct request * ) cqe -> user_data;
        if (ret < 0)
            fatal_error("io_uring_wait_cqe");
        if (cqe -> res < 0) {
            fprintf(stderr, "Async request failed: %s for event: %d\n",
                strerror(-cqe -> res), req -> event_type);
            exit(1);
        }
        switch (req -> event_type) {
        case EVENT_TYPE_ACCEPT:
            add_accept_request(server_socket, & client_addr, & client_addr_len);
            add_read_request(cqe -> res);
            free(req);
            break;
        case EVENT_TYPE_READ:
            if (!cqe -> res) {
                fprintf(stderr, "Empty request!\n");
                break;
            }
            handle_client_request(req);
            free(req -> iov[0].iov_base);
            free(req);
            break;
        case EVENT_TYPE_WRITE:
            for (int i = 0; i < req -> iovec_count; i++) {
                free(req -> iov[i].iov_base);
            }
            close(req -> client_socket);
            free(req);
            break;
        }
        /* Mark this request as processed */
        io_uring_cqe_seen( & ring, cqe);
    }
}

Connection acceptance and request handling are both mixed in a big switch statement. Compared with Go, this is difficult to maintain. When you face real complex business logic, it’s hard to imagine what the code looks like.

With coco using C++20 coroutines, it can be expressed as:

// Note: This is simplified pseudocode to illustrate the concept.
// For a complete working example, see the examples/ directory in the GitHub repo.

co_t accept_connections(int server_socket) {
    while (true) {
        // Accept connection (async operation using io_uring)
        int client_socket = co_await accept_async(server_socket);

        // Spawn coroutine to handle this connection
        auto handler = handle_connection(client_socket);
        handler.resume();  // Runs concurrently with accept loop
    }
}

co_t handle_connection(int client_socket) {
    // Read request (async operation using io_uring)
    auto request = co_await read_request_async(client_socket);

    // Process request (synchronous business logic)
    auto response = handle_request(request);

    // Send response (async operation using io_uring)
    co_await send_response_async(client_socket, response);

    close(client_socket);
}

Much cleaner and easier to understand, right? The business logic flows naturally from top to bottom.

The io_uring event loop does only one thing: schedules the coroutine for resumption.

void event_loop(io_uring* ring) {
    while (true) {
        io_uring_cqe* cqe;
        io_uring_wait_cqe(ring, &cqe);

        // Get the awaiter from completion event
        iouring_awaiter* awaiter = (iouring_awaiter*)cqe->user_data;
        awaiter->res = cqe->res;  // Store the result

        // CRITICAL: Use scheduler to resume the coroutine handle
        if (awaiter->handle) {
            scheduler_t::instance().schedule(awaiter->handle);
        }

        io_uring_cqe_seen(ring, cqe);

        // Run scheduler to process resumed coroutines
        scheduler_t::instance().run();
    }
}

It lets you write C++ code like Go, but without any performance sacrifices.

Key Features and Benefits

Since coco uses native C++20 coroutines, it provides several advantages over traditional callback-based or macro-based approaches:

Native Language Support: Uses C++20’s built-in coroutine support for better performance and debugging
Type Safety: Better compile-time error checking compared to macro-based approaches
Clean Syntax: Natural async/await syntax without complex macros
Automatic State Management: No manual state tracking - the compiler handles coroutine state automatically
Exception Safety: Exceptions can be used normally within coroutines and are properly propagated through join operations
Memory Management: Full RAII compliance - destructors are called properly and resource cleanup works as expected
Debugging Support: Better debugger integration for stepping through coroutine code
Simple Scheduler: FIFO queue-based scheduler for managing multiple coroutines with cooperative multitasking
Coroutine Join: Ability to wait for specific coroutines to complete with exception propagation
Flexible Yielding: Support for both automatic rescheduling (co_yield resched) and manual control (co_yield no_sched). Auto-rescheduling is for fair distribution of work, and manual control is for fine-grained control (we resume it manually only when ready, e.g., io_uring completion)

Performance Characteristics

Zero-cost abstraction: C++20 coroutines compile to efficient state machines with minimal overhead
No heap allocation per operation: Only the coroutine frame is heap-allocated once at creation
Single-threaded: No lock contention or synchronization overhead
Cooperative scheduling: Predictable execution order with FIFO scheduling
Stackless: Lower memory footprint compared to stackful coroutines (like Boost.Context)

For high-performance I/O workloads (e.g., with io_uring), coco provides near-native performance while maintaining clean, readable code.

Important Caveats of C++20 Coroutines and coco Implementation

While C++20 coroutines provide powerful async/await capabilities, there are several important limitations and considerations to keep in mind when using coco:

Scheduler-Based Coordination

The channel implementation uses a scheduler-based approach for coroutine coordination, which prevents stack exhaustion:

// In read_wait_t::await_resume()
void wake_up_writer() {
    if (!ch->wq.empty()) {
        auto writer = ch->wq.front();
        ch->wq.pop();
        if (writer && !writer.done()) {
            scheduler_t::instance().schedule(writer);  // Schedules instead of direct resume
        }
    }
}

Benefit: This prevents recursive execution and stack exhaustion by queuing coroutine handles in the scheduler’s FIFO queue.

Impact: The scheduler processes coroutines in FIFO order, providing cooperative multitasking. All channel operations (read/write) automatically use the scheduler to wake up waiting coroutines.

Coroutine Composition with Join

Important Pattern: You can only use co_await and co_yield in the top-level coroutine function. To compose coroutines, use the join() method.

// ✅ This works - co_await in top-level coroutine function
co_t authenticate_user(const std::string& username) {
    std::cout << "Authenticating " << username << "..." << std::endl;
    co_yield resched;
    std::cout << "Authentication successful" << std::endl;
    co_return;
}

// ✅ Correct: Use join() to compose coroutines
co_t handle_user_request(const std::string& username) {
    // Use go().join() to create, schedule, and join in one expression
    co_await go([&](){ return authenticate_user(username); }).join();
    std::cout << "Request handled!" << std::endl;
    co_return;
}

// ❌ This doesn't work - co_await in helper function
void helper_function() {
    auto result = co_await some_async_operation(); // Compilation error!
}

// ❌ This also doesn't work - mixing return with coroutine keywords
co_t handle_request_WRONG(const std::string& username) {
    if (username.empty()) {
        return authenticate_user(username);  // Compilation error!
    }
    co_yield resched;
    co_return;
}

Solution: Use co_await coroutine.join() to compose coroutines sequentially. The go() helper creates and schedules the coroutine, then join() waits for completion.

Variable Lifetime and RAII

Critical Consideration: C++20 coroutines are stackless - the compiler transforms your coroutine into a state machine with a heap-allocated coroutine frame. RAII works perfectly - objects maintain their identity and resources across suspension points.

However, pointers and references to local variables (stack variables) become invalid across suspension points because the coroutine frame may be relocated in memory.

Key Rules:

RAII objects are preserved perfectly across suspension points
Local variables maintain their identity - no copying or moving occurs during suspension
Destructors are called only when the coroutine completes (co_return)
Heap allocations remain valid (smart pointers, containers’ internal data)
⚠️ Pointers/references to local variables are INVALID after suspension - the frame may be relocated

co_t raii_example() {
    std::unique_ptr<int> resource = std::make_unique<int>(42);
    std::string data = "Hello";           // Lives in coroutine frame
    std::string* ptr = &data;             // Pointer to local variable

    co_await some_operation();

    // ✅ Safe - RAII objects are preserved across suspension
    data += " World";
    *resource = 100;

    // ❌ UNSAFE - ptr may point to old frame location after suspension
    // *ptr += " World";  // Undefined behavior!

    // Resources destroyed only when coroutine completes
    co_return;
}

Why This Happens: The coroutine frame is heap-allocated and may be moved in memory during suspension/resumption. While the objects themselves are preserved (maintaining their state and identity), any pointers or references you created that point to these objects will still point to the old memory location, making them invalid.

How Coroutine Suspension Works

When a coroutine suspends (via co_await or co_yield), the coroutine frame is preserved in heap memory:

Coroutine Frame (heap-allocated):
├── Promise object
├── Parameters
├── Local variables (including RAII objects)
├── Temporary objects
└── Suspension state

Key Points:

No copy/move constructors called during suspension/resumption
No destructors called until the coroutine completes (co_return)
Object identity preserved - the same objects exist before and after suspension
RAII works perfectly - resources are held across suspension points

This is why C++20 coroutines are so efficient - objects maintain their state and identity across suspension points without any copying or moving overhead.

Best Practices

✅ RAII works perfectly - Use RAII objects freely across suspension points
✅ Use value semantics - Local variables are safely preserved in the coroutine frame
⚠️ Avoid pointers/references to local variables across suspension points - They become invalid when the coroutine frame is relocated
✅ Use the scheduler - Always call scheduler_t::instance().run() to execute scheduled coroutines
✅ Yield cooperatively - Use co_yield resched to allow other coroutines to run, especially in loops or long-running operations
✅ Use join for coordination - Use co_await coroutine.join() to wait for specific coroutines to complete and propagate exceptions
✅ Use wg_guard_t for exception safety - Prefer wg_guard_t over manual wg.done() calls when exceptions are possible
✅ Compose with join - Use co_await go(...).join() pattern to compose coroutines sequentially
✅ Exception handling - Exceptions thrown in coroutines are captured and can be propagated via join()

Understanding these patterns is crucial for writing robust coroutine-based code with coco. The key insights are:

Coroutine suspension preserves RAII semantics perfectly - objects maintain their identity across suspension points
Pointers/references to local variables become invalid across suspension points - the coroutine frame may be relocated in memory

Comparison with Other Approaches

Feature	coco (C++20 Coroutines)	Callbacks	Threads	Stackful Coroutines
Readability	✅ Excellent	❌ Poor (callback hell)	✅ Good	✅ Good
Performance	✅ Excellent	✅ Excellent	⚠️ Moderate (context switch overhead)	⚠️ Good (stack overhead)
Memory Usage	✅ Low (stackless)	✅ Low	❌ High (stack per thread)	⚠️ Moderate (stack per coroutine)
Debugging	✅ Good (native support)	❌ Difficult	✅ Good	⚠️ Moderate
Synchronization	✅ Not needed (single-threaded)	✅ Not needed	❌ Required (locks, atomics)	Depends
Exception Handling	✅ Natural	⚠️ Complex	✅ Natural	✅ Natural
Composability	✅ Excellent (co_await)	❌ Poor	⚠️ Moderate	✅ Good

Use Cases

coco is ideal for:

High-performance I/O servers - Web servers, API gateways, proxy servers using io_uring or epoll
Async data processing pipelines - ETL workflows, stream processing with clean sequential logic
Protocol implementations - Network protocols, message parsers with state machines
Game servers - Single-threaded game loops with async I/O operations
Embedded systems - Resource-constrained environments where thread overhead is prohibitive
Any scenario where you want Go-like concurrency in C++ - Without the runtime overhead

Conclusion

Whether you’re building high-performance servers with io_uring, complex async workflows, or concurrent data processing pipelines, coco makes concurrent programming in C++ more expressive and enjoyable. The single-threaded, scheduler-based approach eliminates the need for locks while providing cooperative multitasking capabilities. With only a simple header file, it’s easy to integrate and understand.

Key Takeaways:

Write async C++ code that reads like synchronous code
Leverage native C++20 coroutines for zero-cost abstraction
Use Go-like channels and waitgroups for elegant concurrency patterns
No external dependencies, just include the header and start coding

If you like it, please star the GitHub repo!

GitHub: https://github.com/kingluo/coco

For complete examples, documentation, and the latest updates, visit the repository. Contributions and feedback are welcome!

coco: a simple stackless, single-threaded, and header-only C++11 coroutine library

luajit.io@gmail.com (jinhualuo) — Sun, 22 Dec 2024 00:00:00 +0000

TL;DR

coco is a simple stackless, single-threaded, and header-only C++11 coroutine library.

https://github.com/kingluo/coco

Background

Don’t communicate through shared memory, share memory through communication.

I have always been impressed by Golang’s CSP programming, which is very expressive and couples different business logic together through channels.

In my C++ programming career, I have always struggled with callback hell. Usually, more than 5 callbacks are enough to make me confused even if I am the author, not to mention parallel asynchronous calls. On modern Linux, we use epoll or io_uring for asynchronous programming, but callback handling will make the business logic fragmented. C++20 introduced coroutines, but it is very complicated and far from the simple building blocks in Go that I need. I searched on GitHub, but did not find a corotuine library that is simple enough for my needs.

In Rust, a coroutine is implemented by an implicit state machine generated by the compiler. It is essentially a switch statement. So why not use macros to make coroutines look like Go, but use a state machine behind the scenes? Also, I don’t need multithreaded coroutines (goroutines) because in C++, usually we handle threads ourselves to improve efficiency, and we also don’t need a complex runtime like Go to schedule coroutines, so I hope the coroutine behaves like Lua, that is, it is cooperative and managed by the programmer to make it as flexible as possible.

So, I decided to build one myself (funny enough, I did something similar a decade ago, bringing Java’s reflection, annotations, and object proxies to C++). This is coco, a header file with only 200 code lines.

Design

No depends on C++20 coroutine, C++11 is enough.
No compiler dependencies, simple macros, header-only.
coroutine like Lua, stackless async/await implementation just like Rust.
channel and waitgroup like Go.
single-threaded, no locks.
No performance overhead.

Synopsis

    auto sock_read = new co_t([=](co_t* __self, state_t* _st) {
        auto st = dynamic_cast<sock_read_state_t*>(_st);
        COCO_ASYNC_BEGIN(sock_read);
        for (; st->cnt < 3; st->cnt++) {
            COCO_ASYNC_BEGIN(loop);
            COCO_WRITE_CHAN(write_ch, fs_write_ch, st->cnt);
            COCO_WRITE_CHAN(write_ch2, kafka_produce_ch, st->cnt);
            COCO_ASYNC_END();
        }
        fs_write_ch->close();
        kafka_produce_ch->close();
        COCO_WAIT(wait_all, wg);
        COCO_ASYNC_END();
        COCO_DONE();
    }, new sock_read_state_t);

    while (!sock_read->done())
        sock_read->resume();

Example: A simple webserver based on io_uring

In “Lord of the io_uring”, there is an example that implements a simple webserver. You can see that the callback style makes the code difficult to read and maintain.

void server_loop(int server_socket) {
    struct io_uring_cqe * cqe;
    struct sockaddr_in client_addr;
    socklen_t client_addr_len = sizeof(client_addr);
    add_accept_request(server_socket, & client_addr, & client_addr_len);
    while (1) {
        int ret = io_uring_wait_cqe( & ring, & cqe);
        struct request * req = (struct request * ) cqe -> user_data;
        if (ret < 0)
            fatal_error("io_uring_wait_cqe");
        if (cqe -> res < 0) {
            fprintf(stderr, "Async request failed: %s for event: %d\n",
                strerror(-cqe -> res), req -> event_type);
            exit(1);
        }
        switch (req -> event_type) {
        case EVENT_TYPE_ACCEPT:
            add_accept_request(server_socket, & client_addr, & client_addr_len);
            add_read_request(cqe -> res);
            free(req);
            break;
        case EVENT_TYPE_READ:
            if (!cqe -> res) {
                fprintf(stderr, "Empty request!\n");
                break;
            }
            handle_client_request(req);
            free(req -> iov[0].iov_base);
            free(req);
            break;
        case EVENT_TYPE_WRITE:
            for (int i = 0; i < req -> iovec_count; i++) {
                free(req -> iov[i].iov_base);
            }
            close(req -> client_socket);
            free(req);
            break;
        }
        /* Mark this request as processed */
        io_uring_cqe_seen( & ring, cqe);
    }
}

Connection acceptance and request handling are all mixed in a big switch. Compared with golang, this is terrible, isn’t it? When you face the real complex business logic, it’s hard to imagine what the code looks like.

With coco, it can be expressed as:

    go([=](co_t* __self, state_t* _st) {
        auto st = dynamic_cast<iouring_state_t*>(_st);
        st->co = __self;
        while (true) {
            COCO_ASYNC_BEGIN(loop);
            do_accept(server_socket, nullptr, nullptr, st);
            COCO_YIELD();
            auto sk = st->res;
            go([=](co_t* __self, state_t* _st) {
                auto req = dynamic_cast<conn_state_t*>(_st);
                req->co = __self;
                COCO_ASYNC_BEGIN(process_req);

                read_request(req);
                COCO_YIELD();

                handle_request(req);
                send_response(req);
                COCO_YIELD();
                delete __self;

                COCO_ASYNC_END();
                COCO_DONE();
            }, new conn_state_t(sk));
            COCO_ASYNC_END();
        }
        COCO_DONE();
    }, new iouring_state_t);

Looks much better, right?

And the io_uring callback does only one thing: resumes the coroutine.

void server_loop(int server_socket) {
    while (true) {
        struct io_uring_cqe *cqe;
        int ret = io_uring_wait_cqe(&ring, &cqe);
        if (ret < 0)
            fatal_error("io_uring_wait_cqe");
        if (cqe->res < 0) {
            fprintf(stderr, "Async request failed: %s\n",
                    strerror(-cqe->res));
            close(server_socket);
            exit(1);
        }

        iouring_state_t *st = (iouring_state_t*)cqe->user_data;
        st->res = cqe->res;
        st->co->resume();
        io_uring_cqe_seen(&ring, cqe);
    }
}

It lets you program in C++ like Go, but without any performance sacrifices.

Coding restrictions

Since Coco is simple and has no compiler support, it inevitably has some coding limitations.

coroutine lambda functions should be reentrant, so the statements should be reentrant and produce the same flow after reentry.
yield, wait, or channel r/w should be placed in an async block at the first level.
variables declared cannot escape yield, so if you need some variables to survive from a yield, use the state or global variables.
exceptions must be caught inside the lambda function.
state subclasses must inherit the state_t base class and dynamic_cast it to the subclass when using it.
The state is managed by the coroutine, so the state will be deleted when the coroutine exits.
You must delete all created coroutines, channels, and waitgroups yourself.

Conclusion

Coco is a poor man’s C++11 coroutine library that can improve your productivity when doing asynchronous programming.

If you like it, please star my GitHub repo!

https://github.com/kingluo/coco

envoy asynchronous HTTP filter: lua-resty-ffi vs golang filter

luajit.io@gmail.com (jinhualuo) — Sun, 04 Feb 2024 00:00:00 +0000

In Envoy, both lua-resty-ffi (envoy porting) and the built-in golang filter allow you to use golang to develop asynchronous business logic through goroutine, but which one performs better?

Call flow

When the headers of an incoming request get parsed by envoy, an HTTP filter’s decodeHeaders() is called. If decodeHeaders() returns StopIteration, other header filters in the chain will be skipped, but the request body will still be received and move forward to the data phase.

When a body chunk (partial body, not necessarily in chunk encoding) comes in, Envoy will call the decodeData() of filters one by one. If decodeData() returns StopIterationAndBuffer, the filter manager will buffer this body chunk for the subsequent decodeData() call, i.e. the next time when the decodeData() gets called, it will take accumulated body data. If decodeData() returns StopIterationAndNoBuffer, the filter manager will not buffer this body chunk and continue, i.e. the filter will be responsible for the body block.

Consider a simple scenario where Envoy echoes the request body back to the client. Let’s take a look at how golang filter and lua-resty-ffi-golang work respectively.

golang filter

The golang filter consists of two parts, one is the C++ filter, and the other is the golang filter instance (in go language). There is a subtle point here. When decodeHeaders() creates a goroutine to perform an asynchronous job and returns a StopIteration, the decodeData() calls on the C++ side will be made concurrently with that goroutine, so the C++ filter always buffers the request body chunks for later use. The golang side filter’s decodeData() is called only when the goroutine calls the continue callback (f.callbacks.Continue(api.StopAndBuffer)). That is, the golang filter buffers the entire request body by itself, instead of the filter manager.

There are 2 memory copies here:

#1 copy envoy Buffer to golang string

capi_impl.go

func (c *httpCApiImpl) HttpGetBuffer(r unsafe.Pointer, bufferPtr uint64, length uint64) []byte {
	buf := make([]byte, length)
	res := C.envoyGoFilterHttpGetBuffer(r, C.uint64_t(bufferPtr), unsafe.Pointer(unsafe.SliceData(buf)))
	handleCApiStatus(res)
	return unsafe.Slice(unsafe.SliceData(buf), length)
}

golang_filter.cc

  for (const Buffer::RawSlice& slice : buffer->getRawSlices()) {
    // data is the heap memory of go, and the length is the total length of buffer. So use memcpy is
    // safe.
    memcpy(data, static_cast<const char*>(slice.mem_), slice.len_); // NOLINT(safe-memcpy)
    data += slice.len_;
  }

Someone may wonder why not just use func C.GoString(*C.char) string directly. Well, because the Buffer is not a flattened C string, so copying it is not a trivial job. ;-)

#2 copy golang string to Buffer

It gets the golang string data pointer and length so that it can be copied directly to the C++ side. The latest version uses unsafe.StringData() while previous versions used reflect.StringHeader casting.

capi_impl.go

1
2
3

func (c *httpCApiImpl) HttpSetBufferHelper(r unsafe.Pointer, bufferPtr uint64, value string, action api.BufferAction) {
	c.httpSetBufferHelper(r, bufferPtr, unsafe.Pointer(unsafe.StringData(value)), C.int(len(value)), action)
}

lua-resty-ffi-golang

Lua filters work in different ways. It does not use a callback style like golang filter, which maps C++ interfaces to golang interfaces one by one, such as DecodeData(). Instead, it uses coroutines to run logic (one coroutine per request), and all functions (such as getting the body) are encapsulated in wrapper objects and work in a non-blocking manner. Furthermore, it does not export the phases (header, body, encoding, or decoding) explicitly, the wrapping object methods will do the phase transition implicitly.

With lua-resty-ffi, the golang runtime (main goroutine) is started once. All subsequent messages (or requests) will be sent to that goroutine for processing or dispatch. For the echo logic here, the main goroutine creates a worker goroutine for each message processing. The worker coroutine responds to the Lua coroutine through envoy dispatcher.post(), schedules a closure to be executed in the envoy main thread, and resumes the Lua coroutine there.

Note that we have 8 body copies here!

#1 copy Buffer to lua

wrapper.cc

  // TODO(mattklein123): Reduce copies here by using Lua direct buffer builds.
  std::unique_ptr<char[]> data(new char[length]);
  data_.copyOut(index, length, data.get());
  lua_pushlstring(state, data.get(), length);

One more interesting point here, the body is split into slices in the Buffer, but the lua C API requires a contiguous C string, so similar to the golang filter, it first flattens the Buffer and pushes it to the lua stack. Here involves two memory allocations.

#2 marshal the body and auth header

Unlike golang filters, where every operation is a C function call, lua-resty-ffi needs to use IPC (albeit efficient) to communicate with golang. I tested a lot of marshaling (serialization) methods and formats and found that string concatenation was the most efficient in this demo, which also made me focus on the filter efficiency itself.

envoy.yaml

local auth = req:headers():get("Authorization")
local body = req:body()
body = body and body:getBytes(0, body:length()) or "ok"
local tbl = {auth, body}
local data = table.concat(tbl, "\n")

#3 malloc and copy the message containing the auth header and request body to golang runtime
#4 unmarshal the message in the golang

filter.go

var rlen C.int
r := C.ngx_http_lua_ffi_get_req(task, &rlen)
c := unsafe.Pointer(r)
var i int
for ; i < int(rlen); i++ {
    b := *(*byte)(unsafe.Add(c, uintptr(i)*1))
    if b == '\n' {
        break
    }
}
str1 := unsafe.String((*byte)(unsafe.Pointer(r)), i)
str2 := unsafe.String((*byte)(unsafe.Add(unsafe.Pointer(r), uintptr(i+1)*1)), int(rlen)-i-1)

#5 dispatch the result message to the envoy main thread
#6 in envoy main thread, the message is pushed to the lua stack
#7 respond the HTTP request (echo the request body)

As you can see, unlike golang filters, there is a lua runtime between envoy and golang, and lua-resty-ffi also requires IPC to complete its work.

Therefore, lua-resty-ffi has the following disadvantages:

The direction from envoy to golang brings additional OS thread scheduling costs
To exchange messages you need to marshal and unmarshal the messages and use C malloc as transport, hence the reason for 8 memory allocations and copies in this demo.

But does this mean that lua filters are slower than golang filters? Without benchmarks, you have no answers.

There is an interesting point here that I must point out :-). Since the lua filter returns StopIterationAndBuffer, i.e. it relies on the filter manager to buffer and collect request body chunks, the Buffer parameter of each decodeData() call is the currently collected request body. On the last decodeData() call, it doesn’t have a chance to return StopIterationAndBuffer again, so it returns the buffer to the filter manager and gets the buffer data pointer for later reading and writing.

lua_filter.cc

  } else if (state_ == State::WaitForBody && end_stream_) {
    ENVOY_LOG(debug, "resuming body due to end stream");
    callbacks_.addData(data);
    state_ = State::Running;
    resumeCoroutine(luaBody(coroutine_.luaState()), yield_callback_);

lua_filter.cc

body_wrapper_.reset(
    Filters::Common::Lua::BufferWrapper::create(
	state, headers_, const_cast<Buffer::Instance&>(*callbacks_.bufferedBody())),
    true);

Benchmark

Check the benchmark code here:

https://github.com/kingluo/ffi-benchmark

golang filter snippet:

func (f *filter) DecodeHeaders(header api.RequestHeaderMap, endStream bool) api.StatusType {
	go func() {
		if ok, msg := f.verify(header); !ok {
			f.callbacks.SendLocalReply(401, msg, map[string]string{}, 0, "bad-request")
			return
		}
		// StopAndBuffer will buffer the entrie body for use by DecodeData()
		f.callbacks.Continue(api.StopAndBuffer)
	}()
	return api.Running
}

func (f *filter) DecodeData(buffer api.BufferInstance, endStream bool) api.StatusType {
	data := buffer.String()
	go func() {
		f.callbacks.SendLocalReply(200, data, map[string]string{}, 0, "basic-auth")
	}()
	return api.Running
}

lua filter snippet:

  function envoy_on_request(req)
    require("resty_ffi")
    local demo = ngx.load_ffi("ffi_go_basic_auth",
      '{"user":"fooname","password":"validpassword"}')
    local auth = req:headers():get("Authorization")
    local body = req:body()
    body = body and body:getBytes(0, body:length()) or "ok"
    local tbl = {auth, body}
    local data = table.concat(tbl, "\n")
    local _, rc, res = demo:auth(data, req)
    req:respond({[":status"] = tostring(rc)}, res)
  end

go func() {
	for {
		task := C.ngx_http_lua_ffi_task_poll(tq)
		if task == nil {
			break
		}
		...
		req := &Request{Auth: str1, Body: str2}
		go func() {
			if ok, msg := verify(req.Auth); !ok {
				C.ngx_http_lua_ffi_respond(task, 401, (*C.char)(C.CString(msg)), C.int(len(msg)))
			} else {
				C.ngx_http_lua_ffi_respond(task, 200, (*C.char)(C.CString(req.Body)), C.int(len(req.Body)))
			}
		}()
	}
}()

Envoy and nginx can use the same golang shared library files compiled for lua-resty-ffi, this is one of the advantages of lua-resty-ffi, just like how LSP (Language Server Protocol) works, so I include nginx in the benchmark for reference.

I use k6 as the benchmark tool.

benchmark.js

import http from 'k6/http';

export const options = {
  discardResponseBodies: true,
};

const data = open('/opt/test.data.64k', 'b');
//const data = open('/opt/test.data.1m', 'b');
//const data = open('/opt/test.data.10m', 'b');

export default function () {
  //http.get('http://envoy:10000/get', {headers:{Authorization:'Basic Zm9vbmFtZTp2YWxpZHBhc3N3b3Jk'}});
  http.post('http://envoy:10000/post', data, {
    headers: {
      'Content-Type': 'application/octet-stream',
      Authorization: 'Basic Zm9vbmFtZTp2YWxpZHBhc3N3b3Jk',
      Expect: '',
    }
  });
}

Test:

# build envoy with lua-resty-ffi
git checkout lua-resty-ffi
./ci/run_envoy_docker.sh './ci/do_ci.sh dev.contrib'
ln -sf `find . -name envoy-contrib -executable` /usr/local/bin/envoy

# build openresty with lua-resty-ffi
bash build_openresty.sh

# start golang-filter enabled envoy
GODEBUG=cgocheck=0 envoy -c envoy.yaml --concurrency 1

# start lua-resty-ffi enabled envoy
LD_LIBRARY_PATH=$PWD LUA_PATH='/opt/lua-resty-ffi/?.lua;;' envoy -c envoy.yaml --concurrency 1

# start lua-resty-ffi enabled nginx
LD_LIBRARY_PATH=/opt/ffi-benchmark/lua-filter/ LUA_PATH='/opt/lua-resty-ffi/?.lua;;' \
  nginx -p $PWD -g 'daemon off; error_log /dev/stderr info;'

# start k6 to do benchmark
for ((i=0;i<$((1024*64));i++)); do echo -n 'a' >> /opt/test.data.64k; done
for ((i=0;i<$((1024*1024*1));i++)); do echo -n 'a' >> /opt/test.data.1m; done
for ((i=0;i<$((1024*1024*10));i++)); do echo -n 'a' >> /opt/test.data.10m; done
k6 run -u 1 -d 60s -q /opt/envoy_vs_nginx_ffi.js

We only check the latency metric here, which is http_req_duration.

# golang filter
# cpu: 66%
avg=105.09µs min=75.32µs med=100.85µs max=5.9ms p(90)=120.53µs p(95)=129.46µs

# lua filter
# cpu: 70%
avg=111.78µs min=78.01µs med=108.98µs max=12.62ms p(90)=129.03µs p(95)=136.13µs

# nginx
# cpu: 62%
avg=74.66µs  min=48.19µs med=72.57µs  max=6.36ms p(90)=83.15µs p(95)=86.51µs

POST 64k

# golang filter
# cpu: 77%
avg=218.94µs min=155.97µs med=192.04µs max=12.27ms p(90)=218.93µs p(95)=235.65µs

# lua filter
# cpu: 66%
avg=230.68µs min=170.97µs med=205.81µs max=10.74ms p(90)=235.01µs p(95)=247.15µs

# nginx
# cpu: 60%
avg=164.92µs min=107.25µs med=142.75µs max=10.89ms p(90)=167.14µs p(95)=176.01µs

POST 1m

# golang filter
# cpu: 90%
avg=1.47ms min=1.07ms med=1.18ms max=9.67ms p(90)=1.49ms p(95)=3.32ms

# lua filter
# cpu: 80%
avg=1.82ms min=1.5ms med=1.59ms max=8.21ms p(90)=1.65ms p(95)=4.12ms

# nginx
# cpu: 83%
avg=2.04ms min=1.58ms med=1.77ms max=9.73ms p(90)=2.16ms p(95)=4.52ms

POST 10m

# golang filter
# cpu: 90%
avg=14.06ms min=12.59ms med=13.66ms max=39.31ms p(90)=16.61ms p(95)=17.01ms

# lua filter
# cpu: 90%
avg=20.56ms min=19ms med=20.19ms max=66.24ms p(90)=22.94ms p(95)=23.56ms

# nginx
# cpu: 93%
avg=23.02ms min=21.08ms med=22.57ms max=42.85ms p(90)=24.71ms p(95)=25.99ms

You can see that due to the shortcomings I mentioned before, especially the memory copies, lua-resty-ffi takes more time than the golang filter, but not too much (the biggest time difference is approximately 20% when transferring the 10MB body) and is proportional to the size of the subject.

However, in most use cases we don’t have to move the entire bodies to and from the lua-resty-ffi runtime, because the data is arbitrary and on demand. In other words, this demo represents an extreme situation and is for reference only.

Conclusion

Since both lua-resty-ffi-golang and golang filters can meet the same development needs, which one is better?

In my opinion, the performance overhead of lua-resty-ffi is a bit higher than golang filters, but lua-resty-ffi is a better choice for extending envoy functionality in a hybrid programming approach:

lua-resty-ffi supports multiple languages, not just golang: Rust, Golang, Java, Python, Nodejs, so it can adapt well to different technology stacks
lua-resty-ffi supports both nginx and envoy, so as long as you develop an extension for envoy, it can also run in nginx (even without recompiling!)
lua-resty-ffi is relatively simple, while golang’s filter implementation is more complex. I actually spent some time figuring out from the source code how golang filters work.
Golang filters export the same interface as C++ filters, so it is subtle to make a correct decision about which status code is returned, such as the difference between Continue and StopIteration when calling back envoy in a goroutine. golang filters are error-prone, while lua-resty-ffi provides a generic golang sidecar, you can write any normal golang code like in other projects and the lua filter will handle the correct phase transitions for you.
lua filter supports hot-reload of the source code, while golang filter cannot (unless re-compilation). Moreover, lua-resty-ffi supports language runtime hot-reload, e.g. Python, Java and Nodejs.

Welcome to learn more about lua-resty-ffi:

https://github.com/kingluo/lua-resty-ffi

Asynchronous nginx response header/body filter

luajit.io@gmail.com (jinhualuo) — Thu, 04 Jan 2024 00:00:00 +0000

TL;DR

In openresty, using ngx.location.capture(), we can achieve high-performance asynchronous processing of upstream responses.

Motivation

As we all know, due to the design limit, nginx header/body filters are synchronous and you should not perform blocking operations such as blocking system calls, decrypting, decompressing, or forwarding responses to other servers for further processing.

In contrast, envoy supports asynchronous HTTP filters on the response path.

However, there are some use cases where the response headers and body from the upstream server must be processed asynchronously before being sent downstream, such as by requesting a Vault server to verify the digest of the body, or by scanning the content on a dedicated server for confidentiality reasons.

How to interpret the upstream response asynchronously?

We should access the upstream in advance before the filter phases, that is the rewrite phase, the access stage, or the content phase.

There are two ways to access the upstream asynchronously:

lua-resty-http cosocket-based HTTP client, pure lua
ngx.location.capture() openresty API, based on nginx subrequest

Let’s evaluate them and choose a better approach.

conf/nginx.conf

worker_processes auto;
error_log logs/error.log info;
worker_rlimit_nofile 20480;

events {
    worker_connections 10620;
}

lua {}

http {
    lua_package_path "/usr/local/openresty/luajit/share/luajit-2.1.0-beta3/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;/usr/local/openresty/luajit/share/lua/5.1/?.lua;/usr/local/openresty/luajit/share/lua/5.1/?/init.lua;;";

    underscores_in_headers on;

    resolver 127.0.0.53 ipv6=on;

    keepalive_requests 10000;

    upstream fake_upstream {
        server 127.0.0.1:8080;

        keepalive 320;
        keepalive_requests 10000;
        keepalive_timeout 60s;
    }

    # fake upstream server
    server {
        listen 0.0.0.0:8080 default_server reuseport;
        listen [::]:8080 default_server reuseport;

        location /get {
            return 200 "hello\n";
        }
    }

    server {
        listen 0.0.0.0:80 default_server reuseport;
        listen [::]:80 default_server reuseport;

        # proxy_pass location, internal use only
        location /pass {
            internal;

            # remove the subrequest URL prefix
            rewrite ^/pass/(.*)$ /$1 break;

            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_pass http://fake_upstream;
        }

        location / {
            content_by_lua_block {
                -- async filter via subrequest
                if ngx.var.arg_subreq then
                    local res = ngx.location.capture("/pass" .. ngx.var.uri)
                    ngx.status = res.status
                    for k,v in pairs(res.header) do
                        ngx.header[k] = v
                    end

                    -- process body asynchronously here
                    local body = res.body
                    -- ...

                    ngx.print(body)
                    ngx.exit(ngx.HTTP_OK)

                -- async filter via lua-resty-http
                elseif ngx.var.arg_resty then
                    local httpc = require("resty.http").new()
                    local res, _ = httpc:request_uri("http://127.0.0.1:8080" .. ngx.var.uri)
                    for k,v in pairs(res.headers) do
                        ngx.header[k] = v
                    end
                    local body = res.body

                    -- process body asynchronously here
                    local body = res.body
                    -- ...

                    ngx.print(body)
                    ngx.exit(ngx.HTTP_OK)
                end

                -- fall-through
                ngx.exec("/pass" .. ngx.var.uri)
            }
        }
    }
}

The key point here is that we leverage the internal location (/pass) as a transfer point. When the request is processed at the match-all root location /, it sends a subrequest (ngx.location.capture("/pass" .. ngx.var.uri)) to upstream, /pass removes the prefix and uses the original URL (rewrite ^/pass/(.*)$ /$1 break;), which makes it strictly compatible with the nginx native way.

Note that here I’m using 127.0.0.1 as the only upstream IP, but you can use balancer_by_lua_block to enable dynamic IP set. So one /pass location is enough to access any upstream node, and you can even use DNS names like lua-resty-http, as long as you resolve the DNS names in any async phase (e.g. access phase) before the balancer phase.

Let’s test it:

mkdir -p /opt/subreq/{conf,logs}
cd /opt/subreq/
# create and edit conf/nginx.conf
nginx -p $PWD -c conf/nginx.conf -g 'daemon off; error_log /dev/stderr info;'

# check if it works
curl http://localhost/get -v
* processing: http://localhost/get
*   Trying [::1]:80...
* Connected to localhost (::1) port 80
> GET /get HTTP/1.1
> Host: localhost
> User-Agent: curl/8.3.0-DEV
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: openresty/1.25.1.1
< Date: Thu, 04 Jan 2024 08:56:04 GMT
< Content-Type: text/plain
< Content-Length: 6
< Connection: keep-alive
<
hello

curl http://localhost/get?subreq=1
hello

curl http://localhost/get?resty=1
hello

Performance test

I use k6 to test latency and throughput.

subreq.js

import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
  http.get(`${__ENV.URL}`);
  sleep(`${__ENV.SLEEP}`);
}

Test commands:

# fall-through, i.e. normal path to access upstream
k6 run -u 1 -d 30s -e 'URL=http://dev/get' -e 'SLEEP=0' -q subreq.js

# subrequest
k6 run -u 1 -d 30s -e 'URL=http://dev/get?subreq=1' -e 'SLEEP=0' -q subreq.js

# lua-resty-http
k6 run -u 1 -d 30s -e 'URL=http://dev/get?resty=1' -e 'SLEEP=0' -q subreq.js

Result

You can see that the method of subrequests is almost the same as the native method of nginx, whether it is latency, throughput or CPU.

However, lua-resty-http is much worse than the subrequest in terms of latency, throughput, and CPU. Mainly because it requires pure Lua code to handle HTTP stuff that can’t be fully compiled by the JIT.

Conclusion

lua-resty-http
- Pros:
  - No need for location configuration in nginx.conf
- Cons:
  - poor performance
  - Only supports HTTP/1.1
ngx.location.capture()
- Pros:
  - Supports all upstream protocols of nginx
  - Very good performance with almost zero overhead
- Cons:
  - Requires subtle location configuration in nginx.conf

If you need to use an asynchronous body filter written in Lua code, ngx.location.capture() is the best choice.

burl: A simple but flexible HTTP/3 testing framework based on bash and curl

luajit.io@gmail.com (jinhualuo) — Mon, 01 Jan 2024 00:00:00 +0000

TL;DR

I’ve created a new HTTP/3 testing framework based on bash and curl:

https://github.com/kingluo/burl

Chinese blog version:

https://zhuanlan.zhihu.com/p/675741855

Background

A few months ago, when I ported the QUIC patches from the nginx mainline to APISIX and tried to test it, I found that test::nginx didn’t work very well. It uses the wrong listen directive parameter “http3” instead of “quic” (probably due to version differences).

So I was wondering if I could design a simple testing framework because the latest curl already fully supports HTTP/3.

Many testing frameworks prefer DSLs (Domain Specific Languages) such as test::nginx and Hurl because it is more user-friendly. And it looks more data-driven. However, this is not good for developers looking to extend its functionality as they would need to learn the language behind it , take the time to understand the core and extension APIs by reading the code base of this testing framework. In other words, DSL and the language behind it create a knowledge gap for developers because it is not intuitive and cannot map the functions corresponding to the DSL to the language itself. Especially when the framework itself lacks capabilities and needs to be expanded, this gap will bring significant learning costs.

Take test::nignx as an example, which is implemented in Perl. When I need to verify the response body, decrypt it by pre-shared key, The built-in function doesn’t help as it only supports regular expression matching. So I need to write a bunch of perl code to do this. However, I’m not familiar with Perl, so this isn’t an easy task. As we all know, Perl is not a mainstream language and has a steep learning curve. At that time, I was thinking, how great it would be if I could write code in Python.

The expressive capabilities of test::nginx are limited, even though it already supports sending requests using curl, because it is not a scripting system (Of course, you can extend it using Perl modules, but due to the limitations of the framework design, this is a difficult task for most people).

It assumes your input is a text string, you cannot use other formats such as protobuf, and you cannot compress or encrypt text.
The only output post-processing supported is regular matching, so you cannot parse and verify other forms of content, such as decryption.
You cannot describe complex test procedures, such as sending three requests to trigger the request quota and verify the rate-limiting results.
You cannot make other types of requests such as GRPC, or SOAP.

In my practice, when I have such needs that exceed test::nginx’s capabilities, I have to delegate them to the lua part, which is very troublesome.

For example, when I need to parse and verify the correctness of the CAS Auth plugin, it involves a lot of client-side CAS protocol details, so you have to implement the entire logic in the lua part, which increases the difficulty of maintaining this mix. As we all know, neither Perl nor Lua are popular languages. ;-(

You can see the test case below involves additional lua lib development to test CAS. And, to accommodate test::nginx’s lack of response parsing, you have to do the conversion in the lua environment. Quite annoying, at least to me, because I’ve written a lot of test cases like this over the years.

=== TEST 2: login and logout ok
--- config
    location /t {
        content_by_lua_block {
            local http = require "resty.http"
            local httpc = http.new()
            -- Test-specific CAS lua lib
            local kc = require "lib.keycloak_cas"

            local path = "/uri"
            local uri = "http://127.0.0.1:" .. ngx.var.server_port
            local username = "test"
            local password = "test"

            local res, err, cas_cookie, keycloak_cookie = kc.login_keycloak(uri .. path, username, password)
            if err or res.headers['Location'] ~= path then
                ngx.log(ngx.ERR, err)
                ngx.exit(500)
            end
            res, err = httpc:request_uri(uri .. res.headers['Location'], {
                method = "GET",
                headers = {
                    ["Cookie"] = cas_cookie
                }
            })
            assert(res.status == 200)
            ngx.say(res.body)

            res, err = kc.logout_keycloak(uri .. "/logout", cas_cookie, keycloak_cookie)
            assert(res.status == 200)
        }
    }
--- response_body_like
uri: /uri
cookie: .*
host: 127.0.0.1:1984
user-agent: .*
x-real-ip: 127.0.0.1

Similarly, Hurl also uses a DSL to describe tests. Its backend language is Rust. So in terms of scalability, it seems to be worse than test::nginx because it needs to compile the code. However, Rust’s ecosystem is much better than Perl’s. So you can write code easily. But anyway, you need to write the code in a different language than the DSL.

Is there a simple testing framework that can satisfy the following requirements?

No DSL, just simple shell scripts
Easy to extend and programming language agnostic
Can be used to initialize and test any server, not limited to nginx, such as envoy

Bash scripting

Shell is the most important tool under the terminal in our daily work. Every administrator and developer needs to enter commands to complete their work. The most interesting thing is that the shell is also a programming platform, that is, scripts. When we talk about scripts, we care about two aspects: the power of the shell language itself and the available commands, which act like a standard library for a high-level programming language.

In the ancient times of computer history, the functions of the shell were very weak and the commands were very limited, so Perl was born. But now, things have changed. Many advanced shells have emerged, such as bash and zsh, which provide many excellent language elements for expressing complex logic. There are also a growing number of convenient and elegant commands, such as jq, which parses JSON with compact expressions. In Linux and Mac, the default shell is bash, so it’s right under our noses, so why not use it to complete our testing tasks? Why bother with complicated programming to get the job done, right?

Shell commands look very natural and straightforward. The command line is made up of words, just like human language, so why use a DSL? Furthermore, if a simple command line doesn’t meet your requirements, you can refactor it using variables and flow control statements, just like any mainstream programming language. Note that scripting is optional, i.e. you won’t encounter any complexity as a simple command line does everything, so everything works like a DSL, but better.

For example, if I need to send a GET and check for 404, two simple commands are enough:

# send request
REQ /anything/foobar

# validate the response headers
HEADER -x "HTTP/3 404"

But I need to check if Prometheus is logging 404 events as the correct number, how can I do this? Parses Prometheus metric messages and checks for counter changes in a shell function. Everything still looks simple, right? That’s the beauty of scripting.

count_404() {
    curl http://127.0.0.1:9091/apisix/prometheus/metrics 2>&1 | \
        grep -F 'apisix_http_status{code="404",route="1",matched_uri="/anything/*"' | \
        awk '{print $2}'
}

# send the first request
REQ /anything/foobar

# get current counter
cnt1=`count_404`

# send the second request
REQ /anything/foobar

cnt2=`count_404`

# check if the counter is increased
((cnt2 == cnt1 + 1))

You can use flow control statements to express any business logic.

For example, to test whether the request limiting plugin is working, you can use a loop.

# consume the quota
for ((i=0;i<2;i++)); do
    # send request
    REQ /httpbin/get -X GET --http3-only

    # validate the response headers
    HEADER -ix "HTTP/3 200"
done

# no quota
REQ /httpbin/get -X GET --http3-only
HEADER -x "HTTP/3 503"
HEADER -ix "x-ratelimit-remaining: 0"

burl: bash + curl

burl is a simple but flexible HTTP/3 testing framework based on bash and curl.

Design

The test file contains one or more test cases, and an optional initial part of the file header, such as configuring nginx.conf and starting nginx via template rendering.
Each test case consists of three parts:
1. Construct and send the request, and save the response header and response body to files for subsequent steps.
2. Verify the response headers, for example using “grep”.
3. Parse and validate the response body, for example with the “jq” expression.
Easily extensible, you can validate responses (steps ii and iii) using any command or other advanced script, such as Python.
Failure of any command will stop the testing process (enabled via the “set -euo pipefail” bash option).
The test process is echoed by default (enabled via the “set -x” bash option).

Synopsis

#!/usr/bin/env burl

# Optional initialization here...
# Before all test cases are executed.
# For example, render nginx.conf and start nginx.
SET NGX_CONF_HTTP <<EOF
upstream test_backend {
    server $(dig +short nghttp2.org):443;

    keepalive 320;
    keepalive_requests 1000;
    keepalive_timeout 60s;
}
EOF

SET NGX_CONF <<'EOF'
location / {
    add_header Alt-Svc 'h3=":443"; ma=86400';
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host "nghttp2.org";
    proxy_pass https://test_backend;
}
EOF

START_NGX

TEST 1: test case

# Send request
# REQ is a curl wrapper so you can apply any curl options to suit your needs.
# Check https://curl.se/docs/manpage.html for details.
REQ /httpbin/anything --http3 -d foo=bar -d hello=world

# Validate the response headers
# HEADER is a grep wrapper so you can apply any grep options and regular expressions to suit your needs.
HEADER -x "HTTP/3 200"

# Validate the response body, e.g. JSON body
# JQ is a jq wrapper so you can apply any jq options and jq expression to suit your needs.
JQ '.method=="POST"'
JQ '.form=={"foo":"bar","hello":"world"}'

TEST 2: another test case
# ...

# More test cases...

Examples

APISIX

Test MTLS whitelist

TEST 2: route-level mtls, skip mtls

ADMIN put /ssls/1 -d '{
    "cert": "'"$(<${BURL_ROOT}/examples/certs/server.crt)"'",
    "key": "'"$(<${BURL_ROOT}/examples/certs/server.key)"'",
    "snis": [
        "localhost"
    ],
    "client": {
        "ca": "'"$(<${BURL_ROOT}/examples/certs/ca.crt)"'",
        "depth": 10,
        "skip_mtls_uri_regex": [
            "/httpbin/get"
        ]
    }
}'

sleep 1

REQ /httpbin/get --http3-only

# validate the response headers
HEADER -x "HTTP/3 200"

# validate the response body, e.g. JSON body
JQ '.headers["X-Forwarded-Host"] == "localhost"'

Test HTTP/3 Alt-Svc

ADMIN put /ssls/1 -d '{
    "cert": "'"$(<${BURL_ROOT}/examples/certs/server.crt)"'",
    "key": "'"$(<${BURL_ROOT}/examples/certs/server.key)"'",
    "snis": [
        "localhost"
    ]
}'

ADMIN put /routes/1 -s -d '{
    "uri": "/httpbin/*",
    "upstream": {
        "scheme": "https",
        "type": "roundrobin",
        "nodes": {
            "nghttp2.org": 1
        }
    }
}'



TEST 1: check if alt-svc works

altsvc_cache=$(mktemp)
GC "rm -f ${altsvc_cache}"

REQ /httpbin/get -k --alt-svc ${altsvc_cache}
HEADER -x "HTTP/1.1 200 OK"

REQ /httpbin/get -k --alt-svc ${altsvc_cache}
HEADER -x "HTTP/3 200"

SOAP

Send a SOAP request to the web service and verify the response.

Construct JSON input using jo and validate JSON output using jq expressions.

TEST 1: test a simple Web Service: Add two numbers: 1+2==3

SOAP_REQ \
    'https://ecs.syr.edu/faculty/fawcett/Handouts/cse775/code/calcWebService/Calc.asmx?WSDL' \
    Add `jo a=1 b=2` '.==3'

XML

TEST 2: GET XML

# send request
REQ /httpbin/xml

# validate the response headers
HEADER -x "HTTP/1.1 200 OK"
HEADER -x "Content-Type: application/xml"

# validate the response XML body
XML '.slideshow["@author"]=="Yours Truly"'

Conclusion

I think the HTTP testing framework is not difficult and does not require the use of high-level programming languages to implement. DSL is effectively a meaningless gap between framework implementation and representation because we need to incur translation costs, especially for compiled programming languages, and For testers, learning it is a burden. So why not unify them and script them? With the help of bash (the standard shell used daily in almost all operating systems) and curl, we can handle all our testing efforts easily.

Welcome to discuss and contribute to burl:

https://github.com/kingluo/burl

Make lua-resty-ffi run on Envoy

luajit.io@gmail.com (jinhualuo) — Sun, 09 Jul 2023 00:00:00 +0000

Background

Both nginx and envoy are high-performance proxy server software.

They are written in C/C++ so they run fast. But the C/C++ ecosystem is notoriously weak. Can we do hybrid programming with Go, Java, Python, Rust or NodeJS? so we can use Those rich ecosystems of popular mainstream programming languages?

The envoy officially supports Lua filter, so could we port the lua-resty-ffi to the envoy? Then everything is ok.

Solution

Envoy has a filter mechanism that allows us full control over the request and response flow. And filters allow asynchronous processing, and Lua filters support asynchronous processing based on coroutines.

Envoy is a multi-threaded model where each worker thread accepts requests individually.

The calling direction of lua-resty-ffi has nothing to do with the platform, it uses pthread mutex lock queue and pthread condition notification.
The callback direction of lua-resty-ffi has to be adapted to the envoy callback mechanism, which uses eventfd (via libevent) and a locked queue.

Unlike nginx worker processes, each source block defined in the filter configuration has its own lua vm instance. So lua-resty-ffi may have different multiple instances in each lua vm instance.

Call

We just need to change the post function in resty_ffi.lua.

In the evnoy lua filter, we get a wrapper handle from the entry function, such as envoy_on_request, It stands for the filter in lua realm. But the wrapper is userdata type, and the address of the corresponding C++ object is aligned, so we need to implement a method Returns the C++ address so we can call the C++ method in the C API.

`1`	`local r = wrapper and wrapper:this() or get_request()`

We also need a yield method (because luajit ffi cannot yield):

1
2
3

if wrapper then
    return wrapper:ffiYield()
end

ffiYield() sets the status to FFICall, which in turn suspends this request (Http::FilterHeadersStatus::StopIteration) to wait for ffi to resume:

int StreamHandleWrapper::ffiYield(lua_State* state) {
  state_ = State::FFICall;
  return lua_yield(state, 0);
}

...

Http::FilterHeadersStatus StreamHandleWrapper::start(int function_ref) {
  // We are on the top of the stack.
  coroutine_.start(function_ref, 1, yield_callback_);
  Http::FilterHeadersStatus status = (state_ == State::WaitForBody || state_ == State::FFICall ||
                                      state_ == State::HttpCall || state_ == State::Responded)
                                         ? Http::FilterHeadersStatus::StopIteration
                                         : Http::FilterHeadersStatus::Continue;

  if (status == Http::FilterHeadersStatus::Continue) {
    headers_continued_ = true;
  }

  return status;
}

Callback

The most notable change is the callback path, we need to use Dispatcher::post, then The callback will be invoked in a worker thread and call callbacks_.continueIteration().

void ngx_http_lua_ffi_respond(void* tsk, int rc, char* rsp, int rsp_len) {
  ngx_thread_task_t* task = static_cast<ngx_thread_task_t*>(tsk);
  ffi_task_ctx_t* ctx = static_cast<ffi_task_ctx_t*>(task->ctx);
  pthread_mutex_lock(&ctx->mtx);
  if (!ctx->is_abort) {
    ctx->rc = rc;
    ctx->rsp = rsp;
    ctx->rsp_len = rsp_len;
    auto wrapper = static_cast<StreamHandleWrapper*>(ctx->wrapper);
    wrapper->dispatcher().post([ctx, wrapper] {
      if (!ctx->is_abort) {
        wrapper->onFFICallback(ctx);
      }
    });
  } else {
    ngx_http_lua_ffi_task_free(ctx);
    if (rsp) {
      free(rsp);
    }
  }
  pthread_mutex_unlock(&ctx->mtx);
}

void StreamHandleWrapper::onFFICallback(ffi_task_ctx_t* ffi_ctx) {
  ASSERT(state_ == State::FFICall);
  ...

  try {
    resumeCoroutine(3, yield_callback_);
    markDead();
  } catch (const Filters::Common::Lua::LuaException& e) {
    filter_.scriptError(e);
  }

  if (state_ == State::Running) {
    headers_continued_ = true;
    callbacks_.continueIteration();
  }
}

Demo

envoy.yaml

  http_filters:
  - name: lua_filter_with_custom_name_0
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
      default_source_code:
        inline_string: |
          function envoy_on_request(request_handle)
            require("resty_ffi")
            local demo = ngx.load_ffi("ffi_go_echo")
            local ok, res = demo:foobar("foobar", request_handle)
            assert(ok)
            assert(res == "foobar")
            local demo = ngx.load_ffi("resty_ffi_python", "ffi.echo?,init", {is_global = true})
            local ok, res = demo:echo("hello", request_handle)
            assert(ok)
            assert(res == "hello")
          end

Test:

# run demo
cd /opt/envoy/examples/lua
PYTHONPATH=/opt/lua-resty-ffi/examples/python/ \
LD_LIBRARY_PATH=/opt/lua-resty-ffi/examples/python \
LUA_PATH='/opt/lua-resty-ffi/?.lua;;' \
envoy -c envoy.yaml --concurrency 1

curl -v localhost:8000/httpbin/get

Output:

* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET /httpbin/get HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< date: Sun, 09 Jul 2023 06:05:36 GMT
< content-type: application/json
< content-length: 248
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-backend-header-rtt: 0.004774
< alt-svc: h3=":443"; ma=3600, h3-29=":443"; ma=3600
< server: envoy
< via: 1.1 nghttpx
< x-frame-options: SAMEORIGIN
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< x-envoy-upstream-service-time: 146
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "localhost:8000",
    "User-Agent": "curl/7.68.0",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  },
  "origin": "192.168.0.1",
  "url": "http://localhost:8000/httpbin/get"
}
* Connection #0 to host localhost left intact

Conclusion

With lua-resty-ffi, you could use your favorite mainstream programming language, e.g. Go, Java, Python, Rust, or Node.js, to do development in Nginx and Envoy, so that you could enjoy their rich ecosystem directly.

Welcome to discuss and like my github repo:

https://github.com/kingluo/lua-resty-ffi

OpenResty: record the Lua execution time per request in ngx.ctx via LD_PRELOAD

luajit.io@gmail.com (jinhualuo) — Fri, 21 Apr 2023 00:00:00 +0000

TL;DR

How can I log the Lua execution time of each request (excluding time spent on asynchronous IO) in ngx.ctx in a non-intrusive, zero-cost way?

The Lua execution time is the sum of the following metrics:

lua_resume
- phases
  - REWRITE
  - ACCESS
  - CONTENT
  - BALANCER
lua_pcall
- filters
  - header_filter
  - body_filter

https://github.com/kingluo/ngx-lua-request-time

Issue to measure request time

Nginx has two variables to record request time:

$request_time

request processing time in seconds with a milliseconds resolution (1.3.9, 1.2.6); time elapsed since the first bytes were read from the client

$upstream_response_time

keeps time spent on receiving the response from the upstream server; the time is kept in seconds with millisecond resolution. Times of several responses are separated by commas and colons like addresses in the $upstream_addr variable.

Nginx is a multi-processes and single-threading model. The main thread of each worker process must be fully async-processing, otherwise, the performance is bad.

In OpenResty, our business logic is built mainly in Lua code. If the code makes full use of async API, e.g. cosocket, run_worker_thread, then it meets the requirements of high-performance processing. However, in reality, the code may be huge and complex, and sometimes the downstream/upstream network goes wrong, and sometimes the downstream client and the upstream server may suck in the socket r/w.

So how to distinguish how much time we spend on Lua code?

How much time to handle the network r/w? And whether the lua code is blocking or not?

Could we use $request_time - $upstream_response_time? Unfortunately, the answer is no!

Let’s first take a look at Nginx’s request life cycle:

The most common misunderstanding of nginx is that you thought nginx would read the whole response body from upstream before it sends the body to the downstream, as well as the request body, so the $upstream_response_time is the pure upstream traffic time-cost. But actually, it depends on the parameter configuration. By default, the traffic pipeline between the upstream and downstream of the request and response body is done dynamically and incrementally.

Moreover, nginx will not read the request body from the downstream before all following phases, e.g. rewrite phase. Instead, when the headers are read, it triggers the phase handlers to handle this request, and after the balancer phase chooses an upstream node, it starts to read the request body and send it to the upstream gradually.

Also note that after reading the request headers, Nginx stops reading the request body until the content phase completes or is explicitly triggered in the lua code.

So let’s clarify the time-cost relationship again:

proxy_request_buffering	proxy_buffering	`$upstream_response_time`	`$request_time - $upstream_response_time`
on	on	upstream_time_cost	downstream_time_cost + lua_time_cost
off	on	upstream_time_cost + downstream_request_body_time_cost	downstream_response_body_time_cost + lua_time_cost
on	off	upstream_time_cost + downstream_response_body_time_cost	downstream_request_body_time_cost + lua_time_cost
off	off	upstream_time_cost + downstream_time_cost	lua_time_cost

You can see that the definition of $upstream_response_time changes depending on the proxy_request_buffering and proxy_buffering settings.

So, the motivation is, how to measure Lua time cost without being affected by directives settings?

Important notes:

Interestingly, you may find that request_time is smaller than upstream_response_time! how come?

Here is an example:

127.0.0.1 - - [15/Apr/2023:20:05:32 +0800] 127.0.0.1:9080 "GET /get HTTP/1.1" 200 463 0.001 "-" "curl/7.68.0" 172.30.74.11:80 200 0.002 "http://127.0.0.1:9080"

$request_time is 0.001, while $upstream_response_time is 0.002!

This is caused by the two metrics using different timing sources.

$upstream_response_time calculated by clock_gettime(CLOCK_MONOTONIC_COARSE), and by default it can be in past for 4 milliseconds, on opposite, $request_time calculated by gettimeofday(). So eventually upstream_response_time might be larger, than response_time.

$request_time is generated in real-time, so different values may be obtained at different times.
The access log is written after the Lua log phase handler is completed, so there should be no blocking tasks in the Lua code, otherwise the $request_time in the access log will be larger than the actual value.

proxy_buffering on (by default)

In most use cases, the upstream is in the local network, so the network rate is faster than the downstream. Then we need to use buffering as much as possible so that the upstream could recycle the resource to handle the next request.

Nginx will try to read the response body from the upstream as much as possible into the memory buffers or even disk files, in parallel to writing to the downstream by draining the buffers.

If the downstream is too slow to sync with the upstream, then nginx will close the upstream first. In this case, $upstream_response_time is the time-cost of the upstream traffic and partial downstream traffic.

We could set up a demo to visualize this process.

mock_get.go

Send a GET, but do not receive the response.

package main

import (
	"net"
)

func main() {
	conn, err := net.Dial("tcp", "localhost:9080")
	if err != nil {
		panic(err)
	}

	request := "GET /anything/a HTTP/1.1\r\nHost: httpbin.org\r\n\r\n"

	_, err = conn.Write([]byte(request))
	if err != nil {
		panic(err)
	}

	ch := make(chan struct{})
	<-ch
}

mock_get_big_respnose.go

Send a big response (4MB body, ensure it is over the sum of nginx buffers and kernel sockbuf).

package main

import (
	"fmt"
	"net"
	"os"
)

const (
	CONN_HOST = "localhost"
	CONN_PORT = "3333"
	CONN_TYPE = "tcp"
)

func main() {
	l, err := net.Listen(CONN_TYPE, CONN_HOST+":"+CONN_PORT)
	if err != nil {
		fmt.Println("Error listening:", err.Error())
		os.Exit(1)
	}
	defer l.Close()
	fmt.Println("Listening on " + CONN_HOST + ":" + CONN_PORT)
	for {
		conn, err := l.Accept()
		if err != nil {
			fmt.Println("Error accepting: ", err.Error())
			os.Exit(1)
		}
		go handleRequest(conn)
	}
}

// Handles incoming request
func handleRequest(conn net.Conn) {
	buf := make([]byte, 1024*1024*4)
	_, err := conn.Read(buf)
	if err != nil {
		fmt.Println("Error reading:", err.Error())
	}
	for i := range buf {
		buf[i] = 'A'
	}
	headers := "HTTP/1.1 200 OK\r\nContent-Length: 4194304\r\n\r\n"
	conn.Write([]byte(headers))
	conn.Write(buf)
	conn.Close()
}

nginx.conf

daemon off;
error_log /dev/stderr info;

events {}

http {
    upstream backend {
        server 127.0.0.1:3333;
        keepalive 320;
        keepalive_requests 1000;
        keepalive_timeout 60s;
    }

    log_format main escape=default '$remote_addr - $remote_user [$time_local] $http_host "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent" $upstream_addr $upstream_status $upstream_response_time';

    access_log logs/access.log main;

    send_timeout 10s;

    server {
        listen 20000;
        location / {
            proxy_pass http://backend;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }
    }
}

# in first terminal
go run mock_get_big_respnose.go

# in second terminal
nginx -p $PWD -c nginx.conf

# in third terminal
go run mock_get.go

Watch logs/access.log:

127.0.0.1 - - [20/Apr/2023:18:39:13 +0800] httpbin.org "GET /anything/a HTTP/1.1" 200 2555860 10.016 "-" "-" 127.0.0.1:3333 200 0.015

You can see that $upstream_response_time is 0.015 secs, and the $request_time is 10.016 secs.

TCP flow between the upstream and nginx:

$upstream_response_time is the time-cost from step1 to step2 only.

Check the nginx source code:

static void
ngx_http_upstream_process_request(ngx_http_request_t *r,
    ngx_http_upstream_t *u)
{
...
        //The upstream body reading is finished.
        //Close the upstream request
        if (p->upstream_done || p->upstream_eof || p->upstream_error) {
            ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                           "http upstream exit: %p", p->out);

            if (p->upstream_done
                || (p->upstream_eof && p->length == -1))
            {
                ngx_http_upstream_finalize_request(r, u, 0);
                return;
            }
...
        }
...
}

static void
ngx_http_upstream_finalize_request(ngx_http_request_t *r,
    ngx_http_upstream_t *u, ngx_int_t rc)
{
...
    //Close the associated downstream request
    ngx_http_finalize_request(r, rc);
}

void
ngx_http_finalize_request(ngx_http_request_t *r, ngx_int_t rc)
{
...
    //If the downstream writing is still blocking,
    //set up a postponed handler to continue writing.
    if (r->buffered || c->buffered || r->postponed) {

        if (ngx_http_set_write_handler(r) != NGX_OK) {
            ngx_http_terminate_request(r, 0);
        }

        return;
    }
...
}

proxy_buffering off

Modify nginx.conf:

...
        location / {
            proxy_pass http://backend;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_buffering off;
        }
...

Watch logs/access.log:

127.0.0.1 - - [20/Apr/2023:18:48:46 +0800] httpbin.org "GET /anything/a HTTP/1.1" 200 4157440 10.022 "-" "-" 127.0.0.1:3333 200 10.021

You can see that $upstream_response_time is 10.021 secs, and the $request_time is 10.022 secs, which are almost the same. Now $upstream_response_time fully covers the downstream traffic, i.e. including the step3.

Solution

We need a metric to calculate the execution time purely on lua code, excluding other factors. So that we could ensure our business logic is fully async.

We have a few solutions:

systemtap
- heavyweight, you need to install debug symbols, sometimes it’s not feasible, e.g. lack of symbols for custom kernel
- Non-negligible overhead, at least 5% performance impact
- not container-friendly
- could not write the userspace, i.e. could not set the lua variable
record and calculate in Lua code manually
- it’s hard and tricky to handle lua errors and coroutine yield
- code pollution

Could we have a non-intrusive and zero-cost way to record the Lua execution time?

Maybe LD_PRELOAD is a better choice.

https://github.com/kingluo/ngx-lua-request-time

demo

Use the sleep function from libc to do blocking sleep for 1 second, and log the ngx.ctx.openresty_request_time_us to confirm it’s blocked by 1 second.

nginx.conf

server {
    error_log /dev/stderr info;

    listen 20000;

    location / {
        access_by_lua_block {
            local ffi = require("ffi")
            ffi.cdef[[unsigned int sleep(unsigned int seconds);]]
            ffi.C.sleep(1);
        }
        proxy_pass https://httpbin.org;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }

    log_by_lua_block {
        ngx.log(ngx.WARN, "ngx.ctx.openresty_request_time_us: ", ngx.ctx.openresty_request_time_us)
    }
}

Test:

NGX_LUA_REQUEST_TIME_VAR_NAME=openresty_request_time_us \
LD_PRELOAD=/opt/ngx-lua-request-time/ngx_lua_request_time.so \
openresty -p $PWD -c nginx.conf

curl http://localhost:20000/get

Output:

2023/04/16 09:59:43 [warn] 2868937#2868937: *3 [lua] log_by_lua(server.conf:17):2: ngx.ctx.openresty_request_time_us: 1000120 while logging request, client: 127.0.0.1, server: , request: “GET /get HTTP/1.1”, upstream: “https://34.193.132.77:443/get”, host: “localhost:20000”

The LD_PRELOAD could be also enabled globally via /etc/ld.so.preload. So it’s also non-intrusive for docker usage.

ld.so.preload

/opt/ngx-lua-request-time/ngx_lua_request_time.so

docker run --rm --name openresty \
-v $PWD/server.conf:/opt/bitnami/openresty/nginx/conf/server_blocks/my_server_block.conf:ro \
-v $PWD/../../ld.so.preload:/etc/ld.so.preload:ro \
-v $PWD/../../ngx_lua_request_time.so:/opt/ngx-lua-request-time/ngx_lua_request_time.so:ro \
-e NGX_LUA_REQUEST_TIME_VAR_NAME=openresty_request_time_us \
-p 20000:20000 \
bitnami/openresty:latest

design

It’s worth noting to use LD_PRELOAD, it needs to satisfy below conditions:

The hooked C symbol is public, and the dependent does not link it via offset, i.e. the other symbols in the same elf file use offset to locate this target symbol, although it’s public.
the function input arguments and return value contain enough information to do your analysis
it’s possible to filter unneeded contexts because some low-level function is used by many high-level logics, and some of them are noise

OpenResty enables to inject handlers in different phases of the request life cycle. All async phases (that is, such phase runs your lua code in a coroutine, which could yield for async IO and resume later) use lua_resume() function, and all recursive coroutine resumes are flattened by OpenResty, i.e. when you create a new coroutine to access the LDAP server via cosocket from the async phase handler, it will exit the entry coroutine (the handler itself) and resumes the extra coroutine there. So, calculating the duration in lua_resume() will not overlap with each other.

Note that it excludes the time spent on async IO! It only contains the pure Lua code execution time.

int lua_resume(lua_State *L, int narg) {
    if (!is_enabled()) {
        return lua_resume_f(L, narg);
    }

    ngx_http_request_t *r = ngx_http_lua_get_request_f(L);
    if (!r) {
        return lua_resume_f(L, narg);
    }

    char* err;
    int phase = ngx_http_lua_ffi_get_phase_f(r, &err);
    if (phase == NGX_ERROR ||
		(phase != PHASE_SET &&
			phase != PHASE_REWRITE &&
			phase != PHASE_ACCESS &&
			phase != PHASE_CONTENT)) {
        return lua_resume_f(L, narg);
    }

    timespec tv1;
    clock_gettime(CLOCK_MONOTONIC, &tv1);

    int ret = lua_resume_f(L, narg);

    timespec tv2;
    clock_gettime(CLOCK_MONOTONIC, &tv2);

    record_time(&tv1, &tv2, L);

    return ret;
}

All sync phases, e.g. balancer phase, and header/body filter phase use lua_pcall(), but this is a very low-level function, so to minimize the overhead, you need a cheap method to filter the noise.

int lua_pcall(lua_State *L, int nargs, int nresults, int errfunc) {
    if (!is_enabled()) {
        return lua_pcall_f(L, nargs, nresults, errfunc);
    }

    // all phases uses ngx_http_lua_traceback as the pcall error handler,
    // so that's a great point used to filter out the other `lua_pcall` invocations.
    // And, this checking is cheap!
    if (lua_tocfunction_f(L, 1) != ngx_http_lua_traceback_f) {
        return lua_pcall_f(L, nargs, nresults, errfunc);
    }
    ...
}

After testing, the CPU overhead is small enough to be ignored.

HTTP/3 client library for OpenResty

luajit.io@gmail.com (jinhualuo) — Wed, 05 Apr 2023 10:50:00 +0000

Background

HTTP/3 is the third major version of the Hypertext Transfer Protocol used to exchange information on the World Wide Web, complementing the widely-deployed HTTP/1.1 and HTTP/2.

Both HTTP/1.1 and HTTP/2 use TCP as their transport. HTTP/3 uses QUIC, a transport layer network protocol which uses user space congestion control over the User Datagram Protocol (UDP). The switch to QUIC aims to fix a major problem of HTTP/2 called “head-of-line blocking”: because the parallel nature of HTTP/2’s multiplexing is not visible to TCP’s loss recovery mechanisms, a lost or reordered packet causes all active transactions to experience a stall regardless of whether that transaction was impacted by the lost packet. Because QUIC provides native multiplexing, lost packets only impact the streams where data has been lost.

How to send HTTP/2 and HTTP/3 request from OpenResty/Nginx?

In fact, it’s difficult to support HTTP/2 and HTTP/3, because cosocket has below limitations:

tcpsock:sslhandshake doesn’t support the ALPN.
cosocket instance (connection) is bound to a specific HTTP downstream request, i.e. connection could not be shared globally, which disables HTTP2/HTTP3 multiplexing and concurrency then.

req is a golang library: Simple Go HTTP client with Black Magic.

Highlights:

unique interface for all HTTP versions
auto HTTP version selection
easy download and upload

Why not encapsulate it so that we could reuse it in openresty?

lua-resty-ffi provides an efficient and generic API to do hybrid programming in openresty with mainstream languages (Go, Python, Java, Rust, Nodejs).

lua-resty-ffi-req = lua-resty-ffi + req

Check this repo for source code:

https://github.com/kingluo/lua-resty-ffi-req

Demo

simple get

local client, err = req:new_client()
local ok, res = client:request{
    url = "https://httpbin.org/anything?foo=bar",
    body = "hello",
    args = {
        foo1 = "foo1",
        foo2 = 2,
        foo3 = false,
        foo4 = 2.2,
    },
}
assert(ok)
ngx.say(inspect(res))
ngx.say(inspect(cjson.decode(res.body)))

Output:

-- response
{
  body = '...',
  headers = {
    ["Access-Control-Allow-Credentials"] = { "true" },
    ["Access-Control-Allow-Origin"] = { "*" },
    ["Content-Length"] = { "609" },
    ["Content-Type"] = { "application/json" },
    Date = { "Wed, 05 Apr 2023 09:45:57 GMT" },
    Server = { "gunicorn/19.9.0" }
  },
  proto_major = 2,
  proto_minor = 0,
  status = 200,
  tls_version = 771
}
-- body
{
  args = {
    foo = "bar",
    foo1 = "foo1",
    foo2 = "2",
    foo3 = "false",
    foo4 = "2.2"
  },
  data = "hello",
  files = {},
  form = {},
  headers = {
    ["Accept-Encoding"] = "gzip",
    ["Content-Length"] = "5",
    ["Content-Type"] = "text/plain; charset=utf-8",
    Host = "httpbin.org",
    ["User-Agent"] = "req/v3 (https://github.com/imroc/req)",
    ["X-Amzn-Trace-Id"] = "Root=1-642d4355-2c0076244bb7d61c385efbb4"
  },
  json = <userdata 1>,
  method = "GET",
  origin = "34.193.132.77",
  url = "https://httpbin.org/anything?foo=bar&foo1=foo1&foo2=2&foo3=false&foo4=2.2"
}

request body writer and response body reader

local client, err = req:new_client()
local ok, res = client:request{
    method = req.HTTP_POST,
    url = "https://httpbin.org/anything",
    body = coroutine.wrap(function()
        coroutine.yield("hello")
    end),
    body_reader = true,
}
assert(ok)
ngx.say(inspect(res))

local body = {}
for chunk in res.body_reader do
    table.insert(body, chunk)
end
ngx.say(inspect(cjson.decode(table.concat(body, ""))))

HTTP3

Install Nginx HTTP3 version:

https://www.nginx.com/blog/binary-packages-for-preview-nginx-quic-http3-implementation/

# nginx.conf
http {
    # for better compatibility it's recommended
    # to use the same port for quic and https
    listen 8443 http3 reuseport;
    listen 8443 http2 ssl reuseport;
    ssl_certificate     /opt/certs/ca.crt;
    ssl_certificate_key /opt/certs/ca.key;
    ssl_protocols       TLSv1.3;

    location / {
        if ($arg_http3 = true) {
            add_header Alt-Svc 'h3=":8443"; ma=86400';
        }
        #proxy_pass http://httpbin_backend;
        #proxy_http_version 1.1;
        #proxy_set_header Connection "";
        return 200 ok;
    }
}

Test Alt-Svc:

local ok, res = client:request({url="https://foo.bar:8443/ip?http3=true"})
ngx.say(inspect(res))
ngx.sleep(0.1)
local ok, res = client:request({url="https://foo.bar:8443/ip?http3=true"})
ngx.say(inspect(res))

Output:

-- first response
-- uses HTTP2
{
  body = "ok",
  headers = {
    ["Alt-Svc"] = { 'h3=":8443"; ma=86400' },
    ["Content-Length"] = { "2" },
    ["Content-Type"] = { "application/octet-stream" },
    Date = { "Wed, 05 Apr 2023 09:51:35 GMT" },
    Server = { "nginx/1.23.4" }
  },
  proto_major = 2,
  proto_minor = 0,
  status = 200,
  tls_version = 772
}
-- second response
-- switches to HTTP3
{
  body = "ok",
  headers = {
    ["Alt-Svc"] = { 'h3=":8443"; ma=86400' },
    ["Content-Length"] = { "2" },
    ["Content-Type"] = { "application/octet-stream" },
    Date = { "Wed, 05 Apr 2023 09:51:35 GMT" },
    Server = { "nginx/1.23.4" }
  },
  proto_major = 3,
  proto_minor = 0,
  status = 200,
  tls_version = 772
}

Auto HTTP version selection

req uses TLS ALPN and Alt-Svc header to do auto HTTP version selection.

HTTP1 to H2C (HTTP2 Clear Text)

This method is not supported yet.

HTTP2 ALPN

HTTP3 Alt-Svc

Redirect to the same site:

Redirect to a different site:

Benchmark

Due to golang http client implementation, lua-resty-ffi-req has only 70% of the performance of lua-resty-http, and it consumes 30% more cpu than lua-resty-http.

The performance is not that good yet, but there is room for improvement.

It is still worth sacrificing some performance for the ecosystem:

No need to reinvent the wheel, especially when the wheel is big
Cover maximum functionalities
No need to bugfix yourself, because there is many contributors for popular open source project

FYI, here is the flamegraph of lua-resty-ffi-req (open in new tab page to browse the flamegraph svg):

Conclusion

With lua-resty-ffi, you could use your favorite mainstream programming language, e.g. Go, Java, Python, Rust, or Node.js, to do development in Openresty/Nginx, so that you could enjoy their rich ecosystem directly.

Welcome to discuss and like my github repo:

https://github.com/kingluo/lua-resty-ffi

Access Windows Active Directory Domain Services with Kerberos Authentication from OpenResty

luajit.io@gmail.com (jinhualuo) — Sat, 11 Mar 2023 00:39:21 +0000

Background

Kerberos is a computer network security protocol that authenticates service requests between two or more trusted hosts across an untrusted network, like the internet. It uses secret-key cryptography and a trusted third party for authenticating client-server applications and verifying users’ identities.

Since Windows 2000, Microsoft has used the Kerberos protocol as the default authentication method in Windows, and it is an integral part of the Windows Active Directory (AD) service.

LDAP is commonly used to do authentication and authorization.

Windows ADDS uses LDAP protocol, but normally the enterprise ADDS will use Kerberos as authentication method, instead of simple bind.

How to access ADDS/Kerberos from OpenResty/Nginx?

As known, OpenResty does not have a fully functional LDAP library.

Let’s have a look at the current alternatives:

lualdap
- No SASL auth, simple bind only
- Not based on cosocket, i.e. not async
lua-resty-ldap
- No SASL auth, simple bind only

What about other programming lanuages?

go-ldap
- No GSSAPI/Kerberos support
rust-ldap3
- depends on kinit to get service ticket first
- no keytab, no ad-hoc credential support
python-ldap
- depends on kinit to get service ticket first
- no keytab, no ad-hoc credential support
- not async

After investigation, I think bonsai is the best choice, which is a popular and active python ldap client library.

Highlights:

asyncio support
Full SASL support
- DIGEST-MD5 and NTLM
- GSSAPI and GSS-SPNEGO (keytab, ad-hoc credential support)
- EXTERNAL
simple pythonic design
based on robust and time-tested C libraries, e.g. libldap2, libsasl2, libkrb5

Why not encapsulate it so that we could reuse it in openresty?

lua-resty-ffi provides an efficient and generic API to do hybrid programming in openresty with mainstream languages (Go, Python, Java, Rust, Nodejs).

lua-resty-ffi-ldap = lua-resty-ffi + bonsai

Check this repo for source code:

https://github.com/kingluo/lua-resty-ffi-ldap

Demo

Set up a minimal ADDS

Let’s install a Windows ADDS on Window Server 2022 in VM.

From the server manager, add ADDS feature:

Promote this server to a domain controller:

Add a user chuck:

Add a group foobar:

Attach user chuck to group foobar:

Done!

Use lua-resty-ffi-ldap

Add Windows IP into /etc/hosts:

# Hostname is windows
# Domain is bonsai.test
# IP is local network IP 172.30.90.18
172.30.90.18       windows.bonsai.test bonsai.test

Edit krb5.conf:

[libdefaults]
    default_realm = BONSAI.TEST
    ccache_type = 3
    rdns = false

[realms]
    BONSAI.TEST = {
        kdc = windows.bonsai.test:88
        admin_server = windows.bonsai.test:749
    }

[domain_realm]
    .bonsai.test = BONSAI.TEST
    bonsai.test = BONSAI.TEST

demo.lua

local ldap = require("resty.ffi.ldap")

return function()
    local client = ldap.new({
        url = "ldap://bonsai.test",
        maxconn = 2,
        auth = {
            mechanism="GSSAPI",
            user="chuck",
            password="Foo2023@",
            realm="BONSAI.TEST",
        }
    })
    assert(client)

    local ok, res = client:search({
        base = "cn=chuck,dc=bonsai,dc=test",
        scope = ldap.SCOPE_SUB,
        filter_exp = "(objectclass=user)",
        attrlist = {'memberOf', 'sAMAccountName'},
    })
    assert(ok)
    res = res[1]
    assert(res.dn == "CN=chuck,DC=bonsai,DC=test", "dn mismatch")
    assert(res.memberOf[1] == "CN=foobar,DC=bonsai,DC=test", "memberOf mismatch")
    assert(res.sAMAccountName[1] == "chuck", "sAMAccountName mismatch")

    local ok = client:close()
    assert(ok)

    ngx.say("ok")
end

nginx.conf

daemon off;
error_log /dev/stderr info;
worker_processes auto;
env LD_LIBRARY_PATH;
env PYTHONPATH;
env KRB5_CONFIG;

events {}

http {
    lua_package_path '/opt/lua-resty-ffi-ldap/demo/?.lua;/opt/lua-resty-ffi-ldap/lua/?.lua;/opt/lua-resty-ffi-ldap/lua/?/init.lua;;';

    server {
        listen 20000;

        location /demo {
            content_by_lua_block {
                require("demo")()
            }
        }
    }
}

Test:

# install lua-resty-ffi
# https://github.com/kingluo/lua-resty-ffi#install-lua-resty-ffi-via-luarocks
# set `OR_SRC` to your openresty source path
luarocks config variables.OR_SRC /tmp/tmp.Z2UhJbO1Si/openresty-1.21.4.1
luarocks install lua-resty-ffi

# make lua-resty-ffi python loader library
apt install python3-dev python3-pip libffi-dev
cd /opt
git clone https://github.com/kingluo/lua-resty-ffi
cd /opt/lua-resty-ffi/examples/python
make

apt install libldap2-dev libsasl2-dev heimdal-dev

pip3 install bonsai

cd /opt
git clone https://github.com/kingluo/lua-resty-ffi-ldap

cd /opt/lua-resty-ffi-ldap/demo

# run nginx
KRB5_CONFIG="$PWD/krb5.conf" \
LD_LIBRARY_PATH=/opt/lua-resty-ffi/examples/python:/usr/local/lib/lua/5.1 \
PYTHONPATH=/opt/lua-resty-ffi-ldap \
nginx -p $PWD -c nginx.conf

# in another terminal, trigger demo
curl localhost:20000/demo

Search result in JSON:

[
  {
    "dn":"CN=chuck,DC=bonsai,DC=test",
    "memberOf":[
      "CN=foobar,DC=bonsai,DC=test"
    ],
    "sAMAccountName":[
      "chuck"
    ]
  }
]

Conclusion

Welcome to discuss and like my github repo:

https://github.com/kingluo/lua-resty-ffi

Resolve C static variables of the main executable from the dynamic loaded shared library

luajit.io@gmail.com (jinhualuo) — Sun, 05 Feb 2023 10:51:01 +0000

As known, nginx modules are static linking.

Of course, you could also use dynamic modules, but they are limited to use dynamic symbols from nginx executable.

What if I need to use static variables and functions from nginx executable in my code?

In lua-resty-ffi, for the sake of extreme efficiency, I need to inject the task directly into the global thread pool done queue, which involves some static variables within ngx_thread_pool.c.

Avoid patching?

The traditional way is of course patching the nginx and recompile it.

But the shorages are obvious:

you need to integrate lua-resty-ffi into your build, which is cumbersome
you can not use lua-resty-ffi in old releases of your product, because the binary should not be replaced
each time lua-resty-ffi changes, you need to recompile it with your product

All in all, could we use a non-intrusive way? i.e. dynamic loading.

Resolve static variables

Let’s check what is static variable in the elf binary.

# nm /usr/local/openresty-debug/nginx/sbin/nginx
000000000027f890 b ngx_thread_pool_done
000000000027f8a0 b ngx_thread_pool_done_lock
000000000028c690 B ngx_cycle
00000000000841a0 t ngx_thread_pool_handler
000000000007b920 T ngx_calloc

Name	ELF Type	ELF Section	C semantic
`ngx_thread_pool_done_lock`	local symbol	`.bss/.symtab`	static variable
`ngx_cycle`	global symbol	`.bss/.symtab/.dynsym`	external variable
`ngx_thread_pool_handler`	local symbol	`.text/.symtab`	static function
`ngx_calloc`	global symbol	`.text/.symtab/.dynsym`	external function

C static variables are visable only inside the same compile unit, i.e. source file. You have no way to access them elsewhere in C world. Without patching the source file, how could I use it in my dynamic loaded shared library?

In fact, all defined symbols in the same final binary output (executable or library) are referenced with relative addresses¹. All undefined symbols are referenced via GOT, which will be resloved by dynamic linker at runtime (function call is a bit special, which use plt stubs to do support lazy binding).

Let’s confirm it in assembly code.

objdump -D /usr/local/openresty-debug/nginx/sbin/nginx

0000000000083f40 <ngx_thread_pool_cycle>:
   ...
   lea    0x1fb83a(%rip),%rdi        # 27f8a0 <ngx_thread_pool_done_lock>
   ...
   mov    0x1fb826(%rip),%rax        # 27f898 <ngx_thread_pool_done+0x8>
...
00000000000733a0 <ngx_open_file_add_event>:
   ...
   callq  7b920 <ngx_calloc>
   ...
   mov    0x219239(%rip),%rax        # 28c690 <ngx_cycle>
   ...
   lea    0x127(%rip),%rdi        # 841a0 <ngx_thread_pool_handler>

You could see that it uses %rip relative addressing to locate the defined symbols, no matter whether it’s global or not, e.g. 27f8a0 <ngx_thread_pool_done_lock>, here 27f8a0 is the offset in .text section, which is 0x1fb83a away from %rip.

So, if we use global defined symbol in the nginx exectuable as anchor, plus a fixed offset, we could calculate the absolute address of the static variable!

However, we should be aware that the offsets are strictly bound to specific nginx executable. If the exectuable changes, the offsets may be invalid.

So it comes another question, how to guard that the nginx executable at runtime is identical to the version at compile time?

In fact, besides offsets, the C type (e.g. structure) definition and ABI/API compatibility also requires we should match the nginx executable version correctly. Nginx does not guarantee compatiblity among different version and builds, so even nginx dynamic modules also need to take care of this fact.

Retrieve build-id

Build id is what gcc used to reflect the identity of specific build. When you change the source code or compile options, the build id changes.

By default, gcc puts the generated build id into the .note.gnu.build-id section.

binutils’ ld has supported the –build-id=… option since version 2.18 (released 2007). When used, with a sha1 or md5 argument it directs ld to insert an ELF section .note.gnu.build-id into the binary containing a hash of the normative parts of the output—that is, an identifier that uniquely identifies the output file.

Let’s check our nginx executable:

$ file $(which nginx)
/usr/local/openresty/nginx/sbin/nginx: ELF 64-bit LSB shared object,
x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2,
BuildID[sha1]=bda5fd746456c2453605499e4d4372c90bde73eb,
for GNU/Linux 3.2.0, with debug_info, not stripped

From ELF 64-bit LSB shared object, we know it’s a PIE executable, and the build id is bda5fd746456c2453605499e4d4372c90bde73eb.

So, retrieve the build id of the nginx executable at runtime and compare it with the build id we gather at compile time, we could ensure our static variables references are correct and safe!

We could just simply use API from libdl.so from libc6 to retrieve the build id.

Build shared library

Gather offsets and build-id at compile time

In the build script, we uses nm and file commands to gather necessary infomation into a C header file:

// symbols.h
#define NGX_THREAD_POOL_DONE_OFFSET 0xFFFFFFFFFFFF3200
#define NGX_THREAD_POOL_DONE_LOCK_OFFSET 0xFFFFFFFFFFFF3210
#define NGX_THREAD_POOL_HANDLER_OFFSET 0x8880
#define NGX_BUILD_ID "bda5fd746456c2453605499e4d4372c90bde73eb"

Somebody may ask, why not inspect the nginx executable at runtime to resolve addresses, after all the offsets are always in the .symtab section, then we could fit any version of nginx?

No, because besides symbol offsets, we also depend on the type definition and ABI/API compatibility of nginx. So the build of lua-resty-ffi shared library is one-to-one bound to the specific version of nginx executable. Moreover, inspecting elf information at runtime is not a trivial job, which requires to use API from libbfd or libelf.

Resolve symbols at runtime

// ngx_http_lua_ffi.c
int
lua_resty_ffi_init()
{
...
    // resolve symbols
    char* addr;
    addr = (char*)&ngx_cycle;
    addr += NGX_THREAD_POOL_DONE_OFFSET;
    ngx_thread_pool_done = (ngx_thread_pool_queue_t*)addr;

    addr = (char*)&ngx_cycle;
    addr += NGX_THREAD_POOL_DONE_LOCK_OFFSET;
    ngx_thread_pool_done_lock = (ngx_atomic_t*)addr;

    addr = (char*)&ngx_calloc;
    addr += NGX_THREAD_POOL_HANDLER_OFFSET;
    ngx_thread_pool_handler = (ngx_event_handler_pt)addr;
...
}

Make within the openresty build context

We should enter the context of your openresty build and build lua-resty-ffi there, which ensures we have the same compile options as your product build, i.e. ensure lua-resty-ffi uses correct type definitions and API prototypes.

In fact, it’s same to what you develop nginx dynamic modules.

specify your openresty source path in variable $OR_SRC
ensure openresty source are already configured and built according to your product release

`1`	`OR_SRC=/tmp/tmp.Z2UhJbO1Si/openresty-1.21.4.1 ./build_shared.sh`

# build_shared.sh
...
export NGX_LUA_SRC="$(find $OR_SRC/build/ -maxdepth 1 -name 'ngx_lua-*' -type d)/src"
export NGX_OBJS_MAKEFILE="$NGX_BUILD/objs/Makefile"

export SRC=$PWD
(cd $NGX_BUILD; make -f $SRC/Makefile)

# Makefile
.DEFAULT_GOAL := libresty_ffi.so

include $(NGX_OBJS_MAKEFILE)

libresty_ffi.so:
    $(CC) $(CFLAGS) $(ALL_INCS) -I$(NGX_LUA_SRC) -shared -fPIC -DSHARED_OBJECT -ldl \
        $(SRC)/build-id.c $(SRC)/ngx_http_lua_ffi.c -o $(SRC)/$@

Review the output

0000000000002b50 <lua_resty_ffi_init>:
    ...
    mov    0x2387(%rip),%rax        # 4fc0 <ngx_cycle>
    mov    %rbp,%rdi
    lea    -0xce00(%rax),%rdx
    sub    $0xcdf0,%rax
    mov    %rax,0x24e8(%rip)        # 5138 <ngx_thread_pool_done_lock>
    ...

ngx_cycle is undefined dynamic symbol in libresty_ffi.so, and it’s placed in .got section, which will be resolved by dynamic linker to the absolute address in nginx executable. Plus correct offset, we resolve the absolute address of ngx_thread_pool_done_lock, which is saved in the .bss section.

You could confirm the .bss and .got section address range via readelf:

# readelf --sections libresty_ffi.so
  [22] .got              PROGBITS         0000000000004fb8  00003fb8
       0000000000000048  0000000000000008  WA       0     0     8
  [25] .bss              NOBITS           0000000000005120  00004120
       0000000000000020  0000000000000000  WA       0     0     8

Adjust resty_ffi.lua to fit both ways

If it could not find lua-resty-ffi symbols in the nginx executable, e.g. ngx_http_lua_ffi_create_task_queue, it tries to load libresty_ffi.so.

Note that it must load libresty_ffi.so in global namespace, because the APIs of lua-resty-ffi must be available for runtimes powered by lua-resty-ffi later.

-- resty_ffi.lua
...
ngx.load_ffi = function(lib, cfg, opts)
    if not init then
        if not pcall(test_symbol, "ngx_http_lua_ffi_create_task_queue") then
            local handle = ffi.load("resty_ffi", true)
            if handle.lua_resty_ffi_init() ~= 0 then
                error("lua_resty_ffi_init() failed: nginx build-id not found or mismatch")
            end
            resty_ffi = handle
        end
        init = true
    end
...

Package it via luarocks

Now, you could install lua-resty-ffi via luarocks at ease:

1
2

luarocks config variables.OR_SRC /tmp/tmp.Z2UhJbO1Si/openresty-1.21.4.1
luarocks install lua-resty-ffi

# luarocks show lua-resty-ffi
...
Modules:
        libresty_ffi (/usr/local/lib/lua/5.1/libresty_ffi.so)
        resty_ffi (/usr/local/share/lua/5.1/resty_ffi.lua)

Use lua-resty-ffi shared library

Since we do not patch lua-resty-core, we need to require lua-resty-ffi before we use ngx.load_ffi().

require("resty_ffi")
local demo = ngx.load_ffi("ffi_go_echo")
local ok, res = demo:echo("foobar")
assert(ok)
assert(res == "foobar")

Conclusion

With some “black magic”, we could build lua-resty-ffi as shared library, then no need to patch your openresty/nginx and rebuild it anymore.

lua-resty-ffi enables you to use your favorite mainstream programming language, e.g. Go, Java, Python, Rust, or Node.js, to do development in Openresty/Nginx, so that you could enjoy their rich ecosystem directly.

https://github.com/kingluo/lua-resty-ffi

In ancient ages, the executable, even shared library, uses absolute addressing. But nowadays, PIE and PIC is default option, as well as ASLR. In brief, the start address of the mapping of exectuable and library is determined randomly at startup or loading. All symbol references use relative addressing instead. ↩︎

Learn Async Rust From Assembly Code

luajit.io@gmail.com (jinhualuo) — Sun, 29 Jan 2023 12:14:34 +0000

Asynchronous programming or async for short, is a concurrent programming model supported by an increasing number of programming languages. It lets you run a large number of concurrent tasks on a small number of OS threads, while preserving much of the look and feel of ordinary synchronous programming, through the async/await syntax.

How rust implements the asynchronous programming?

Useful references:

https://rust-lang.github.io/async-book/01_getting_started/01_chapter.html

https://cfsamson.github.io/books-futures-explained/introduction.html

OS threading

The most common and easy way is OS threading.

But the most disadvantage is resource consumption, especially context switching overhead.

Green thread (stackful coroutine)

Just like Golang, rust provides green thread before it hits 1.0.

Callback/Promise

Just like Netty and Javascript, the callback based dispatching is a main method to implement async.

But callback hell, fragmented program logic and data sharing are the nightmare.

Async/Await (stackless coroutine)

Just like Python and Javascript, async/await is a popular async way.

Advantages:

The programming logics are linear, which is line with human thinking habits
No need to manage stack grow and shrink
No need to save CPU state
Reuse generator mechanism

Future trait

The Future trait is at the center of asynchronous programming in Rust.

A Future is an asynchronous computation that can produce a value.

pub trait Future {
    type Output;

    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>;
}

So, all implementations of async runtime, e.g. tokio, take Future as the schedule unit:

// Function tokio::spawn
pub fn spawn<T>(future: T) -> JoinHandle<T::Output>
where
    T: Future + Send + 'static,
    T::Output: Send + 'static,

Future itself provides Promise style async programming, aka combinators.

In fact, before async/await borns, the crate Futures 0.1 used combinators.

Example:

let future = Connection::connect(conn_str).and_then(|conn| {
    conn.query("somerequest").map(|row|{
        SomeStruct::from(row)
    }).collect::<Vec<SomeStruct>>()
});

let rows: Result<Vec<SomeStruct>, SomeLibraryError> = block_on(future);

Moreover, you can wrap futures into a high-level future, to implement new logic.

For example, write a Select top-level Future to run two sub-Futures simultaneously:

pub struct Select<A, B> {
    inner: Option<(A, B)>,
}

impl<A, B> Future for Select<A, B>
where
    A: Future + Unpin,
    B: Future + Unpin,
{
    type Output = Either<(A::Output, B), (B::Output, A)>;

    fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
        let (mut a, mut b) = self.inner.take().expect("cannot poll Select twice");

        if let Poll::Ready(val) = a.poll_unpin(cx) {
            return Poll::Ready(Either::Left((val, b)));
        }

        if let Poll::Ready(val) = b.poll_unpin(cx) {
            return Poll::Ready(Either::Right((val, a)));
        }

        self.inner = Some((a, b));
        Poll::Pending
    }
}

How Async/Await looks like in assembly code?

Async function is just a more friendly presentation of Future. Combined with .await, you can compose different Futures together at ease.

Each async function would be compiled into a special generator, which implements Future trait.

State machine

In the compiler, generators are currently compiled as state machines. Each yield expression will correspond to a different state that stores all live variables over that suspension point. Resumption of a generator will dispatch on the current state and then execute internally until a yield is reached, at which point all state is saved off in the generator and a value is returned.

Let’s take a simple example:

use futures::executor::block_on;

async fn foo() -> i32 {
    return 32
}

async fn bar() -> i32 {
    return 32
}

async fn foobar() {
    let str = "foo".to_string();
    let val = foo().await;
    let str2 = "bar".to_string();
    let val2 = bar().await;
    println!("hello, world! &str: {:p}, &val: {:p} &str2: {:p}, &val2: {:p}", &str, &val, &str2, &val2);
}

fn main() {
    let future = foobar();
    block_on(future);
}

We have a top-level async function foobar(), which calls async functions foo() and bar().

Let’s use playground to compile the code into assembly code in debug mode.

Check the assembly code of foobar() generator:

playground::foobar::{{closure}}:
	subq	$488, %rsp
	movq	%rsi, 176(%rsp)
	movq	%rdi, 192(%rsp)
	movq	%rsi, 416(%rsp)
	movq	192(%rsp), %rax
	movzbl	28(%rax), %eax
	movq	%rax, 184(%rsp)
	movq	184(%rsp), %rax
	leaq	.LJTI38_0(%rip), %rcx
	movslq	(%rcx,%rax,4), %rax
	addq	%rcx, %rax
	jmpq	*%rax
	ud2

State data structure

The first argument %rdi is the generator type itself, which contains state data, saved in the stack 192(%rsp).

This type has a number of states (represented here as an enum) corresponding to each of the conceptual states of the generator. At the beginning we’re closing over our outer variable foo and then that variable is also live over the yield point, so it’s stored in both states.

What’s the structure of the state data? Well, let’s check it in llvm IR:

%"[async fn body@src/main.rs:7:23: 9:2]" = type { i8 }
%"[async fn body@src/main.rs:3:23: 5:2]" = type { i8 }
%"[async fn body@src/main.rs:11:19: 17:2]::Suspend1" =
type { %"alloc::string::String", i32, [4 x i8], %"[async fn body@src/main.rs:7:23: 9:2]", [7 x i8], %"alloc::string::String" }
%"[async fn body@src/main.rs:11:19: 17:2]::Suspend0" =
type { %"alloc::string::String", [8 x i8], %"[async fn body@src/main.rs:3:23: 5:2]", [7 x i8] }

Suspend0 is the enum variant lives across the first yield point foo().await, and Suspend1 is the enum variant lives across the second yield point bar().await.

In my program, all local variables lives till the end, so Suspend1 is a superset of Suspend0, so it’s enough to check Suspend1. Note that embedded generator is stored in the parent generator, and because they only live before their yield points perspectively and of the same size, embedded generators reuse the same field of the foobar generator.

In assembly code, you can see that field mapping to variable is:

+0 : %"alloc::string::String" -> str
+24 : i32 -> val
+28 : [4 x i8] -> state discriminator
async fn body -> embedded generator for foo() or bar() 1 byte
[7 x i8] padding
+40 : %"alloc::string::String" -> str2

Note that val2 is a variable after the last yield point, so it’s on the stack but not in the state.

From the program output, you can confirm the address relationship too:

hello, world! &str: 0x7ffc9e68c9b8, &val: 0x7ffc9e68c9d0 &str2: 0x7ffc9e68c9e0, &val2: 0x7ffc9e68c6ec

State resume/yield

.LJTI38_0 is the memory label where stores the offsets of state resume entries.

.LJTI38_0:
	.long	.LBB38_2-.LJTI38_0 ; first resume entry
	.long	.LBB38_3-.LJTI38_0 ; resumed after completion
	.long	.LBB38_4-.LJTI38_0 ; resumed after panicking
	.long	.LBB38_5-.LJTI38_0 ; yield point1: foo().await
	.long	.LBB38_6-.LJTI38_0 ; yield point2: bar().await

The assembly code uses %rip relative addressing to locate the entry address:

; get discriminator and save it in %rax
movq	192(%rsp), %rax
movzbl	28(%rax), %eax
...
; get .LJTI38_0 address
leaq	.LJTI38_0(%rip), %rcx
; get entry offset relative to .LJTI38_0, each offset value is in 32 bit size
; i.e. offset = *(%rcx + %rax * 4)
movslq	(%rcx,%rax,4), %rax
; get entry absolute address = .LJTI38_0 + offset
addq	%rcx, %rax

Let’s check .LBB38_2:

.LBB38_2:
	movq	176(%rsp), %rax
	movq	%rax, 408(%rsp)
	movq	192(%rsp), %rdi
	leaq	.L__unnamed_12(%rip), %rsi
	movl	$3, %edx
	callq	<str as alloc::string::ToString>::to_string
	jmp	.LBB38_13
...
.LBB38_13:
	callq	playground::foo
	movb	%al, 159(%rsp)
	jmp	.LBB38_16

You can see that to_string() saves the result in 192(%rsp), where is field address of str.

Then it calls foo() to get and poll its generator.

.LBB38_16:
	...
	callq	<F as core::future::into_future::IntoFuture>::into_future
	movb	%al, 158(%rsp)
	jmp	.LBB38_17

.LBB38_17:
	...
	jmp	.LBB38_8

.LBB38_18:
	movq	408(%rsp), %rdi
	callq	core::future::get_context
	movq	%rax, 144(%rsp)
	jmp	.LBB38_19

.LBB38_19:
	...
	; foo().await
	callq	playground::foo::{{closure}}
	movl	%edx, 136(%rsp)
	movl	%eax, 140(%rsp)
	jmp	.LBB38_20

.LBB38_20:
	movl	136(%rsp), %eax
	movl	140(%rsp), %ecx
	movl	%ecx, 224(%rsp)
	movl	%eax, 228(%rsp)
	movl	224(%rsp), %eax
	; check poll result
	; 0: ready
	; 1: pending
	cmpq	$0, %rax
	; if pending, yield
	jne	.LBB38_22
	; if ready, continue next statement
	; i.e. `let str2 = "bar".to_string();`
	movl	228(%rsp), %ecx
	movl	%ecx, 444(%rsp)
	movq	192(%rsp), %rax
	; set foo() result to `val`
	movl	%ecx, 24(%rax)
	; set result address to `str2`
	movq	192(%rsp), %rdi
	addq	$40, %rdi
	leaq	.L__unnamed_14(%rip), %rsi
	movl	$3, %edx
	callq	<str as alloc::string::ToString>::to_string
	; continue next statement
	jmp	.LBB38_23

.LBB38_22:
	...
	; foo() not finished yet,
	; set discriminator to 3, i.e. `.LBB38_5`,
	; so that next resume will re-run foo().
	movq	192(%rsp), %rax
	movb	$3, 28(%rax)
	...
	retq

After the completion of the foobar(), it sets the discriminator to 1.

.LBB38_41:
	...
	movq	192(%rsp), %rax
	movb	$1, 28(%rax)
	...
	retq

Offset 1 means the next resume will jump to .LBB38_3, which will panic the program.

.LBB38_3:
	xorl	%eax, %eax
	testb	$1, %al
	jne	.LBB38_3
	jmp	.LBB38_10
...
.LBB38_10:
	leaq	str.0(%rip), %rdi
	leaq	.L__unnamed_13(%rip), %rdx
	movq	core::panicking::panic@GOTPCREL(%rip), %rax
	movl	$35, %esi
	callq	*%rax
	ud2
...
str.0:
	.ascii	"`async fn` resumed after completion"

Multi-variant layouts for generators

As said, each state of generator has its own variant structure layout. For non-overalpped fields of two consecutive states, memory will be reused.

In async function, the compiler needs to determine the local variables as stack variables or state fields, via liveness analysis from MIR.

stack variables will not be saved in the state
some state fields may be gone after this state.

Useful references:

https://tmandry.gitlab.io/blog/posts/optimizing-await-2/

https://github.com/rust-lang/rust/pull/59897

Let’s talk about some common cases.

Scoped

Normally, variables live until the end of a function, even if you do not use it after declaration. So, for temporary variables you don’t want to save in the state, you need to scope them.

For example, if you don’t use str after printing it, you need to scope it:

async fn foobar() {
    {
        let str = "foo".to_string();
        println!("{}", str);
    }
    let val = foo().await;
    let str2 = "bar".to_string();
    let val2 = bar().await;
    println!("hello, world! &val: {:p} &str2: {:p}, &val2: {:p}",  &val, &str2, &val2);
}

Then str is no longer one of the fields in the state.

1
2

Suspend0 = type { [8 x i8], %"[async fn body@src/main.rs:3:23: 5:2]", [3 x i8] }
Suspend1 = type { i32, [4 x i8], %"[async fn body@src/main.rs:7:23: 9:2]", [7 x i8], %"alloc::string::String" }

It’s especially meaningful for thread-safe generator, which means the async function will be scheduler by multi-threading executor.

Traits like Send and Sync are automatically implemented for a Generator depending on the captured variables of the environment. Unlike closures, generators also depend on variables live across suspension points. This means that although the ambient environment may be Send or Sync, the generator itself may not be due to internal variables live across yield points being not-Send or not-Sync.

For example, MutexGuard is not Send, then it cannot be used with tokio multi-thread scheduler.

You need to scope it:

{
    let s = some_mutex.lock();
    // use `s` here
}
// we don't own the lock anymore.
yield;

Move

When you move out a variable, it will not be saved in the state anymore.

Interestingly, drop is actually a move, so when you drop a variable manually, it’s execlued from the state.

let str = "foo".to_string();
drop(str);
// str is gone
let val = foo().await;

Copy

So far, rust compiler will keep copied variables in the state even if they are actually not used later.

You could test it:

async fn foobar() {
    let str = "foo".to_string();
    let val = foo().await;
    drop(val);
    let str2 = "bar".to_string();
    let val2 = bar().await;
    println!("hello, world! &val: {:p} &str2: {:p}, &val2: {:p}",  &val, &str2, &val2);
}

Yes, drop will copy for types implements Copy trait, although it’s no-op.

This effectively does nothing for types which implement Copy, e.g. integers. Such values are copied and then moved into the function, so the value persists after this function call.

From llvm IR, we could see that val is still there:

1
2

Suspend1 = type { %"alloc::string::String", i32, [4 x i8], %"[async fn body@src/main.rs:7:23: 9:2]", [7 x i8], %"alloc::string::String" }
Suspend0 = type { %"alloc::string::String", [8 x i8], %"[async fn body@src/main.rs:3:23: 5:2]", [7 x i8] }

shadowed variables

As known, shadowed variables will not be dropped until they go out of scope.

async fn foobar() {
    let str = "foo".to_string();
    let str = 233;
    let val = foo().await;
    let str2 = "bar".to_string();
    let val2 = bar().await;
    println!("hello, world! &str: {:p}, &val: {:p} &str2: {:p}, &val2: {:p}", &str, &val, &str2, &val2);
}

From the llvm IR, we could see that shadowed value "foo" still occupies one field of the state:

1
2

Suspend0 = type { %"alloc::string::String", i32, [12 x i8], %"[async fn body@src/main.rs:3:23: 5:2]", [7 x i8] }
Suspend1 = type { %"alloc::string::String", i32, i32, [8 x i8], %"[async fn body@src/main.rs:7:23: 9:2]", [7 x i8], %"alloc::string::String" }

How Pin looks like in assembly code?

Pin is some kind of smart pointer, which applies some constraint that the compiler should not move its address or swap its content, so that self-referenced struct is feasible, which is common case in async function, because we most likely use variable references in the function body.

Pin does not appears in the assembly code, even Deref and field projection will be optimized out.

Java Class Reloading

luajit.io@gmail.com (jinhualuo) — Sun, 15 Jan 2023 09:24:13 +0000

lua-resty-ffi provides an efficient and generic API to do hybrid programming in openresty with mainstream languages (Go, Python, Java, Rust, Nodejs, etc.).

https://github.com/kingluo/lua-resty-ffi

I already implement code hot-reload for Python and Nodejs in lua-resty-ffi.

The Java class reloading is the last piece of the puzzle.

No out-of-box API to do class reloading, so it’s more tricky.

What is classloader?

All classes in a Java application are loaded using some subclass of java.lang.ClassLoader.

Loading classes are on-demand and lazy, i.e. when the java program starts, it doesn’t load all classes constructing this program.

When a class is loaded, all classes it references are loaded too. This is done by the same classloader for the entry class. This class loading pattern happens recursively, until all classes needed are loaded. This may not be all classes in the application. Unreferenced classes are not loaded until the time they are referenced.

The instances of the same class loaded by different class loaders is in different type.

Classloader chain

Class loaders in Java are organized into a hierarchy.

The default class loader chain of openjdk-11:

org.codehaus.mojo.exec.URLClassLoaderBuilder$ExecJavaClassLoader@5aae8eb5
jdk.internal.loader.ClassLoaders$AppClassLoader@659e0bfd
jdk.internal.loader.ClassLoaders$PlatformClassLoader@25b342cd

Note that the bootstrap classloader (parent of PlatformClassLoader) is null, so it’s not shown in the chain.

Parent delegation

As known, the process of class loading applies the parent delegation model.

When the class is not found, the class loader would try to load class from its parent class loader, if not ok, it turns to do it itself, and so on.

Pseudo code:

Class loadClass(String name) {
    // if loaded
    Class cls = findFromLoaded(name);
    if(cls == null) {
        // try parent first
        cls = this.parent.loadClass(name)
    }
    if(cls == null) {
        // if parent failes, try to do on its own
        cls = this.findClass(name);
    }
    return cls;
}

Class Reloading

By default, a loaded class is cached forever by JVM.

To reload a class, it has two conditions:

crate a new class loader to load class each time
the class loader breaks the parent delegation model

I implement such class loader for lua-resty-ffi, based on this blog.

Let’s see how to satisfy above two conditions.

Check the source code here.

new ClassLoader to load new class

// determine where to load classes
// here it uses the CLASSPATH env var
var cfgItems = cfg.split(",");
var clsName = cfgItems[0];
clsName = clsName.substring(0, clsName.length() - 1);
String classpath = System.getenv("CLASSPATH");
var paths = classpath.split(":");

// determine the package of init class
var tmp = clsName.split("\\.");
var prefix = String.join(".", Arrays.copyOfRange(tmp, 0, tmp.length-1));

// reload the init class
Class<?> cls = new DynamicClassLoader(paths, prefix).load(clsName);
var method = cls.getDeclaredMethod(cfgItems[1], String.class, long.class);
Object ret = method.invoke(null, cfgItems.length > 2 ? cfgItems[2] : "", tq);

load classes from the child first

@Override
public Class<?> loadClass(String name) throws ClassNotFoundException {
    // if I don't care about this class, pass it to the parent completely,
    // otherwise, load it by myself first.
    // in practice, lua-resty-ffi only cares about the classes within
    // the same package of the init class, but other dependent classes,
    // e.g. gson, freemarker classes should be handled in the deafult JVM way,
    // so that they would be loaded once.
    if (!name.startsWith(prefix)) {
        return super.loadClass(name);
    }
...
    // load classes from class directory or JAR file.
    byte[] newClassData = loadNewClass(name);
...
}

With new class loader, the chain becomes:

DynamicClassLoader@55731ba7
jdk.internal.loader.ClassLoaders$AppClassLoader@659e0bfd
jdk.internal.loader.ClassLoaders$PlatformClassLoader@23a187ca

Why not Thread.contextClassLoader?

I thought by setting contextClassLoader of the polling thread, every new class from that thread would use that context class loader automatically.

But I am wrong. The context class loader should be used explicitly, otherwise, it’s useless.

The fact that the linked classes will be loaded by the same class loader perfectly meets my need, so that I do not need to change the code of each JAVA app, i.e. the class reloading is non-intrusive.

How to use?

Put a question mark as the class suffix when you calls ngx.load_ffi():

-- java
-- libffi_java.so
-- demo/http2/App.class
local demo = ngx.load_ffi("ffi_java", "demo.http2.App?,init,")

Please look at the doc in lua-resty-ffi for more details:

https://github.com/kingluo/lua-resty-ffi/blob/main/hot-reload.md

Use Nodejs to extend Openresty/Nginx

luajit.io@gmail.com (jinhualuo) — Wed, 14 Dec 2022 10:51:01 +0000

As known, Node.js is a popular javascript runtime based on V8 and libuv, with many third-party packages managed by npm.

How to enable nodejs in openresty, so that we could reuse its rich ecosystem?

Check https://github.com/kingluo/lua-resty-ffi/tree/main/examples/nodejs for source code.

lua-resty-ffi

https://github.com/kingluo/lua-resty-ffi

lua-resty-ffi provides an efficient and generic API to do hybrid programming in openresty with mainstream languages (Go, Python, Java, Rust, Node.js, etc.).

Features:

nonblocking, in coroutine way
simple but extensible interface, supports any C ABI compliant language
once and for all, no need to write C/Lua codes to do coupling anymore
high performance, faster than unix domain socket way
generic loader library for python/java/nodejs
any serialization message format you like

Just like Python, we could implement a loader library to embed nodejs in each nginx worker process.

Runtime initialization

In lua-resty-ffi, libffi_init() is defined in language specific shared library, used to initialize a ffi runtime (you could have multiple runtimes upon the same shared library, but with different configurations). One instance of runtime is normally one poll thread and other threads. How to implement libffi_init() in nodejs?

nodejs allows only one instance in one process, i.e. node::Environment, although you could have multiple instances of v8, i.e. v8::Isolate. So all ffi runtimes share this nodejs instance. Unlike python, nodejs does not provide C API to interact with nodejs when it is running, so we have to figure out some way to do init.

I choose unix domain socket.

The main thread of nodejs creates an unix socket server when libffi_init() was called for the first time. When nginx calls libffi_init() each time, it sends an init message, which contains the task queue pointer address and configuration string to the unix socket server. When the nodejs receives the init message, it would load the specific js module to do the real init, and finally, it returns the result to nginx.

Note that the unix socket is only used to do ffi runtime init, but not request-response flow between nginx and ffi runtime later, which uses lua-resty-ffi IPC there.

Threading Model

Node.js is single threading. The main thread uses libuv to provide event-driven nonblocking I/O processing. But nodejs supports worker thread, which is usually used to do CPU-intensive or other blocking tasks. Polling tasks from nginx is blocking, so we can take advantage of one worker thread to do polling. Once the polling thread gets one task, it passes the task to the main thread via parentPort, which is MessagePort allowing communication with the parent thread.

Wrap lua-resty-ffi API

Unlike python cffi, nodejs lacks a good ffi library.

The most famous one is node-ffi-napi, but I don’t know how to use it to convert a long integer into pointer, which is exactly what I need for task queue pointer conversion. And this project said that it has a non-trivial call overhead, so it is not suitable for busy requests flows.

Luckily, in nodejs, you could develop an addon module to couple with C at ease, based on Node-API.

Node-API (formerly N-API) is an API for building native Addons. It is independent from the underlying JavaScript runtime (for example, V8) and is maintained as part of Node.js itself. This API will be Application Binary Interface (ABI) stable across versions of Node.js.

The function prototype is flexible, you can get arbitrary numbers and types of input arguments from javascript, and return any type of output argument to javascript.

For example, I wrap ngx_http_lua_ffi_get_req() into get_req() function of Javascript, which returns a string to represent the request from nginx.

napi_value get_req(napi_env env, napi_callback_info args) {
    size_t argc = 1;
    napi_value argv[1];
    napi_status status = napi_get_cb_info(env, args, &argc, argv, NULL, NULL);
    if (status != napi_ok || argc != 1) {
        napi_throw_type_error(env, NULL, "Wrong number of arguments.");
        return NULL;
    }

    uint64_t ptr = 0;
    bool lossless;
    napi_get_value_bigint_uint64(env, argv[0], &ptr, &lossless);
    (void)lossless;

    void* tsk = (void*)ptr;
    int len;
    char* req = ngx_http_lua_ffi_get_req(tsk, &len);
    if (req) {
        napi_value ret;
        status = napi_create_string_utf8(env, req, len, &ret);
        if (status != napi_ok) return nullptr;
        return ret;
    }

    return nullptr;
}

How to use it?

This is the simplest example:

demo/echo.js

const { Worker, isMainThread, parentPort } = require('node:worker_threads');
const ffi = require("resty_ffi")

if (isMainThread) {
  // main thread logic
  // define a module method to do init
  exports.init = (cfg, tq) => {
    // create a worker thread
    // note that the worker thread also requires this module
    var worker = new Worker(__filename);
    // get a new task from nginx
    worker.on('message', (tsk) => {
      const req = ffi.get_req(tsk)
      // just echo back to nginx, rc=0
      ffi.respond(tsk, 0, req)
    });
    // tell the worker thread to do polling
    worker.postMessage(tq);
    return 0;
  };
} else {
  // worker thread logic
  parentPort.once('message', (tq) => {
    while (true) {
      // poll task from nginx
      const tsk = ffi.poll_task(tq)
      // if task queue is done, i.e. __unload() by lua, then exit
      if (tsk == 0) {
        console.log('exit ffi nodejs echo runtime');
        break;
      }
      // notify the main thread to handle the request
      parentPort.postMessage(tsk);
    }
  });
}

Specify the entry module and function in lua and use it:

-- note that the nodejs loader library symbols must be opened in global
local demo = ngx.load_ffi("ffi_nodejs", "demo/echo,init", {is_global=true})
local ok, res = demo:echo("hello")
assert(ok)
assert(res == "hello")

-- You could use '?' as module suffix to indicate that the module is hot-reload.
-- When the module gets loaded next time (after previous unload),
-- it would reload the module instead of the VM cached version
local demo = ngx.load_ffi("ffi_nodejs", "demo/echo?,init", {is_global=true})

Conclusion

Welcome to discuss and like my github repo:

https://github.com/kingluo/lua-resty-ffi

Implement Grpc Client in Rust for Openresty

luajit.io@gmail.com (jinhualuo) — Sat, 26 Nov 2022 10:53:49 +0000

Back in 2019, when I work as individual postgresql/nginx consultant. Many clients ask me if they could do hybrid programming in nginx.

They said, nginx/openresty is powerful and high performance, but the most significant drawback is it lacks of ecosystem of mainstream programming languages, especially when they have to interact with popular frameworks, e.g. kafka, consul, k8s, etc. The C/Lua ecosystem is slim, even if they bite the bullet and write down C/Lua modules, they face the awkward fact that most frameworks do not provide C/Lua API.

As known, nginx is multi-proceesses and single threading. In the main thread, it does nonblocking async event processing. Openresty wraps it in Lua API, combines the coroutine and epoll into coscoket concept.

Meanwhile, each mainstream programming language has its own threading model. So the problem is, how to couple with them in coroutine way? For example, when I call the function in Rust (maybe runs in a thread, or maybe in tokio async task, whatever), how to make it nonblocking in Lua land, and make the call/return in yield/resume way?

Of course, the traditional way is to spawn an individual process to run Rust, and at the lua side, use unix domain socket to do the communication, as known as proxy model. But that way is low-efficient (either development or runtime). You need to encode/decode the messsage in TLV (type-length-value) format, and, you need to maintain nginx and proxy process separately, as well as failover and load-balance topics.

Could I make it simple like FFI call but with high performance? The answer is yes, I created lua-resty-ffi.

lua-resty-ffi

https://github.com/kingluo/lua-resty-ffi

lua-resty-ffi provides an efficient and generic API to do hybrid programming in openresty with mainstream languages (Go, Python, Java, Rust, etc.).

Features:

nonblcking, in coroutine way
simple but extensible interface, supports any C ABI compliant language
once and for all, no need to write C/Lua codes to do coupling anymore
high performance, faster than unix domain socket way
generic loader library for python/java
any serialization message format you like

Architecture

The combination of library and configuration would init a new runtime, which represents some threads or goroutines to do jobs.

lua-resty-ffi has a high performance IPC mechanism, based on request-response model.

The request from lua is appended to a queue. This queue is protected by pthread mutex. The language runtime polls this queue, if it’s empty, then wait on the queue via pthread_cond_wait(). In busy scenario, almost all enqueue/dequeue happens in userspace, without futex system calls.

lua-resty-ffi makes full use of nginx nonblocking event loop to dispatch the response from the language runtime. The response would be injected into the global done queue of nginx, and notify the nginx main thread via eventfd to handle it. In main thread, the response handler would setup the return value and resume the coroutine waiting on the response.

As known, linux eventfd is high performance. It’s just an accumulated counter, so multiple responses would be folded into one event.

Both request and response data are exchanged in userspace.

What it looks like?

Take python as example.

ffi/echo.py

from cffi import FFI
ffi = FFI()
ffi.cdef("""
char *strdup(const char *s);
void* ngx_http_lua_ffi_task_poll(void *p);
char* ngx_http_lua_ffi_get_req(void *tsk, int *len);
void ngx_http_lua_ffi_respond(void *tsk, int rc, char* rsp, int rsp_len);
""")
C = ffi.dlopen(None)

import threading

class State:
    def poll(self, tq):
        while True:
            task = C.ngx_http_lua_ffi_task_poll(ffi.cast("void*", tq))
            if task == ffi.NULL:
                break
            req = C.ngx_http_lua_ffi_get_req(task, ffi.NULL)
            res = C.strdup(req)
            C.ngx_http_lua_ffi_respond(task, 0, res, 0)
        print("exit python echo runtime")

def init(cfg, tq):
    st = State()
    t = threading.Thread(target=st.poll, args=(tq,))
    t.daemon = True
    t.start()
    return 0

Calls it like normal function call in your lua coroutine:

local demo = ngx.load_ffi("ffi_python3", "ffi.echo?,init",
    {is_global = true, unpin = true})
local ok, res = demo:echo("hello")
assert(ok)
assert(res == "hello")

Benchmark

Send 100,000 requests, in length of 512B, 4096B, 10KB, 100KB respectively. The result is in seconds, lower is better.

You could see that lua-resty-ffi is faster than unix domain socket, and the difference is proportional to the length.

Check benchmark in detail.

Develop GRPC client in Rust

Let’s use lua-resty-ffi to do some useful stuff.

In k8s, GRPC is standard RPC. How to make GRPC calls in lua?

Let’s use Rust to develop a simple but complete GRPC client for lua, covering unary, client/server/bidirectional streaming calls, as well as TLS/MTLS.

In this example, you could also see the high development efficiency using lua-resty-ffi.

Check https://github.com/kingluo/lua-resty-ffi/tree/main/examples/rust for code.

tonic

tonic is a gRPC over HTTP/2 implementation focused on high performance, interoperability, and flexibility. This library was created to have first class support of async/await and to act as a core building block for production systems written in Rust.

tonic is my favourite rust library. It’s based on hyper and tokio. Everything works in async/await way.

Use low-level APIs

Normal steps to develop grpc client:

write a .proto file
compiles it into high-level API for use, via build.rs
calls generated API

For example:

helloworld.proto

package helloworld;

// The greeting service definition.
service Greeter {
  // Sends a greeting
  rpc SayHello (HelloRequest) returns (HelloReply) {}
}

// The request message containing the user's name.
message HelloRequest {
  string name = 1;
}

// The response message containing the greetings
message HelloReply {
  string message = 1;
}

client.rs

let request = tonic::Request::new(HelloRequest {
    name: "Tonic".into(),
});
let response = client.say_hello(request).await?;

But it’s not suitable for lua, because it’s expected to call arbitrary protobuf interface and does not need to compile .proto file.

So we have to look at low-level API provided by tonic.

Let’s check the generated file:

helloworld.rs

#[derive(Clone, PartialEq, ::prost::Message)]
pub struct HelloRequest {
    #[prost(string, tag="1")]
    pub name: ::prost::alloc::string::String,
}
...
pub async fn say_hello(
    &mut self,
    request: impl tonic::IntoRequest<super::HelloRequest>,
) -> Result<tonic::Response<super::HelloReply>, tonic::Status> {
...
    let codec = tonic::codec::ProstCodec::default();
    let path = http::uri::PathAndQuery::from_static(
        "/helloworld.Greeter/SayHello",
    );
    self.inner.unary(request.into_request(), path, codec).await
}

message encode/decode

You could see that the request structure is marked with ::prost::Message attribute. And it uses tonic::codec::ProstCodec::default() to do the encoding.

This part could be done in lua using lua-protobuf. It could do encode/decode based on the .proto file, just like JSON encode/decode.

Check grpc_client.lua for detail.

But wait, if you do encoding in lua, then how to tell rust to bypass it?

The answer is customized codec, instead of ProstCodec, so that we could transfer the byte array “AS IS”.

struct EncodedBytes(*mut c_char, usize, bool);
...
impl Encoder for MyCodec {
    type Item = EncodedBytes;
    type Error = Status;

    fn encode(&mut self, item: Self::Item, dst: &mut EncodeBuf<'_>) -> Result<(), Self::Error> {
        let slice = unsafe { slice::from_raw_parts(item.0 as *const u8, item.1) };
        dst.put_slice(slice);
...
    }
}

impl Decoder for MyCodec {
    type Item = EncodedBytes;
    type Error = Status;

    fn decode(&mut self, buf: &mut DecodeBuf<'_>) -> Result<Option<Self::Item>, Self::Error> {
        let chunk = buf.chunk();
        let len = chunk.len();
        if len > 0 {
            let cbuf = unsafe { libc::malloc(len) };
            unsafe { libc::memcpy(cbuf, chunk.as_ptr() as *const c_void, len) };
            buf.advance(len);
            Ok(Some(EncodedBytes(cbuf as *mut c_char, len, false)))
...
    }
}

For encode, we put the original byte array encoded by lua into the request.

For decode, we copy the byte array to lua and let lua decode it later.

Then, we could use unary directly.

connection management

Each GRPC request is called upon the connection object (interestingly, the connection in tonic has retry mechanism). I use hashmap to store connections, and assign each connection a unique string, so that the lua side could index it. In Lua, you could explicitly close it or let the lua gc to handle the close.

Each connection could be configured with TLS/MTLS info, e.g. CA certificate, private key.

message format of request and response

Now, let’s look at the message format between lua and rust, which is most important part.

request

#[derive(Deserialize, Debug)]
pub struct GrpcCommand {
    cmd: i32,
    key: String,
    host: Option<String>,
    ca: Option<String>,
    cert: Option<String>,
    priv_key: Option<String>,
    path: Option<String>,
    payload: Option<Base64>,
}

Since we need to handle different kinds of functions, e.g. create a connection, GRPC unary call, etc. So it’s obviously we need a structure to describe all information.

cmd: the command indicator, e.g. NEW_CONNECTION
key: depends on cmd, could be connection URL, connection id or streaming id.
host: HTTP2 host header, used for TLS
ca, cert, priv_key: TLS related stuff
path: full-qualified GRPC method string, e.g. /helloworld.Greeter/SayHello
payload: the based64 and protobuf encoded requset byte array by lua

Note that, all optional fields use Option<>.

JSON is high performance in lua, so no need to worry about the overhead.

Check my blog post for benchmark between JSON and protobuf if you are interested in this part.

response

This part is more straightforward than the request. Because for each request type, you could expect only one kind of result.

For example, for NEW_CONNECTION, it returns connection id string; for UNARY, it returns the protobuf encoded response from the network.

So, returning the malloc() byte array is ok, no need to wrap it in JSON.

GRPC streaming

Under the hood, no matter which type the call is, unary request, client-side streaming, server-side streaming or bidirectional streaming, tonic encapsulates the request and response into Stream object.

So we just need to establish the bridge between lua and Stream, then we could support GRPC streaming in lua.

send direction

We create a channel pair, wrap the receiver part into request Stream.

let (send_tx, send_rx) = mpsc::unbounded_channel::<EncodedBytes>();
let send_stream = tokio_stream::wrappers::UnboundedReceiverStream::new(send_rx);
...
let mut stream = cli
    .streaming(tonic::Request::new(send_stream), path, MyCodec)
    .await
    .unwrap()
    .into_inner();

Then the call upon the stream would be transfer to the network directly.

recv direction

We create an async task to handle the response retrieval.

tokio::spawn(async move {
...
    while let Some(task) = recv_rx.recv().await {
        if let Some(res) = stream.message().await.unwrap() {
            unsafe {
                ngx_http_lua_ffi_respond(task.0, 0, res.0, res.1 as i32);
            }
...
    }
});

close the send direction

Thanks to the channel characteristic, when we drop the send part of the channel pair, the receiver part would be closed and in turn close the send direction of GRPC stream.

Final look

Unary call:

local ok, conn = grpc.connect("[::1]:50051",
    {
        host = "example.com",
        ca = grpc.readfile("/opt/tonic/examples/data/tls/ca.pem"),
        cert = grpc.readfile("/opt/tonic/examples/data/tls/client1.pem"),
        priv_key = grpc.readfile("/opt/tonic/examples/data/tls/client1.key"),
    }
)
assert(ok)

local ok, res = conn:unary(
    "/grpc.examples.echo.Echo/UnaryEcho",
    {message = "hello"}
)
assert(ok)
assert(res.message == "hello")

conn:close()

Streaming calls:

local ok, conn = grpc.connect("[::1]:10000")
assert(ok)

-- A Bidirectional streaming RPC.
local ok, stream = conn:new_stream("/routeguide.RouteGuide/RouteChat")
assert(ok)

for i=1,3 do
    local note = {
        location = {latitude = 409146138 + i*100, longitude = -746188906 + i*50},
        message = string.format("note-%d", i),
    }
    local ok = stream:send(note)
    assert(ok)

    local ok, res = stream:recv()
    assert(ok)
    ngx.say(cjson.encode(res))
end

local ok = stream:close()
assert(ok)

local ok = conn:close()
assert(ok)

Benchmark

The most important question is, since we wrap the tonic, so how much is the overhead?

I use AWS EC2 t2.medium (Ubuntu 22.04) to do the benchmark.

Send 100,000 calls, use lua-resty-ffi and tonic helloworld client example respectively.

The result is in seconds, lower is better.

You could see that the overhead is 20% to 30%, depending the CPU affinity. No too bad, right?

Conclusion

With lua-resty-ffi, you could use your favourite mainstream programming language, e.g. Go, Java, Python or Rust, to do development in Openresty/Nginx, so that you could enjoy their rich ecosystem directly.

I just use 400 code lines (including lua and rust) to implement a complete GRPC client in OpenResty. And this is just an example, you could use lua-resty-ffi to do anything you want!

Welcome to discuss and like my github repo:

https://github.com/kingluo/lua-resty-ffi

Learn Rust From Assembly Code

luajit.io@gmail.com (jinhualuo) — Sun, 13 Nov 2022 14:55:14 +0000

Rust is a complex programming language, with a lot of concepts. When you find the concept hard to understand, perhaps checking the assembly code would be helpful. It’s my favourite way to learn Rust.

Move

In Rust, we have to use move in most time. It is used in variable assignment, function arguments and closure. move means ownship transfer, i.e. when you move one object, you could not access it from the original variable binding. But what happens in assembly code?

Let’s take a simple example:

use std::env;

#[derive(Debug)]
struct Foobar {
    a: i8,
    b: u8,
    c: i64,
    d: i64,
    e: i64,
    f: i64,
    g: i64,
    h: i64,
    i: i64,
}

#[inline(never)]
fn echo(bar: Foobar) {
    println!("{:?}", bar);
}

fn main() {
    let arg1 = env::args().nth(1).unwrap().parse::<i8>().unwrap();
    let arg2 = env::args().nth(2).unwrap().parse::<u8>().unwrap();
    let arg3 = env::args().nth(3).unwrap().parse::<i64>().unwrap();
    let arg4 = env::args().nth(4).unwrap().parse::<i64>().unwrap();
    let arg5 = env::args().nth(5).unwrap().parse::<i64>().unwrap();
    let arg6 = env::args().nth(6).unwrap().parse::<i64>().unwrap();
    let arg7 = env::args().nth(7).unwrap().parse::<i64>().unwrap();
    let arg8 = env::args().nth(8).unwrap().parse::<i64>().unwrap();
    let arg9 = env::args().nth(9).unwrap().parse::<i64>().unwrap();

    let bar = Foobar {
        a: arg1,
        b: arg2,
        c: arg3,
        d: arg4,
        e: arg5,
        f: arg6,
        g: arg7,
        h: arg8,
        i: arg9,
    };
    echo(bar);
}

Note that I define Foobar with more than two fields, so that llvm would not unroll the struct and pass the fields to echo field by field. And, I disable inline of echo, so that we could check move via real function call.

I use Rust playground to generate the assembly code.

Assembly code in debug mode:

...
	callq	core::str::<impl str>::parse
	movw	%ax, 206(%rsp)
	jmp	.LBB80_9

.LBB80_9:
	movw	206(%rsp), %ax
	movw	%ax, 1212(%rsp)
	movw	1212(%rsp), %ax
	movw	%ax, 224(%rsp)
	movzwl	224(%rsp), %edi
	leaq	.L__unnamed_22(%rip), %rsi
	callq	core::result::Result<T,E>::unwrap
	movb	%al, 205(%rsp)
	jmp	.LBB80_10

.LBB80_109:
	leaq	1032(%rsp), %rdi
	callq	core::ptr::drop_in_place<std::env::Args>
	movq	8(%rsp), %rax
...
	movb	205(%rsp), %r11b
	movb	%r11b, 1120(%rsp)
	movb	%r10b, 1121(%rsp)
...
	movq	%rax, 1112(%rsp)
	leaq	1128(%rsp), %rdi
	leaq	1064(%rsp), %rsi
	movl	$64, %edx
	callq	memcpy@PLT
	leaq	1128(%rsp), %rdi
	callq	playground::echo

You could see that it uses memcpy to copy the struct and pass to echo.

206(%rsp) is one of the parsed argument, after unwrap(), it becomes 205(%rsp), and finally as the source item 1120(%rsp) of memcpy.

Let’s check the release version:

...
	callq	*<std::env::Args as core::iter::traits::iterator::Iterator>::next@GOTPCREL(%rip)
...
	callq	*core::num::<impl core::str::traits::FromStr for i64>::from_str@GOTPCREL(%rip)
	cmpb	$0, 8(%rsp)
	jne	.LBB11_72
	movq	16(%rsp), %rax
	movq	%rax, 144(%rsp)

.LBB11_352:
	movq	144(%rsp), %rax
	movq	%rax, 40(%rsp)
	movq	136(%rsp), %rax
...
	leaq	40(%rsp), %rdi
	callq	playground::echo

No copy in release mode! It just pass the struct pointer to echo.

144(%rsp) is one of the parsed argument, get assigned to the first item of Foobar struct, 40(%rsp). And then, leaq 40(%rsp), %rdi gets the struct address. Interestingly, you could also see that the fields are reordered by the llvm.

So no need to worry about the performance of move.

BTW, let’s change the code a bit and use reference instead.

`1`	`echo(&bar);`

Then no surprise, it uses pointer, no matter debug or release mode.

1
2

	leaq	1064(%rsp), %rdi
	callq	playground::echo

Copy

How about struct with copy trait?

#[derive(Debug, Copy, Clone)]
struct Foobar {
    a: i8,
    b: u8,
    c: i64,
    d: i64,
    e: i64,
    f: i64,
    g: i64,
    h: i64,
    i: i64,
}

Check the debug version of assembly code:

	movq	%rcx, 1104(%rsp)
	movq	%rax, 1112(%rsp)
	leaq	1128(%rsp), %rdi
	leaq	1064(%rsp), %rsi
	movl	$64, %edx
	callq	memcpy@PLT
	leaq	1128(%rsp), %rdi
	callq	playground::echo

Well, memcpy happens.

Check the release version of assembly code:

1
2

	leaq	40(%rsp), %rdi
	callq	playground::echo

No copy happens! Just like move, llvm does not do stupid copy even if the copy trait is implemented.

Even if you call echo twice, it happens the same.

We could change the code and check again:

    let mut bar = ...
    echo(bar);
    bar.a = 99;
    echo(bar);

Check the debug version of assembly code:

	movq	%rax, 1112(%rsp)
	leaq	1128(%rsp), %rdi
	leaq	1064(%rsp), %rsi
	movl	$64, %edx
	callq	memcpy@PLT
	leaq	1128(%rsp), %rdi
	callq	playground::echo
	movb	$99, 1120(%rsp)
	leaq	1192(%rsp), %rdi
	leaq	1064(%rsp), %rsi
	movl	$64, %edx
	callq	memcpy@PLT
	leaq	1192(%rsp), %rdi
	callq	playground::echo

Copy one to echo, and modify a field in place, and copy another to second echo.

Check the release version of assembly code:

	leaq	40(%rsp), %rdi
	callq	playground::echo
	movq	%rbp, 40(%rsp)
	movq	%r14, 48(%rsp)
	movq	%r12, 56(%rsp)
	movq	%r13, 64(%rsp)
	movq	%r15, 72(%rsp)
	movq	120(%rsp), %rax
	movq	%rax, 80(%rsp)
	movq	112(%rsp), %rax
	movq	%rax, 88(%rsp)
	movb	$99, 96(%rsp)
	movb	%bl, 97(%rsp)
	leaq	40(%rsp), %rdi
	callq	playground::echo

The release assembly code reuses the same memory block in the stack to hold the struct content and passes the address to echo. Perfect!

Move in thread::spawn

Because function calls happen in the same stack frame chains, so Rust could optimize them without question. But for the closure case, esepcially for threading, the copy is unavoidable, because the closure invocation would happen in different context or even different OS thread!

Let’s confirm it.

1
2
3

    std::thread::spawn(|| {
        echo(bar);
    });

We only check the release assembly code:

...
	callq	*core::num::<impl core::str::traits::FromStr for i64>::from_str@GOTPCREL(%rip)
	cmpb	$0, 8(%rsp)
	jne	.LBB41_345
	movq	16(%rsp), %r12
...
	movq	%r12, 72(%rsp)
...
	movl	$88, %edi
	movl	$8, %esi
	callq	*__rust_alloc@GOTPCREL(%rip)
	testq	%rax, %rax
	je	.LBB41_397
	movq	88(%rsp), %rcx
	movq	%rcx, 80(%rax)
	movups	72(%rsp), %xmm0
	movups	%xmm0, 64(%rax)
	movdqu	8(%rsp), %xmm0
	movups	24(%rsp), %xmm1
	movups	40(%rsp), %xmm2
	movups	56(%rsp), %xmm3
	movups	%xmm3, 48(%rax)
	movups	%xmm2, 32(%rax)
	movups	%xmm1, 16(%rax)
	movdqu	%xmm0, (%rax)
	leaq	.L__unnamed_23(%rip), %rcx
	leaq	112(%rsp), %rdi
	movq	%r15, %rsi
	movq	%rax, %rdx
	callq	*std::sys::unix::thread::Thread::new@GOTPCREL(%rip)

You could see that it allocates on the heap and copy from the stack.

Take one field as example: %r12 -> 72(%rsp) -> %xmm0 -> 64(%rax).

Box

Box is used to manage memory on heap. In fact, Rust perfers stack. As known, struct/enum allocation and initialization only happen on the stack. Even if you wrap it with Box::new(), you still need to construct a struct instance on the stack first.

Then we have a question, does Rust optimize it so that the construction done on the heap directly?

Let’s check, we reuse above source code and change it:

#[inline(never)]
fn echo(bar: Box<Foobar>) {
    println!("{:?}", bar);
}
...
    let bar = Box::new(Foobar {
        a: arg1,
        b: arg2,
        c: arg3,
        d: arg4,
        e: arg5,
        f: arg6,
        g: arg7,
        h: arg8,
        i: arg9,
    });
    echo(bar);

Check the debug assembly code:

	callq	alloc::alloc::exchange_malloc
...
	movq	(%rsp), %rdi
	leaq	1072(%rsp), %rsi
	movl	$64, %edx
	callq	memcpy@PLT
	movq	(%rsp), %rax
	movq	%rax, 1216(%rsp)
	movq	(%rsp), %rdi
	callq	playground::echo

Yes, as expected, it constructs the struct on stack, and copy it to the heap.

What about release version?

	callq	*__rust_alloc@GOTPCREL(%rip)
	testq	%rax, %rax
	movl	76(%rsp), %ecx
	movzbl	75(%rsp), %edx
	je	.LBB13_357
	movq	112(%rsp), %rsi
	movq	%rsi, (%rax)
	movq	104(%rsp), %rsi
	movq	%rsi, 8(%rax)
	movq	96(%rsp), %rsi
	movq	%rsi, 16(%rax)
	movq	88(%rsp), %rsi
	movq	%rsi, 24(%rax)
	movq	80(%rsp), %rsi
	movq	%rsi, 32(%rax)
	movq	%r12, 40(%rax)
	movq	%r15, 48(%rax)
	movb	%cl, 56(%rax)
	movb	%dl, 57(%rax)
	movq	%rax, %rdi
	callq	playground::echo

Wow! It allocates and initializes the struct on the heap directly!

match

match is like C switch. In fact, simple constant switch would generate linear comparison branches, instead of jump table!

#[inline(never)]
fn echo(bar: Foobar) {
    match bar.i {
        3 => println!("{:?}", bar),
        99 => println!("99 {:?}", bar),
        88 => println!("88 {:?}", bar),
        188 => println!("188 {:?}", bar),
        288 => println!("288 {:?}", bar),
        388 => println!("388 {:?}", bar),
        488 => println!("488 {:?}", bar),
        588 => println!("588 {:?}", bar),
        688 => println!("688 {:?}", bar),
        788 => println!("788 {:?}", bar),
        888 => println!("888 {:?}", bar),
        _ => todo!(),
    }
}

Check the debug version of assembly code:

playground::echo:
	subq	$904, %rsp
	movq	%rdi, 184(%rsp)
	movq	48(%rdi), %rax
	movq	%rax, 192(%rsp)
	subq	$3, %rax
	je	.LBB79_2
	jmp	.LBB79_36

.LBB79_36:
	movq	192(%rsp), %rax
	subq	$88, %rax
	je	.LBB79_4
	jmp	.LBB79_37

.LBB79_37:
	movq	192(%rsp), %rax
	subq	$99, %rax
	je	.LBB79_3
	jmp	.LBB79_38

.LBB79_38:
	movq	192(%rsp), %rax
	subq	$188, %rax
	je	.LBB79_5
	jmp	.LBB79_39

Check the release version of assembly code:

playground::echo:
	subq	$72, %rsp
	movq	48(%rdi), %rax
	cmpq	$387, %rax
	jle	.LBB10_1
	cmpq	$687, %rax
	jg	.LBB10_14
	cmpq	$388, %rax
	je	.LBB10_21
	cmpq	$488, %rax
	je	.LBB10_22
	cmpq	$588, %rax
	jne	.LBB10_17
	movq	%rdi, 8(%rsp)
	leaq	<playground::Foobar as core::fmt::Debug>::fmt(%rip), %rax
	movq	%rax, 16(%rsp)
	leaq	.L__unnamed_2(%rip), %rax
	jmp	.LBB10_26

.LBB10_1:
	cmpq	$98, %rax
	jle	.LBB10_2
	cmpq	$99, %rax
	je	.LBB10_19
	cmpq	$188, %rax
	je	.LBB10_20
	cmpq	$288, %rax
	jne	.LBB10_17

Interestingly, the release version of assembly code splits the comparison branches into ranges to speed up the branch selection.

Json vs Flatbuffers vs Protobuf in Lua

luajit.io@gmail.com (jinhualuo) — Fri, 21 Oct 2022 10:57:06 +0000

flatBuffers is an efficient cross platform serialization library for C++, C#, C, Go, Java, Kotlin, JavaScript, Lobster, Lua, TypeScript, PHP, Python, Rust and Swift. It was originally created at Google for game development and other performance-critical applications.

In lua/luajit, is flatbuffers really faster than json?

I make a simple benchmark. Let’s see what’s the result.

Source code

test.fbs

namespace Test;

table TextEntry {
    name:string;
    value:string;
}

table Req {
    conf:[TextEntry];
    key:string;
}

runjson.lua

local cjson = require"cjson"

local t = {
    key = "hello",
    conf = {
        {name="foo", value="foo"},
        {name="bar", value="bar"},
        {name="fz", value="fz"},
        {name="baz", value="baz"},
    }
}

local _M = {}

function _M.marshal_bench()
    local cnt = 0
    for i = 1, 9000000 do
        cnt = cnt + #cjson.encode(t)
    end
    print(cnt)
end

function _M.unmarshal_bench()
    local data = cjson.encode(t)
    local cnt = 0
    for i = 1, 9000000 do
        cnt = cnt + #cjson.decode(data).key
    end
    print(cnt)
end

return _M

runfb.lua

local flatbuffers = require("flatbuffers")
local test_req = require("Test.Req")
local text_entry = require("Test.TextEntry")
local tblnew = require"table.new"

local builder = flatbuffers.Builder(0)

local t = {
    key = "hello",
    conf = {
        {name="foo", value="foo"},
        {name="bar", value="bar"},
        {name="fz", value="fz"},
        {name="baz", value="baz"},
    }
}

local _M ={}

local function fb_marshal()
    builder:Clear()

    local key = builder:CreateString(t.key)
    local len = #t.conf
    local textEntries = tblnew(len, 0)
    for i = 1, len do
        local name = builder:CreateString(t.conf[i].name)
        local value = builder:CreateString(t.conf[i].value)
        text_entry.Start(builder)
        text_entry.AddName(builder, name)
        text_entry.AddValue(builder, value)
        local c = text_entry.End(builder)
        textEntries[i] = c
    end
    test_req.StartConfVector(builder, len)
    for i = len, 1, -1 do
        builder:PrependUOffsetTRelative(textEntries[i])
    end
    local conf_vec = builder:EndVector(len)

    test_req.Start(builder)
    test_req.AddKey(builder, key)
    test_req.AddConf(builder, conf_vec)
    local req = test_req.End(builder)
    builder:Finish(req)

    return builder:Output()
end

local function fb_unmarshal(data)
    local buf = flatbuffers.binaryArray.New(data)
    local req = test_req.GetRootAsReq(buf, 0)
    local len = req:ConfLength()
    local res = {key = req:Key(), conf = tblnew(len, 0)}
    for i = 1, len do
        local entry = req:Conf(i)
        res.conf[i] = {name = entry:Name(), value = entry:Value()}
    end
    return res
end

function _M.marshal_bench()
    local cnt = 0
    for i = 1, 9000000 do
        cnt = cnt + #fb_marshal()
    end
    print(cnt)
end

function _M.unmarshal_bench()
    local data = fb_marshal()
    local cnt = 0
    for i = 1, 9000000 do
        cnt = cnt + #fb_unmarshal(data).key
    end
    print(cnt)
end

return _M

runpb.lua

local pb = require "pb"
local protoc = require "protoc"

local t = {
    key = "hello",
    conf = {
        {name="foo", value="foo"},
        {name="bar", value="bar"},
        {name="fz", value="fz"},
        {name="baz", value="baz"},
    }
}

assert(protoc:load [[
message Conf {
    required string name  = 1;
    required string value = 2;
}
message Test {
    required string key   = 1;
    repeated Conf   conf  = 2;
}
]])

local _M = {}

function _M.marshal_bench()
    local cnt = 0
    for i = 1, 9000000 do
        cnt = cnt + #pb.encode("Test", t)
    end
    print(cnt)
end

function _M.unmarshal_bench()
    local data = pb.encode("Test", t)
    local cnt = 0
    for i = 1, 9000000 do
        cnt = cnt + #pb.decode("Test", data).key
    end
    print(cnt)
end

return _M

Benchmark

mkdir /tmp/fb_bench
cd $_

git clone https://github.com/google/flatbuffers flatbuffers.repo
cp -a flatbuffers.repo/lua/flatbuffers* .

apt install -y flatbuffers-compiler
flatc --lua test.fbs

luarocks install lua-cjson lua-protobuf

# serialization

time luajit -e "require('runfb').marshal_bench()"
1584000000

real    2m18.694s
user    2m14.777s
sys     0m0.446s

time luajit -e "require('runjson').marshal_bench()"
1242000000

real    0m13.712s
user    0m13.438s
sys     0m0.030s

time luajit -e "require('runpb').marshal_bench()"
477000000

real    0m29.054s
user    0m28.377s
sys     0m0.060s

# deserialization

time luajit -e "require('runfb').unmarshal_bench()"
45000000

real    1m59.111s
user    1m55.956s
sys     0m0.392s

time luajit -e "require('runjson').unmarshal_bench()"
45000000

real    0m28.408s
user    0m27.686s
sys     0m0.092s

time luajit -e "require('runpb').unmarshal_bench()"
45000000

real    0m28.245s
user    0m27.531s
sys     0m0.089s

Conclusion

You could see that json is faster than flatbuffers a lot, even the serialization size is better a bit.

And the protobuf is half of speed in serialization compared to JSON, but deserialization is almost the same.

Of course, it’s for lua only, and my test schema only contains string types. You need to make a similar benchmark in your scenario.

I do not have time to investigate the reason, but I guess it’s due to the lua implementation of flatbuffers: it handle each field of struct via lua code, including table iteration, but cjson handles everything in C land, as well as lua-protobuf.

JSON is my favourite format.

Besides performance, it has below advantages:

no schema
no code generation
human readable

If you’re worrying about the size of JSON for huge data, you could use lz4 to do compression, which has low CPU overhead.

What’s your opinion? Welcome to discuss!

My Shell Tips

luajit.io@gmail.com (jinhualuo) — Wed, 19 Oct 2022 14:39:21 +0000

I work on a remote terminal most of the time. This article collects some tips from my experience.

bash

multiple line editing

Sometimes a complex command consists of multiple lines and you need to edit them before executing it. The lines may be in any format, e.g. json.

use quotes

curl http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance -d '
some_metric{label="val1"} 42
another_metric 2398.283
'

Here Documents

cat <<EOF | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance
some_metric{label="val1"} 42
another_metric 2398.283
EOF

edit via vim

Above ways are inconvenient, because you could not jump back and forth the lines to edit.

Now you need vim.

`1`	`export EDITOR=vim`

Just press C-x C-e to enter vim, and bash would create a temp file to edit, and when you save and exit vim, bash will execute the file content.

You could even paste an string in bash and then press C-x C-e, bash would pass that string to vim as initial content and let you continue editing.

history customization

# increase history size
export HISTSIZE=10000
export HISTFILESIZE=10000

# bash history is not automatically saved by default.
# below setting would save history after each command execution.
PROMPT_COMMAND='$PROMPT_COMMAND; history -a'

Process Substitution

Some command requires file arguments only, then process substitution is the solution.

For example, diff two curl responses:

diff -aNur <(curl -s httpbin.org/get) <(curl -s httpbin.org/get)
--- /dev/fd/63  2022-10-18 16:02:18.633005560 +0800
+++ /dev/fd/62  2022-10-18 16:02:18.633005560 +0800
@@ -4,7 +4,7 @@
     "Accept": "*/*",
     "Host": "httpbin.org",
     "User-Agent": "curl/7.68.0",
-    "X-Amzn-Trace-Id": "Root=1-634e5d8b-1eb31b42468a62f11ca13683"
+    "X-Amzn-Trace-Id": "Root=1-634e5d8b-3d27e89e310ea9fb56e8c0a8"
   },

alias completion

Normally we prefer short alias instead of long command, but bash doesn’t support command completion for alias. Then you could use complete-alias plugin.

`1`	`git clone https://github.com/cykerway/complete-alias ~/.complete-alias`

For example, alias kubectl as k.

~/.bashrc

alias k=kubectl
source <(kubectl completion bash)
. ~/.complete-alias/complete_alias
complete -F _complete_alias k

long string literal

Use concatenation

1
2
3

export FOOBAR="/foo/abc"
FOOBAR+=":/bar/xyz"
FOOBAR+=":/foobar"

Use backward slash

Note that spaces from the line beginning will be included in the string.

1
2
3

export FOOBAR="/foo/abc\
:/bar/xyz\
:/foobar"

tmux

tmux is a brilliant terminal tool you have to use. With tmux, you could parallel your works in one terminal.

Most useful features:

multiple tmux sessions, each tmux session contains multiple windows
tmux session runs in background
- no worry about ssh connection gets broken and you could resume where you were
- run server programs in the foreground but left them continue running after detaching
- shared with other user to watch or edit the screen together

attach last session

It’s useful when your ssh connection is broken, you login again and try to attach last session you were in.

tmux a

set session name

Set a meaningful session name: C-b $, e.g. dev.

You could use this name to attach later.

`1`	`tmux att -t dev`

session jump

Show all tmux session/window in tree view and jump to any arbitary window.

Setup shortcut:

~/.tmux.conf

unbind t
bind t choose-tree

Press C-b t to show the windows tree:

(0)   - dev: 8 windows (attached)
(1)   ├─> 0: bash* (1 panes) "foo"
(2)   ├─> 1: bash (1 panes) "foo"
(3)   ├─> 2: bash (1 panes) "foo"
(4)   ├─> 3: bash (1 panes) "foo"
(5)   ├─> 4: bash- (1 panes) "foo"
(6)   ├─> 5: vim (1 panes) "foo"
(7)   ├─> 6: bash (1 panes) "foo"
(8)   └─> 7: bash (1 panes) "foo"
(9)   - ops: 1 windows
(M-a) └─> 0: bash* (1 panes) "foo"

Each window shown in the tree has a key:

for 0-9, press the number key will jump to the corresponding window
for M-*, press Alt+* will jump to the corresponding window

jump between two last windows

It’s useful to swtich between your last two windows: C-b l

Tmux Resurrect

With this tmux plugin, you could resume everything after reboot or shutdown, e.g. power off.

https://github.com/tmux-plugins/tmux-resurrect

tmux-resurrect saves all the little details from your tmux environment so it can be completely restored after a system restart (or when you feel like it). No configuration is required. You should feel like you never quit tmux.

fzf

Command history search in bash has no fuzzy search and no history list. fzf could help you there.

Install fzf:

1
2
3

git clone --depth 1 https://github.com/junegunn/fzf.git ~/.fzf
~/.fzf/install
git clone https://github.com/4z3/fzf-plugins ~/.fzf-plugins

Edit ~/.bashrc:

[ -f ~/.fzf.bash ] && source ~/.fzf.bash

# when you select one command, press `C-e` to edit it instead of executing it directly:
FZF_CTRL_R_EDIT_KEY=ctrl-e
FZF_CTRL_R_EXEC_KEY=enter
source ~/.fzf-plugins/history-exec.bash

Press C-r to search history interactively in fuzzy way.

zoxide

The builtin cd of bash is very inconvenient, it has no history and fuzzy jump.

I prefer the tool zoxide, which could improve your work productivity.

https://github.com/ajeetdsouza/zoxide

It remembers which directories you use most frequently, so you can “jump” to them in just a few keystrokes.

Example:

root@myhost:~# z /tmp/
root@myhost:~# mkdir -p foo/bar/{apple,orange/{abc-bi,xyz},food/{abc,xyz}}
root@myhost:/tmp# tree foo
foo
└── bar
    ├── apple
    ├── food
    │   ├── abc
    │   └── xyz
    └── orange
        ├── abc-bi
        └── xyz

8 directories, 0 files

root@myhost:/tmp# z foo/bar/apple/
root@myhost:/tmp/foo/bar/apple# z ../food/abc/
root@myhost:/tmp/foo/bar/food/abc# z /tmp/foo/bar/orange/abc-bi/

# input `abc` only would match `/tmp/foo/bar/food/abc`
root@myhost:/tmp/foo/bar/orange/abc-bi# z abc

# more specific word, with `-` suffix, then jump to `abc-bi`
root@myhost:/tmp/foo/bar/food/abc# z abc-

root@myhost:/tmp/foo/bar/orange/abc-bi# z /tmp/foo/bar/food/xyz/
root@myhost:/tmp/foo/bar/food/xyz# z /tmp/foo/bar/orange/xyz

# specify multiple path segments
root@myhost:/tmp/foo/bar/orange/xyz# z food xyz

# choose alternatives for `xyz` interactively
root@myhost:/tmp/foo/bar/food/xyz# zi xyz

ssh

socks5 proxy sever

`1`	`ssh -o ServerAliveInterval=60 -N -D 127.0.0.1:10000 root@vpn -p 10022 &`

Then use 127.0.0.1:10000 as your proxy address, and the traffic would go through the safe ssh connection.

port forwarding

Sometimes you need to let two machines access to each other, but for security, you could not export ports except ssh port. In this case, you could use port forwarding.

For example, you need to access foo web server from the same port 1984 but with different IP address bindings, because you need to simulate sending different cookies.

1
2

ssh -N -L 127.0.0.1:1984:127.0.0.1:1984 foo &
ssh -N -L 127.0.0.2:1984:127.0.0.2:1984 foo &

jump host

Some servers could only be accessed in internal network. You could use gateway server as jump host, then no need to login gateway first and run ssh from there. Another benefit is you don’t have to store credential, e.g. private keys in the gateway machine, and everything works just like you login the target machine directly.

~/.ssh/config

Host gateway
    ...

Host foobar
    HostName internal.machine
    User foobar
    IdentityFile ~/.ssh/foobar.pem
    ProxyJump gateway

git

Configure ~/.gitconfig to improve work productivity.

commit history tree

git tree

1
2

[alias]
    tree = log --graph --decorate --oneline --all

exclude some files changes temporarily

[alias]
    ignore = !git update-index --assume-unchanged
    unignore = !git update-index --no-assume-unchanged
    ignored = !git ls-files -v | grep ^[a-z]

make diff fancy, e.g. hightlight different parts of two lines

[core]
    pager = /root/diff-so-fancy/diff-so-fancy | less --tabs=4 -RFX
[interactive]
    diffFilter = /root/diff-so-fancy/diff-so-fancy --patch
[diff-so-fancy]
    changeHunkIndicators = false
    stripLeadingSymbols = false

credential cache, then no need to input access token for a period

1
2

[credential]
    helper = cache --timeout 7200

find

Sometimes you need to check which files you modified within some period.

For example, backup all json files modified in last hour:

`1`	`find dir -mmin -60 -name '*.json' -type f -exec mv {} /tmp/backup/ \;`

less

less is a necessary tool to view a file.

tail -f

Press F to watch new changes of the file (e.g. log file) from the tail constantly.

filter lines

Press &<pattern> to show lines only match the pattern.

Press &!<pattern> to show lines not matching the pattern.

jq

jq is a brilliant tool to analyze json in the terminal.

https://stedolan.github.io/jq/manual/

Prettify JSON

etcdctl get --print-value-only /apisix/routes/test | jq
{
  "status": 1,
  "plugins": {
    "proxy-rewrite": {
      "uri": "/get",
      "use_real_request_uri_unsafe": false
    }
  },
  "id": "test",
  "priority": 0,
  "update_time": 1666154686,
  "upstream": {
    "hash_on": "vars",
    "nodes": {
      "httpbin.org": 1
    },
    "scheme": "http",
    "pass_host": "pass",
    "type": "roundrobin"
  },
  "uri": "/foo",
  "methods": [
    "GET"
  ],
  "create_time": 1666154686
}

Analyze JSON

Analyze file

jq '.[] | select(.color=="yellow" and .price>=0.5)' fruits.json

Analyze command output

For example, we use kcadmin.sh to do keycloak provisioning. We need to determine one client’s id and use this id to configure its profile.

Output from kcadm.sh:

[ {
    "id" : "0895c867-8fc1-4817-9bf3-06dddb6eace8",
    "clientId" : "account",
    "name" : "${client_account}",
  }, {
    "id" : "3b51e2a9-1eec-483a-b62d-d056fb7b622b",
    "clientId" : "account-console",
    "name" : "${client_account-console}",
  },
  ...
]

Use jq to get the desired id:

1
2

kcadm.sh get clients -r test --fields id,clientId 2>/dev/null | \
     jq -r '.[] | select(.clientId=="account") | .id'

Merge JSON

Example1:

`1`	`jq -s '.[0] * .[1] \| {somekey: .somekey}' <file1> <file2>`

Example2:

echo '[ {"a":1}, {"b":2} ]' | \
jq --argjson input1 '[ { "c":3 } ]' \
   --argjson input2 '[ { "d":4 }, { "e": 5} ]' \
   '. = $input1 + . +  $input2'

Vim Minimalist

luajit.io@gmail.com (jinhualuo) — Sat, 15 Oct 2022 07:56:38 +0000

Vim, the god of editor, is popular in developers and hackers. I use vim over two decade (when I am in the college, I already use vim to write VHDL code to programming the chip).

A lot of people install many plugins into vim, and even turn vim into IDE, which is cumbersome and unnecessary in my eye. In my daily life, vim is only suitable for terminal usage (on Windows, I would rather use vscode and pycharm), so why not just keep it simple and be a minimalist? Just reuse anything from vanilla vim, and add plugins only if you have to. Currently, I only install two plugins: ctrlp and vim-fugitive.

Keep the screen clean (even turning off syntax highlight), for me, it could keep me focus on the code only.

Vim has many many topics, and I only talk about my own experience.

Plugin manager

Since version 8, vim has builtin plugin manager, so just ignore all sorts of wild ones.

For example, to install ctrlp, download it into specific directory is enough.

git clone https://github.com/ctrlpvim/ctrlp.vim ~/.vim/pack/plugins/start/ctrlp

Basic settings

set nocompatible

" import default settings
source $VIMRUNTIME/vimrc_example.vim

" use 4 spaces instead of tab
set expandtab
set shiftwidth=4
set softtabstop=4
set tabstop=4

" While searching though a file incrementally highlight matching characters as you type.
set incsearch

" Override the ignorecase option if searching for capital letters.
" This will allow you to search specifically for capital letters.
set smartcase

" Use highlighting when doing a search.
set hlsearch

" when you press *, just highlight and stay still
nnoremap * :keepjumps normal! mi*`i<CR>

" wrap lines
autocmd OptionSet * set wrap

" highlight spaces at the end of line
highlight WhiteSpaceEOL ctermbg=darkgreen guibg=lightgreen
match WhiteSpaceEOL /\s$/
autocmd WinEnter * match WhiteSpaceEOL /\s$/

" vim default leader key is \
" press \p when you paste text from outside to keep indents,
" and press it again to disable paste mode
nnoremap <Leader>p :set paste!<Cr>

" press \r to avoid misoperation
" and press again if you really need to do modification
nnoremap <Leader>r :set readonly!<Cr>

" press ctrl-l to clean search highlight
nnoremap <silent> <C-l> :<C-u>nohlsearch<CR><C-l>

" after you paste something, press gb to select it in visual mode
" so that you could format it or do something else
nnoremap <expr> gb '`[' . strpart(getregtype(), 0, 1) . '`]'

File management

netrw is vim’s builtin file explorer. It’s good enough. No need to install nerdtree or other plugins.

It’s boring when you need to traverse deep directory tree and do a lot of create/move/rename/delete file operations in the shell. Then netrw is your best friend.

Just press :E to enter netrw window. Like normal window, you could press /pattern to locate the desired file.

Check this gist for detailed shortcuts:

https://gist.github.com/danidiaz/37a69305e2ed3319bfff9631175c5d0f

But netrw does not support deleteing non-empty directory.

Here is the workaround, so that you could press FF to do so.

function! NetrwRemoveRecursive()
  if &filetype ==# 'netrw'
    cnoremap <buffer> <CR> rm -r<CR>
    normal mu
    normal mf
    try
      normal mx
    catch
      echo "Canceled"
    endtry

    cunmap <buffer> <CR>
  endif
endfunction

function! NetrwMapping()
    nmap <buffer> FF :call NetrwRemoveRecursive()<CR>
endfunction

augroup netrw_mapping
    autocmd!
    autocmd filetype netrw call NetrwMapping()
augroup END

ctrlp

https://github.com/ctrlpvim/ctrlp.vim

Press ctrl-p to open file in fuzzy way.

let g:ctrlp_map = '<c-p>'
let g:ctrlp_cmd = 'CtrlPMixed'
let g:ctrlp_user_command = ['.git', 'cd %s && git ls-files -co --exclude-standard']
let g:ctrlp_open_multiple_files = 'i'

There is a ctrlp plugin I have to mention, which is very useful to locate function in source code. That is ctrlp-funky. No ctag files needed and regex based search is good enough. Just press \fu to open the function window.

1
2
3

nnoremap <Leader>fu :CtrlPFunky<Cr>
" narrow the list down with a word under cursor
nnoremap <Leader>fU :execute 'CtrlPFunky ' . expand('<cword>')<Cr>

Buffer management

The builtin buffer mangement is good enough, you only need some shortcuts:

" open buffer list
" then press `:b<N>` to jump to the desired buffer
nnoremap <silent> bl :ls<CR>

" jump to next buffer
nnoremap <silent> bn :bn<CR>

" jump to previous buffer
nnoremap <silent> bp :bp<CR>

" jump to the alternative buffer
" this is brilliant useful!
" press it again and again to switch between last two buffers you use,
" just like what tmux `ctrl-l` does
nnoremap <silent> bo :b#<CR>

Search and Replace

quickfix window shortcuts

nnoremap <silent> co :copen<CR>
nnoremap <silent> cn :cn<CR>
nnoremap <silent> cp :cp<CR>
nnoremap <silent> ccl :ccl<CR>

Search current buffer

`1`	`:vim /pattern/ % \| cw`

Search in all open buffers

Press :Vim <pattern>, show results in quickfix window.

function! Vimgrepall(pattern)
  call ClearQuickfixList()
  exe 'bufdo vimgrepadd ' . a:pattern . ' %'
  copen
endfunction

command! -nargs=1 Vim call Vimgrepall(<f-args>)

Clear the quickfix window when finished:

`1`	`:call setqflist([])`

Replace in all buffers

`1`	`:bufdo %s/search/replace/g`

Replace in arbitrary file tree

Take a little example I did. I need to modify tons of **/*.t files (directory depth is random):

remove pattern only at the end of the file

--- no_error_log
[error]

remove pattern (if there is an empty line before the pattern, it should be removed too)

    if ... {
        $block->set_value("no_error_log", "[error]");
    }

Multi-lines pattern matching is very tricky for trandition tools like sed, even when you write perl script, it’s complex and error-prone. In fact, use vim, you could handle it easily. What you see is what you get, you could improve the pattern step by step, and you could even do selective replacement interactively. The vim regular expression is powerful.

Here I use quickfix list to do search and replace:

:vimgrep /--- no_error_log\n\[error\]\%$/ t/**/*.t
:silent cfdo %s/--- no_error_log\n\[error\]\%$//g | update
:call setqflist([])
:vimgrep /\n\?\s*if\s\+.*\n.*\$block->set_value("no_error_log", "\[error\]");.*\n\s*}\s*\n/ t/**/*.t
:silent cfdo %s/\n\?\s*if\s\+.*\n.*\$block->set_value("no_error_log", "\[error\]");.*\n\s*}\s*\n//g | update
:call setqflist([])

Misc

" show matched count
:%s/pattern//ng

" list all matches to have a whole view
:g/pattern/#

" To delete the lines containing either "one" or "two", run:
:g/one\|two/d

" If we want to substitute all "bad"s into "nice",
" but only on the lines that contain "good", we can do:
:g/good/s/bad/nice/g

" In normal mode
" Press `S`, vim will remove the current line,
" indent corresponding to current context and enter insert mode.

" figure out buffer changes since last save
:w !diff % -
" if git is installed, use git to color the diff
:w !git diff % -

vim-fugitive

When you have many files modified, when you need to:

revert some of them
stage/unstage some of them
diff some of them to confirm changes before commit

It’s boring to copy/paste file name and execute git command one by one. vim-fugitive would help.

Press :G to enter git window.

Check shortcuts here:

https://github.com/tpope/vim-fugitive/blob/master/doc/fugitive.txt

Most useful shorcuts:

s                       Stage (add) the file or hunk under the cursor.
u                       Unstage (reset) the file or hunk under the cursor.
X                       Discard the change under the cursor.
dd                      Perform a |:Gdiffsplit| on the file under the cursor.
cc                      Create a commit.

Conclusion

Vim is powerful editor, insist vanilla vim if possible, add plugins gradually if you have to. Don’t use it as IDE, which is not original intention of vim, in my opinion. Keep focus on your source code, but not the fancy looks and addons.

Nginx Blocking

luajit.io@gmail.com (jinhualuo) — Fri, 14 Oct 2022 16:21:46 +0000

What’s blocking?

Nginx has one master process and multiple worker processes, and each worker process is single threading.

It’s well known that NGINX uses an asynchronous, event‑driven approach to handling connections. This means that instead of creating another dedicated process or thread for each request (like servers with a traditional architecture), it handles multiple connections and requests in one worker process. To achieve this, NGINX works with sockets in a non‑blocking mode and uses efficient methods such as epoll and kqueue.

However, once the connection is accepted in one worker process, it could not be moved to other processes, even when that process is busy. Not like golang, it has no work stealing. And, only socket is asynchronous, the cpu-intensive tasks and file io tasks is still synchronous.

Every phase handler for an http request runs in a single thread. OpenResty allows you to run lua code in each phase. The rewrite, access, content phases, as well as the timer (implemented as faked connection) are running lua code within an individual lua coroutine. Within the coroutine, you could use cosocket to communicate with the world. The cosocket is 100% nonblocking out of the box. When you call cosocket API and it needs to wait for more data, it would yield the coroutine, register corresponding event in epoll and let the event-loop handles other events.

As the name suggests, coroutines use the CPU in cooperative way. When you block a long time within one coroutine, other coroutines have no way to get executed, because all coroutines reside in the one thread, which results in uneven request handling and long request delay.

When I work in Kugou Music, my team encounters blocking issue, and I wrote a simple tool to diagnose it. This tool makes use of LD_PRELOAD to hook the lua_resume and check the function execution time, and print the url and entry lua function for long execution.

In fact, this tool is not good, because it could not print the backtrace and does not work for compiled lua code.

Locate the blocking source

The best tool to check blocking should be systemtap.

The blocking comes from two things:

cpu-intensive tasks
blocking system calls

So just check the execution time, if the time exceed a threshold, print the backtraces collected during that time.

You could find the example code here:

https://gist.github.com/kingluo/50b8589069f5f11a78d2136c1155d613

workload.lua

The workload simulates both cpu-intensive tasks (random string generation and MD5 calculation) and calls blocking system calls (blocking socket to access httpbin).

local http = require("socket.http")
local cjson = require("cjson")

math.randomseed(os.time())

local _M = {}

local function random_str(length)
    local res = ""
    for i = 1, length do
        res = res .. string.char(math.random(97, 122))
    end
    return res
end

local function run_md5()
    local cnt = 0
    local buf = random_str(81920)
    for _= 1,1000 do
        cnt = cnt + #ngx.md5(buf)
    end
    return cnt
end

function _M.run()
    local cnt = run_md5()
    ngx.say(cnt)
    local r, c, h = http.request {
        method = "GET",
        url = "http://httpbin.org/delay/1"
    }
    assert(r == 1)
    ngx.say("status=", c)
    ngx.say(cjson.encode(h))
end

return _M

nginx.conf

error_log /dev/stderr info;
worker_processes 1;

events {}

http {
    server {
        listen 10000;

        location /run {
            content_by_lua_block {
                require("workload").run()
            }
        }
    }
}

blocking-bt.sxx

It tracks the lua_resume (the entry of coroutine) and syscalls.

...
probe process("$^libluajit_path").function("lua_resume").return
{
    if (tid() == target()) {
        tracing = 0
        elapsed = gettimeofday_ms() - start_time

        if (elapsed >= $^arg_min_elapsed :default(200)) {
            printf("--- lua_resume block %d ms\n", elapsed)
            banner = 0
            foreach (bt in bts- limit $^arg_limit :default(3)) {
                cnt = @count(bts[bt])
                if (cnt <= $^arg_min_bt :default(2)) {
                    break
                }
                if (banner == 0) {
                    printf(">>> Hot CPU backtraces\n")
                    banner = 1
                }
                printf("%s\t%d\n", bt, cnt)
            }

            banner = 0
            foreach (lbt in syscall_bts- limit $^arg_syscall_limit :default(2)) {
                total_us = @sum(syscall_bts[lbt])
                if (total_us >= 1000) {
                    if (banner == 0) {
                        printf(">>> Time-consuming syscall backtraces\n")
                        banner = 1
                    }
                    printf("%s\t%d us\n", lbt, total_us)
                }
            }
            printf("\n")
        }

        delete bts
        delete syscall_bts
    }
}

probe kernel.trace("sys_enter") {
    if (tid() == target() && tracing) {
        syscall_start_time = gettimeofday_us()
    }
}

probe kernel.trace("sys_exit") {
    if (tid() == target() && tracing) {
        elapsed = gettimeofday_us() - syscall_start_time
        if (elapsed >= $^arg_syscall_min_elapsed_us :default(400)) {
            syscall_bts[lua_bt()] <<< elapsed
        }
    }
}

Test

# run openresty
/usr/local/openresty/nginx/sbin/nginx -p $PWD -c nginx.conf -g "daemon off;"

# run systemtap script
./samples/blocking-bt.sxx -D STP_NO_OVERLOAD --arg time=60 --skip-badvars -x 144081

# trigger the workload
time curl localhost:10000/run
32000
status=200
{"access-control-allow-credentials":"true","connection":"close","content-type":"application\/json","server":"gunicorn\/19.9.0","access-control-allow-origin":"*","date":"Thu, 29 Sep 2022 10:30:42 GMT","content-length":"287"}

real    0m3.090s
user    0m0.006s
sys     0m0.006s

# then you could see the systemtap output
--- lua_resume block 2424 ms
>>> Hot CPU backtraces
__memcpy_avx_unaligned_erms
@./workload.lua:8
@./workload.lua:18
@./workload.lua:26
=content_by_lua(nginx.conf:17):2
        367
ngx_md5_body
@./workload.lua:16
@./workload.lua:26
=content_by_lua(nginx.conf:17):2
        169
propagatemark
@./workload.lua:8
@./workload.lua:18
@./workload.lua:26
=content_by_lua(nginx.conf:17):2
        38
>>> Time-consuming syscall backtraces
0x0
C:meth_connect
@/usr/local/share/lua/5.1/socket/http.lua:130
@/usr/local/share/lua/5.1/socket/http.lua:354
C:protected_
@./workload.lua:28
=content_by_lua(nginx.conf:17):2
        235173 us
0x0
C:meth_receive
@/usr/local/share/lua/5.1/socket/http.lua:160
@/usr/local/share/lua/5.1/socket/http.lua:362
C:protected_
@./workload.lua:28
=content_by_lua(nginx.conf:17):2
        1472811 us

We get the expected blocking source: the blocking time is about 3 seconds. From the backtrace, we know the blocking comes from MD5 calculation, random string generation and connect()/receive() syscalls.

How to slove blocking?

Since the main thread is blocking sensitive, obviously, we should delegate blocking jobs to other threads.

The nginx provides the thread pool, which is originally used for serving static files more efficiently. I wrap them in lua, so that you could use nonblocking API to execute blocking stuff.

I contribute my works to OpenResty official, that is ngx.run_worker_thread() API:

https://github.com/openresty/lua-nginx-module#ngxrun_worker_thread

Each thread in the thread pool would be associated an individual luajit vm. ngx.run_worker_thread() would send the task to one thread to get executed, yield the coroutine, and when the task get done, it would resume the coroutine.

This API sloves blocking headaches for many companys.

However, it’s far from perfect:

the lua vm allocation is random, for each execution, you get a brand new environment, so no way to cache resident resources, e.g. keepalive tcp connection. Every time, you need to init everything.
the throughput is limited by the number of the thread pool, and context switch overhead can not be ignored.
only small parts (string, hashing, shared dict) of ngx APIs could be used in the separate luajit vm

Could we make a perfect way? So that we could both relax the main thread and make use of ecosystem from other mainstream programming languages?

The answer is lua-resty-nonblocking-ffi library. It provides an efficient, generic and noblocking API to do hybrid programming in openresty with mainstream languages (Go, Python, Java, etc.).

Features:

simple but extensible interface, supports any C ABI compliant language
once and for all, no need to write C/Lua codes to do coupling anymore
high performance, 3~4 times faster than unix domain socket way
shim for python/java, once and for all
any serialization message format you like

This library has been used successfully by several commercial companies for over two years. My customers come from different domains, including AI, IOT, Edge computing.

I would open source this library in near future, please stay tuned.

Conclusion

The blocking is nginx specific issue. Be aware of it, otherwise, you would not benefit high performance from OpenResty.

For the time being, use ngx.run_worker_thread() to offload your blocking tasks. In future, lua-resty-nonblocking-ffi is your best choice.

Analyze Lua Memory Leak With Systemtap

luajit.io@gmail.com (jinhualuo) — Thu, 13 Oct 2022 14:30:37 +0000

Use systemtap to analyze the memory leak of lua code

The memory used by lua code is managed by the GC, not calling malloc/free/mmap directly. The luajit GC uses mark-and-sweep algorithm. In simple words, it links all allocated gc objects in a global list. When the memory is in pressure, it would trigger recycle procedures upon that list. Note that because of single-threading design, the GC is interspersed in the flow of lua code (either compiled or interpretered), e.g. after string allocation. GC would try to check for each gc object if it is still in used, e.g. as an upvalue, reside in a stack, etc. The GC would sweep all dangling gc objects at the end, and the corresponding memory would be freed.

In luajit, lj_alloc_malloc and lj_alloc_realloc would allocate gc objects, and lj_alloc_free would free them. So naturally, we could trace the invocations of these functions to check memory leak.

Note that triggering gc is a bit uncertain and the gc steps would be splited in different time slices. It would not free unused memory immediately. Moreover, the lua code may use caches. So it’s necessary to check the source code to distinguish the real memory leak.

systemtap script

I would use stapxx to analyze lua code running in openresty.

lua-leak.sxx

https://gist.github.com/kingluo/1df4736665a3381ce745f26cf17181d8#file-lua-leak-sxx

...
probe process("$^libluajit_path").function("lj_alloc_realloc").return,
      process("$^libluajit_path").function("lj_alloc_malloc").return
{
    if (gc_size_begin == 0) {
        gc_size_begin = get_gc_size()
    }

    gc_size_end = get_gc_size()

    if (tid() == target() && !quit) {
        ptr = returnval()
        bt = get_bt()
        if (ptr && bt != "") {
            if (alloc_cnt >= 100000) {
                report()
            }

            if (ptr2bt[ptr] == "") {
                alloc_cnt++
            }

            size = @entry($nsize)
            ptr2bt[ptr] = bt
            ptr2size[ptr] = size
            bt_stats[bt] <<< size
        }
    }
}

probe process("$^libluajit_path").function("lj_alloc_free")
{
    ptr = pointer_arg(2)
    if (tid() == target() && ptr && !quit) {
        bt = ptr2bt[ptr]
        delete ptr2bt[ptr]

        bytes = ptr2size[ptr]
        delete ptr2size[ptr]

        if (bt != "") {
            alloc_cnt--
            bt_stats[bt] <<< -bytes
            if (@sum(bt_stats[bt]) == 0) {
                delete bt_stats[bt]
            }
        }
    }
}
...

This script collects all memory not freed during the tracing period and corresponding backtraces, which supports both compiled code and interpretered lua code.

Note that it could not check memory allocated via ffi call. If lua code is clean, you could turn to use sample-bt-leaks.sxx to check possible memory leak outside lua code.

There is an interesting point here. Those alloc/free functions are not only called by the lua code, but also the lua vm itself. We could assume the vm itself has no memory leak, so how to exclude these noises?

In fact, the tapset luajit.sxx already does it for us.

The entrance of lua code from C is either lua_resume(), lua_pcall() or lua_call(). The exit of lua code is either lua_yield() or return.

The laujit vm has mainly below states:

LJ_VMST_INTERP default state
LJ_VMST_C, LJ_VMST_GC already in lua code
compiled already denotes you are in lua code, because each compiled trace are atomic execution.

The LJ_VMST_INTERP is a bit tricky, it does not means the lua vm is actually running lua code. When resume or call, it would set cframe, and when yield or return, cframe would be cleared.

Let’s check vm_x86.dasc (laujit x86/x86_64 assembly interpreter):

  |->vm_resume:				// Setup C frame and resume thread.
  ...
  |  lea KBASEa, [esp+CFRAME_RESUME]
  ...
  |  mov L:RB->cframe, KBASEa
  
  ...
  
  |.ffunc coroutine_yield
  ...
  |  xor RD, RD
  |  mov aword L:RB->cframe, RDa
  ...

The tapset would check the vm states and cframe to determine whether vm is excuting lua code, if not, the backtrace is returned as empty.

if (vmstate == -1 && !$*L->cframe) {
    return ""
}

if (vmstate == -1 || vmstate == -2 || vmstate == -3) {
    base = $*L->base
    bt .= luajit_debug_dumpstack(L, 0, $^arg_depth, base, simple)
}

A real case scenario of memory leak

A user found a memory leak issue.

1. Confirm

We should watch the memory usage in a whole first:

`1`	`watch -n 3 'ps -p 91107 -o cmd,vsz,rss'`

If over times, the memory keeps growing, then let’s check if the leak comes from the lua code:

./samples/lj-gc.sxx -D STP_NO_OVERLOAD -D MAXMAPENTRIES=150000 -D MAXACTION=10000 --skip-badvars -x 91107
...
Total GC count: 24407837 bytes

If the GC memory keeps growing, then it possibly comes from lua code.

2. Narrow down the scope

Use exclusion principle to narrow down the leak scope. If you are lucky, you could even solve the issue at this step.

3. Setup reproduce environment

It’s better to abstract a minimal and stable setup to reproduce the memory leak, e.g. constructing a minimal client workload, excluding unrelated dependencies, mocking a simple upstream server, etc.

4. Use systemtap to analyze

# mock the upstream
/usr/local/openresty/nginx/sbin/nginx -p $PWD -c nginx.conf -g "daemon off;"

# run workload
go run leak.go

# When the leak is stable, run systemtap script
./samples/lua-leaks.sxx -D STP_NO_OVERLOAD -D MAXMAPENTRIES=150000 -D MAXACTION=10000 --skip-badvars -arg time=60 -x 91107

The systemtap script outputs:

 T:@/opt/apisix//deps/share/lua/5.1/prometheus.lua:346
@/opt/apisix//deps/share/lua/5.1/prometheus.lua:346
@/opt/apisix//deps/share/lua/5.1/prometheus.lua:540
@/opt/apisix/apisix/plugins/prometheus/exporter.lua:210
@/opt/apisix/apisix/plugin.lua:884
@/opt/apisix/apisix/plugin.lua:961
@/opt/apisix/apisix/init.lua:314
@/opt/apisix/apisix/init.lua:725
=log_by_lua(nginx.conf:410):0
 : 0x0
        total 544320 bytes
 C:json_decode
C:json_protect_conversion
@/opt/apisix//deps/share/lua/5.1/resty/etcd/v3.lua:770
@/opt/apisix/apisix/core/config_etcd.lua:123
@/opt/apisix/apisix/core/config_etcd.lua:298
@/opt/apisix/apisix/core/config_etcd.lua:538
builtin#22
@/opt/apisix/apisix/core/config_etcd.lua:516
 : 0xc
        total 358888 bytes
 builtin#177
=coroutine_api:7
@/opt/apisix//deps/share/lua/5.1/resty/http.lua:495
 : 0xb
        total 323136 bytes

It points out the possible memory leak sources!

With the hints, I check the source code and figure out the real leak reason:

In nginx-lua-prometheus library, each metric instance maintains a lookup table to cache the full metric name of each label values combination. The prometheus plugin would update metrics at the log phase of each request and in turn update the lookup table. Because the workload generates new routes repeatlly, where the route id in random string each time, and old routes gets discarded, so the lookup table is inserted with new items over and over (and those items would not be freeed even if they are no used anymore), which causes memory leak.

Based on the memory leak reason, we could make a bugfix easily.

Bonus topic: performance difference between systemtap and gdb

It’s been a long history of linux tracing. Linux has many tracing sources. But here we only talk about two of them:

uprobe/kprobe (most popular ones)
ptrace (from the ancient ages)

The relevant front-end tools:

systemtap

The systemtap script has awk-like syntax. The script would be compiled into kernel module. It mainly uses kernel events and uprobe/kprobe as breakpoints.

It makes use of ptrace to hook the target program/process.

In simplicity, interrupts from both uprobe/kprobe and ptrace obey similar procedures:

Replace PC with INT3 (x86 debug instruction)
trigger SIGTRAP, handled by gdb or kernel
access the address space and registers of the target
recovers PC from the original instrcution and continue

Although uprobe/kprobe and ptrace are similar, but they have completely different efficiency.

gdb handles SIGTRAP in user space, interacts with the target in PEEK-POKE way, which involves a lot of context switchs. gdb aims to be an interactive debug tool (the gdb script is also low efficiency).

But systemtap works in kernel space only, and it runs in assembly code.

Let’s take an simple example to visualize the runtime efficiency.

nginx.conf

error_log /dev/stderr info;
worker_processes 1;

events {}

http {
    server {
        listen 10000;

        location /run {
            content_by_lua_block {
                local t = {}
                for i=1,33 do
                    for i=1,500 do
                        table.insert(t, {i=i})
                    end
                end
            }
        }
    }
}

lua-leak.sxx

#!/usr/bin/env stap++

probe process("$^libluajit_path").function("lj_alloc_realloc").return,
      process("$^libluajit_path").function("lj_alloc_malloc").return
{
}

probe process("$^libluajit_path").function("lj_alloc_free")
{
}

probe timer.s($^arg_time)
{
    exit()
}

leak.gdb

set logging off
b lj_gc.c:875
command 1
    continue
end

b lj_alloc_free
command 2
    continue
end

continue

Test:

# run nginx
/usr/local/openresty/nginx/sbin/nginx -p $PWD -c nginx.conf -g "daemon off;"

# run without systemtap and gdb

time curl http://localhost:9080/leak
ok

real    0m0.019s
user    0m0.006s
sys     0m0.006s

# run systemtap

./samples/lua-leaks.sxx -x 36998

time curl http://localhost:9080/leak
ok

real    0m0.119s
user    0m0.007s
sys     0m0.007s

# run gdb
gdb -iex 'set pagination off' -batch-silent -x leak.gdb -p 36998

time curl http://localhost:9080/leak
ok

real    0m16.866s
user    0m0.006s
sys     0m0.006s

systemtap slows down the program by 10x, but gdb is 1600x! Of course, here the workload is dedicated and trigger breakpoints frequently. In other cases, e.g. CPU flamegraph, the systemtap overhead is small and acceptable.

Conclusion

Systemtap is a useful profiling tool to find out and analyze the issues, e.g. CPU flamegraph, context inspect, memory leak, etc.

Gdb Black Magics

luajit.io@gmail.com (jinhualuo) — Thu, 06 Oct 2022 10:55:36 +0000

Without questioin, gdb is a brilliant weapon to debug our program. However, the reality is complex. We need black magics to handle these cases.

Magic1: optimized out?

Your program is normally compiled in optimized mode (e.g. -O2), sometimes even without debug info!

With optimized option, gcc would most likely use registers instead of the stack to store the function arguments and local variables. In some circumstances, gdb would said they are <optimized out>, then you could not inspect their values. Even worse, in the flow of the code, those registers would be overrided, so it’s impossible to check their original values anymore!

Normally, to tackle these issues, you have to recompile your program without optimized options.

But what if you could not? Think about you’re facing your client’s production environment.

You would be frustrated, isn’t it?

Then the only way is to read the assembly codes and inspect everything you need at the right places!

Input registers

From the x86-64 System V ABI manual, these register store the function arguments:

%rdi first argument
%rsi second argument
%rdx third argument
%rcx fourth argument
...

So if the arguments are optimized out, check the registers!

Take an example, ngx_http_core_run_phases runs all phase handlers to handle the http request.

(gdb) b ngx_http_lua_content_handler
Breakpoint 2 at 0x564ab5dcf4e0: file ../ngx_lua-0.10.21/src/ngx_http_lua_contentby.c, line 158.
(gdb) c
Continuing.

Breakpoint 2, ngx_http_lua_content_handler (r=0x564ab7545900) at ../ngx_lua-0.10.21/src/ngx_http_lua_contentby.c:158
158     {
(gdb) bt
#0  ngx_http_lua_content_handler (r=0x564ab7545900) at ../ngx_lua-0.10.21/src/ngx_http_lua_contentby.c:158
#1  0x0000564ab5d178c6 in ngx_http_core_content_phase (r=0x564ab7545900, ph=<optimized out>) at src/http/ngx_http_core_module.c:1277
#2  0x0000564ab5d11bed in ngx_http_core_run_phases (r=r@entry=0x564ab7545900) at src/http/ngx_http_core_module.c:888

The ph is optimized out.

Let’s check the assembly codes:

(gdb) disas /s ngx_http_core_content_phase
Dump of assembler code for function ngx_http_core_content_phase:
src/http/ngx_http_core_module.c:
1270    {
   0x0000564ab5d17890 <+0>:     endbr64
   0x0000564ab5d17894 <+4>:     push   %rbp
   0x0000564ab5d17895 <+5>:     mov    %rdi,%rbp
   0x0000564ab5d17898 <+8>:     push   %rbx
   0x0000564ab5d17899 <+9>:     sub    $0x38,%rsp
   0x0000564ab5d1789d <+13>:    mov    %fs:0x28,%rax
   0x0000564ab5d178a6 <+22>:    mov    %rax,0x28(%rsp)
   0x0000564ab5d178ab <+27>:    xor    %eax,%eax

1271        size_t     root;
1272        ngx_int_t  rc;
1273        ngx_str_t  path;
1274
1275        if (r->content_handler) {
   0x0000564ab5d178ad <+29>:    mov    0x498(%rdi),%rax
   0x0000564ab5d178b4 <+36>:    test   %rax,%rax
   0x0000564ab5d178b7 <+39>:    je     0x564ab5d178f0 <ngx_http_core_content_phase+96>

1276            r->write_event_handler = ngx_http_request_empty_handler;
   0x0000564ab5d178b9 <+41>:    lea    0x20c0(%rip),%rdx        # 0x564ab5d19980 <ngx_http_request_empty_handler>
   0x0000564ab5d178c0 <+48>:    mov    %rdx,0x38(%rdi)

1277            ngx_http_finalize_request(r, r->content_handler(r));
   0x0000564ab5d178c4 <+52>:    callq  *%rax
=> 0x0000564ab5d178c6 <+54>:    mov    %rbp,%rdi
   0x0000564ab5d178c9 <+57>:    mov    %rax,%rsi
   0x0000564ab5d178cc <+60>:    callq  0x564ab5d1cf60 <ngx_http_finalize_request>

You could see that gdb does not save %rsi into the stack, so at the frame position => 0x0000564ab5d178c6 <+54>: mov %rbp,%rdi, it said ph is unknown, but in fact, luckily, the %rsi has not been overrided, so just print it!

(gdb) p /x $rsi
$2 = 0x564ab7571a60

Someone would say why do you think you get the correct value?

Well, let’s break at ngx_http_core_content_phase directly and retrigger the breakpint again:

(gdb) b ngx_http_core_content_phase
Breakpoint 1 at 0x564ab5d17890: file src/http/ngx_http_core_module.c, line 1270.
(gdb) c
Continuing.

Breakpoint 1, ngx_http_core_content_phase (r=0x564ab7545900, ph=0x564ab7571a60) at src/http/ngx_http_core_module.c:1270
1270    {
(gdb) bt 3
#0  ngx_http_core_content_phase (r=0x564ab7545900, ph=0x564ab7571a60) at src/http/ngx_http_core_module.c:1270
#1  0x0000564ab5d11bed in ngx_http_core_run_phases (r=r@entry=0x564ab7545900) at src/http/ngx_http_core_module.c:888
#2  0x0000564ab5d11cc4 in ngx_http_handler (r=r@entry=0x564ab7545900) at src/http/ngx_http_core_module.c:871
(More stack frames follow...)

In this case, the ph is not optimized out, because it’s the entry of ngx_http_core_content_phase. We could verify we get the correct value.

As a side note, gdb sometimes behaves jumpy for optimized out variables, so if we have trouble, we may change a suitable breakpoint and we may have surprise there.

Check the stack or history

Sometimes at some position, the register get overrided, and gdb reports a fake <optimized out>.

Take ngx_http_lua_run_thread as example:

Breakpoint 1, ngx_http_lua_run_thread (L=L@entry=0x7f64d244f380, r=r@entry=0x564ab7545900, ctx=ctx@entry=0x564ab7546700, nrets=nrets@entry=0) at ../ngx_lua-0.10.21/src/
ngx_http_lua_util.c:1112
1112    {
(gdb) p nrets
$1 = 0
(gdb) n
1123        ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
(gdb)
1128        lua_atpanic(L, ngx_http_lua_atpanic);
(gdb)
1130        NGX_LUA_EXCEPTION_TRY {
(gdb)
1140            if (ctx->cur_co_ctx->thread_spawn_yielded) {
(gdb)
1155                old_pool = ngx_http_lua_pcre_malloc_init(r->pool);
(gdb) p nrets
$2 = <optimized out>

Check the assembly codes:

(gdb) disas ngx_http_lua_run_thread,+300
Dump of assembler code from 0x564ab5dcb2a0 to 0x564ab5dcb3cc:
...
   0x0000564ab5dcb2c3 <ngx_http_lua_run_thread+35>:     mov    %ecx,0x5c(%rsp)
...
   0x0000564ab5dcb352 <ngx_http_lua_run_thread+178>:    mov    0x30(%rsp),%rcx
   0x0000564ab5dcb357 <ngx_http_lua_run_thread+183>:    mov    0x18(%rcx),%rcx
...
   0x0000564ab5dcb39d <ngx_http_lua_run_thread+253>:    nopl   (%rax)
=> 0x0000564ab5dcb3a0 <ngx_http_lua_run_thread+256>:    mov    (%rsp),%rax
   0x0000564ab5dcb3a4 <ngx_http_lua_run_thread+260>:    mov    0x58(%rax),%rdi
   0x0000564ab5dcb3a8 <ngx_http_lua_run_thread+264>:    callq  0x564ab5dd1e60 <ngx_http_lua_pcre_malloc_init>

The %ecx (the fourth argument, remember?) gets overrided, but 0x5c(%rsp) also stores the nrets value.

(gdb) p /x *(int*)(int64_t *)($rsp+0x5c)
$3 = 0x0

Bingo!

But wait, how about both stack and register get overrided? Well, let’s use record magic! We could take the time machine to find out the historical value we need.

Breakpoint 1, ngx_http_lua_run_thread (L=L@entry=0x7f64d244f380, r=r@entry=0x564ab7545900, ctx=ctx@entry=0x564ab7546700, nrets=nrets@entry=0) at ../ngx_lua-0.10.21/src/ngx_http_lua_util.c:1112
1112    {
(gdb) record
(gdb) n
1123        ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
(gdb)
1128        lua_atpanic(L, ngx_http_lua_atpanic);
(gdb)
1130        NGX_LUA_EXCEPTION_TRY {
(gdb)
1140            if (ctx->cur_co_ctx->thread_spawn_yielded) {
(gdb)
1155                old_pool = ngx_http_lua_pcre_malloc_init(r->pool);
(gdb) p nrets
$4 = <optimized out>
(gdb) reverse-next
1140            if (ctx->cur_co_ctx->thread_spawn_yielded) {
(gdb) p nrets
$5 = 0

Inline function

Inline function has no function frame, the function call does not really happen. gdb just simulate the call. In optimized code, there are two issues:

function arguments are optimized out
break at weird instruction

Let’s take a practical example:

(gdb) b lj_mem_free
Breakpoint 1 at 0x7fb7daa7178f: lj_mem_free. (40 locations)
(gdb) c
Continuing.

Breakpoint 1, lj_mem_free (osize=<optimized out>, p=<optimized out>, g=<optimized out>) at lj_tab.c:291
291         lj_mem_freevec(g, oldnode, oldhmask+1, Node);
(gdb) bt 3
#0  lj_mem_free (osize=<optimized out>, p=<optimized out>, g=<optimized out>) at lj_tab.c:291
#1  lj_tab_resize (L=0x7fb7d95e3380, t=0x7fb7d95fb800, asize=<optimized out>, hbits=<optimized out>) at lj_tab.c:291
#2  0x00007fb7daa73e9e in rehashtab (L=L@entry=0x7fb7d95e3380, t=t@entry=0x7fb7d95fb800, ek=ek@entry=0x7ffc0f6d6688) at lj_tab.c:369
(More stack frames follow...)

lj_mem_free is an inline function.

static LJ_AINLINE void lj_mem_free(global_State *g, void *p, size_t osize)
{
  g->gc.total -= (GCSize)osize;
  g->allocf(g->allocd, p, osize, 0);
}

lj_tab_resize is the caller.

/* Resize a table to fit the new array/hash part sizes. */
void lj_tab_resize(lua_State *L, GCtab *t, uint32_t asize, uint32_t hbits)
{
    ...
    g = G(L);
    lj_mem_freevec(g, oldnode, oldhmask+1, Node);
  }
}

#define lj_mem_freevec(g, p, n, t)	lj_mem_free(g, (p), (n)*sizeof(t))

You could see all arguments are optimized out, and even the argument order is wrong.

Let’s dig in the assembly code:

(gdb) disas lj_tab_resize
Dump of assembler code for function lj_tab_resize:
...
   0x00007fb7daa73b29 <+617>:   mov    0xc(%rsp),%esi
   0x00007fb7daa73b2d <+621>:   mov    0x10(%rbp),%rax
=> 0x00007fb7daa73b31 <+625>:   xor    %ecx,%ecx
   0x00007fb7daa73b33 <+627>:   lea    0x1(%rsi),%edx
   0x00007fb7daa73b36 <+630>:   mov    0x8(%rax),%rdi
   0x00007fb7daa73b3a <+634>:   mov    %r15,%rsi
   0x00007fb7daa73b3d <+637>:   lea    (%rdx,%rdx,2),%rdx
   0x00007fb7daa73b41 <+641>:   shl    $0x3,%rdx
   0x00007fb7daa73b45 <+645>:   sub    %rdx,0x10(%rax)
   0x00007fb7daa73b49 <+649>:   mov    (%rax),%rax
   0x00007fb7daa73b4c <+652>:   add    $0x28,%rsp
   0x00007fb7daa73b50 <+656>:   pop    %rbx
   0x00007fb7daa73b51 <+657>:   pop    %rbp
   0x00007fb7daa73b52 <+658>:   pop    %r12
   0x00007fb7daa73b54 <+660>:   pop    %r13
   0x00007fb7daa73b56 <+662>:   pop    %r14
   0x00007fb7daa73b58 <+664>:   pop    %r15
   0x00007fb7daa73b5a <+666>:   jmpq   *%rax
...

It’s interesting that lj_mem_free also calls g->allocf, which points to lj_alloc_f, so lj_tab_resize actually prepare the arguments for lj_alloc_f, and lj_mem_free is embedded in the between.

It breaks at xor %ecx,%ecx, in this time point, no arguments are prepared, so gdb breaks at the wrong place! For example, the void *p is %rsi, but currectly not set. But from the codes, you could see that it woule be set from r15. So the simplest way to check *p is:

(gdb) p /x $r15
$2 = 0x7fb7d95a9620
(gdb) f 1
#1  lj_tab_resize (L=0x7fb7d95e3380, t=0x7fb7d95fb800, asize=<optimized out>, hbits=<optimized out>) at lj_tab.c:291
291         lj_mem_freevec(g, oldnode, oldhmask+1, Node);
(gdb) p oldnode
$3 = (Node *) 0x7fb7d95a9620

Jump to the parent frame (frame 1), we could validate it’s correct.

But sometimes, the mapping between register and argument is not so obvious. We need to step forward a bit and check the value again. jmpq *%rax is the instruction to call lj_alloc_f, so let’s break it.

(gdb) b *0x00007fb7daa73b5a
Breakpoint 2 at 0x7fb7daa73b5a: file lj_gc.h, line 122.
(gdb) c
Continuing.

Breakpoint 2, 0x00007fb7daa73b5a in lj_mem_free (osize=48, p=0x7fb7d95a9620, g=<optimized out>) at lj_gc.h:122
122       g->allocf(g->allocd, p, osize, 0);

Now you get the value of *p! Amazing, right? :)

Magic2: gdb script

gdb has builtin scripting. You could use it to define variables to r/w the program symbols and do some control flows upon them. It’s really useful when you want to inspect complex context infomation.

Take a practical example, a few weeks ago, I found a bug in openresty.

If you do ngx.say without ngx.flush, and use cosockets to send messages, the message buffers would be appended to the busy buffer list and do not get released until the main request finish. The busy buffer list would become bigger and bigger, and slow down the send operation dramatically.

ngx_chain_update_chains has a buggy loop (verified via perf top):

       │    for (cl = *busy; cl->next; cl = cl->next) { /* void */ }
  5.95 │20:   mov     %rax,%r9
  0.02 │      mov     0x8(%rax),%rax
 93.94 │      test    %rax,%rax
       │    ↑ jne     20

The busy buffer list is shared by the main request and cosockets created from the same main request. If the head part of the list (unflushed ngx.say) keeps still, the following of the list would not be cleared and get accumulated when you send more and more message via the cosocket).

How to verify I am correct? Someone may say how about adding codes to openresty?

But how to achieve the same goal in non-intrusive way?

Because mostly, we could not touch the source codes of the target process. Think about you’re facing the client’s environment

The answer is gdb script:

b ngx_chain_update_chains
    command 1
    set $v = *busy
    set $cnt = 0
    while ($v != 0)
        set $cnt++
        set $v=$v->next
    end
    p $cnt
    continue
end

Script outout:

Breakpoint 1, ngx_chain_update_chains (p=0x55a53413e850, free=free@entry=0x55a53413f7b8, busy=busy@entry=0x55a53413f7c0, out=out@entry=0x7f80f353fe50, tag=0x55a5326887a0 <ngx_http_lua_module>) at src/core/ngx_buf.c:187
187     {
$11 = 80399

Breakpoint 1, ngx_chain_update_chains (p=0x55a53413e850, free=free@entry=0x55a53413f7b8, busy=busy@entry=0x55a53413f7c0, out=out@entry=0x7f80f353fe50, tag=0x55a5326887a0 <ngx_http_lua_module>) at src/core/ngx_buf.c:187
187     {
$12 = 80400

Breakpoint 1, ngx_chain_update_chains (p=0x55a53413e850, free=free@entry=0x55a53413f7b8, busy=busy@entry=0x55a53413f7c0, out=out@entry=0x7f80f353fe50, tag=0x55a5326887a0 <ngx_http_lua_module>) at src/core/ngx_buf.c:187
187     {
$13 = 80401

With a simple script, the reason gets confirmd, and we do not touch any source codes!

Magic3: gdb python

Besides basic scripting, if you want more magic, use python!

Python could be used to define customized command, breakpoint or function.

Here I just take a function example. Assume I need to ignore some breakpoint events if the backtrace contains some functions.

py
class should_skip_f(gdb.Function):
    def __init__ (self):
        super (should_skip_f, self).__init__("should_skip")

    def invoke(self):
        try:
            frame = gdb.selected_frame()
            while frame is not None:
                if "lj_BC_FUNCC" in frame.name():
                    return False
                frame = frame.older()
        except Exception as e:
            print(e)
            return True
        return True
end
py should_skip_f()

b lj_gc.c:875 if !$should_skip()
    command 1
    ...
end

If the breakpint event get skipped, the commands attached to the breakpint would not get executed, pefect!

Magic4: set source path

Mostly it’s necessary to map the debug steps to the lines of source codes.

In fact, normally, your binary, if not stripped, would contains DWARF sections, which contains the path of the source codes. So all you need to do is to download the matching source codes and specify a correct perfix.

Determine the DW_AT_comp_dir from the DWARF of the binary first:

1
2

objdump -Wi /usr/local/openresty-debug/nginx/sbin/nginx |grep DW_AT_comp_dir -m 1
    <15>   DW_AT_comp_dir    : (indirect string, offset: 0x1b20): /tmp/tmp.FthtgW2A3R/openresty-1.21.4.1/build/nginx-1.21.4

Substitute the prefix with the path of the corresponding source codes:

~/.gdbinit

set substitute-path /tmp/tmp.FthtgW2A3R /path/to/source

Conclusion

gdb is a powerful tool, when we encounter issues, just stay clarm and think about if we make a full use of gdb or not. Assembly codes, scripting and python extentions are your friends.

Luajit String Interning

luajit.io@gmail.com (jinhualuo) — Wed, 05 Oct 2022 13:58:25 +0000

Why string interning?

String Interning is a common optimization in most of programming languages.

As the wiki said:

In computer science, string interning is a method of storing only one copy of each distinct string value, which must be immutable.[1] Interning strings makes some string processing tasks more time- or space-efficient at the cost of requiring more time when the string is created or interned. The distinct values are stored in a string intern pool.

It has below advantages in luajit:

compare string via pointers directly
use string as table key efficiently, because the hash is calculated at alloc stage and use pointer as uniqueness
save memory for duplicated strings

Drawback

Meanwhile, it brings in extra time-consuming tasks:

Hash
when collision, memcmp all strings in the same hash slot
rehash when collision is high and/or load factor is low

No matter whether you want to use string as table key or not, whether you want to do comparison or not, it would do string interning for all string allocations. So it would not suitable for below usecase:

You want to cache strings in openresty, e.g. save most frequent used redis key-values in memory. In this case, it would cause a lot of CPU in memcmp and rehash, especially when the size of value is big and similar.

When I work in Kugou Music, we save a lot of file segments in redis cluster (each segment is 1~4 MBytes in average). When we need to cache them in memory or shared memory, the string interning blocks our way.

Someone would say that why not store string in ffi cdata? In fact, you could not avoid storing data in string, especially in OpenResty, because almost all data comes from cosocket, and cosocket is implemented in standard lua API and only support string, no ffi there. That is, no matter which data source, even from C ffi, you have to convert it into string before you could communicate with the outer world.

In contrast, most of mainstream programming languages only do string interning for literals or constant expression.

For example, Python only interns for “Variables, Constants, and Function Names” and “Dictionary Keys”. It would distinguish whether the string is interned or not, and you could intern it explicitly.

Check the source code

/* Intern a string and return string object. */
GCstr *lj_str_new(lua_State *L, const char *str, size_t lenx)
{
  global_State *g = G(L);
  if (lenx-1 < LJ_MAX_STR-1) {
    MSize len = (MSize)lenx;
    StrHash hash = hash_sparse(g->str.seed, str, len);
    MSize coll = 0;
    int hashalg = 0;
    /* Check if the string has already been interned. */
    GCobj *o = gcref(g->str.tab[hash & g->str.mask]);
#if LUAJIT_SECURITY_STRHASH
    if (LJ_UNLIKELY((uintptr_t)o & 1)) {  /* Secondary hash for this chain? */
      hashalg = 1;
      hash = hash_dense(g->str.seed, hash, str, len);
      o = (GCobj *)(gcrefu(g->str.tab[hash & g->str.mask]) & ~(uintptr_t)1);
    }
#endif
    while (o != NULL) {
      GCstr *sx = gco2str(o);
      if (sx->hash == hash && sx->len == len) {
	if (memcmp(str, strdata(sx), len) == 0) {
	  if (isdead(g, o)) flipwhite(o);  /* Resurrect if dead. */
	  return sx;  /* Return existing string. */
	}
	coll++;
      }
      coll++;
      o = gcnext(o);
    }
#if LUAJIT_SECURITY_STRHASH
    /* Rehash chain if there are too many collisions. */
    if (LJ_UNLIKELY(coll > LJ_STR_MAXCOLL) && !hashalg) {
      return lj_str_rehash_chain(L, hash, str, len);
    }
#endif
    /* Otherwise allocate a new string. */
    return lj_str_alloc(L, str, len, hash, hashalg);
  } else {
    if (lenx)
      lj_err_msg(L, LJ_ERR_STROV);
    return &g->strempty;
  }
}

First, it would hash the string with hash_sparse. This function only picks some bytes from the string and do a quick hash and determine the hash slot in the hash array. It’s fast but the hash collision is high.

Second, compare with each string in the hash slot (linked list), and try to find the target. If there is too many collision in this step, it would switch to use hash_dense to rehash all strings in that hash slot, and mark them in dense mode (if new string comes in the first step and found dense hash slot, it would hash it with hash_dense in the early first step).

hash_dense iterates all characters of the strings to calculate the hash value, which is more expensive than hash_sparse.

Finally, if the same string is found in the pool, just returns. Otherwise, it would allocate a new string object. At this step, it may enlarge the hash array size if the load factor is not 100%.

Now, we get possible time-consuming places:

if your have many similar strings, then hash_sparse would results in the same hash value, then causes high hash collisions, do a lot of expensive memcmp and switch to use expensive hash_dense
When the number of strings increases, luajit would reallocate a array vector (twice of the old size), and keep in mind that it would rehash all exist strings in the current table!

So avoid similar and big strings and warm up the cache at startup.

Analyze the overhead of string interning

How expensive is memcmp?

memcmp_perf.c

    ...
    const int SZ = 1024*1024*1;
    char* foo = malloc(SZ);
    randstr(foo, SZ);
    char* bar = malloc(SZ);
    memcpy(bar, foo, SZ);
    bar[SZ-1] = 'A';
    int cnt = 0;
    struct timeval  tv1, tv2;
    gettimeofday(&tv1, NULL);
    for (int i = 0; i < atoi(argv[1]); i++) {
        cnt += memcmp(foo, bar, SZ);
    }
    gettimeofday(&tv2, NULL);
    ...

In a signle CPU (Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz), for each string with 1MB, only the last char is different, it could only do memcmp 90 times per second at most.

How expensive is hash collision?

We construct a workload to simulate the cache usecase (only set kv, no get).

Each http request inserts a key-value in the lua table. The key is integer and the value is string of 1MB size. Note that the key is not string, but luajit still does string interning for each string.

The http client would choose to use unique random strings or similar strings.

We use systemtap script to check the lj_str_new time cost differences.

nginx.conf

https://gist.github.com/kingluo/ca30b8f3b76d2af95f98dde93b91d049#file-nginx-conf

error_log /dev/stderr info;
worker_processes 1;

events {}

http {
    server {
        client_max_body_size 2M;
        client_body_buffer_size 2M;
        listen 10000;

        location /run {
            content_by_lua_block {
                require("workload").run()
            }
        }

        location /clean {
            content_by_lua_block {
                require("workload").clean()
            }
        }
    }
}

workload.lua

https://gist.github.com/kingluo/ca30b8f3b76d2af95f98dde93b91d049#file-workload-lua

local cjson = require("cjson")

local _M = {}

local MAX_CNT = 800
local DEL = 300

local cnt = 0
local cache = {}

function _M.run()
    ngx.req.read_body()
    local data = ngx.req.get_body_data()
    assert(data)

    local kv = cjson.decode(data)
    if not cache[kv.k] then
        cnt = cnt + 1
        if cnt > MAX_CNT then
            local del = DEL
            cnt = cnt - del
            for k, v in pairs(cache) do
                cache[k] = nil
                del = del - 1
                if del == 0 then
                    break
                end
            end
        end
    end
    cache[kv.k] = kv.v
end

function _M.clean()
    cache = {}
    cnt = 0
    collectgarbage()
end

return _M

bench.go

https://gist.github.com/kingluo/ca30b8f3b76d2af95f98dde93b91d049#file-bench-go

...
if *uniqFlag {
    kv.Value = randomString(MAX_STR_SIZE)
} else {
    buf[j%len(buf)] = byte(int('a') + j)
    kv.Value = string(buf)
}
data, err := json.Marshal(kv)
if err != nil {
    panic(err)
}
start := time.Now()
resp, err := http.Post(url, "text/plain", bytes.NewBuffer(data))
if err != nil {
    panic(err)
}
if resp.StatusCode != 200 {
    panic("post failed")
}
elapsed := time.Now().Sub(start)
...

lj_str_new.sxx

https://gist.github.com/kingluo/ca30b8f3b76d2af95f98dde93b91d049#file-lj_str_new-sxx

...
probe process("$^libluajit_path").function("lj_str_new") {
    if (tid() == target()) {
        start_time = gettimeofday_us()
    }
}

probe process("$^libluajit_path").function("lj_str_new").return {
    if (tid() == target()) {
        if (start_time) {
            latencies <<< gettimeofday_us() - start_time
        }
    }
}
...

Test

Let’s check the scenario of similar strings first:

# In terminal1
/usr/local/openresty/nginx/sbin/nginx -p $PWD -c nginx.conf -g "daemon off;"

# In terminal2
# get the pid of nginx worker process with `ps auxf |grep nginx`, e.g. 172147
./samples/lj_str_new.sxx -D STP_NO_OVERLOAD --arg time=30 --skip-badvars -x 172147
...
min/avg/max: 5/813/78809 us

# In terminal3
go run bench.go
...
avg rtt: 22014 microsec

The average of lj_str_new is almost 1ms!

Now, check the scenario of unique strings:

./samples/lj_str_new.sxx -D STP_NO_OVERLOAD --arg time=60 --skip-badvars -x 172147
...
min/avg/max: 5/653/5996 us


go run bench.go -uniq
...
avg rtt: 13745 microsec

The unique random strings cause less hash collision and so that less memcmp, so the result is better.

Could we do dynamic string interning?

We could do string interning only when the string is used to:

use as table key
string comparison

In fact, the lua official has an idea too: http://lua-users.org/wiki/SpeedingUpStrings

However, you need to modify a lot of codes in luajit.

Interpreter

luajit uses dynasm to generate interpreter for different CPU Architecture. So you need to change each interpreter file.

Take vm_x86.dasc as example:

    if (op == BC_ISEQV || op == BC_ISNEV) {
      ...
      |  // Same types and not a primitive type. Compare GCobj or pvalue.
      |  mov RA, [BASE+RA*8]
      |  mov RD, [BASE+RD*8]
      |  cmp RA, RD
      |  je <1				// Same GCobjs or pvalues?

You could see that it uses the address to do comparison directly.

Table key

Note that here strV gets address of the string object.

TValue *lj_tab_setstr(lua_State *L, GCtab *t, const GCstr *key)
{
  TValue k;
  Node *n = hashstr(t, key);
  do {
    if (tvisstr(&n->key) && strV(&n->key) == key)
      return &n->val;
  } while ((n = nextnode(n)));
  setstrV(L, &k, key);
  return lj_tab_newkey(L, t, &k);
}

At sweep stage of GC (gc_sweepstr), it would iterate the string pool to remove all dangle objects. If we do not intern the strings, then those strings must be stored elsewhere and handle it in GC too. Moreover, we need an extra pointer to point to the main GC object, and flag to check the object if interned.

Tracing and compilation

This is the most difficult part, check lj_obj_equal and lj_record_ins.

There may be other trivial places to change.

All in all, luajit assumes and hardcodes string interning everywhere. It is a tough job to optimize.

Conclusion

String interning in luajit is rough and straightforward. You must be careful to avoid similar strings and hash collision to get better performance. If luajit supports dynamic string interning in future, it would be great.