Writing a simple Javascript async message queue server - Part III

Marton Trencseni - Mon 17 June 2024 - Javascript

Introduction

In previous posts, I wrote a simple async message queue server in Python, C++ and Javascript, and a library of unit tests:

In this final post on the toy Javascript async message queue server implementation, I make it compatible with the Python version. I use the library of unit tests developed previously to ensure identical behaviour between the two codebases.

The code is up on Github.

Improvements

I want to make all 3 versions (Python, C++ and Javascript) wire-compatible, so I standardized on JSON messages. Overall, the changes I made in this final version:

JSON wire protocol
support the following commands:
- subscribe
  - last_seen (optional, int, default 0): if supplied, only send messages not yet seen
  - cache (optional, bool, default true): if supplied and true, don't send caches messages on subscribe
- unsubscribe
- send
  - delivery: all or one
  - cache (optional, bool, default true): if supplied and false, do not cache this messages, only send to currently connected subscribers, overriding delivery semantics
server always sends JSON responses, including error messages
add reasonable error handling to the code, but avoid cascading exceptions where possible
add Python type hints
refactor code for small wins

Porting from Python

Since the code base was ported from Python, two helper classes defaultdict and deque that exist in Python had to be hand-coded:

class DefaultDict {
    constructor(defaultInit) {
        return new Proxy({}, {
            get: (target, name) => name in target ?
                target[name] :
                (target[name] = typeof defaultInit === 'function' ?
                    defaultInit() :
                    defaultInit)
        });
    }
}

class Deque extends Array {
    constructor(maxlen) {
        super();
        this.maxlen = maxlen;
    }

    push(...args) {
        if (args.length + this.length > this.maxlen) {
            this.splice(0, args.length + this.length - this.maxlen);
        }
        return super.push(...args);
    }
}

Code similarity

The code ended up being very similar to the Python code. To illustrate this, I will show the mini-architecture for validating JSON. In both languages, I use a dictionary to define valid JSON objects. First, in Python:

template_send: Dict[str, Dict[str, Union[type, bool, list]]] = {
    "command": {"type": str, "required": True},
    "topic": {"type": str, "required": True},
    "msg": {"type": str, "required": True},
    "delivery": {"type": str, "required": True, "values": ["all", "one"]},
    "cache": {"type": bool, "required": False},
}

The same in Javascript:

const template_send = {
    command: { type: "string", required: true },
    topic: { type: "string", required: true },
    msg: { type: "string", required: true },
    delivery: { type: "string", required: true, values: ["all", "one"] },
    cache: { type: "boolean", required: false }
};

The helper function to validate objects, first in Python:

def template_match(template: Dict[str, Dict[str, Any]], obj: Dict[str, Any]) -> bool:
    if not set(obj.keys()).issubset(template.keys()):
        return False
    for key, props in template.items():
        if props["required"] and key not in obj:
            return False
        if key in obj and not isinstance(obj[key], props["type"]):
            return False
        if "values" in props and obj[key] not in props["values"]:
            return False
    return True

The same in Javascript:

function templateMatch(template, obj) {
    const objKeys = new Set(Object.keys(obj));
    const templateKeys = new Set(Object.keys(template));
    if (![...objKeys].every(key => templateKeys.has(key)))
        return false;
    for (const [key, props] of Object.entries(template)) {
        if (props.required && !(key in obj))
            return false;
        if (key in obj && typeof obj[key] !== props.type)
            return false;
        if ('values' in props && !props.values.includes(obj[key]))
            return false;
    }
    return true;
}

Interesting aspects of the code

Overall, noteworthy or interesting things about the code:

Custom Data Structures: The DefaultDict class emulates Python's defaultdict, dynamically creating default values for undefined keys. This is particularly useful for managing collections like topics, topics_reverse, caches, and indexs without worrying about initialization.
Deque Implementation: The Deque class extends the native JavaScript array to support a maximum length, automatically removing the oldest elements when the limit is exceeded. This is useful for caching where you want to maintain a fixed size of recent items.
UTF-8 Decoding: The use of StringDecoder ensures that the byte streams are correctly interpreted as UTF-8 encoded strings. This is crucial for correctly handling multi-byte characters in network communication.
Command Handling Architecture: The code uses a map of handlers (handlers) to dynamically dispatch different commands (subscribe, unsubscribe, send). This design pattern makes the code extensible and easy to manage, as adding new commands only requires adding a new handler function.
Template Matching for Validation: The templateMatch function validates command objects against predefined templates, ensuring they have the correct structure and types. This provides a robust mechanism for input validation, crucial for preventing malformed data from causing runtime errors.
Graceful Error Handling: By listening to the error event on sockets and implementing the unhandledRejection handler, the code robustly handles runtime errors and unhandled promise rejections. This helps maintain server stability and provides useful logging for debugging.
Dynamic Caching Control: The sendCached function not only sends cached messages but also dynamically manages the cache based on command properties. This ensures that clients receive relevant historical data while keeping the cache within its size limit.
Client Cleanup: The cleanupSocket function centralizes the logic for removing a client's subscriptions and reversing the topics upon disconnection. This avoids code duplication and ensures that resources are freed correctly.
Command Verification: The verifyCommand function checks if commands conform to their respective templates before processing them. This step is critical for ensuring that only valid commands are executed, enhancing security and stability.
Port and Cache Size Parsing: The code robustly parses and validates command-line arguments for the port and cache size, falling back to defaults if necessary. This makes the server configuration flexible while preventing invalid configurations from causing issues.

Unit tests

I used the unit tests from the previous post to verify that the server is correct and matches the Python version. As I was working on this, I also made minor improvements to the unit tests themselves. I added a --target flag, so both Javascript and Python server's can be tested:

$ pytest-3 -v unittests.py --target=javascript --full-trace -x

Conclusion

My goal with writing these toy message queue servers is to keep myself somewhat sharp as a programmer on mainline programming languages. Also, I wanted to learn how to write async code in some of these languages, as I have only written async servers in (old style) C++ before. I was also curious to see how similar these implementations end up! The Python and Javascript versions ended up very similar, more than I anticipated.

It's also interesting how good ChatGPT has become with the latest ChatGPT-4o version. Here is the log of using ChatGPT for pair-programming, to get it to translate the Python code to Javascript, and write the 10 points above.