Router Concurrency

Rules

The router is non-blocking in that:

1. Reading and writing one session will never block another session.
2. Writing to a session will not affect reading from that same session.
3. Reading from a session will not affect writing to that same session.

This means that:

• Different publishers can dispatch concurrently to one subscriber.
• One publisher can dispatch concurrently to different subscribers.
• One publisher dispatches serially to one subscriber.
• Different callers can concurrently call a callee.

Design Concept

• Floorplan Diagram

The nexus WAMP router operates as a set of concurrent processing stages, that does not block on I/O at any stage (src-client -> router/realm -> broker -> sendQueue -> dst-client). Messages received from clients are dispatched to the appropriate handlers for routing and are then written to the outbound message queues of the receiving clients. This way the router never delays processing of messages waiting for I/O with clients.

      Router
        Reaml1
          sessions
  invk <--- session1............(outHandler) <--------+
   evt <--- session2............(outHandler) <-----+  |
   evt <--- session3............(outHandler) <--+  |  |
   pub ---> session4........(inHandler)         |  |  |
  call ---> session5..(inHandler)  |            |  |  |
                          |        |pub         |  |  |
                      call|        |            |  |  |
                          |        V            |  |  |
          Broker..........)..(Handler)          |  |  |
            subscribers   |     |   |     event |  |  |
              *session2   |     |   +-----------+  |  |
              *session3   |     +------------------+  |
                          |                           |
                          V            invocation     |
          Dealer.......(Handler)----------------------+
            callees
              *session1

          Roles
            callee: session1
            subscriber: session2, session3
            publisher: session4
            caller: session5

Operation

Each (xHandler) is a singe goroutine+channel that persists for the lifetime of the associated object. Therefore, order is preserved for messages between any two sessions regardless of goroutine scheduling order.

Broker and Dealer cannot dispatch an event to a new goroutine, because this would mean that messages bound for the same destination session would be sent in separate goroutines, and delivery order would be affected by goroutine scheduling order. Instead they dispatch to the receiving session's out channel+goroutine.

Broker and Dealer have their own handler goroutines to:

1. Safely access subscription or call maps.
2. Not make session's input-handler wait for Broker when it could be dispatching other messages to Dealer.

NOTE: Sessions have separate in (recv) and out (send) handlers so that processing incoming messages is not blocked while waiting for outgoing messages to finish being sent. Also, the router is not blocked waiting for a session to send a message to a client.

Handling Message Overflow

If there are too many messages sent to the same session before the session can consume them, the session's channel that the router is writing may become full and block. When this happens the broker or dealer will need to drop messages to be able to continue processing.

The current implementation drops messages in this situation. If a RPC invocation message is dropped, an error is returned to the caller.

This may be addressed by putting the outgoing messages on a dynamically sized output queue for the session. See: https://github.com/gammazero/bigchan. This approach was not chosen as dropping messages will not lead to unbounded memory use.