Apache Qpid Dispatch Router waypoints

What are waypoints?

A waypoint is an intermediate stopping point in your journey. You fly from Miami to Hong Kong but decide to stop over in London. You can consider London to be your waypoint before you reach your final destination which is Hong Kong. You can have more than one waypoint if you like. You can fly from Miami to London to Vienna to Shanghai and finally to Hong Kong. The more waypoints the merrier.

What are Qpid Dispatch Router waypoints?

You can set up waypoints in the Dispatch Router and have messages travel from source to destination making one or more stops along the way. For example, the message source can be a producer and the message destination can be a consumer and the waypoints being one or more brokers which have store and forward capability.

Waypoint Scenarios

We will explore a few waypoint scenarios. A simple scenario will help you understand how waypoints work. Progressively more complex scenarios demonstrate how router waypoints solve tough messaging problems.

Single Router vs Network of Routers

Waypoints operate across a network of routers the same as they do for a single router. In the following scenarios the simplest examples will show only a single router. More complex scenarios that follow will feature a network of routers.

Scenario 1 - Using the broker as a waypoint

The topology (Figure 1) consists of a router, a broker and a producer/consumer. The producer and consumer connect to the router not the broker. The broker has a work_orders queue that stores all the work orders. The producer produces to the work_orders queue and the consumer consumes from the same queue. The store-and-forward broker acts as a waypoint for a message that travels from producer to consumer. The broker queue is our London. Get it ? In Figure 1, observe how the messages traveld from the producer to the consumer via the broker's work_orders queue.

Figure 1

Figure 2 is similar to Figure 1 but it shows the internal address used by the router to make waypoints work. A quick look at addressing scheme that is internal to the router is coming up next.

Figure 2

Let's carefully look at the router configuration file that helps make this scenario possible.

router { mode: edge # mode can be standalone as well. } # The consumer and producer will connect on this listener # and start consuming and producing messages. listener { saslMechanisms: ANONYMOUS host: 0.0.0.0 authenticatePeer: no port: 77777 } # This connector entity will prompt the router to make an outbound connection # to Broker # It is important to remember that even though the router # is initiating the connection, traffic flows both ways (duplex connection) connector { name: Broker host: 0.0.0.0 port: 61699 role: route-container sasl-mechanisms: ANONYMOUS } # Autolinks are used to attach to remote containers like brokers. # The router initiates the attach to the remote container. # The direction of the link is 'out' meaning messages on this link # flow from the router outwards to the broker. autoLink { addr: work_orders connection: Broker direction: out # phase defaults to 0 for 'out' autoLinks. Showing it here for clarity phase: 0 } # The direction of the link is 'in' meaning messages on this link # flow from the broker inwards (toward) the router. autoLink { addr: work_orders connection: Broker direction: in # phase defaults to 1 for 'in' autoLinks. Showing it here for clarity phase: 1 } # This helps set the ingressPhase and the egressPhase when a producer and # consumer respectively connect to this address address { prefix: work_orders ingressPhase: 0 egressPhase: 1 } # The above address entity is equivalent to the following address entity. The # ingressPhase defaults to 0 and egressPhase defaults to 1 if you # set the waypoint flag to 'yes' # address { # prefix: work_orders # waypoint: yes # }

How does the router make this happen internally?

The router schema has many entities like router, connector, autoLink, address etc. as seen from the above config. See here to understand more about the router schema.

Here is the workflow -

To begin with, the router makes a duplex connection to the broker based on the information provided in the router config's connector entity. Messages flow in both directions over this connection.
Next, based on the router config's autoLink entities, the router creates one producing link and one consuming link (autoLink) to the broker's work_orders queue. The broker queue is now a consumer of messages on address work_orders. But internal to the router, the broker queue is a consumer on the address M0work_orders ('M' is for mobile address, '0' is from the phase field of the direction: out autoLink and work_orders is a queue name (addr of autoLink)). Similarly, the router becomes a consumer of work_orders queue in the broker which is represented internally on the router as address M1work_orders ('M' is for mobile address, '1' is from the phase field of the direction: in autoLink and work_orders is a queue name(addr of autoLink)). What is the deal with the M0s and M1s ? Keep reading.
The producer connects to the router on the listener port 77777 and says it wants to start producing messages to address work_orders. Internally the router creates a producer to the address M0work_orders ('M' is for mobile address, '0' is from the ingressPhase attribute of the address entity and work_orders is the name of the queue that the producer produces to). Remember from the previous step that there is already a consumer for the address M0work_orders. So any messages sent by the producer to address work_orders (M0work_orders internally) will enter the router and will be forwarded to the work_orders queue in the broker and all this happens due to the magical internal address scheme that is created up by the router.
A consumer now shows up and wants to consume messages from work_orders queue and connects to the router. Internally, the router sets up a consumer on the work_orders queue M1work_orders ('M' is for mobile address, '1' is from the egressPhase attribute of the address entity and work_orders is the name of the queue that the consumer consumes from).
To recap, the sender sends messages to the router on address work_orders which in turn forwards the message to the work_orders queue in the broker because the queue is consuming on the same address the producer is producing on. Next, the message is sent by the broker from the broker's work_orders queue back to the router because the router is a consumer to queue work_orders internal address M1work_orders which in turn forwards it to the consumer, which is receiving on M1work_orders. The message thus completes its journey from producer to consumer while making a stop on the broker queue waypoint. All this is possible due to the internal address representation generated by the router using the phase/ingressPhase/egressPhase attributes specified in the router. The producers and consumers are unaware that their messages have been routed through a broker. The producers and consumers simply connect to the router and produce to and consume from the work_orders queue. Cool, huh ?

Scenario 2 - Multiple waypoints - Moving messages from one broker to another

Let's look at a more involved two waypoint scenario which uses more phases to move messages from one broker to another broker via a router. Yes, we are going to move messages in one broker's queue to another broker's queue with the same name without the broker's knowledge and without requiring any complex configuration on the brokers...mind blown?. The two brokers act as two waypoints for the messages traveling from a producer to a consumer. Consider that there are two brokers in our topology. One broker is called the LocalBroker and the other one is called a GlobalBroker. Both brokers have a queue called work_orders. The producer produces a message to the router which forwards the messages to the LocalBroker's work_order queue. This is the first waypoint. This messages leaves the queue in LocalBroker and heads back to the router which forwards the message to the GlobalBroker's work_orders queue. This is the second waypoint. The message then leaves the GlobalBroker's work_order queue and comes back to the router where a consumer consumes the message as shown in Figure 3. The internal address representations are shown in Figure 3.

Figure 3

Let's quickly look at the router configuration that helps make this scenario possible

router {
mode: edge # mode can be standalone as well.
}

# The Consumer will connect on this listener
# and start receiving messages from all 3 queues
listener {
saslMechanisms: ANONYMOUS
host: 0.0.0.0
authenticatePeer: no
port: 77777
}

# This will prompt the router to make an outbound connection
# to LocalBroker
# It is important to remember that even though the router
# is initiating the connection, traffic flows both ways
connector {
name: LocalBroker
host: 0.0.0.0
port: 61698
role: route-container
sasl-mechanisms: ANONYMOUS
}

# router establishes connection to GlobalBroker
connector {
name: GlobalBroker
host: 0.0.0.0
port: 61699
role: route-container
sasl-mechanisms: ANONYMOUS
}

# autoLinks are used to attach to remote containers.
# The router initiates the attach
# The direction of the link is 'in' meaning messages on this link
# flow from the LocalBroker toward the router.

# These autoLinks are in the LocalBroker connection
autoLink {
addr: work_orders
connection: LocalBroker
direction: in
phase: 1
}

autoLink {
addr: work_orders
connection: LocalBroker
direction: out
phase: 0
}

# These autoLinks are in the GlobalBroker connection
autoLink {
addr: work_orders
connection: GlobalBroker
direction: in
phase: 2
}

autoLink {
addr: work_orders
connection: GlobalBroker
direction: out
phase: 1
}

# Notice how the egressPhase phase is set to 2
address {
prefix: work_orders
ingressPhase: 0
egressPhase: 2
}

How does the router make this happen internally?

The router makes a duplex connection (traffic flows in both directions) to the two brokers based on the information provided in the connector entities.
After the connections to the brokers have been successfully made, the router creates producing and consuming links to both brokers to the work_orders queue. Internally, the router uses the phase field of the autoLink to create addresses in its address table.
In Figure 3, if you look at the red outbound link from the router to the broker, the broker queue is a consumer on the address work_orders but internally in the router it is actually consuming from address M0work_orders ('M' is for mobile address, '0' is from the phase field of the direction: out autoLink and work_orders is a queue name(addr of autoLink))
Now, when a producer comes along and starts producing messages on the listener port 77777 to the work_orders queue, it is actually producing to the router internal address M0work_orders (M' is for mobile address, '0' is from the ingressPhase attribute of the address entity and work_orders is the name of the queue that the producer produces to). The messages from the producer ultimately ends up in the work_orders queue of the LocalBroker since the queue is a consumer on internal address M0work_orders which is the same address the producer is producing to.
Once the message reaches the work_order queue in the LocalBroker, it travels on the outbound green link from the LocalBroker's work_orders queue back to the router since the router is a consumer of the work_order queue using the internal address M1work_orders ('M' is for mobile address, '1' is from the phase field of the direction: in autoLink and work_orders is a queue name) . When the message reaches the router, it sees that there is a consumer link on the same address M1work_orders (green outbound link) and it sends the message down that link where it ultimately reaches the work order queue of the GlobalBroker. You can now see that the message has traveled from the work_orders queue of the LocalBroker to the work_orders queue of the GlobalBroker. How cool is that? We are not done yet.
Now a consumer can connect to the router and can consume from the work_orders queue of the GlobalBroker. The consumer attaches to the M2work_orders address('M' is for mobile address, '2' is from the egressPhase attribute of the address entity and work_orders is the name of the queue that the consumer consumes from).
All this is possible due to the clever internal address representation generated by the router using the configured phase/ingressPhase/egressPhase which the broker and consumer/producer have no idea about. The producers and consumers simply produce to and consume from the work_orders queue.

We can expand out this scenario into a topology seen in Figure 4 where producers produce to their local edge routers which are part of a larger network of routers. The edge routers are connected to LocalBrokers and to interior routers. The interior routers are connected to the GlobalBroker. This setup allows for the producer to keep producing to the LocalBroker by connecting to its edge router even if the GlobalBroker is offline. Once the GlobalBroker comes online, the routers will move the messages from the LocalBroker to the GlobalBroker using the same internal addressing technique described above. Several LocalBrokers can feed one GlobalBroker with work orders and a consumer can consume all work orders from any router in the network. The possibilities are endless.

Figure 4

Scenario 3 - Consuming from multiple queues (waypointed address)

Consider the following simple scenario where we have a consumer that wants to consume from three sharded queues, called work_orders across three different brokers. A sharded queue, also known as a partitioned queue, is a queue with the same name (work_orders) present in every broker but data is unique across queues. The consumer wants to know about every work order in every queue. Unfortunately, the consumer can only connect to one broker at a time to consume messages from one work_order queue. After consuming a fixed number of messages from one queue/broker, the consumer will have to switch over to the next broker and so on. This switching logic complicates the client's code. The consumer wishes that it could just consumer from one message source that would deliver all messages from the three queues. In Figure 5, the consumer is expressing its frustration via its thought bubble.

Figure 5
Solution: Enter Dispatch Router Waypoint
Let's introduce a single Dispatch Router with a waypointed address (work_orders) between the consumer and the broker. (This single Dispatch Router can be replaced with a network of routers but lets keep it simple stupid to start with).

Figure 6

The consumer now connects to the the router and asks for messages from the address work_order which in turn requests messages from all three work_order queues. The messages from the queues reach the router (which can be thought of as a waypoint) which in turn sends them to the consumer. Miraculously, the consumer can now connect to a single source (router) and start consuming messages from all three queues without needing to jump around from broker to broker (mind blown?). If a new broker is added to the network, the consumer will start receiving from that broker as well (router can connect to new broker without needing a restart). The consumer is unaware of anything that is happening behind the router and it shouldn't not care. The consumer can now focus on what it does best, consuming messages and delegating them to appropriate services that deal with business logic.

Router configuration using waypointed address

Let's quickly look at the router configuration necessary to make this possible.

router {

mode: edge # mode can be standalone as well.

}

# The Consumer will connect on this listener port

# and start receiving messages from all 3 queues

listener {

saslMechanisms: ANONYMOUS

host: 0.0.0.0

authenticatePeer: no

port: 77777

}

# This will prompt the router to make an outbound connection

# to Broker A

# It is important to remember that even though the router

# is initiating the connection, traffic flows both ways

# like a duplex connection

connector {

name: Broker A

host: 0.0.0.0

port: 61697

role: route-container

sasl-mechanisms: ANONYMOUS

}

# router establishes connection to Broker B

connector {

name: Broker B

host: 0.0.0.0

port: 61698

role: route-container

sasl-mechanisms: ANONYMOUS

}

# router establishes connection to Broker C

connector {

name: Broker C

host: 0.0.0.0

port: 61699

role: route-container

sasl-mechanisms: ANONYMOUS

}

# AutoLinks are used to attach to remote containers like brokers.

# The router initiates the attach

# The direction of the link is 'in' meaning messages on this link

# flow from the Broker.A toward the router.

# Also notice that the autoLink references a specific connection

autoLink {

addr: work_orders

connection: Broker A

direction: in

}

# autoLink from Broker.B to the router

autoLink {

addr: work_orders

connection: Broker.B

direction: in

}

# autoLink from Broker.C to the router

autoLink {

addr: work_orders

connection: Broker.C

direction: in

}

# Notice the waypoint flag is set to yes indicating to

# the router that this is a waypointed address

address {

prefix: work_orders

waypoint: yes

}

How does the Router make this happen internally?

Here is the workflow of how the consumer ends up receiving messages from all three queues.

The router makes duplex connections (traffic flows in both directions) to all 3 brokers based on the information provided in the three connector entities.
Then the router runs through the autoLink definitions. The router tells each broker that it wants to be a consumer for the queue work_orders (notice autoLink's direction of in which means coming into the router from the broker). This opens a consuming link from each broker to the router. The messages from the broker queues don't start flowing into the router soon after the consuming links are opened because the router has not issued any credit on these consuming links because there are no consumers yet on the router to which the messages can be sent to.
A consumer connects on the listener port 77777 and asks to consume messages from the address work_orders by opening a consuming link and sending credit to the router.
On seeing that the consumer is asking for messages on address work_orders, the router issues credit on its consuming link to all three brokers so they can start sending messages to the router. The brokers start sending messages from their work_orders queue to the router which in turn forwards those messages to the consumer. This will continue as long as the consumer keeps replenishing the credit to the router.
Internally the router uses what are called phases to construct the full address which are stored in the router's address hashtable. The phase field of the autoLink entity defaults to '0' for out links and '1' for in links.". When the router opens an inbound link (inbound into the router) from a broker queue, it stores the address M1work_orders ('M' stands for mobile address, '1' stands for the address phase and 'work_orders' is the queue name). in its address table. Then the consumer comes along and wants to consume from the same address, M1work_orders.

If another consumer connects to the router and also wants to consume from the work_order queue, the router will send some messages to it. The router might divvy up the messages between consumers if they consume equally fast. The messages that these two consumers get are not duplicates. If you want both consumers to get every message, set the address 'distribution' to be multicast like here ('distribution' defaults to balanced) -

address { prefix: work_orders waypoint: yes distribution: multicast }

Conclusion

There is a lot to gain by using waypoints. Meeting new people, learning new cultures and languages and exposing yourselves to a whole range of messaging possibilities. So, get out there, travel and solve your messaging problems by using the Dispatch Router and one of its many many unique features like waypoints.

Search This Blog

Thoughts about messaging