Table Of Contents

Overview & tutorial
Radar client
Radar server
REST API
Example source code on Github
Full source code on Github
Zendesk is hiring Radar developers

radar

High level API and backend for writing web apps that use push messaging

Features

  • More than just pub/sub: a resource-based API for presence, messaging and push notifications via a Javascript client library
  • Written in Javascript/Node.js, and uses engine.io (the new, low-level complement to socket.io)
  • Backend to multiple front-facing servers
  • REST API for working with web apps that don't use Node (presently, rework in progress)

What is Radar and how is it different from (Socket.io|Sockjs|Faye|Meteor)?

Radar is built on top of engine.io, the next-generation backend for socket.io. It uses Redis for backend storage, though the assumption is that this is only for storing currently active data.

Radar solves a number of real-world problems, and it reduces the need for specialized backend services by providing good primitive operations.

  • Radar has high-level APIs that cover basic use-cases such as presence, and persisted message channels. Many push notification frameworks only expose message passing and do not explicitly handle having multiple backend servers, leaving the implementation for their users. Radar provides high-level constructs, so that you are for example:

    • subscribing to notifications about a user going online/offline
    • subscribing to notifications about changes to a shared variable in a database
    • sending messages to a channel

    ... rather than just passing messages. More complex systems can be built by combining the Radar API resources.

  • REST API for interacting with Radar resources from non-Node web frameworks.
  • Configurable authentication for resources. You can restrict access based on a token.
  • Robust recovery. Radar takes care of re-establishing subscriptions in the event of a server error. Other frameworks can recover a connection, but not the application-level subscriptions and state.
  • Multi-server support via shared Redis instance. Add capacity by adding new server instances behind a load balancer.
  • Persistence. Messages can be stored (for long-ish terms) in the backend. For example, you might want to send recent messages to new users joining a chat - this can be configured via a policy.
  • Library, not a framework. Doesn't require code changes or structural changes beyond responding to the events from the backend.

Tutorial: let's write a Radar application

This tutorial will walk you through the process of implementing a chat server using Radar. You can also find another example application bundled in the radar (server) repository under ./sample, which uses Radar to present a UI.

The example source code used below (except that which you enter in the javascript console of your browser) is available in the Example source code on Github.

1. Setting up the server

Let's start by getting the Radar server up and running.

Create a package.json file by running npm init for your new project; then run npm install --save radar. This installs the Radar server library and also adds the dependency to the package.json file.

Now, let's require the Radar server library and attach it to a HTTP server:

var fs = require('fs'),
    url = require('url'),
    http = require('http'),
    Radar = require('radar').server;

var server = http.createServer(function(req, res) {
  console.log('404', req.url);
  res.statusCode = 404;
  res.end();
});

// attach Radar server to the http server
var radar = new Radar();

radar.attach(server, {
  redis_host: 'localhost',
  redis_port: 6379
});

server.listen(8000);
console.log('Server listening on localhost:8000');

Note that Redis must be running for Radar to work. Save this as "server.js" and run it with "node server.js".

2. Setting up the client

First, add the radar client to our package.json:

npm install --save radar_client

The Radar client has two dependencies:

  • engine.io-client: since Radar uses engine.io internally, it needs the client file to be available
  • minilog: Radar logs events on the client side using Minilog; that's mostly just because I wanted a logger that works on both the client and the server and is small (~80 lines)

There are many ways in which you could include these dependencies:

  • you can just copy the distribution files from ./dist/ for the dependencies and make sure they are included on the page before the Radar client (e.g. with regular script tags)
  • you can create a single file that you can distribute

The first one is probably easy to figure out, so let's do the second one. Here is a Makefile that does that:

build:
  @echo 'Building public/radar_client.js'
  @mkdir -p public
  @cat ./node_modules/radar_client/node_modules/minilog/dist/minilog.js > public/radar_client.js
  @cat ./node_modules/radar_client/node_modules/engine.io-client/engine.io.js >> public/radar_client.js
  @cat ./node_modules/radar_client/dist/radar_client.js >> public/radar_client.js
  # uncomment if you want to use uglifyJS to further minify the file
  # @uglifyjs --overwrite public/radar_client.js
  @echo 'Wrote public/radar_client.js'

.PHONY: build

Note that GNU make requires that you use tabs for indentation and it will not be helpful in telling you to that.

To generate the build, run make build.

3. Putting the two together

Let's set up the server to serve the files, and a minimal HTML page that initializes the Radar client.

Rather than building anything more complicated like a UI, let's just take advantage of the developer console that all good modern browsers have, and use that to create a chat. Create the following public/index.html:

<!doctype html>
<html>
  <head>
    <title>Test</title>
    <script src="/radar_client.js"></script>
    <script>
      Minilog.enable();
      RadarClient.configure({
        host: window.location.hostname,
        port: 8000,

        userId: Math.floor(Math.random()*100),
        userType: 0,
        accountName: 'dev'
      });
      RadarClient.on('message:in', function(message) {
        console.log(JSON.stringify({incoming: message}));
      })
      RadarClient.on('message:out', function(message) {
        console.log(JSON.stringify({outgoing: message}));
      })
    </script>
  </head>
  <body>
    <p>Open the developer console...</p>
  </body>
</html>

Also add support for serving the two files we created earlier, changing the server's HTTP request handler to:

var server = http.createServer(function(req, res) {
  var pathname = url.parse(req.url).pathname;

  if (/^\/radar_client.js$/.test(pathname)) {
    res.setHeader('content-type', 'text/javascript');
    res.end(fs.readFileSync('./public/radar_client.js'));
  } else if (pathname == '/') {
    res.setHeader('content-type', 'text/html');
    res.end(fs.readFileSync('./public/index.html'));
  } else {
    console.log('404', req.url);
    res.statusCode = 404;
    res.end();
  }
});

Open up http://localhost:8000/ in your browser and open the developer console.

4. What's in the Radar client configuration?

The code in index.html contains two function calls:

  • Minilog.enable() which turns on all logging and includes Radar's internal logging

  • RadarClient.configure(), which configures the host and the port of the Radar server - and three other important pieces of information:
    • userId: any number or string that uniquely identifies a user
    • userType: any number or string that represents a user type
    • accountName: any string

Every user needs an account, a user id and a user type. Radar was initially built for Zendesk's use and data for every Zendesk user has these fields (and more). It is likely that most other applications will have the same or similar constructs, so there was no point in getting rid of these fields once we open sourced Radar.

These fields represent business data, which you can manage any way you choose. There is no "user management" in Radar, and Radar doesn't care about the values you use. From Radar's perspective, these are opaque application-level data. Sometimes these values are used for key names - for example, all Radar data in Redis contains the account name as a part of the key so you can identify data for a specific account. It's up to you to determine what makes sense for your application.

Lastly, there are two event handlers that are triggered by message:in and message:out events, respectively.

OK, with that, let's get started.

5. Using alloc() to connect

First, let's connect to the server by calling alloc. Copy-paste this into your developer console after loading the page from localhost:

RadarClient.alloc('example', function() {  console.log('Radar is ready'); });

alloc(scopename, [callback]) is used to connect to the server. The scope name ("example") is just a name for the functionality you are using. The nice part is that when you have an app consisting of multiple independent features that use Radar, each can uniquely identify itself as either needing a Radar connection or not needing a Radar connection (via dealloc(scopename)).

So the connection is initialized the first time you call alloc() - any subsequent calls to alloc will use the existing connection rather than create a new one. Also, the connection is disconnected only when each name passed to alloc has made a corresponding dealloc call.

The callback is called when the connection is established (when the connection already exists, the callback is called immediately). In our example, the callback is an anonymous function that sends 'Radar is ready' to the console log.

In the developer console, the call to alloc generates a number log statements (logging is enabled by the call to Minilog.enable()) in index.html.

radar_state event-state-connect, from: opened, to: connecting ["connect", "opened", "connecting"]
radar_state before-connect, from: opened, to: connecting ["connect", "opened", "connecting"]
Client {_ackCounter: 1, _channelSyncTimes: Object, _users: Object, _presences: Object, _subscriptions: Object…}
radar_client socket open DeeyNr57kfKqA7oOAAAD
radar_state event-state-established, from: connecting, to: connected ["established", "connecting", "connected"]
radar_state event-state-authenticate, from: connected, to: authenticating ["authenticate", "connected", "authenticating"]
radar_state before-authenticate, from: connected, to: authenticating ["authenticate", "connected", "authenticating"]
radar_state event-state-activate, from: authenticating, to: activated ["activate", "authenticating", "activated"]
radar_state before-activate, from: authenticating, to: activated ["activate", "authenticating", "activated"]
Radar is ready
radar_client ready:  example
radar_state before-established, from: connecting, to: connected ["established", "connecting", "connected"]
{"incoming":{"server":"6319-username.local","cid":"DeeyNr57kfKqA7oOAAAD"}}

Here you can see that the Radar client transitions through the following states: connecting, connected, authenticating, and activated.

The callback runs, and you also get back a message with the hostname of the server you're on and the randomly generated client id for this Radar session.

6. Creating a chat - message resource

What is chat? It's a history of messages, and information about who is online.

We'll use a message scope to store a channel of messages, and a presence scope to track people going online and offline.

Let's set that up by subscribing to a message scope:

RadarClient.message('chat/1').on(function(message) {
  console.log('Chat:', message.value);
}).sync();

If you run that you'll see something like this:

{"outgoing":{"op":"sync","to":"message:/dev/chat/1"}}
Scope {prefix: "message:/dev/chat/1", client: Client, set: function, get: function, subscribe: function…}
{"incoming":{"op":"sync","to":"message:/dev/chat/1","value":[],"time":1416292363640}} 

So, there are two parts: the on() handler, which is triggered when messages are received, and the sync() call, which reads all old messages from a message resource, and subscribes to any new messages on that channel. If there had been any messages on that channel, the message handler would have been triggered.

Now, let's send a message:

RadarClient.message('chat/1').publish('Hello world 1');

You should see the message echoed back to you, since your current session is subscribed to the "chat/1" message resource.

OK, open a second tab and keep it open. Run this code to initialize your second client session:

RadarClient.alloc('example', function() {
  RadarClient.message('chat/1').on(function(message) {
    console.log('Chat:', message.value);
  }).sync();
});

Now, enter the following in the second tab, and you should see the message Hello World 2 arrive in both tabs.

RadarClient.message('chat/1').publish('Hello world 2');

7. Message history

Here is the cool part: you can run multiple Radar servers that use the same Redis server, and they just work.

When two or more clients are on different Radar servers, and those servers use the same backend Redis server, then messages will be routed correctly via Redis and, in conjuction with Radar, messages will be routed correctly to the listening clients. The only caveat is that you need source-IP sticky load balancing if you put a load balancer in front of Radar. The need for sticky load balancing is a limitation inherent in how client id-based transports work.

The ability to access message history is part of chat or any message channel application.

Radar has the ability to cache a configurable number of messages. In the following example, the server is configured to store up to 300 messages for each connection. Note that this is configured through the Type system:

var Type = require('radar').core.Type;

Type.register('chatMessage', {
    expr: new RegExp('^message:/.+/chat/.+$'),
    type: 'MessageList',
    policy: { cache: true, maxCount: 300 }
});

Here we're registering a new type of channel for any channel whose name matches the regular expression defined by expr. The policy key defines that the data should be cached - which is what we want so that messages are kept around; maxCount says that we will keep up to 300 messages (see the server docs for the other options).

Once you add the code above to server.js, and then restart the server, you'll see that previously published messages are now synchronized back to your client. This makes it a lot easier to implement a chat with history. Since the persistence policy is determined via a RegExp, you can set different policies for different use cases.

8. Creating a chat - presence resource

At this point two users have joined a chat channel, and we have configured the caching policy to keep old messages around.

Another useful feature of a chat application is one that allows a given user to know when other users come online, are online, and go offline. We call this feature presence. An application can use Radar presence, in particular, to route messages to other users who have indicated they are interested in such messages.

Radar has a presence resource type that is built specifically to track users who are online - that's the purpose of the userId information in the original client configuration in index.html.

In the example below, the sync method directs Radar to send the client up to maxCount messages, and in addition, subscribes the client to all future messages on this channel. Note also that we are interested only in the online and offline messages and not, for example, in the client_online or client_offline messages.

Run this snippet of code in each of the tabs:

RadarClient.alloc('example', function() {
  RadarClient.presence('chat/1').on(function(message) {
    if (!message.op || !message.value) { return; }
    for (var userId in message.value) {
      if (!message.value.hasOwnProperty(userId)) { continue; }
      if (message.op == 'online' || message.op == 'offline') {
        console.log('User ' + userId+' is now '+message.op);
      }
    }
  }).sync();
});

Then, run this code in the first tab:

RadarClient.presence('chat/1').set('online');

You will see output like this in the first tab:

{"outgoing":{"op":"set","to":"presence:/dev/chat/1","value":"online","key":80,"type":0}}
Scope {prefix: "presence:/dev/chat/1", client: Client, set: function, get: function, subscribe: function…}
{"incoming":{"to":"presence:/dev/chat/1","op":"online","value":{"80":0}}}
{"incoming":{"to":"presence:/dev/chat/1","op":"client_online","value":{"userId":80,"clientId":"Ms1o2g3MqK3wS4Z6AAAC"}}}
User 80 is now online 

and you will see output like this in the second tab:

{"incoming":{"to":"presence:/dev/chat/1","op":"online","value":{"80":0}}}
{"incoming":{"to":"presence:/dev/chat/1","op":"client_online","value":{"userId":80,"clientId":"Ms1o2g3MqK3wS4Z6AAAC"}}}
User 80 is now online 

Run this (i.e. the same) code in the second tab:

RadarClient.presence('chat/1').set('online');

You will see additional output like this in the first tab:

{"incoming":{"to":"presence:/dev/chat/1","op":"online","value":{"55":0}}}
{"incoming":{"to":"presence:/dev/chat/1","op":"client_online","value":{"userId":55,"clientId":"8btE85e0agO5pevrAAAD"}}}
User 55 is now online 

and you will see additional output like this in the second tab:

{"outgoing":{"op":"set","to":"presence:/dev/chat/1","value":"online","key":55,"type":0}}
Scope {prefix: "presence:/dev/chat/1", client: Client, set: function, get: function, subscribe: function…}
{"incoming":{"to":"presence:/dev/chat/1","op":"online","value":{"55":0}}}
{"incoming":{"to":"presence:/dev/chat/1","op":"client_online","value":{"userId":55,"clientId":"8btE85e0agO5pevrAAAD"}}}
User 55 is now online 

In the Radar client configuration in Step 3, we set the userId option to a random number between 0 and 100, so you'll get a different user id for each client (tab).

Now, close the second tab. In the first tab you will see something like this:

{"incoming":{"to":"presence:/dev/chat/1","op":"client_offline","explicit":false,"value":{"userId":55,"clientId":"8btE85e0agO5pevrAAAD"}}}

This says that a client_offline message arrived for userId 55, but the offline was not explicit, in which case the user remains online for a configurable timeout period (e.g. for 15 seconds.)

Once the timeout period expires, additional debug text shows up in the first tab:

{"incoming":{"to":"presence:/dev/chat/1","op":"offline","value":{"55":0}}}
User 55 is now offline 

Now user 55 is truly offline. Note that in the second tab you could have instead issued the command:

RadarClient.presence('chat/1').set('offline');

and this would have triggered an "offline" message immediately - the 15 second delay only applies if you close the tab without telling Radar that you're going offline (an ungraceful exit).

Though not demonstrable in this test scenario, a single user can have multiple client sessions, and only when the last client session has ended is the user considered a candidate for being "offline". When the last client session for a user is implicitly made offline, there is a timeout - or grace period - during which the user can create a new client session that will keep the user "online". If no new client session is created, then the user goes offline once the timeout expires. In contrast, an explicit offline ignores the timeout period, and puts the user in an offline state immediately.

Radar supports two granularities of events:

  • client_online/client_offline messages are triggered when a client session goes offline. They are less reliable, but quicker to trigger.

  • online/offline messages are triggered conservatively. They represent users, rather than client sessions. The difference is important when you have multiple Radar servers: then you don't want to consider a user offline until there are no client sessions that are active for that user on any of the Radar servers. There is also a grace period of 15 seconds (configurable) - this allows users to experience short-term network issues, or to reload a Radar-enabled page without being immediately considered offline.

Usually, you want to use the "online" and "offline" events unless you want to specifically track client sessions and do the work mentioned above on the client side.

Lastly, in some of the debug output above, you can clearly see that userId and clientId are two different values. A single userId can be associated with 1 to many clientIds, but a clientId is associated with 1 and only 1 userId.

9. Status resources and the REST API

TODO

Radar client

Read the client docs for the details.

Radar server

Read the server docs for the details.

REST API

Read the REST API docs for the details.

Copyright and License

Copyright 2012, Zendesk Inc.

Licensed under the Apache License Version 2.0, http://www.apache.org/licenses/LICENSE-2.0