Realtime Web Stats Using Node.js, Socket.IO and Redis

In today’s bigdata analytics world, it is very important and easy to get real-time stats exposed either to end users to have better user experience or to internal dashboards to drive business insights further.

Few example candidates for real-time stats:

Current active visitors
Currently logged-in users
When a friend comes online or goes offline
When someone likes or comments on your status
Send notification when an event happens to subscribed users
How many people are actively playing a particular game, watching online webinar/event, etc
Total transactions, purchases, impressions, messages, clicks till day and so on

Two approaches on how this data can be exposed to the browser:

Front-end browser(s) pulling the data from back-end at fixed regular intervals.
Back-end pushing the change to front-end browser(s) as and when the change happens or when desired.

Both has its own advantages and dis-advantages; and in general one need to pick the right solution (push or pull) that fits the performance and scalability needs (how often the data is getting changed, how many clients are connected, etc).

ARCHITECTURE

Here is the simple push architecture using Node.js, Redis and Socket.IO:

Redis is picked as common source as it supports both pub-sub model (used for push mechanism) and regular key look-ups (for pull mechanism). This can also be implemented using any data store or queue system; but as it needs to be real-time and shouldn’t introduce any latency; redis fits the best.

Application can be either regular app that is serving the main content or it can be a event processing module that works on top of event logs in real-time.

How it works:

Application publishes the event/counter to a dedicated channel in redis as and when the event happens.
Node.js is subscribed to redis channel and gets all the events as and when it happens along with any other subscribers that are subscribed to redis channel.
Node.js also has socket.io implementation; upon receiving the event from redis channel, node.js posts the event back to socket.io.
Browser (javascript) is connected to node.js server and subscribes to the same channel from socket.io.
Browser gets the event as and when application publishes the event to redis (application -> redis -> node.js -> socket.io -> browser).

Working Example

Here is the screenshot of working model where node.js is started from server.js script and client.html is opened in browser. Any event that gets published to ‘realtime’ channel in redis; browser get to see immediately.

Source code

Here is the source code for the above example and can also be browsed from github.

server.js

/*
Node.js server script
Required node packages: express, redis, socket.io
*/
const PORT = 3000;
const HOST = 'localhost';
var express = require('express'),
http = require('http'), 
server = http.createServer(app);
var app = express();
const redis = require('redis');
const client = redis.createClient();
const io = require('socket.io');
if (!module.parent) {
server.listen(PORT, HOST);
const socket  = io.listen(server);
socket.on('connection', function(client) {
const subscribe = redis.createClient()
subscribe.subscribe('realtime');
subscribe.on("message", function(channel, message) {
client.send(message);
});
client.on('message', function(msg) {
});
client.on('disconnect', function() {
subscribe.quit();
});
});
}

Node.js server script

Required node packages: express, redis, socket.io

const PORT = 3000;

const HOST = 'localhost';

var express = require('express'),

http = require('http'),

server = http.createServer(app);

var app = express();

const redis = require('redis');

const client = redis.createClient();

const io = require('socket.io');

if (!module.parent) {

server.listen(PORT, HOST);

const socket = io.listen(server);

socket.on('connection', function(client) {

const subscribe = redis.createClient()

subscribe.subscribe('realtime');

subscribe.on("message", function(channel, message) {

client.send(message);

});

client.on('message', function(msg) {

});

client.on('disconnect', function() {

subscribe.quit();

});

}

client.html

<html>
<head>
<title>REAL TIME Node.js, Redis and Socket.IO</title>
<meta http-equiv="Content-Type" content="text/html;charset=UTF-8"/>
<!--Load the AJAX API-->
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js"></script>
<script type="text/javascript" src="https://localhost:3000/socket.io/socket.io.js"></script>
<script type="text/javascript">
var socket = io.connect('https://localhost:3000/');
socket.on('connect', function(data){
setStatus('connected');
socket.emit('subscribe', {channel:'realtime'});
});
socket.on('reconnecting', function(data){
setStatus('reconnecting');
});
socket.on('message', function (data) {
console.log('received a message: ', data);
addMessage(data);
});
function addMessage(data) {
$('#online').html(data);
}
function setStatus(msg) {
console.log('Connection Status : ' + msg);
}
</script>
</head>
<body>
<table>
<tr style="width:100%; height: 200px;"> 
<td align="center" style="width: 30%; height: 100%; border: 1px solid #000;" >
<p id="online" style="font-size: 100px;">0</p>
</td>
</tr>
</table>
</body>
</html>

<html>

<head>

<title>REAL TIME Node.js, Redis and Socket.IO</title>

var socket = io.connect('https://localhost:3000/');

socket.on('connect', function(data){

setStatus('connected');

socket.emit('subscribe', {channel:'realtime'});

});

socket.on('reconnecting', function(data){

setStatus('reconnecting');

});

socket.on('message', function (data) {

console.log('received a message: ', data);

addMessage(data);

});

function addMessage(data) {

$('#online').html(data);

}

function setStatus(msg) {

console.log('Connection Status : ' + msg);

}

</script>

</head>

<body>

<table>

</td>

</tr>

</table>

</body>

</html>

Tags: how to node.js redis socket.io how to push changes from backend to browser How to use node.js with socket.io node.js node.js socket.io redis realtime analytics using redis realtime data processing using node.js realtime stats redis socket.io socket.io for realtime

33 Comments

Venu Anuganti says:

June 26, 2013 at 2:40 pm

Realtime Web Stats Using Node.js, Socket.IO and Redis – https://t.co/JEvoqbXo3Q https://t.co/JEvoqbXo3Q
Pablo Angeletti says:

June 27, 2013 at 1:10 am

Realtime Web Stats Using Node.js, Socket.IO and Redis https://t.co/hOY14LKLKg
nodejs-news says:

June 27, 2013 at 4:50 am

Realtime Web Stats with Node.js,Socket.IO,Redis #nodejs #redis via @NewsNodejs https://t.co/1L2sDD6K6w
Yosuke FURUKAWA says:

June 27, 2013 at 4:54 am

[redis][socket.io]Redisã¨socket.ioã‚’ä½¿ã£ãŸrealtimeæŠ•ç¥¨ / â€œRealtime Web Stats Using Node.js, Socket.IO and Redis | Venu Anuâ€¦â€ https://t.co/RodOyk1ZFS
alexander knorr says:

June 27, 2013 at 5:15 am

Realtime Web Stats Using Node.js, Socket.IO and Redis https://t.co/CNOetdbKoW
Didier Rano says:

June 27, 2013 at 6:45 am

RT @vanuganti: Realtime Web Stats Using Node.js, Socket.IO and Redis – https://t.co/JEvoqbXo3Q https://t.co/JEvoqbXo3Q
Chaitanya Kuber says:

June 27, 2013 at 2:29 pm

Realtime Web Stats Using Node.js, Socket.IO and Redis https://t.co/h8b8fgSKdb — nice tool for #hackathons
Sososo says:

June 27, 2013 at 4:45 pm

RT @opexxx: Realtime Web Stats Using Node.js, Socket.IO and Redis https://t.co/CNOetdbKoW
Miriam Parkinson says:

June 28, 2013 at 3:51 am

Realtime Web Stats Using Node.js, Socket.IO and Redis https://t.co/LmY6nl1aNk via @zite
Fabio Franzini says:

June 28, 2013 at 7:33 am

Realtime Web Stats Using Node.js, Socket.IO and Redis https://t.co/IK5GdIhIwy
JIFASNIF says:

June 28, 2013 at 8:00 am

Realtime Web Stats Using Node.js, Socket.IO and Redis | Venu Anuganti Blog https://t.co/dofpmMRlSL
Realtime Web Stats Using Node.js, Socket.IO and Redis - PDG Technologies::Outsourcing company Smartphone Web Windows 8 Development says:

June 30, 2013 at 11:27 am

[…] more here ← How to create, develop, and distribute iOS Static Frameworks quickly and […]
Realtime Web Stats Using Node.js, Socket.IO and... says:

July 5, 2013 at 8:19 pm

[…] In today's bigdata analytics world, it is very important and easy to get real-time stats exposed either to end users to have better user experience or to internal dashboards to drive business insights further. […]
@IraklisAlex says:

August 19, 2013 at 11:20 pm

Realtime Web Stats Using Node.js, Socket.IO and Redis: https://t.co/mGHAJI7JtM
@tgrip says:

August 19, 2013 at 11:53 pm

RT @IraklisAlex: Realtime Web Stats Using Node.js, Socket.IO and Redis: https://t.co/mGHAJI7JtM
Realtime Web Stats Using Node.js, Socket.IO and... says:

August 29, 2013 at 6:54 am

[…] Node.js also has socket.io implementation; upon receiving the event from redis channel, node.js posts the event back to socket.io. Browser (javascript) is connected to node.js server and subscribes to the same channel from … […]
Oleg says:

October 21, 2013 at 1:05 pm

How i can create some channels for user, for example, user can subcsribe on category of web site (articles, comments).
This info is stored at DB mysql for each user. So i can select all names of channels users via PHP script and how create this channels to user and transmit control to Node.js? Or all possible channels are created in advance, but how do I have to send this information to the user or not?
http://obtainaquote.co.uk/page/ugg-boots-sale.html says:

November 11, 2013 at 9:17 am

Hi, this is a great article.
ugg boots uk says:

November 26, 2013 at 12:31 am

Hi there, I am new to running a blog and websites in general and was wondering how you got the “www” included in your domain name? I see your domain name, ” has the www and my domain looks like, “http://mydomain.com”. Do you know the best way I can alter this? I’m using WordPress. Thanks
ugg boots uk http://www.uggukboots2012.com
@cayasso says:

December 13, 2013 at 8:59 pm

Realtime Web Stats Using Node.js, Socket.IO and Redis: https://t.co/npTtNFTN0Y
micheal says:

January 29, 2014 at 11:11 pm

thanks for this great article ,
i downloaded code form git and installed required module and also installed redis,
>node server.js
john@ubuntu:~/All_nodejs/realtime-master/node.js$ node-dev server.js
info – connected to redis server
info – socket.io started
opened another terminal and entered
http://localhost:3000/client
and opened another terminal
redis-cli publish realtime 99

after all this, there is no stat ouptu on the browser

please help

and
@deromemont says:

August 17, 2014 at 8:49 pm

To use php with socket io, suscribe to redis in php and pzss trough nodejs https://t.co/OxW1tAUoMA
@ameanmbot says:

August 17, 2014 at 8:49 pm

RT @deromemont: To use php with socket io, suscribe to redis in php and pzss trough nodejs https://t.co/OxW1tAUoMA
@agoeng_es says:

September 4, 2015 at 5:47 pm

bro @hikmahgumelar maaf lum sempet ubek2 sourcenya, tapi coba ini liat2 node & socket.io https://t.co/2eB1EAhYSm
@ameanmbot says:

January 27, 2016 at 5:24 am

RT @wiwer77: Realtime Web Stats Using #Nodejs, Socket.IO and #Redis
https://t.co/LFVNFBUJy4
#BigData https://t.co/98H7J2wB4B
@AdnaneMahmoudi says:

January 27, 2016 at 6:10 am

RT @wiwer77: Realtime Web Stats Using #Nodejs, Socket.IO and #Redis
https://t.co/LFVNFBUJy4
#BigData https://t.co/98H7J2wB4B
@wiwer77 says:

March 14, 2016 at 4:25 pm

Realtime Web Stats Using #NodeJS, Socket.IO and #Redis
https://t.co/LFVNFBUJy4 #BigData https://t.co/k7cY2CB7eX
@BigDataTweetBot says:

March 14, 2016 at 4:41 pm

RT @wiwer77: Realtime Web Stats Using #NodeJS, Socket.IO and #Redis
https://t.co/LFVNFBUJy4 #BigData https://t.co/k7cY2CB7eX
@AdnaneMahmoudi says:

March 15, 2016 at 8:20 am

RT @wiwer77: Realtime Web Stats Using #NodeJS, Socket.IO and #Redis
https://t.co/LFVNFBUJy4 #BigData https://t.co/k7cY2CB7eX
@wiwer77 says:

July 5, 2017 at 4:58 pm

RT @wiwer77: Realtime Web Stats Using #Nodejs, Socket.IO and #Redis
https://t.co/LFVNFBUJy4
#BigData https://t.co/98H7J2wB4B
@wiwer77 says:

July 23, 2018 at 3:50 am

Realtime Web Stats Using #NodeJS, https://t.co/xU9DlQ4YmB and Redis:
https://t.co/LFVNFBUJy4
#BigData
@PhoenixGyaan says:

July 23, 2018 at 10:18 am

RT @wiwer77: Realtime Web Stats Using #NodeJS, https://t.co/xU9DlQ4YmB and Redis:
https://t.co/LFVNFBUJy4
#BigData
@lucasoft_co_uk says:

July 30, 2018 at 9:25 am

RT @PhoenixGyaan RT @wiwer77: Realtime Web Stats Using #NodeJS, https://t.co/MhJ5fry61B and Redis:â€¦ https://t.co/2t8fDU6Lxu

Realtime Web Stats Using Node.js, Socket.IO and Redis

ARCHITECTURE

How it works:

Working Example