For this first post I would like to write about a chat project I started a few years ago. What does it actually do? (Here we go for the buzz words)

• Real time chat with Socket-io using the new pubsub feature backed up by redis
• Streaming Upload in MongoDB GridFs
• MVC layer over Express
• Linearly scalable architecture ?

You can try the live demo here.

1 – Chat

A chat application is the probably the simplest way of testing a distributed push application. Messages are passed from one to many like it could be in a video game or any multi-user interactive application. Thus once you have done your chat application and tested it’s performance you can confidently build your next-gen video game.

But making a performant distributed message passing application is not the simplest job to accomplish.

So, why node.js ?

Node.js is a low level API for evented I/O which allows developpers to build performant client-server application in Javascript. It means you have access to a very thin layer of abstraction and can glue together your new killer features! This is exactly what we need to build a distributed push application.

When I started this project in 2010, I had to investigate on push technologies and I finally implemented a simple long polling technique. The Node.js API (with express.js) made it very simple. But as the code was getting bigger, and as it was also my first JavaScript project, I remember having difficulties in mixing OOP with the callback programming style node.js had just choosen.

I restarted the project last year and wanted to keep up with all the new technologies which had grown with node.js. I replaced my long polling by socket-io and it worked pretty well!

2 – File sharing

My first goal was to build a realtime imageboard where users could share images, videos and more. So basically a chat with file sharing capability.

To make it harder (and cooler ;)) I wanted to access the file stream directly and make it available to all users as it was uploaded. Making this work on a large scale is a difficult problem because we need to replicate file chunks as soon as they are sent to a node. I trusted an open source distributed filesystem available with MongoDB called GridFS. This latter enables you to store and retrive files in your MongoDB cluster.

I wrote a small middleware for node.js to access chunks of data when uploading a file along with a library to easily store growing files into MongoDB.

3 – MVC Layer

I wanted to see if it makes sense to follow the MVC pattern while using an evented library like node.js. For this I wrote my own little framework to :
– Autoload controllers, models and views
– Have a django style urls.js file which defines the routes
– Load a config.js file for configuration preferences such as database host, view engine or session distributed cache engine
– Define socket.io sockets mapping with the routes

I found it hard to stuck to my MVC framework implementation when integrating socket.io because it doesn’t use HTTP routes and sessions handled by Express. I choosed to write a bit of glue code to automatically load the corresponding client’s HTTP session from the session store. I haven’t load tested it but I’m afraid it can really impact performance.
To be honest, it’s not as clean as others Node.js MVC frameworks but I have the avantage of knowing the code and beeing able to tweak it to my needs.

4 – Scalability

For an application to run in the cloud you need to export everything statefull to a service known to run in cluster. I chosed to use MongoDB to save messages and files.

The problem is, file sharing requires a lot of ressources which I could not afford. I wanted to make a free or ad-paid service but the cost of bandwidth and storage involved by deploying on Amazon EC2 instances was too high. So I decided to deploy it on normal servers with unlimited download capability (see OVH offers). Then came the devops part of the project which I probably underevaluated and is the main reason I did not bring this project to the public. This will be the subject of another post!