Ad-hoc static file web server for development use with Node.js

There is often need to serve local files through web server so you can access them with browser at http://localhost:8000/ for development and debugging.

Python has good tool, SimpleHTTPServer that can run standalone simply by command python -m SimpleHTTPServer in the directory you want to serve.

Node.js has something similar, http-server that works essentially in the same way.

That said, Today I thought that what internet needs is yet another example of http server with Node.js. Instead of the tools above I tend to use my own one-file implementation as it’s easy to expand and modify for testing purposes, for example to add some mock up Ajax endpoints.
The code also demonstrates some common node.js programming patterns so learning programmers might find it useful.

Code

Server is based on express framework and uses send and async modules to serve index.html and directory listings.

Minimum viable static file server is only few lines in express.

var express = require('express'),
    path = require('path');

var mainapp = express();
mainapp.use(express.static( process.cwd() ));
mainapp.listen(8000);

In case file is not found, we need fallback handler for checking the index.html or if that does not exists then build directory listing. Note that this listens only HTTP GET requests, not POST or HEADs.

mainapp.get('*', function(req, res) {
   var pathname = url.parse(req.url).pathname;
   pathname = path.join(dir, pathname);

    fs.stat(pathname, function(err, stat) {
        // Check if path is directory
        if ( !stat || !stat.isDirectory() ) return res.send(404);

        // check for index.html
        var indexpath = path.join(pathname, 'index.html');
        fs.stat(indexpath, function(err, stat) {
           if ( stat && stat.isFile() ) {
               // index.html was found, serve that
               send(res, indexpath)
                   .pipe(res);
               return;

           } else {
               // No index.html found, build directory listing
               fs.readdir(pathname, function(err, list) {
                  if ( err ) return res.send(404);
                  return directoryHTML( res, req.url, pathname, list );
               });
           }
        });
    });
});

There is to default 404 handler, as express does this automatically.

Function that builds the HTML page from directory listing is surprisingly messy, as it’s not easy to do directory iteration with asynchronous node.js fs api. Directory listing use same HTML layout as python’s SimpleHTTPServer.

function directoryHTML( res, urldir, pathname, list ) {
    var ulist = [];

    function sendHTML( list ) {
        res.setHeader('Content-Type', 'text/html');
        res.send('<!DOCTYPE html>' +
            '<html>\n' +
            '<title>Directory listing for '+urldir+'</title>\n' +
            '<body>\n' +
            '<h2>Directory listing for '+urldir+'</h2>\n' +
            '<hr><ul>\n' +
            list.join('\n') +
            '</ul><hr>\n' +
            '</body>\n' +
            '</html>');
    }

    if ( !list.length ) {
        // Nothing to resolve
        return sendHTML( ulist );
    }

    // Check for each file if it's a directory or a file
    var q = async.queue(function(item, cb) {
        fs.stat(path.join(pathname, item), function(err, stat) {
           if ( !stat ) cb();
           if ( stat.isDirectory() ) {
               ulist.push('<li><a href="'+item+'/">'+item+'/</a></li>')
           } else {
               ulist.push('<li><a href="'+item+'">'+item+'</a></li>')
           }
            cb();
        });
    }, 4); // 4 parallel tasks
    // Push directory listing in workqueue
    list.forEach(function(item) {
        q.push(item);
    });
    // Set drain handler that is called when all tasks are completed
    q.drain = function() {
       sendHTML(ulist);
    };
}

Function uses async librarys queue to control the execution flow.

Running

Running directly with node. Server accepts web root directory as argument.
Download and see full code in Github.

$ git clone git://github.com/tikonen/blog.git
$ cd blog/simplehttpserver
$ node simplehttpserver.js ~/work
Listening port 8000 root dir /Users/teemuikonen/work

Server prints out request access log for debugging purposes.

127.0.0.1 - - [Sun, 2 Apr 2013 12:48:08 GMT] "GET / HTTP/1.1" 200 730 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:20.0) Gecko/20100101 Firefox/20.0"

You also install the server globally either from the checkout or from NPM repository. Install globally with

$ npm install -g simplehttpserver
$ simplehttpserver ~/work
Listening port 8000 root dir /Users/teemuikonen/work

Test server for Apple push notification and feedback integration

Here is a test server that you can use to verify your integration to Apple Push notification service and push notification feedback server. It should be good enough for testing out your application behavior and pdu format. Server helps you to get more debug info than just connecting directly to apple.

See these blog posts on details how to send push notifications

Test server runs under node.js and listens both SSL and plain ports where your application can connect. Get code from Github and generate the SSL keys (see quick howto in cert-howto.txt) and start up server

$ node server.js 
Waiting for connections in ports
Listening feedback port 2296 
Listening feedback port 2196 SSL
Listening push port 2295 
Listening push port 2195 SSL

Note that you may need to install binary module to run the server. Use npm install binary.

Successful push notification sending should look like following. The server dumps the data from your app in binary format for debugging and tries to parse it as Push PDU. Server prints out the fields so you can verify the data.

Accepted push connection 2195 1 SSL
=== RECEIVED DATA (1) ====
00000000: 0100 0000 0151 7261 1d00 206b 4628 de93  .....Qra...kF(^.
00000010: 17c8 0edd 1c79 1640 b58f dfc4 6d21 d0d2  .H.].y.@5._Dm!PR
00000020: d135 1687 239c 44d8 e30a b100 1e7b 2261  Q5..#.DXc.1..{"a
00000030: 7073 223a 7b22 616c 6572 7422 3a22 4865  ps":{"alert":"He
00000040: 6c6c 6f20 5075 7368 227d 7d              llo.Push"}}

=== PDU ====
{ command: 1,
  pduid: 1,
  expiry: Sat Apr 20 2013 17:34:21 GMT+0800 (SGT),
  tokenlength: 32,
  token: '6B4628DE9317C80EDD1C791640B58FDFC46D21D0D2D1351687239C44D8E30AB1',
  payloadlength: 30,
  payload: { aps: { alert: 'Hello Push' } } }

Test server does not validate the data, but it tries to parse JSON message in push notifications and prints error if it fails. Also if command was not set to 1, it sends back error pdu and closes connection. This should be good enough for testing. For example here I made HTTP request to the server to get some error output.

Accepted push connection 2295 1 SSL
=== RECEIVED DATA (1) ====
00000000: 4745 5420 2f20 4854 5450 2f31 2e31 0d0a  GET./.HTTP/1.1..
00000010: 5573 6572 2d41 6765 6e74 3a20 6375 726c  User-Agent:.curl
00000020: 2f37 2e32 392e 300d 0a48 6f73 743a 206c  /7.29.0..Host:.l
00000030: 6f63 616c 686f 7374 3a32 3239 350d 0a41  ocalhost:2295..A
00000040: 6363 6570 743a 202a 2f2a 0d0a 0d0a       ccept:.*/*....

=== PDU ====
{ command: 71,
  pduid: 1163141167,
  expiry: Sun Mar 01 1987 23:31:32 GMT+0800 (SGT),
  tokenlength: 20527,
  token: '312E310D0A557365722D4167656E743A206375726C2F372E32392E300D0A486F73743A206C6F63616C686F73743A323239350D0A4163636570743A202A2F2A0D0A0D0A',
  payloadlength: null,
  payload: 'ERROR: INVALID JSON PAYLOAD [SyntaxError: Unexpected end of input]' }
=== SEND ERROR: 08014554202F
Connection terminated 1

When your app connects successfully to the feedback test service, it sends back few feedback tokens and closes connection after one minute. Edit the actual tokens in the server source code.

Accepted feedback connection 2296 1
SEND: 2696A21000207518B1C2C7686D3B5DCAC8232313D5D0047CF0DC0ED5D753C017FFB64AD25B60
SEND: 2696A21100207518B1C2C7686D3B5DCAC8232313D5D0047CF0DC0ED5D753C017FFB64AD25B60
SEND: 2696A21100207518B1C2C7686D3B5DCAC8232313D5D0047CF0DC0ED5D753C017FFB64AD25B60

Source code is available in Github.

Running external worker process in Node.js

This is example how to properly spawn external worker process in Node.js with error checking. This example runs a ImageMagick convert tool to transform .png image files to 8-bit .png file.

Function spawns ‘convert’ with original file as input and temporary destination path as output and checks the result. If everything seems to be in order, it renames the temporary file over original one.

This also prints out all output from the spawned process to stdout for diagnostics.

var fs = require('fs'),
    util = require('util'),
    spawn = require('child_process').spawn;

function package_image( path, next ) {

    // temporary file name. use process PID as part of the name so there
    // wont' be conflicts if two processes run in parallel
    var tmpfile = path + '.tmp.' + process.pid;

    var cmd = 'convert';
    // executes command 'convert path -type Palette png8:path.tmp'
    var convert = spawn(cmd, [
        path,
        '-type', 'Palette',
	'png8:' + tmpfile ]);

    // capture stdout and stderr. Note that convert does not have any output on success
    convert.stdout.setEncoding('utf8');
    convert.stdout.on('data', function (data) {
        console.log( cmd + ': stdout '+ path + ' ' + data.trim() );
    });

    convert.stderr.setEncoding('utf8');
    convert.stderr.on('data', function (data) {
        if (/^execvp\(\)/.test(data)) {
            // we get here if 'convert' command was not found or could
            // not be executed
	    console.log( cmd + ': failed to start: ' + data );
	} else {
	    console.log( cmd + ': stderr '+ path + ' ' + data.trim() );
	}
    });

    // hook on process exit
    convert.on('exit', function( code ) {
        if ( code ) {
	    // 127 means spawn error, command could not be executed
   	    console.log(cmd + ': error '+ path + ' ' + code );
	    return next(code);
	}
        // check if output file exists
        fs.stat( tmpfile, function(err, info) {
	    if ( err ) {
		// no file found? something went wrong. Just ignore
		console.log( cmd + ': output file not found ' + util.inspect(err));
		return next( err );
	    }
	    // check output file size
	    if ( !info.size || !info.isFile() ) {
		console.log( cmd + ': out file 0 bytes or not a file '+ tmpfile);
		fs.unlink( tmpfile ); // remove output file
		return next( true );
	    }

	    // rename temporary file over original one
	    fs.rename( tmpfile, path, function(err) {
	         if ( err ) {
		    fs.unlink( tmpfile );
		    console.log( cmd + ': can not rename '+ tmpfile + ' ' + util.inspect(err));
		 }
                 // done
		 return next( err );
	    });
	});
    });
}

Example of usage

package_image( '/tmp/some.png', function(err) { 
    if ( err ) { 
       console.log('Image conversion failed', err );
    }
});

Polling Apple Push Notification feedback service with Node.js

Apple assumes that your app polls the Push Notification Feedback Service to get information about App uninstalls so you can stop sending  notifications to those devices. Service returns list of tokens with uninstall timestamps.

This is how you can do it with Node.js. First, check TLS Example and how to create SSL certs to understand basics of how to make secure SSL connection with Node.js.

First simple utility to convert binary data to hex

function bintohex(buf) {

    var hexbuf = new Buffer(buf.length * 2);

    function nibble(b) {
	if (b <= 0x09) return 0x30 + b;
	return 0x41 + (b - 10);
    }

    for(var i=0; i < buf.length; i++) {
        hexbuf[i*2] = nibble(buf[i] >> 4);
        hexbuf[i*2+1] = nibble(buf[i] & 0x0F);
    }
    return hexbuf.toString('ascii', 0, hexbuf.length);
}

Then define function that will be called with push device tokens

function processTokens( pdtokens ) {

    // send or process tokens here
    pdtokens.forEach( function( pd  ){
	console.log('TOKEN ' + pd.token ' INVALIDATED AT ' + new Date( pd.timestamp ) );
    });
}

Then polling function that does it all, note that it assumes function connectAPN that needs simply create SSL connection to ‘feedback.push.apple.com’ port 2196 with your apps client cert and private key.

function pollAPNFeedback() {

    console.log("Connecting APN feedback service");

    connectAPN(function( apnc ) {

	var bufferlist = [];
	apnc.on('data', function(data) {
	    // APN feedback starts sending data immediately on successful connect
	    bufferlist.push(data)
	});

	apnc.on('end', function() {
	    console.log("APN Connection closed");

	    // tokens are parsed to object
	    var pdtokens = []

	    // concatenate all asynchronously collected buffers, and parse the PDU's from them
	    var parsebuf;
	    var totall = 0;
	    for(var i=0; i < bufferlist.length; i++) {
	        totall += bufferlist[i].length;
	    }
	    parsebuf = new Buffer(totall);
	    var offset = 0;
	    for(var i=0; i < bufferlist.length; i++) {
		bufferlist[i].copy(parsebuf, offset, 0);
		offset += bufferlist[i].length;
	    }

	    var count = 0;
	    for(var k = 0; k < parsebuf.length; ) {
		count++;
                 // parse timestamp
		var ts = ((parsebuf[k] << 24) + (parsebuf[k+1] << 16) + (parsebuf[k+2] << 8) + parsebuf[k+3]) >>> 0;
		k += 4;
		var l = ((parsebuf[k] << 8) + parsebuf[k]) >>> 0;
                  k += 2;
                  var pdtokenbuf = new Buffer(l);
		parsebuf.copy(pdtokenbuf, 0, k, k + l);
		k += l;

		var hextoken = bintohex(pdtokenbuf);
		pdtokens.push({timestamp: ts, token:hextoken});

		// process tokens in 100 item batches
	        if( count >= 100 ) {
		    processTokens( pdtokens );
		    count = 0;
		    pdtokens = [];
		}
	}
	processTokens(pdtokens);
         setTimeout(pollAPNFeedback, 1000*60*60*4);
    });
}

pollAPNFeedback();

The function read raw binary data from connection and after Feedback services closes the socket (this happens immediately after it has sent the data) it parses tokens and calls processing function to handle them. Function connects to server every 4 hours.

Note that feedback service host and port is different for sandbox (testing environment) that also needs its own SSL certificates.

CouchDB cleanup script for purging old docs

CouchDB does not have straightforward ways to clean up old data. This is one simple way do delete entries by date, but it requires that

  • Your documents have date or timestamp property
  • There is view for each database to fetch documents for that property

Prerequisities

  1. Node.js
  2. jss module, i.e. ‘npm install jss’.

1. Prepare views for Cleanup

Define view in each database that needs regular cleanup. Use something like this where emitted key field is timestamp in seconds.

views: {
    created: {
	map: function(doc) {
		if ( doc.created ) {
			emit(doc.created, doc._rev);
		}
	}
    }
    ....
}

2. The Cleanup script

Script queries old doc ids from the cleanup view and marks them as deleted. Documents are not deleted immediately but are removed physically on next CouchDB compact. CouchDB 1.2.0 supports autocompact so just enable it and don’t worry about it.

#!/bin/bash

DBHOST=localhost

# Get key for entries that are over 6 months old. This assumes that created view can be queried using timestamps as keys.
if uname -a | grep -i darwin > /dev/null
then
	TODAY=$(date '+%Y-%m-%d')
	MONTHSAGO=$(date -v -24w '+%Y-%m-%d')
	MONTHSAGO_E=$(date -v -24w '+%s')
else
	TODAY=$(date '+%Y-%m-%d')
	MONTHSAGO=$(date -d '24 weeks ago' '+%Y-%m-%d')
	MONTHSAGO_E=$(date -d '24 weeks ago' '+%s')
fi

PATH=$PATH:/usr/local/bin

# JSON scripting tool
JSS=$(npm bin)/jss

cleanup() {
	DATABASE=$1
	DESIGN=$2

	echo "Cleaning $DATABASE/$DESIGN"
	curl --silent -S http://$DBHOST:5984/$DATABASE/_design/$DESIGN/_view/created?endkey=$MONTHSAGO_E | \
		$JSS --bulk_docs '$.id' '{_id: $.id, _rev:$.value, _deleted:true}' | \
		curl --silent -S -X POST -d @-  -H "Content-Type:application/json" http://$HOST:$PORT/$DATABASE/_bulk_docs | \
		sed 's/\({[^}]*}\),/\1\n/g' | tr -d '[]' | \
		$JSS '$.ok != true'
}

echo "STATS CLEANUP <= $MONTHSAGO - Start" `date`

# Put databases and views here
cleanup somedb1 someview
cleanup somedb1 otherview
cleanup somedb2 alsoview

echo "STATS CLEANUP - Done" `date`

The script does this

  1. Get expired docs, e.g. curl ‘http://localhost:5984/mydatabase/_design/mydesign/_view/created?endkey=1337049581&#8217;
  2. Build bulk doc delete request (jss)
  3. Issue delete bulk request (curl post)
  4. sanitize couchdb output, i.e. add newlines and remove brackets (sed)
  5. print failed ones

Note that default version of jss doesn’t output proper JSON if no documents are found, use my fork to workaround this problem if you dont want to see errors in logs.

npm install https://github.com/tikonen/jss/tarball/master

Node.js Application Configuration Files

What is the best practice to make configuration file for your Node.js application? Writing property file parser or passing parameters at command line is cumbersome.

Eval

One easy way to separate configuration and application code is by using eval statement. Define your configuration as simple Javascript associative array and load and evaluage it on app startup.

Example configuration file myconfig.js

settings = {
    a: 10,
    // this is used for something
    SOME_FILE: "/tmp/something"
}

Then at start of your application

var fs = require('fs');
eval(fs.readFileSync('myconfig.js', encoding="ascii"));

Now settings object can be used as your program settings. e.g.

var mydata = fs.readFileSync(settings.SOME_FILE);
for( i = 0 ; i < settings.a ; i++) {
   // do something
}

Require

Another alternative to load configuration, as stated in comments, is to define configuration as module file and require it.

//-- configuration.js
module.exports = {
  a: 10,
  SOME_FILE: '/tmp/foo'
}

In application code then require file

var settings = require('./configuration');

This prevents other global variable creeping in global scope, but it’s hackier to do dynamic configuration reloading. If you detect that file has changed, and would want to reload it at runtime you must delete entry from require cache and re-require the file. Another minor complication is that require uses its own search path (that you can override with NODE_PATH env. variable) so it’s more work to define dynamic location for configuration file in your app. (e.g. set it from command line).

// to reload file with require
var path = require('path');
var filename = path.resolve('./configuration.js');
delete require.cache[filename];
var tools = require('./configuration');

Plain javascript as configure file has benefit (and downside) that it’s possible to run any javascript in the settings. For example.

settings = {
    started: new Date(),
    nonce: ~~(1E6 * Math.random()),
    a: 10,
    SOME_FILE: "/tmp/something"
}

Both of these methods are mostly matter of taste. Eval is bit riskier as it allows leaking variables to global namespace but you’ll never have anything “stupid” in the configuration files anyway. Right?

JSON file

I’m not fan of using JSON as configuration format as it’s cumbersome to write and most editors are not able to show syntax errors in it. JSON also does not support comments that can be a big problem in more complicated configuration files.
Example configuration file myconfig.json

{
   "a":10,
   "SOME_FILE":"/tmp/something"
}

Then at start of your application read json file and parse it to object.

var fs = require('fs');
var settings = JSON.parse(fs.readFileSync('myconfig.json', encoding="ascii"));

And then use settings as usual

var mydata = fs.readFileSync(settings.SOME_FILE);
for( i = 0 ; i < settings.a ; i++) {
   // do something
}

Merging configuration files

One way to simplify significantly configuration management is to do hierarchical configuration. For example have single base configuration file and then define overrides for developer, testing and production use.

For this we need merge function.

// merges o2 properties to o1
var merge = exports.merge = function(o1, o2) {
    for (var prop in o2) {
         var val = o2[prop];
         if (o1.hasOwnProperty(prop)) {
             if (typeof val == 'object') {
                 if (val && val.constructor != Array) { // not array
                     val = merge(o1[prop], val);
                 }
             }
         }
         o1[prop] = val; // copy and override
    }
    return o1;
}

You can use merge to combine configurations. For example, lets have these two configuration objects.

// base configuration from baseconf.js
var baseconfig = {
    a: "someval",
    env: {
        name "base",
        code: 1
    }
}

// test config from localconf.js
var localconfig = {
    env: { 
        name "test"
        db: 'localhost'
    },
    test: true
}

Now it’s possible to merge these easily

var settings = merge( baseconfig, localconfig );

console.log( settings. a ); // prints 'someval'
console.log( settings.env.name ); // prints 'test'
console.log( settings.env.code ); // prints '1'
console.log( settings.env.db ); // prints 'localhost'
console.log( settings.env.test ); // prints 'true'

Socket Pooling on Node.js

UPDATE: This post is pretty old and nowadays there are quite a few socket pooling implementations for Node.js. I recommend taking look into Jackpot (https://github.com/3rd-Eden/jackpot). It’s simple and does the job.

I was looking for connection pooling for my Apple Push notification proxy but couldn’t find proper pool implementation, just some experiments that didn’t do proper error handling or could not handle basic real life requirements. I also needed a pool that could be called by both blocking and non-blocking mode.

Basic functionality requirements of a typical connection pool

  • Ensure that connections are available and no more than maximum number of parallel connections exists
  • Do not keep unnecessary connections open for long. Less traffic, less connections.
  • Handle connection decay. For example if connection gets closed by remote peer while it’s waiting in pool.
  • Sane retry logic. Most importantly do not retry connections as fast as CPU allows in case of error.
  • Can either guarantee waiting time, or supports way of checking connection availability.

Lets define first the interface as Node.js module that exports reserve and release methods.

exports.ConnectionPool = function(factory) {
 var self = {};
 var waitlist = [] ;   // callbacks waiting for connection
 var connections = [];  // unused connections currently in pool
 var ccount = 0;  // number of current connections

 // called to reserve connection from pool. Calls callback without connection if wait is false
 self.reserve = function(callback, wait) { ... }
 // returns connection to the pool. Connection is destroyed if destroy is true
 self.release = function(connection, destroy) { ... }
}

Pool requires user provided factory that is dictionary that defines 3 functions and one property.

factory = {
create : function(callback) { ... }  // create connection and call callback with it
validate: function(validate) { .. } // return true or false for connection
destroy: function(connection) { .. }  // destroy connection
max: 5
}

Pool implementation needs several methods for house keeping. checkWaiters() is called to create new connections for waiting callbacks.

function checkWaiters() {
  if(waitlist.length > 0 && ccount < factory.max) {
     ccount++;
     factory.create(function(connection) {          
       if(!connection) {
         ccount--; // failed                    
       } else {
         if(waitlist.length > 0)
            waitlist.shift()(connection);
         else
           connections.push(connection);
       }
    });                           
  }      
 }

Then its counterpart, destroyConnection() that removes connection from pool for good. This function can be called from several places and situations so it adds “deleted” flag to the connection to avoid duplicate processing. It also tries to recreate new connection immediately (if needed) by calling checkWaiters().

function destroyConnection(connection) {
    if(connection.destroyed) {
       return;
    }
    connection.destroyed = true;
    for(var i=0; i < connections.length; i++) {
        // remove from pool if it's there                                
        if(connection == connections[i]) {
           clearTimeout(connection.timeoutid);
           connections.splice(i,1);
        }
    }
    ccount--;
    factory.destroy(connection);

    // connection was lost, we need to create new one if there are       
    // waiting requests                                                  
    checkWaiters();
 }

Then the actual reserve interface method. Function has two modes, if wait is false it returns immediately if it fails to give connection, otherwise the callback goes to the waiting list.  Connections from pool are validated with factory before they are passed to the callback.

self.reserve = function(callback, wait) {
    if (wait == undefined) {
        wait = true;
    }
    if(connections.length > 0) {  // pool has available connections
        connection = connections.shift();
        if(factory.validate(connection)) {  // is it still valid
           clearTimeout(connection.timeoutid);  // cancel the cleanup timeout
           callback(connection);
           return;
        } else {
            destroyConnection(connection);  // stale connection
        }
    }
    if(ccount >= factory.max) { // maximum number of connections created
       if(!wait) {
          callback();
       } else {
          waitlist.push(callback);
       }
       return;
    }
    ccount++;  // try to create connection
    factory.create(function(connection) {
        if(!connection) {
           ccount--; // failed                                          
           if(!wait) {
              callback();
           } else {
              waitlist.push(callback);
           }
        } else {
          callback(connection); // connection created successfully
        }
    });
 }

And the release method. Release method destroys connection if requested and forgets about it then completely. Otherwise it tries to find callback from waiting list and passes the connection for immediate reuse. In case there is nobody waiting, it puts the connection back  to pool and times the connection cleanup event in 10 seconds.

self.release = function(connection, destroy) {
    if(destroy) {
       destroyConnection(connection);
    } else {
       if(waitlist.length > 0) {
          waitlist.shift()(connection);
         return;
       }
       connections.push(connection);
       connection.timeoutid = setTimeout(function() {
           destroyConnection(connection);
       }, 10000);
    }
 }

And finally the background polling that is responsible mainly of connection retry logic. It polls the waiting list once per second, as you remember that function creates connections if there is anyone waiting for it. The connections are normally created on demand in reserve() call.

function poll() {
   checkWaiters()
   setTimeout(poll, 1000);
}
setTimeout(poll, 1000);  // start poller

How this then handles the different error cases?

  • Idle connections are cleaned by timeout call that is set on release()
  • User code can request connection delete by setting the destroy flag to true in call to release()
  • Connections that go bad while in pool are (hopefully) intercepted by user provided factory.validate()
  • Counter keeps track of maximum number of created connections and because its increased before creating connection it also limits maximum number of parallel connection attempts!
  • When connection creation fails, the callback goes to waiting list and poll per second tries to create new parallel connections.
  • User code that needs process immediately can set wait flag to false when reserving connection. Code could be also changed to timed out callback call, so wait could be defined as milliseconds instead of instant fail v.s. infinite wait as its done now.

Then how to use it.

HTTP Pool Example

var pool = require('./pool');
 var http = require('http');
var httppool = pool.ConnectionPool({
 create: function(callback) { callback(http.createClient(80, "www.google.com")); },
 validate: function(connection) { return true; /* no need to validate  */ },
 destroy : function(httpclient) {  /* nothing to destroy */},
 max: settings.couchdbmax
 });

Using the pool

 httppool.reserve(function(connection) {
   var req = connection.request("GET", "/index.html");
   req.on('error', function(error) {
     httppool.release(connection, true);
   });
   req.on('response', function(response) {
     body = '';
     response.on('data', function(data) {
     body += data;
   });
   response.on('end', function() {
     console.log(body)
     httppool.release(connection);
   });
 });

Client socket pool

var pool = require('./pool');
var net = require('net');
var apnpool = pool.ConnectionPool({
 create: function(callback) {
   function errorcb(error) {  // error handler
     require('util').puts(error.stack);
     callback();
   }
   connection = net.createConnection(12345, 'server.example.com');
   connection.once('error', errorcb); 
   connection.on('connect', function() {
     connection.removeListener('error', errorcb); // clear error handler before passing forward
     callback(connection);
   });
 },
 validate: function(connection) { return connection.writable; },
 destroy : function(connection) { connection.end(); },
 max: 5,
 });

Reuse is little bit tricky with sockets, as you need probably set and clear ‘error’ and  ‘data’ event handlers for reading the responses in each worker.