Send data from database to client without templating engine (node.js)

As you allude to in your answer, part of the problem is the speed of the templating engine itself; you've found out that Jade is not the fastest — in fact, it's one of the slowest.

My favorite engine is doT. In the performance test I linked to, doT can render the template 5.4 million times per second. Jade can render a similar template only 29,000 times per second. It's not even a contest.

However, templating engine speed is only a small part of the issue here. I believe your real problem is the Mongo driver you're using appears to be poorly designed for Node's asynchronous model. (Disclaimer: I've never actually used Mongous; I just spent a few minutes looking over the code.)

Node is meant to work with streams of data. In other words, you're supposed to operating on very small chunks of data at a time. In contrast, it looks like Mongous processes the entire dataset and returns it to your code as one JSON object.

This is convenient and fine for small datasets, but completely falls apart when working with large amounts of data like you are (10,000 records). Node will be completely locked up while parsing and handling that much data (which is very, very bad since it won't be able to handle any incoming connections), and the V8 memory management system isn't optimized for large heap allocations like that.

To work with large datasets properly, you have to use a Mongo driver that streams records to your code, like node-mongodb-native, or mongoskin, which makes the API a little easier to deal with.

maerics' answer was on the right track, but is wrong because it uses toArray, which creates the same problem you have under Mongous: the entire dataset is collated into an in-memory array. Instead, just use the cursor's each method, which asynchronously calls your callback function for each returned record as it comes in. This way, the entire resultset is never in memory; you only work with one at a time, allowing the records you've already processed to be discarded and garbage collected if necessary.

Now that we've established how to get your data out of the database, we need to figure out how to get it to the client.

The problem here is that an Express' view system expects you to have all your data available up-front so that the templating engine can render out a single string to be sent to the client. As we discussed above, this isn't such a good idea if you're dealing with thousands of records. The proper way to do this is to stream the data we get from Mongo directly to the client. Unfortunately, we can't really do this within an Express view — they're not designed to be asynchronous.

Instead, you're going to have to write a custom handler. You're already on that path with Hippo's answer and your own attempt, but what you really need to use is res.write(), not res.send. Like res.render, res.send expects you to have a complete response when you call it because it internally calls res.end, ending the HTTP response. In contrast, res.write simply sends data over the network, leaving the HTTP response open and ready to send more data — in other words, streaming your response. (Keep in mind that you have to set any HTTP headers before you start streaming. For example, res.contentType('text/html');)

Just because you're manually handling the response (foregoing the view rendering system) does not preclude you from taking advantage of a templating engine. You can use a template for the header and footer of your document and one for each record. Let's use doT to put everything together.

First, let's declare our templates. In real life, you might load these from files (or even hack Express to load them for you as views and get the template source), but we'll just declare them in our code.

var header = doT.template('<!DOCTYPE html><html><head><title>{{=it.title}}</title></head><body>');
var record = doT.template('<p>{{=it.fieldname}}, ...</p>');
var footer = doT.template('</body></html>');

(doT.template returns a function which generates HTML from the template we gave above and an object you pass to that returned function when you invoke it. For example, now we can call header({title:'Test'});)

Now we do the real work in the request handler: (assumes we already have a collection from the Mongo driver)

app.get('/', function(req, res){
    res.contentType('text/html');
    res.write(header({title:'Test'}));

    collection.find({}).each(function(err, doc) {
        if (err) return res.end('error! ' + err); // bail if there's an error
        if (doc) {
            res.write(record(doc));
        } else { // `doc` will be null if we've reached the end of the resultset.
            res.end(footer());
        }
    });
});

Answers

If you stick to express and some kind of database layer, use res.send() instead of res.render.

Also keep in my mind that node.js is quite new, so not every library is stable or fast as you are used from other languages. (E.g.: You might be faster using another approach for accessing mongo, or not using express.)

Answers

Well this is best what I was able to come up with. It is of course based on Hippos's answer but with a little bit more stuff. However, now I am doing something really wrong, because now it seems to be even slower than with templateing engine, much slower...

The code in question:

//--------------------------------------------------
app.get('/', function(req, res){
    mongous(dbCollection).find(10000,function(output){
        var o = output.documents,
            str = JSON.stringify(o);
        res.send(str);
    });
});
//--------------------------------------------------

Sigh...

Answers

If you're interested in a really raw performance test between PHP and Node.js then you should consider ditching all the middleware on the Node stack (e.g. Express.js, and Mongous) for the direct approach using a MongoDB native driver and raw HTML generation. Something like this (untested):

var http = require('http')
  , mongo = require('mongodb')
  , db = new mongo.Db('foo', new mongo.Server('localhost', 27017, {}));

db.open(function(err, db) {
  if (err) throw err;
  db.collection('people', function(err, collection) {
    var server = http.createServer(function(req, res) {
      res.writeHead('200', {'Content-Type': 'text/html'});
      res.write('<html><head><title>Node.js Output</title></head>');
      res.write('<body><div id="data">');
      collection.find(function(err, cursor) {
        cursor.each(function(err, doc) {
          if (doc) {
            res.write('<div>' + doc.name + '</div>');
          } else {
            res.end('</div></body></html>');
            cursor.close();
          }
        });
      });
    }).listen(8080, 'localhost');
    console.log('OK: listening on http://localhost:8080/');
  });
});

[Edit] Updated the use of the mongo driver to use a cursor instead of the "toArray" method.

A key choice here is the Node MongoDB driver; in particular, you want one that is as non-blocking as possible. Surprisingly, there doesn't seem to be a standout one right now (IMHO).