node.js - How do I stream JSON from node? -

i'm working on node/express (sails, technically) based service used retrieve large number of items. several calls need return many thousands of items json serialized array.

inside node basic control loop retrieve items in pages. each page retrieved, have minor processing performed, , have items returned client.

at present i'm doing "store , forward" approach, in each page's items concat() results array, , once items have been retrieved results returned.

what more of yield or streaming approach, in items added response they're ready--avoiding need build large in-memory collection , beginning send usable data possible.

if need generate 1 large json string data, you're pretty stuck storing in memory , json.stringify-ing it.

an alternative have each item separate json string, using newline character delimiter. way, have processed item, can stringify , pipe response using node streams, , in client can process data receive line line well. go this:

// each page of data get, loop on items item in dataset // yes that's coffeescript    // manipulate item need, make json string out of   jsonstr = json.stringify(item) + '\n'    // pipe string http response   jsonstr.tostream().pipe(res) // assuming 'res' express response object 

this not way approach issue, if use json 1 of easiest implementations. don't have experience sails though, imagine basic implementation same.

i hope answer helps you, if not feel free comment.

note: .tostream() method comes module streammagic there of course other ways make stream out of string.