5 ms·

Aspects of Building a Node.js Application

65 points by gnw 14y ago

hackerboos 14y ago
Can't you just use Fabric or Capistrano for deployment? I can't believe I used to SSH into my server to restart an app when I can now just run `cap unicorn_APPNAME:restart`
- simontabor 14y ago
  As said in the post, Capistrano is a good option but we found we much preferred deliver, it was easier to setup and meant we didn't need ruby dependencies on node.js apps. We also used to SSH into our servers and do everything manually. Crazy times.
  - crazygringo 14y ago
    Genuine question (since I'm doing the same with AWS right now) -- I wrote a script which makes an API call to Amazon to find the name of each of my instances, then SSH's into them automatically to do the equivalent of a git clone or update. Seems to work so far -- any reason why I might want to be using a more advanced tool instead?
    - latchkey 14y ago
      Over time, you will add features to that script and it'll grow and eventually become 'a more advanced tool'. Everything starts from somewhere.
    - gnw 14y ago
      If it works for you for now then no real need to change it. A proper deployment tool usually makes it easier to deploy apps of different types and languages using a consistent set of tools. What I do when working with SSH'ing to AWS instances is to give them a readable name (like nginx-1, nginx-2) tag, and then place those names as hostnames in my /etc/hosts, so I can then just run ssh nginx-1. I use a script around https://github.com/bjeanes/ghost https://github.com/bjeanes/ghost to automate this.
      - mnutt 14y ago
        The script looks nice, though if you don't want to have to use sudo to update /etc/hosts you can also just add entries in ~/.ssh/config. Downsides are that it only works for ssh-based activities.
  - mqzaidi 14y ago
    We use a slightly modified version of deploy from visionmedia (https://github.com/visionmedia/deploy https://github.com/visionmedia/deploy). Shell script, and works very well for us. Of course, when doing releases that affect the database schema, we still have to login to the servers.
  - mscdex 14y ago
    I generally use a node script (sometimes a CLI script like [1]) with the ssh2 module [2] for most deployment tasks. [1] https://gist.github.com/4281277 https://gist.github.com/4281277 [2] https://github.com/mscdex/ssh2 https://github.com/mscdex/ssh2
jeswin 14y ago
First of all, nodejs is a great choice. Having to worry about just one programming language helps a lot. I'll add a few things I found out along my ongoing experiment. 1. Tests are essential in any project using a dynamic language; but don't write regular unit tests. Instead write a setup script. A script that sets up basic data in the store by calling various APIs. For example, setup your test users POSTing to /users. Whenever you need to go back to a clean database (which is, often) you can run this script. And of course, every time you want to test all the APIs. 2. Automate everything in scripts. Script your compile, test and deploy steps. Script emailing. Script your database backups, tar your image folders. 3. Refactor every time you get a chance. It is like having a bath. You feel like working on your app, code feels fresh. 4. Optionally, try coffeescript. The language often promotes good, declarative programming. Once you get used to the shorthand, you won't go back. Don't discard it after spending a couple of days on it; it might take a little longer depending on your experience level. Another minor benefit, since there is a compile step some typos are picked up by the compiler. 5. One of the big benefits of Node is that you could move things like rendering from server to the client (or the other way round) relatively easily. But still it is cleaner to just have REST services on the server and handle rendering on the client (with say, Backbone).
- gnw 14y ago
  Great pointers, setup scripts and automating project chores are essential for creating a good workflow, saving time, and saving colleagues some hair when they get started with the app. I've tried coffeescript but it didn't catch on with me. Perhaps I didn't give it enough time, but I got the feeling that it could get quite semantically ambiguous at times whereas if you stick to JS' native C-style form, the structure of the code is imperative and easy to follow. I also felt uneasy about what code was actually executing at the other end of the transpiler.
  - latchkey 14y ago
    a) You can always see the source output and it is easily readable. b) CS really doesn't do anything magic in the output. Give it more time, the gains for CS are worth any negatives you can conjure up. do ->, classes, simple for loops, equality, etc.
KaoruAoiShiho 14y ago
Why mention newrelic, doesn't even integrate into nodejs.
- gnw 14y ago
  newrelic can be used for server monitoring, and they've got a nodejs probe in development http://try.newrelic.com/nodejs.html http://try.newrelic.com/nodejs.html Downside: expensive
latchkey 14y ago
In working with my app, I've found that Domains http://nodejs.org/api/domain.html http://nodejs.org/api/domain.html are crucial for wrapping 3rd party async code that can crash on errors.
- gnw 14y ago
  Absolutely, although Domains are a new feature in 0.8 and will need some time for maturity.
  - latchkey 14y ago
    When your nodejs instances are crashing several times an hour due to facebook randomly returning html instead of json and the 3rd party library crashing as a result, who has time for maturity? =)
    - doomslice 14y ago
      Wouldn't a simple uncaughtException handler be sufficient to prevent most crashes? process.on('uncaughtException', function (err) { // log it });
      - latchkey 14y ago
        From the documentation: "Don't use it, use domains instead. If you do use it, restart your application after every unhandled exception!" http://nodejs.org/api/process.html#process_event_uncaughtexception http://nodejs.org/api/process.html#process_event_uncaughtexc...
- aroman 14y ago
  I've had a similar need in my app, which has been in use since 0.6. I ended up basically abusing child_process.fork() -- every call to a certain rather long-running, possibly exploding async function is made not from the main node.js process, but is actually spawned as a separate process from node. So if it explodes, the core app instance isn't affected. Of course I do some intelligent things such as kill these "worker forks" if they take longer than 30 seconds, and I check for various UNIX signals to watch if they die and react appropriately. I also tie the worker PID to each user's session, so they can't spawn an infinite number of workers and cause serious strain on the server -- the server just kills the previous worker and spawns a new one if it's asked to run the function again. I was actually thinking about opensourcing this, but I guess it's pointless now that domains basically do what I hacked together.
cjfont 14y ago
It's probably worth pointing out that there is now an increasing trend toward higher level frameworks like Meteor and DerbyJS which help blur the lines between the server and client and simplify the creation of real-time apps. I think in a couple more years they will start to replace Express as the de facto framework to use for medium to large sized projects.

65 points by gnw 14y ago

Can't you just use Fabric or Capistrano for deployment? I can't believe I used to SSH into my server to restart an app when I can now just run `cap unicorn_APPNAME:restart`

simontabor 14y ago
As said in the post, Capistrano is a good option but we found we much preferred deliver, it was easier to setup and meant we didn't need ruby dependencies on node.js apps. We also used to SSH into our servers and do everything manually. Crazy times.
- crazygringo 14y ago
  Genuine question (since I'm doing the same with AWS right now) -- I wrote a script which makes an API call to Amazon to find the name of each of my instances, then SSH's into them automatically to do the equivalent of a git clone or update. Seems to work so far -- any reason why I might want to be using a more advanced tool instead?
  - latchkey 14y ago
    Over time, you will add features to that script and it'll grow and eventually become 'a more advanced tool'. Everything starts from somewhere.
  - gnw 14y ago
    If it works for you for now then no real need to change it. A proper deployment tool usually makes it easier to deploy apps of different types and languages using a consistent set of tools. What I do when working with SSH'ing to AWS instances is to give them a readable name (like nginx-1, nginx-2) tag, and then place those names as hostnames in my /etc/hosts, so I can then just run ssh nginx-1. I use a script around https://github.com/bjeanes/ghost https://github.com/bjeanes/ghost to automate this.
    - mnutt 14y ago
      The script looks nice, though if you don't want to have to use sudo to update /etc/hosts you can also just add entries in ~/.ssh/config. Downsides are that it only works for ssh-based activities.
- mqzaidi 14y ago
  We use a slightly modified version of deploy from visionmedia (https://github.com/visionmedia/deploy https://github.com/visionmedia/deploy). Shell script, and works very well for us. Of course, when doing releases that affect the database schema, we still have to login to the servers.
- mscdex 14y ago
  I generally use a node script (sometimes a CLI script like [1]) with the ssh2 module [2] for most deployment tasks. [1] https://gist.github.com/4281277 https://gist.github.com/4281277 [2] https://github.com/mscdex/ssh2 https://github.com/mscdex/ssh2

jeswin 14y ago

First of all, nodejs is a great choice. Having to worry about just one programming language helps a lot. I'll add a few things I found out along my ongoing experiment. 1. Tests are essential in any project using a dynamic language; but don't write regular unit tests. Instead write a setup script. A script that sets up basic data in the store by calling various APIs. For example, setup your test users POSTing to /users. Whenever you need to go back to a clean database (which is, often) you can run this script. And of course, every time you want to test all the APIs. 2. Automate everything in scripts. Script your compile, test and deploy steps. Script emailing. Script your database backups, tar your image folders. 3. Refactor every time you get a chance. It is like having a bath. You feel like working on your app, code feels fresh. 4. Optionally, try coffeescript. The language often promotes good, declarative programming. Once you get used to the shorthand, you won't go back. Don't discard it after spending a couple of days on it; it might take a little longer depending on your experience level. Another minor benefit, since there is a compile step some typos are picked up by the compiler. 5. One of the big benefits of Node is that you could move things like rendering from server to the client (or the other way round) relatively easily. But still it is cleaner to just have REST services on the server and handle rendering on the client (with say, Backbone).

gnw 14y ago
Great pointers, setup scripts and automating project chores are essential for creating a good workflow, saving time, and saving colleagues some hair when they get started with the app. I've tried coffeescript but it didn't catch on with me. Perhaps I didn't give it enough time, but I got the feeling that it could get quite semantically ambiguous at times whereas if you stick to JS' native C-style form, the structure of the code is imperative and easy to follow. I also felt uneasy about what code was actually executing at the other end of the transpiler.
- latchkey 14y ago
  a) You can always see the source output and it is easily readable. b) CS really doesn't do anything magic in the output. Give it more time, the gains for CS are worth any negatives you can conjure up. do ->, classes, simple for loops, equality, etc.

KaoruAoiShiho 14y ago

Why mention newrelic, doesn't even integrate into nodejs.

gnw 14y ago
newrelic can be used for server monitoring, and they've got a nodejs probe in development http://try.newrelic.com/nodejs.html http://try.newrelic.com/nodejs.html Downside: expensive

latchkey 14y ago

In working with my app, I've found that Domains http://nodejs.org/api/domain.html http://nodejs.org/api/domain.html are crucial for wrapping 3rd party async code that can crash on errors.

gnw 14y ago
Absolutely, although Domains are a new feature in 0.8 and will need some time for maturity.
- latchkey 14y ago
  When your nodejs instances are crashing several times an hour due to facebook randomly returning html instead of json and the 3rd party library crashing as a result, who has time for maturity? =)
  - doomslice 14y ago
    Wouldn't a simple uncaughtException handler be sufficient to prevent most crashes? process.on('uncaughtException', function (err) { // log it });
    - latchkey 14y ago
      From the documentation: "Don't use it, use domains instead. If you do use it, restart your application after every unhandled exception!" http://nodejs.org/api/process.html#process_event_uncaughtexception http://nodejs.org/api/process.html#process_event_uncaughtexc...
aroman 14y ago
I've had a similar need in my app, which has been in use since 0.6. I ended up basically abusing child_process.fork() -- every call to a certain rather long-running, possibly exploding async function is made not from the main node.js process, but is actually spawned as a separate process from node. So if it explodes, the core app instance isn't affected. Of course I do some intelligent things such as kill these "worker forks" if they take longer than 30 seconds, and I check for various UNIX signals to watch if they die and react appropriately. I also tie the worker PID to each user's session, so they can't spawn an infinite number of workers and cause serious strain on the server -- the server just kills the previous worker and spawns a new one if it's asked to run the function again. I was actually thinking about opensourcing this, but I guess it's pointless now that domains basically do what I hacked together.

cjfont 14y ago

It's probably worth pointing out that there is now an increasing trend toward higher level frameworks like Meteor and DerbyJS which help blur the lines between the server and client and simplify the creation of real-time apps. I think in a couple more years they will start to replace Express as the de facto framework to use for medium to large sized projects.