Trace Node.js Monitoring Start my free trial!

npm

Yarn vs npm - The State of Node.js Package Managers

Yarn vs npm - The State of Node.js Package Managers

With the v7.4 release, npm 4 became the bundled, default package manager for Node.js. In the meantime, Facebook released their own package manager solution, called Yarn.

Let's take a look at the state of Node.js package managers, what they can do for you, and when you should pick which one!

Yarn - the new kid on the block

Fast, reliable and secure dependency management - this is the promise of Yarn, the new dependency manager created by the engineers of Facebook.

But can Yarn live up to the expectations?

Yarn - the node.js package manager

Installing Yarn

There are several ways of installing Yarn. If you have npm installed, you can just install Yarn with npm:

npm install yarn --global  

However, the recommended way by the Yarn team is to install it via your native OS package manager - if you are on a Mac, probably it will be brew:

brew update  
brew install yarn  

Yarn Under the Hood

Yarn has a lot of performance and security improvements under the hood. Let's see what these are!

Offline cache

When you install a package using Yarn (using yarn add packagename), it places the package on your disk. During the next install, this package will be used instead of sending an HTTP request to get the tarball from the registry.

Your cached module will be put into ~/.yarn-cache, and will be prefixed with the registry name, and postfixed with the modules version.

This means that if you install the 4.4.5 version of express with Yarn, it will be put into ~/.yarn-cache/npm-express-4.4.5.

Deterministic Installs

Yarn uses lockfiles (yarn.lock) and a deterministic install algorithm. We can say goodbye to the "but it works on my machine" bugs.

The lockfile looks like something like this:

Yarn lockfile

It contains the exact version numbers of all your dependencies - just like with an npm shrinkwrap file.

License checks

Yarn comes with a handy license checker, which can become really powerful in case you have to check the licenses of all the modules you depend on.

yarn licenses

Potential issues/questions

Yarn is still in its early days, so it’s no surprise that there are some questions arising when you start using it.

What’s going on with the default registry?

By default, the Yarn CLI uses a different registry, and not the original one: https://registry.yarnpkg.com. So far there is no explanation on why it does not use the same registry.

Does Facebook have plans to make incompatible API changes and split the community?

Contributing back to npm?

One the most logical questions that can come up when talking about Yarn is: Why don’t you talk with the CLI team at npm, and work together?

If the problem is speed, I am sure all npm users would like to get those improvements as well.

When we talk about deterministic installs, instead of coming up with a lockfile, the npm-shrinkwrap.json should have been fixed.

Why the strange versioning?

In the world of Node.js and npm, versions starts with 1.0.0.

At the time of writing this article, Yarn is at 0.18.1.

Is something missing to make Yarn stable? Does Yarn simply not follow semver?

npm 4

npm is the default package manager we all know, and it is bundled with each Node.js release since v7.4.

Updating npm

To start using npm version 4, you just have to update your current CLI version:

npm install npm -g  

At the time of writing this article, this command will install npm version 4.1.1, which was released on 12/11/2016. Let's see what changed in this version!

Changes since version 3

  • npm search is now reimplemented to stream results, and sorting is no longer supported,
  • npm scripts no longer prepend the path of the node executable used to run npm before running scripts,
  • prepublish has been deprecated - you should use prepare from now on,
  • npm outdated returns 1 if it finds outdated packages,
  • partial shrinkwraps are no longer supported - the npm-shrinkwrap.json is considered a complete manifest,
  • Node.js 0.10 and 0.12 are no longer supported,
  • npm doctor, which diagnose user's environment and let the user know some recommended solutions if they potentially have any problems related to npm

As you can see, the team at npm was quite busy as well - both npm and Yarn made great progress in the past months.

Conclusion

It is great to see a new, open-source npm client - no doubt, a lot of effort went into making Yarn great!

Hopefully, we will see the improvements of Yarn incorporated into npm as well, so both users will benefit from the improvements of the others.

Yarn vs. npm - Which one to pick?

If you are working on proprietary software, it does not really matter which one you use. With npm, you can use npm-shrinkwrap.js, while you can use yarn.lock with Yarn.

The team at Yarn published a great article on why lockfiles should be committed all the time, I recommend checking it out: https://yarnpkg.com/blog/2016/11/24/lockfiles-for-all


npm Publishing Tutorial - Node.js at Scale

npm Publishing Tutorial - Node.js at Scale

With Node.js at Scale we are creating a collection of articles focusing on the needs of companies with bigger Node.js installations, and developers who already learned the basics of Node.

In this second chapter of Node.js at Scale you are going to learn how to expand the npm registry with your own modules. This tutorial is also going to explain how versioning works.

Upcoming chapters for the Node.js at Scale series:


npm Module Publishing

When writing Node.js apps, there are so many things on npm that can help us being more productive. We don't have to deal with low-level things like padding a string from the left because there are already existing modules that are (eventually) available on the npm registry.

Where do these modules come from?

The modules are stored in a huge registry which is powered by a CouchDB instance.

The official public npm registry is at https://registry.npmjs.org/. It is powered by a CouchDB database, which has a public mirror at https://skimdb.npmjs.com/registry. The code for the couchapp is available at https://github.com/npm/npm-registry-couchapp.

How do modules make it to the registry?

People like you write them for themselves or for their co-workers and they share the code with their fellow JavaScript developers.

When should I consider publishing?

  • If you want to share code between projects,
  • if you think that others might run into the very same problem and you'd like to help them,
  • if you have a bit (or even more) code that you think you can make use of later.

Creating a module

First let's create a module: npm init -y should take care of it, as you've learned in the previous post.

{
  "name": "npm-publishing",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "repository": {
    "type": "git",
    "url": "git+https://github.com/author/modulename"
  },
  "bugs": {
    "url": "https://github.com/caolan/async/issues"
  },
  "license": "ISC"
}

Let's break this down really quick. These fields in your package.json are mandatory when you're building a module for others to use.

First, you should give your module a distinct name because it has to be unique in the npm registry. Make sure it does not collide with any trademarks out there! main describes which file will be returned when your users do a require('modulename'). You can leave it as default or set it to any file in your project, but make sure you actually point it to a valid filename.

keywords should also be included because npm is going to index your package based on those fields and people will be able to find your module if they search those keywords in npm's search, or in any third party npm search site.

author, well obviously that's going to be you, but if anyone helps you develop your project be so kind to include them too! :) Also, it is very important to include where can people contact you if they'd like to.

In the repository field, you can see where the code is hosted and the bugs section tells you where can you file bugs if you find one in the package. To quickly jump to the bug report site you can use npm bug modulename.

#1 Licensing

Solid license and licenses adoption helps Node adoption by large companies. Code is a valuable resource, and sharing it has it's own costs.

Licensing is a really hard, but this site can help you pick one that fits your needs.

Generally when people publish modules to npm they use the MIT license.

The MIT License is a permissive free software license originating at the Massachusetts Institute of Technology (MIT). As a permissive license, it puts only very limited restriction on reuse and has therefore an excellent license compatibility.

#2 Semantic Versioning

Versioning is so important that it deserves its own section.

Most of the modules in the npm registry follow the specification called semantic versioning. Semantic versioning describes the version of a software as 3 numbers separated by "."-s. It describes how this version number has to change when changes are made to the software itself.

Given a version number MAJOR.MINOR.PATCH, increment the:

  • MAJOR version when you make incompatible API changes,
  • MINOR version when you add functionality in a backwards-compatible manner, and
  • PATCH version when you make backwards-compatible bug fixes.

Additional labels for the pre-release and the build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

These numbers are for machines, not for humans! Don't assume that people will be discouraged from using your libraries when you often change the major version.

"If you break your API, think about your users and BUMP THAT MAJOR!" via @RisingStack #nodejs #semver

Click To Tweet

You have to start versioning at 1.0!

Most people think that doing changes while the software is still in "beta" phase should not respect the semantic versioning. They are wrong! It is really important to communicate breaking changes to your users even in beta phase. Always think about your users who want to experiment with your project.



Need help with enterprise-grade Node.js Development?
Hire the experts of RisingStack!


#3 Documentation

Having a proper documentation is imperative if you’d like to share your code with others. Putting a README.md file in your project’s root folder is usually enough, and if you publish it to the registry npm will generate a site like this one. It's all done automatically and it helps other people when they try to use your code.

Before publishing, make sure you have all documentation in place and up to date.

#4 Keeping secret files out of your package

Using a specific file called .npmignore will keep your secret or private files from publishing. Use that to your advantage, add files to .npmignore that you wish to not upload.

If you use .gitignore npm will use that too by default. Like git, npm looks for .npmignore and .gitignore files in all subdirectories of your package, not only in the root directory.

#5 Encouraging contributions

When you open up your code to the public, you should consider adding some guidelines for them on how to contribute. Make sure they know how to help you dealing with software bugs and adding new features to your module.

There are a few of these available, but in general you should consider using github's issue and pull-request templates.

npm publish

Now you understand everything that's necessary to publish your first module. To do so, you can type: npm publish and the npm-cli will upload the code to the registry.

Congratulations, your module is now public on the npm registry! Visit
www.npmjs.com/package/yourpackagename for the public URL.

If you published something public to npm, it's going to stay there forever. There is little you can do to make it non-discoverable. Once it hits the public registry, every other replica that's connected to it will copy all the data. Be careful when publishing.

I published something that I didn't mean to.

We're human. We make mistakes, but what can be done now? Since the recent leftpad scandal, npm changed the unpublish policy. If there is no package on the registry that depends on your package, then you're fine to unpublish it, but remember all the replicas will copy all the data so someone somewhere will always be able to get it. If it contained any secrets, make sure you change them after the act, and remember to add them to the .npmignore file for the next publish.

"If you accidentally published secrets to #npm change & add them to the .npmignore file!" via @RisingStack #nodejs

Click To Tweet

Private Scoped Packages

If you don't want or you're not allowed to publish code to a public registry (for any corporate reasons), npm allows organizations to open an organization account so that they can push to the registry without being public. This way you can share private code between you and your co-workers.

Further read on how to set it up: https://docs.npmjs.com/misc/scope

npm enterprise

If you'd like to further tighten your security by running a registry by yourself, you can do that pretty easily. npm has an on-premise version that can be run behind corporate firewalls. Read more about setting up npm enterprise.

Download the whole Learn using npm series as a single pdf

Build something!

Now that you know all these things, go and build something. If you’re up for a little bragging, make sure you tweet us (@risingstack) the name of the package this tutorial helped you to build! If you have any questions, you’ll find me in the comments.

Happy publishing!

In the next part of the Node.js at Scale series, you're going to learn about the Node.js module system and require.


npm Best Practices - Node.js at Scale

npm Best Practices - Node.js at Scale

Node Hero was a Node.js tutorial series with focusing on teaching the most essential Node.js best practices, so one can start developing applications using it.

With our new series, called Node.js at Scale, we are creating a collection of articles focusing on the needs of companies with bigger Node.js installations, and developers who already learned the basics of Node.

In the first chapter of Node.js at Scale you are going to learn the best practices on using npm as well as tips and tricks that can save you a lot of time on a daily basis.

Upcoming chapters for the Node.js at Scale series:


npm Best Practices

npm install is the most common way of using the npm cli - but it has a lot more to offer! In this chapter of Node.js at Scale you will learn how npm can help you during the full lifecycle of your application - from starting a new project through development and deployment.

#0 Know your npm

Before diving into the topics, let's see some commands that help you with what version of npm you are running, or what commands are available.

npm versions

To get the version of the npm cli you are actively using, you can do the following:

$ npm --version
2.13.2  

npm can return a lot more than just its own version - it can return the version of the current package, the Node.js version you are using and OpenSSL or V8 versions:

$ npm version
{ bleak: '1.0.4',
  npm: '2.15.0',
  ares: '1.10.1-DEV',
  http_parser: '2.5.2',
  icu: '56.1',
  modules: '46',
  node: '4.4.2',
  openssl: '1.0.2g',
  uv: '1.8.0',
  v8: '4.5.103.35',
  zlib: '1.2.8' }

npm help

As most cli toolkits, npm has a great built-in help functionality as well. Description and synopsis are always available. These are essentially man-pages.

$ npm help test
NAME  
       npm-test - Test a package

SYNOPSIS  
           npm test [-- <args>]

           aliases: t, tst

DESCRIPTION  
       This runs a package's "test" script, if one was provided.

       To run tests as a condition of installation, set the npat config to true.

"9 npm best practices - a must-read collection for #nodejs developers" via @RisingStack

Click To Tweet

#1 Start new projects with npm init

When starting a new project npm init can help you a lot by interactively creating a package.json file. This will prompt questions for example on the project's name or description. However, there is a quicker solution!

$ npm init --yes

If you use npm init --yes, it won't prompt for anything, just create a package.json with your defaults. To set these defaults, you can use the following commands:

npm config set init.author.name YOUR_NAME  
npm config set init.author.email YOUR_EMAIL  



Need help with enterprise-grade Node.js Development?
Hire the experts of RisingStack!


#2 Finding npm packages

Finding the right packages can be quite challenging - there are hundreds of thousands of modules you can choose from. We know this from experience, and developers participating in our latest Node.js survey also told us that selecting the right npm package is frustrating. Let's try to pick a module that helps us sending HTTP requests!

One website that makes the task a lot easier is npms.io. It shows metrics like quality, popularity and maintenance. These are calculated based on whether a module has outdated dependencies, does it have linters configured, is it covered with tests or when the most recent commit was made.

finding npm packages

#3 Investigate npm packages

Once we picked our module (which will be the request module in our example), we should take a look at the documentation, and check out the open issues to get a better picture of what we are going to require into our application. Don’t forget that the more npm packages you use, the higher the risk of having a vulnerable or malicious one. If you’d like to read more on npm-related security risks, read our related guideline.

If you'd like to open the homepage of the module from the cli you can do:

$ npm home request

To check open issues or the publicly available roadmap (if there’s any), you can try this:

$ npm bugs request

Alternatively, if you'd just like to check a module's git repository, type this:

$ npm repo request

#4 Saving dependencies

Once you found the package you want to include in your project, you have to install and save it. The most common way of doing that is by using npm install request.

If you'd like to take that one step forward and automatically add it to your package.json file, you can do:

$ npm install request --save

npm will save your dependencies with the ^ prefix by default. It means that during the next npm install the latest module without a major version bump will be installed. To change this behaviour, you can:

$ npm config set save-prefix='~'

In case you'd like to save the exact version, you can try:

$ npm config set save-exact true

#5 Lock down dependencies

Even if you save modules with exact version numbers as shown in the previous section, you should be aware that most npm module authors don't. It’s totally fine, they do it to get patches and features automatically.

The situation can easily become problematic for production deployments: It’s possible to have different versions locally then on production, if in the meantime someone just released a new version. The problem will arise, when this new version has some bug which will affect your production system.

To solve this issue, you may want to use npm shrinkwrap. It will generate an npm-shrinkwrap.json that contains not just the exact versions of the modules installed on your machine, but also the version of its dependencies, and so on. Once you have this file in place, npm install will use it to reproduce the same dependency tree.

#6 Check for outdated dependencies

To check for outdated dependencies, npm comes with a built-in tool method the npm outdated command. You have to run in the project's directory which you'd like to check.

$ npm outdated
conventional-changelog    0.5.3   0.5.3   1.1.0  @risingstack/docker-node  
eslint-config-standard    4.4.0   4.4.0   6.0.1  @risingstack/docker-node  
eslint-plugin-standard    1.3.1   1.3.1   2.0.0  @risingstack/docker-node  
rimraf                    2.5.1   2.5.1   2.5.4  @risingstack/docker-node  

Once you maintain more projects, it can become an overwhelming task to keep all your dependencies up to date in each of your projects. To automate this task, you can use Greenkeeper which will automatically send pull requests to your repositories once a dependency is updated.

#7 No devDepenendencies in production

Development dependencies are called development dependencies for a reason - you don't have to install them in production. It makes your deployment artifacts smaller and more secure, as you will have less modules in production which can have security problems.

To install production dependencies only, run this:

$ npm install --production

Alternatively, you can set the NODE_ENV environment variable to production:

$ NODE_ENV=production npm install

"Don't install development dependencies in production" via @RisingStack #nodejs

Click To Tweet

#8 Secure your projects and tokens

In case of using npm with a logged in user, your npm token will be placed in the .npmrc file. As a lot of developers store dotfiles on GitHub, sometimes these tokens get published by accident. Currently, there are thousands of results when searching for the .npmrc file on GitHub, with a huge percentage containing tokens. If you have dotfiles in your repositories, double check that your credentials are not pushed!

Another source of possible security issues are the files which are published to npm by accident. By default npm respects the .gitignore file, and files matching those rules won't be published. However, if you add an .npmignore file, it will override the content of .gitignore - so they won't be merged.

#9 Developing packages

When developing packages locally, you usually want to try them out with one of your projects before publish to npm. This is where npm link comes to the rescue.

What npm link does is that it creates a symlink in the global folder that links to the package where the npm link was executed.

You can run npm link package-name from another location, to create a symbolic link from the globally installed package-name to the /node_modules directory of the current folder.

"Use npm link to test packages locally" via @RisingStack #nodejs

Click To Tweet

Let's see it in action!

# create a symlink to the global folder
/projects/request $ npm link

# link request to the current node_modules
/projects/my-server $ npm link request

# after running this project, the require('request') 
# will include the module from projects/request

Download the whole Learn using npm series as a single pdf

Next up on Node.js at Scale: SemVer and Module Publishing

The next article in the Node.js at Scale series will be a SemVer deep dive with how to publish Node.js modules.

Let me know if you have any questions in the comments!

Controlling the Node.js security risk of npm dependencies

Controlling the Node.js security risk of npm dependencies

This article is a guest post from Guy Podjarny, CEO at Snyk, building dev tools to fix known vulnerabilities in open source components

Open source packages - and npm specifically - are undoubtedly awesome. They make developers extremely productive by giving each of us a wealth of existing functionality just waiting to be consumed. If we had to write all this functionality ourselves, we'd struggle to create a fraction of what we do today.

As a result, a typical Node.js application today consumes LOTS of npm packages, often hundreds or thousands of them. What we often overlook, however, is that each of these packages, alongside its functionality, also pulls in its Node.js security risks. Many packages open new ports, thus increasing the attack surface. Roughly 76% of Node shops use vulnerable packages, some of which are extremely severe; and open source projects regularly grow stale, neglecting to fix security flaws.

Inevitably, using npm packages will expose you to security risks. Fortunately, there are several questions you can ask which can reduce your risk substantially. This post outlines these questions, and how to get them answered.

#1: Which packages am I using?

The more packages you use, the higher the risk of having a vulnerable or malicious package amongst them. This holds true not only for the packages you use directly but also for the indirect dependencies they use.

Discovering your dependencies is as easy as running npm ls in your application’s parent folder, which lists the packages you use. You can use the --prod argument to only display production dependencies (which impact your security the most), and add --long to get a short description of each package. Check out this post to better understand how you can slice and dice your npm depedencies.

~/proj/node_redis $ npm ls --prod --long
[email protected]  
│ /Users/guypod/localproj/playground/node_redis
│ Redis client library
│ git://github.com/NodeRedis/node_redis.git
│ https://github.com/NodeRedis/node_redis
├── [email protected]
│   Extremely fast double-ended queue implementation
│   git://github.com/petkaantonov/deque.git
│   https://github.com/petkaantonov/deque
├── [email protected]
│   Redis commands
│   git+https://github.com/NodeRedis/redis-commands.git
│   https://github.com/NodeRedis/redis-commonds
└── [email protected]
    Javascript Redis protocol (RESP) parser
    git+https://github.com/NodeRedis/node-redis-parser.git
    https://github.com/NodeRedis/node-redis-parser#readme

Figure: Inventorying Node’s redis few dependencies

A new crop of Dependency Management services, such as bitHound and VersionEye, can also list the dependencies you use, as well as track some of the information below.

"The more #npm packages you use, the higher the risk of having a vulnerable or malicious one." via @RisingStack

Click To Tweet

Now that you know what you have, you can ask a few questions to assess the risk each package involves. Below are a few examples of questions you should ask, why you should ask them, and suggestions on how you can get them answered.



See the communication between your Node.js services - check out Trace by RisingStack!


#2: Am I still using this package?

As time goes by and your code changes, you’re likely to stop using certain packages and add new ones instead. However, developers don’t typically remove a package from the project when they stop using it, as some other part of the code may need it.

As a result, projects have a tendency to accumulate unused dependencies. While not directly a security concern, these dependencies unnecessarily grow your attack surface and add clutter to the code. For instance, an attacker may trick one package into loading an unused package with a more severe vulnerability, escalating the potential damage.

"Projects have a tendency to accumulate unused dependencies and quickly grow your attack surface!" via @RisingStack

Click To Tweet

Checking for unused dependencies is most easily done using the depcheck tool. depcheck scans your code for requires and import commands, correlate those with the packages installed or mentioned in your package.json, and provide a report. The command can be tweaked in various ways using command flags, making it easy to automate checking for unused dep’s.

~/proj/Hardy $ depcheck
Unused dependencies  
* cucumber
* selenium-standalone
Unused devDependencies  
* jasmine-node

Figure: Checking for unused dependencies on the Hardy project

#3: Are other developers using this package?

Packages used by many are also more closely watched. The likelihood of someone having already encountered and addressed a security issue on them is higher than in a less utilized package.

For example, the secure-compare package was created to support string comparison that was not susceptible to a timing attack. However, a fundamental flaw in the package led to achieving the exact opposite, making certain comparisons extremely time sensitive (and incorrect).

If you looked more closely, you’d see this package is very lightly used, downloaded only 20 times a day. If this were a more popular package, odds are someone would have found and reported the functional flaw sooner.

The easiest way to assess package usage is its download rate, indicated in the “Stats” section of the npm’s package page. You can extract those stats automatically using the npm stats API, or browse historic stats on npm-stat.com. Alternatively, you can look at the number of “Dependent” packages – other packages that use the current one.

Node.js Security: Download stats for the `redis` package

#4: Am I using the latest version of this package?

Bugs, including security bugs, are constantly found and – hopefully – fixed. Also, it’s quite common to see newly reported vulnerabilities fixed only on the newest major branch of a project.

For instance, in early 2016, a Regular Expression Denial of Service (ReDoS) vulnerability was reported on the HMAC package hawk. ReDoS is a vulnerability where a long or carefully crafted input causes a regular expression match to take a very long time to compute. The processing thread does not serve new requests in the meantime, enabling a denial of service attack with just a small number of requests.

The vulnerability in hawk was quickly fixed in its latest major version stream, 4.x, but left older versions without a fix. Specifically, it left an unfixed vulnerability in the widely used request package, which used [email protected] The author later accepted Snyk’s pull-request with a fix for the 3.x branch, but request users were exposed for a while, and the issue still exists in the older major release branches. This is just one example, but as a general rule, your dependencies are less likely to have security bugs if they’re on the latest version.

You can find out whether or not you’re using the latest version using the npm outdated command. This command also supports the --prod flag to ignore dev dependencies, as well as --json to simplify automation. You can also use Greenkeeper to proactively inform you when you're not using the latest version.

~/proj/handlebars.js $ npm outdated --prod
Package     Current  Wanted  Latest  Location  
async         1.5.2   1.5.2   2.0.1  handlebars  
source-map    0.4.4   0.4.4   0.5.6  handlebars  
uglify-js     2.6.2   2.7.3   2.7.3  handlebars  
yargs        3.32.0  3.32.0   5.0.0  handlebars  

Figure: npm outdated on handlebars prod dependencies

#5: When was this package last updated?

Creating an open source project, including npm packages, is fun. Many talented developers create such projects in their spare time, investing a lot of time and energy in making them good. Over time, however, the excitement often wears off, and life changes can make it hard to find the needed time.

As a result, npm packages often grow stale, not adding features and fixing bugs slowly – if at all. This reality isn’t great for functionality, but it’s especially problematic for security. Functional bugs typically only get in your way when you’re building something new, allowing some leeway for how quickly they’re addressed. Fixing security vulnerabilities is more urgent – once they become known, attackers may exploit them, and so time to fix is critical.

"Fixing security vulnerabilities is urgent – once they become known, attackers will exploit them!" via @RisingStack

Click To Tweet

A good example of this case is a Cross-Site Scripting vulnerability in the marked package. Marked is a popular markdown parsing package, downloaded nearly 2M times a month. Initially released in mid-2011, Marked evolved rapidly over the next couple of years, but the pace slowed in 2014 and work stopped completely in mid-2015.

The XSS vulnerability was disclosed around the same time, and it has remained untouched ever since. The only way to protect yourself from the issue is to stop using marked, or use a Snyk patch, as explained below.

Inspecting your packages for their last update date is a good way to reduce the change you’ll find yourself in such a predicament. You can do so via the npm UI or by running npm view <package> time.modified.

$ npm view marked time.modified
2016-07-30T03:10:20.053Z  

Figure: checking last modified time on marked

#6: How many maintainers do these packages have?

Many npm packages only have a single maintainer, or a very small number of those. While there’s nothing specifically wrong with that, those packages do have a higher risk of getting abandoned. In addition, larger teams are more likely to have at least some members who better understand and care more about security.

Identifying the packages that have only a few maintainers is a good way to assess your risk. Tracking npm maintainers is easily automated by using npm view <pkg> maintainers.

$ npm view express maintainers

[ 'dougwilson <[email protected]>',
  'hacksparrow <[email protected]>',
  'jasnell <[email protected]>',
  'mikeal <[email protected]>' ]

Figure: maintainers of the express package, per npm

However, many packages with a full team behind them publish automatically through a single npm account. Therefore, you’ll do well to also inspect the GitHub repository used to develop this package (the vast majority of npm packages are developed on GitHub). In the example above, you'll find there are 192 contributors to the express repo. Many only made one or two commits, but that's still quite a difference from the 4 listed npm maintainers.

You can find the relevant GitHub repository by running npm view <pkg> repository, and then subsequently run curl https://api.github.com/repos/<repo-user>/<repo-name>/contributors. For instance, for the marked package, you would first run npm view marked repository, and then curl https://api.github.com/repos/chjj/marked/contributors. Alternatively, you can easily see the maintainers, GitHub repository and its contributors via the npm and GitHub web UI.

#7: Does this package have known security vulnerabilities?

The questions above primarily reflect the risk of a future problem. However, your dependencies may be bringing in some security flaws right now! Roughly 15% of packages carry a known vulnerability, in either in their own code or in the dependencies they in turn bring in. According to Snyk's data, about 76% of Node shops use vulnerable dependencies in their applications.

You can easily find such vulnerable packages using Snyk. You can run snyk test in your terminal, or quickly test your GitHub repositories for vulnerable dependencies through the web UI. Snyk's test page holds other testing options.

"15% of #npm packages carry a known vulnerability, 76% of Node shops use vulnerable dependencies!" via @RisingStack

Click To Tweet

Snyk also makes it easy to fix the found issues, using snyk wizard in the terminal or an automated fix pull request. Fixes are done using guided upgrades or open source patches. Snyk creates these patches by back-porting the original fix, and they are stored as part of its Open Source vulnerability database.

Node.js Security: Test git repos with Snyk

Once you're free of vulnerabilities, you should ensure code changes don't make you vulnerable again. If you're using Snyk, you can test if pull requests introduce a vulnerable dependency, or add a test such as snyk test to your build process.

Lastly, when a new vulnerability is disclosed, you'd want to learn about it before attackers do. New vulnerabilities are independently of your code changes, so a CI test is not sufficient. To get an email (and a fix pull request) from Snyk whenever a new vulnerability affects you, click "Watch" on the "Test my Repositories" page, or run snyk monitor when you deploy new code.

Solving Node.js Security

npm packages are amazing, and let us build software at an unprecedented pace. You should definitely keep using npm packages - but there's no reason to do so blindly. We covered 7 questions you can easily answer to understand better, and reduce your security exposure:

  1. Which packages am I using? And for each one...
  2. Am I still using this package?
  3. Are other developers using this package?
  4. Am I using the latest version of this package?
  5. When was this package last updated?
  6. How many maintainers do these packages have?
  7. Does this package have known security vulnerabilities?

Answer those, and you'll be both productive and secure!


If you have any thoughts or questions on the topic, please share them in the comments.

This article is a guest post from Guy Podjarny, CEO at Snyk, building dev tools to fix known vulnerabilities in open source components


Node Hero - Node.js Request Module Tutorial

This is the 6th part of the tutorial series called Node Hero - in these chapters, you can learn how to get started with Node.js and deliver software products using it.

In the following tutorial, you will learn the basics of HTTP, and how you can fetch resources from external sources using the Node.js request module.

Upcoming and past chapters:

  1. Getting started with Node.js
  2. Using NPM
  3. Understanding async programming
  4. Your first Node.js HTTP server
  5. Node.js database tutorial
  6. Node.js request module tutorial [you are reading it now]
  7. Node.js project structure tutorial
  8. Node.js authentication using Passport.js
  9. Node.js unit testing tutorial
  10. Debugging Node.js applications
  11. Node.js Security Tutorial
  12. How to Deploy Node.js Applications
  13. Monitoring Node.js Applications

What's HTTP?

HTTP stands for Hypertext Transfer Protocol. HTTP functions as a request–response protocol in the client–server computing model.

HTTP Status Codes

Before diving into the communication with other APIs, let's review the HTTP status codes we may encounter during the process. They describe the outcome of our requests and are essential for error handling.

  • 1xx - Informational

  • 2xx - Success: These status codes indicate that our request was received and processed correctly. The most common success codes are 200 OK, 201 Created and 204 No Content.

  • 3xx - Redirection: This group shows that the client had to do an additional action to complete the request. The most common redirection codes are 301 Moved Permanently, 304 Not Modified.

  • 4xx - Client Error: This class of status codes is used when the request sent by the client was faulty in some way. The server response usually contains the explanation of the error. The most common client error codes are 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 409 Conflict. ​

  • 5xx - Server Error: These codes are sent when the server failed to fulfill a valid request due to some error. The cause may be a bug in the code or some temporary or permanent incapability. The most common server error codes are 500 Internal Server Error, 503 Service Unavailable. ​ If you'd like to learn more about HTTP status codes, you can find a detailed explanation about them here.

Sending Requests to External APIs

Connecting to external APIs is easy in Node. You can just require the core HTTP module and start sending requests.

Of course, there are much better ways to call an external endpoint. On NPM you can find multiple modules that can make this process easier for you. Fo​r example, the two most popular ones are the request and superagent modules.

Both of these modules have an error-first callback interface that can lead to some issues (I bet you've heard about Callback-Hell), but luckily we have access to the promise-wrapped versions.



See the communication between your Node.js services - check out Trace by RisingStack!


Using the Node.js Request Module

Using the request-promise module is simple. After installing it from NPM, you just have to require it:

const request = require('request-promise')  

​ Sending a GET request is as simple as:

const options = {  
  method: 'GET',
  uri: 'https://risingstack.com'
}
​
request(options)  
  .then(function (response) {
    // Request was successful, use the response object at will
  })
  .catch(function (err) {
    // Something bad happened, handle the error
  })

If you are calling a JSON API, you may want the request-promise to parse the response automatically. In this case, just add this to the request options:

json: true  

​ POST requests work in a similar way:

const options = {  
  method: 'POST',
  uri: 'https://risingstack.com/login',
  body: {
    foo: 'bar'
  },
  json: true 
    // JSON stringifies the body automatically
}
​
request(options)  
  .then(function (response) {
    // Handle the response
  })
  .catch(function (err) {
    // Deal with the error
  })

​ To add query string parameters you just have to add the qs property to the options object:

const options = {  
  method: 'GET',
  uri: 'https://risingstack.com',
  qs: {
    limit: 10,
    skip: 20,
    sort: 'asc'
  }
}

This will make your request URL: https://risingstack.com?limit=10&skip=20&sort=asc.
​ You can also define any header the same way we added the query parameters:

const options = {  
  method: 'GET',
  uri: 'https://risingstack.com',
  headers: {
    'User-Agent': 'Request-Promise',
    'Authorization': 'Basic QWxhZGRpbjpPcGVuU2VzYW1l'
  }
}

Error handling

Error handling is an essential part of making requests to external APIs, as we can never be sure what will happen to them. Apart from our client errors the server may respond with an error or just send data in a wrong or inconsistent format. Keep these in mind when you try handling the response. Also, using catch for every request is a good way to avoid the external service crashing our server.

Putting it together

As you have already learned how to spin up a Node.js HTTP server, how to render HTML pages, and how to get data from external APIs, it is time to put them together!

In this example, we are going to create a small Express application that can render the current weather conditions based on​ city names.

(To get your AccuWeather​ API key, please visit their developer site)

const express = require('express')  
const rp = require('request-promise')  
const exphbs = require('express-handlebars')

const app = express()

app.engine('.hbs', exphbs({  
  defaultLayout: 'main',
  extname: '.hbs',
  layoutsDir: path.join(__dirname, 'views/layouts')
}))
app.set('view engine', '.hbs')  
app.set('views', path.join(__dirname, 'views'))

app.get('/:city', (req, res) => {  
  rp({
    uri: 'http://apidev.accuweather.com/locations/v1/search',
    qs: {
      q: req.params.city,
      apiKey: 'api-key'
         // Use your accuweather API key here
    },
    json: true
  })
    .then((data) => {
      res.render('index', data)
    })
    .catch((err) => {
      console.log(err)
      res.render('error')
    })
})

app.listen(3000)  

The example above does the following:

  • creates an Express server
  • sets up the handlebars structure - for the .hbs file please refer to the Node.js HTTP tutorial
  • sends a request to the external API
    • if everything is ok, it renders the page
    • otherwise, it shows the error page and logs the error

Download the whole Node Hero series as a single pdf

Next up

In the next chapter of Node Hero you are going to learn how to structure your Node.js projects correctly.

In the meantime try out integrating with different API providers and if you run into problems or questions, don't hesitate to share them in the comment section!


Node Hero - Using NPM: Tutorial

Node Hero - Using NPM: Tutorial

This is the second post of the tutorial series called Node Hero - in these chapters you can learn how to get started with Node.js and deliver software products using it. Planned and past chapters:

  1. Getting started with Node.js
  2. Using NPM [you are reading it now]
  3. Understanding async programming
  4. Your first Node.js server
  5. Node.js Database Tutorial
  6. Node.js request module tutorial
  7. Node.js project structure tutorial
  8. Node.js authentication using Passport.js
  9. Node.js unit testing tutorial
  10. Debugging Node.js applications
  11. Node.js Security Tutorial
  12. How to Deploy Node.js Applications
  13. Monitoring Node.js Applications

In this chapter, you'll learn what NPM is and how to use it. Let's get started!

NPM in a Nutshell

NPM is the package manager used by Node.js applications - you can find a ton of modules here, so you don't have to reinvent the wheel. It is like Maven for Java or Composer for PHP. There are two primary interfaces you will interact with - the NPM website and the NPM command line toolkit.

Both the website and the CLI uses the same registry to show modules and search for them.

The Website

The NPM website can be found at https://npmjs.com. Here you can sign up as a new user or search for packages.

npm website for Node Hero using npm tutorial

The Command Line Interface

To run the CLI you can run it simply with:

npm  

Note, that NPM is bundled with the Node.js binary, so you don't have to install it - however, if you want to use a specific npm version, you can update it. If you want to install npm version 3, you can do that with: npm install [email protected] -g.



Node.js Debugging Made Easy - check out Trace by RisingStack!

Using NPM: Tutorial

You have already met NPM in the last article on Getting started with Node.js when you created the package.json file. Let's extend that knowledge!

Adding Dependencies

In this section you are going to learn how to add runtime dependencies to your application.

Once you have your package.json file you can add dependencies to your application. Let's add one! Try the following:

npm install lodash --save  

With this single command we achieved two things: first of all, lodash is downloaded and placed into the node_modules folder. This is the folder where all your external dependencies will be put. Usually, you don't want to add this folder to your source control, so if you are using git make sure to add it to the .gitignore file.

This can be a good starting point for your .gitignore

Let's take a look at what happened in the package.json file! A new property called dependencies have appeared:

"dependencies": {
  "lodash": "4.6.1"
}

This means that lodash with version 4.6.1 is now installed and ready to be used. Note, that NPM follows SemVer to version packages.

Given a version number MAJOR.MINOR.PATCH, increment the MAJOR version when you make incompatible API changes, MINOR version when you add functionality in a backwards-compatible manner, and PATCH version when you make backwards-compatible bug fixes. For more information: http://semver.org/

As lodash is ready to be used, let's see how we can do so! You can do it in the same way you did it with your own module except now you don't have to define the path, only the name of the module:

// index.js
const _ = require('lodash')

_.assign({ 'a': 1 }, { 'b': 2 }, { 'c': 3 });  
// → { 'a': 1, 'b': 2, 'c': 3 }

Adding Development Dependencies

In this section you are going to learn how to add build-time dependencies to your application.

When you are building web applications, you may need to minify your JavaScript files, concatenating CSS files and so on. The modules that will do it will be only ran during the building of the assets, so the running application doesn't need them.

You can install such scripts with:

npm install mocha --save-dev  

Once you do that, a new section will appear in your package.json file called devDependencies. All the modules you install with --save-dev will be placed there - also, they will be put in the very same node_modules directory.

NPM Scripts

NPM script is a very powerful concept - with the help of them you can build small utilities or even compose complex build systems.

The most common ones are the start and the test scripts. With the start you can define how one should start your application, while test is for running tests. In your package.json they can look something like this:

  "scripts": {
    "start": "node index.js",
    "test": "mocha test",
    "your-custom-script": "echo npm"
  }

Things to notice here:

  • start: pretty straightforward, it just describes the starting point of your application, it can be invoked with npm start
  • test: the purpose of this one is to run your tests - one gotcha here is that in this case mocha doesn't need to be installed globally, as npm will look for it in the node_modules/.bin folder, and mocha will be placed there as well. It can be invoked with: npm test.
  • your-custom-script: anything that you want, you can pick any name. It can be invoked with npm run your-custom-script - don't forget the run part!

Scoped / Private Packages

Originally NPM had a global shared namespace for module names - with more than 250.000 modules in the registry most of the simple names are already taken. Also, the global namespace contains public modules only.

NPM addressed this problem with the introduction of scoped packages. Scoped packages has the following naming pattern:

@myorg/mypackage

You can install scoped packages the same way as you did before:

npm install @myorg/mypackage --save-dev  

It will show up in your package.json in the following way:

"dependencies": {
  "@myorg/mypackage": "^1.0.0"
}

Requiring scoped packages works as expected:

require([email protected]/mypackage')  

For more information refer to the NPM scoped module docs.

Download the whole Node Hero series as a single pdf

Next Up: Async Programming

In the next chapter, you can learn the principles of async programming using callbacks and Promises - subscribe to our newsletter for updates.

In the meantime if you have any questions, don't hesitate to ask!

Using the Private NPM Registry from Docker

It has been a while since NPM had announced support for private modules - still, to make it with Docker you need some work to do. In this quick tip, you can learn how to use private Node.js modules with Docker.

During this guide I am going to use the RisingStack Alpine Node.js Docker image.

Setting up the Dockerfile

Let's start with the following simple Dockerfile:

It doesn't do anything extraordinary, just grabs the Alpine image, copies over the package.json file, installs the dependencies, copies the source files and starts the process.

Adding the NPM token

To access the private modules in NPM, we need to pass the NPM_TOKEN environment variable to the Docker image.

The naive approach would be to add it using the ENV:

ENV NPM_TOKEN=token  

However, it does not work. The variables set with ENV are for runtime only.

Luckily since Docker v1.9 there is a new flag available for passing in build-time environment variables: the --build-arg. To make it work we have to modify our Dockerfile:

Note a couple of changes:

  • ARG: with this we can tell Docker, that we will use the NPM_TOKEN build argument
  • COPY .npmrc: using this line we add an .npmrc file to the project. Its content: //registry.npmjs.org/:_authToken=${NPM_TOKEN}
  • rm -f .npmrc: simply remove the .npmrc file after npm install is done

To build the image using this image and the token, you can run Docker:

docker build --build-arg NPM_TOKEN=${NPM_TOKEN} .  

One thing to keep in mind: even if you delete the .npmrc it will be kept in the commit history - to clean your secret entirely up make sure to squash them.

Let me know if you have any questions or comments.


Private NPM Modules

Private NPM is here. Congratulations to the NPM team for shipping it!

NPM just announced the long-awaited support for private modules - you can try them today!

Setup

First you have sign up here. After setting it up, you are good to go.

Next thing is to make sure you npm login with the account you just upgraded.

Creating Your First Private NPM Module

After you are done with the previous tasks, it is pretty much the same as before: you can create your module using npm init - the only thing you have to pay attention to is the naming convention of your package. All private packages are scoped, which is your username. So to create one you have to enter @usernam/package-name as the name of the module:

gergelyke ~/Development/risingstack/test-private $ npm init  
This utility will walk you through creating a package.json file.  
It only covers the most common items and tries to guess sane defaults.

See `npm help json` for definitive documentation on these fields  
and exactly what they do.

Use `npm install <pkg> --save` afterward to install a package and save it as a dependency in the package.json file.

Press ^C at any time to quit.  
name: (test-private) @risingstack/test-private  
version: (1.0.0)  
description:  
entry point: (index.js)  
test command:  
git repository:  
keywords:  
author:  
license: (ISC)  
About to write to /Users/gergelyke/Development/risingstack/test-private/package.json:

{
  "name": "@risingstack/test-private",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}

You are all set, it is time to implement your module! If you need a tutorial on how to get started doing that, I encourage you to read our Publishing your first Browserify/Node module

Publishing

Same as always - npm publish.

Access Control

You add readonly and read/write access to contributors using either the CLI or the NPM website. With access control we arrive to a currently missing functionality: organizations are not supported for now, but they are on the roadmap.

Outro

Should you start using it today? If your team is small, then yes. If you are managing a bigger engineering team then currently it is not for you - you should wait for the organization support.

Publishing your first Browserify/Node module

NPM has just surpassed the 100.000-packages milestone while more and more packages can be run in the browser as well (the so called isomorphic JavaScript). If you have not published a module before, this article is definitely for you.

To be able to run node packages in the browser you need either Browserify or Webpack. In this article we will go with Browserify.

To install Browserify open a terminal and type:

npm install -g browserify  

Hello Browserify

To get started with your first Browserify module, let's create a file containing:

// main.js
var $ = require('jquery');  
$('body').append('<p>Hello Browserify!</p>');

Then run:

npm install jquery --save  
browserify main.js -o bundle.js  

Wow, what just happened? First, we required jQuery, which will be looked up from the node_modules directory. So to place jQuery there, we installed it from NPM. The last thing is to create a bundle with the browserify command.

Also, if you want to create your bundle on each file change you can use watchify.

After this, create an HTML file and include the script:

<script src="bundle.js"></script>  

The Hello Browserify should be appended to the body of the HTML page.

Setting up a package to publish

After you have just created your very first browserify module, it is time to set up a project - a project that can be published to NPM.

To init a project with NPM, you should use:

npm init  

This will ask for the project name, description and some basics. Fill them out! :)

Something like this should be in your package.json:

{
  "name": "MyFirstModule",
  "version": "1.0.0",
  "description": "Gonna be great",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Gergely Nemeth",
  "license": "ISC"
}

Adding functionality

In the generated package.json the main property tells NPM which is the entry point of the application. In this case it is the index.js file. The logic of your module should be here - it is also common to put everything under ./lib/ - in this case your index.js would only be like this:

module.exports = require('./lib');  

Test-all-the-things!

Make sure to provide test cases to your library - it not just helps people to contribute to it but can easily become the documentation for it as well.

More on testing can be read at our guest post at the Joyent blog.

Run your tests

Running your tests not just against Node.js but different browsers is essential to make sure everything works everywhere.

For testing let's use tape.

For Node.js

To run your tests in Node.js Travis is a great choice - also it is free for open source projects.

Setting up just takes a .travis.yml file:

node_js:  
  - "0.10"
  - "0.11"

It will run npm test, so make sure the scripts part of your package.json has one for testing, like:

"scripts": {
    "test": "tape test.js"
  }

The only thing left to do is to set up a commit hook on GitHub to enable Travis. (Settings > Webhooks & Services)

For the browser

Testling is kind of the same thing like Travis, but tailored for the browser.

Setting up takes a little bit more effort as you have to define the browsers to run against:

"testling": {
    "files": "test/*.js",
    "browsers": [
      "ie/6..latest",
      "chrome/22..latest",
      "firefox/16..latest",
      "safari/latest",
      "opera/11.0..latest",
      "iphone/6",
      "ipad/6",
      "android-browser/latest"
    ]
  },

To enable Testling the only thing is left is to enable webhooks on GitHub like we did with Travis. If you need more info, check the official site.

Publishing

We have the tests, we have functionality - publish it!

To do so you only have to give this command to npm:

npm publish  

This will create a tar from your module and publish it to NPM.

Things to consider:

  • it is great that you have tests, but to run your module you may not need them - for this reason you can use the .npmignore file
  • you cannot republish the same version - if you change something please bump the version number in your package.json accordingly to Semver.

Example project

An example project can be found in our GitHub repository: https://github.com/RisingStack/your-first-browserify-module

If you have any questions, do not hesitate to create an issue there or comment here!

Setting Up NPM Registries

Node is great, NPM may be even greater. But would not it be awesome to have your own repository/mirror, so you will not be dependent on the public repository?

Let’s take a look at your options:

The old-school way

This setup is like how NPM was a few years back and uses the same stack. For this, you will need:

  • CouchDB
  • Node.js server

After you have the prerequisites, what you have to do is configure the CouchDB to sync from the official public repository.

After it is finished, you only have to deploy npmjs, which means cloning from git, and make some configuration.

With a proxy

Yammer created a lazy NPM mirror which caches the responses from the official registry. For this, you will need:

  • nginx

After you have a functioning nginx proxy, a cache zone must be set:

 # this is the npm zone, things stay active for 3 days
proxy_cache_path /var/cache/npm/data levels=1:2 keys_zone=npm:20m max_size=1000m inactive=3d;  
proxy_temp_path /var/cache/npm/tmp;  

Then site configuration have to be added to:

server {  
 listen 80;
 server_name npm.example.com;
  location / {
  proxy_pass https://registry.npmjs.org/;
  proxy_cache npm;
  proxy_cache_valid 200 302 3d;
  proxy_cache_valid 404 1m;
  sub_filter 'registry.npmjs.org' 'npm.example.com';
  sub_filter_once off;
  sub_filter_types application/json;
  }
 }

That’s it, you should be good to go.

With Sinopia

Sinopia is a private/caching npm repository server. For Sinopia you do not need any database, only Node.js — it uses file system as storage.

After installing and launching it, it is good to go. Another great thing about Sinopia is that it has a Chef cookbook: https://github.com/BarthV/sinopia-cookbook

More info: https://github.com/rlidwka/sinopia

With cnpmjs

Private npm registry and web for Enterprise” — at least this is what cnpmjs states. It uses MySQL, and Simple Store Service. Also, cnpm is written using Koa, which uses the new ES6 generators — to run this, you will need at least Node.js version 0.11. This is the system that powers the Chinese NPM mirror.

More info and installation: https://github.com/cnpm/cnpmjs.org

So far so nice. Would not it be awesome to easily switch between the public repository and yours?

As of 23th May (2014) npm supports per-project .npmrc files.

When working locally in a project, a .npmrc file in the root of the project (ie, a sibling of node_modules and package.json) will set config values specific to this project.

Note that this only applies to the root of the project that you're running npm in. It has no effect when your module is published. For example, you can't publish a module that forces itself to install globally, or in a different location.

More on .npmrc at: https://www.npmjs.org/doc/files/npmrc.html