Expert Node.js Support
Learn more

debugging

Announcing Free Node.js Monitoring & Debugging with Trace

Announcing Free Node.js Monitoring & Debugging with Trace

Today, we’re excited to announce that Trace, our Node.js monitoring & debugging tool is now free for open-source projects.

What is Trace?

We launched Trace a year ago with the intention of helping developers looking for a Node.js specific APM which is easy to use and helps with the most difficult aspects of building Node projects, like..

  • finding memory leaks in a production environment
  • profiling CPU usage to find bottlenecks
  • tracing distributed call-chains
  • avoiding security leaks & bad npm packages

.. and so on.

Node.js Monitoring with Trace by RisingStack - Performance Metrics chart

Why are we giving it away for free?

We use a ton of open-source technology every day, and we are also the maintainers of some.

We know from experience that developing an open-source project is hard work, which requires a lot of knowledge and persistence.

Trace will save a lot of time for those who use Node for their open-source projects.

How to get started with Trace?

  1. Visit trace.risingstack.com and sign up - it's free.
  2. Connect your app with Trace.
  3. Head over to this form and tell us a little bit about your project.

Done. Your open-source project will be monitored for free as a result.

If you need help with Node.js Monitoring & Debugging..

Just drop us a tweet at @RisingStack if you have any additional questions about the tool or the process.

If you'd like to read a little bit more about the topic, I recommend to read our previous article The Definitive Guide for Monitoring Node.js Applications.

One more thing

At the same time of making Trace available for open-source projects, we're announcing our new line of business at RisingStack:

Commercial Node.js support, aimed at enterprises with Node.js applications running in a production environment.

RisingStack now helps to bootstrap and operate Node.js apps - no matter what life cycle they are in.


Disclaimer: We retain the exclusive right to accept or deny your application to use Trace by RisingStack for free.

Case Study: Finding a Node.js Memory Leak in Ghost

Case Study: Finding a Node.js Memory Leak in Ghost

At RisingStack, we have been using Ghost from the very beginning, and we love it! As of today we have more than 125 blogposts, with thousands of unique visitors every day, and with 1.5 million pageviews in 2016 overall.

In this post I’m going to share the story of how we discovered a node.js memory leak in [email protected], and what role Trace played in the process of detecting and fixing it.

What's Ghost?

Just a blogging platform

Node.js Memory Leak - The Ghost blogging platforms logo

Ghost is a fully open-source publishing platform written entirely in JavaScript. It uses Node.js for the backend, Ember.js for the admin side and Handlebars.js to power the rendering.

Ghost is actively developed - in the last 30 days, it had 10 authors with 66 commits to the master branch. The project's roadmap can be found here: https://trello.com/b/EceUgtCL/ghost-roadmap.

You can open an account at https://ghost.org/ and start writing instantly - or you can host your own version of Ghost, just like we do.

Our Ghost Deployment

Firstly, I'd like to give you a quick overview of how we deploy and use Ghost in production at RisingStack. We use Ghost as a npm module, required into a bigger project, something like this:

// adding Trace to monitor the blog
require([email protected]/trace')  
const path = require('path')  
const ghost = require('ghost')

ghost({  
  config: path.join(__dirname, 'config.js')
}).then(function (ghostServer) {
  ghostServer.start()
})

Deployments are done using Circle CI which creates a Docker image, pushes it to a Docker registry and deploys it to a staging environment. If everything looks good, the updates are moved to the production blog you are reading now. As a backing database, the blog uses PostgreSQL.

The Node.js Memory Leak

As we like to keep our dependencies up-to-date, we updated to [email protected] as soon as it came out. Once we did this, our alerts started to fire, as memory usage started to grow:

Node.js Memory leak in ghost - Trace memory metrics

Luckily, we had alerts set up for memory usage in Trace, which notified us that something is not right. As Trace integrates with Opsgenie and Pagerduty seamlessly, we could have set up alerts for those channels.

Node.js Monitoring and Debugging from the Experts of RisingStack

Set up alerts for your Node.js deployments using Trace
Learn more

We set up alerts for the blog service at 180 and 220 Mb because usually it consumes around 150 Mb when everything’s all right.

Setting up alerts for Node.js memory leaks in Trace

What was even better, is that the alerting was set up in a way that it triggered actions on the collector level. What does this mean? It means, that Trace could create a memory heapdump automatically, without human intervention. Once we started to investigate the issue, the memory heapdump was already in the Profiler section of Trace in the format that's supported by the Google Chrome DevTools.

This enabled us to start looking at the problem instantly, and in a way it happened in the production system, not by trying to reproduce the issue in a local development environment.

Also, as we could take multiple heapdumps from the application itself, we could compare them using the comparison view of the DevTools.

Memory heapshot comparison with Trace and Chrome's Devtools

How to use the comparison view to find the source of a problem? On the picture above, you can see that I compared the heapdump that Trace automatically collected when the alert was triggered with a heapdump that was requested earlier, when everything was ok with the service.

"You can use 2 heapdumps & Chromes comparison view to investigate a Node.js memory leak!" via @RisingStack #nodejs

Click To Tweet

What you have to look for is the #Delta, which shows +772 in our case. This means that at the time our high memory usage alert was triggered the heapdump had an extra 772 objects in it. On the bottom of the picture you can see what were these elements, and that they have something to do with the lodash module.

"The "delta" shows the extra objects in your memory heapdump in case of a leak!" via @RisingStack #nodejs

Click To Tweet

Figuring this out otherwise would be extremely challenging since you’d have to reproduce the issue in a local environment - which is tricky if you don’t even know what caused it.

Should I update? Well..

The final cause of the leak was found by Katharina Irrgang, a core Ghost contributor. To check out the whole thread you can take a look at the GitHub issue: https://github.com/TryGhost/Ghost/issues/7189 . A fix was shipped with 0.10.1. - but updating to it will cause another issue: slow response times.

Slow Response Times

Once we upgraded to the new version, we ran into a new problem - our blog's response time started to degrade. The 95 percentile grew from 100ms to almost 300ms. It instantly triggered our alerts set for response times.

Slow response time graph from Trace

For the slow response time we started to take CPU profiles using Trace. For now, we are still investigating the exact reason, but so far we suspect something is off with how moment.js is used.

CPU profile analysis with Trace

We will update the post once we found why it happens.

Conclusion

I hope this article helped you to figure out what to do in case you're experiencing memory leaks in your Node.js applications. If you'd like to get memory heapdumps automatically in a case like this, connect your services with Trace and enable alerting just like we did earlier.

If you have any additional questions, you can reach me in the comments section!

How Developers use Node.js - Survey Results

How Developers use Node.js - Survey Results

RisingStack, the provider of Trace - a next-gen Node.js debugging and performance monitoring solution and silver member of the Node Foundation conducted a survey during 2016 Summer to find out how developers use Node.js and what technologies they prefer with it. This article summarizes the results.

The results show that MongoDB, RabbitMQ, AWS, Jenkins, Docker and Amazon Container Services are the go-to choices for developing, containerizing and shipping Node.js applications.

The survey also let us find out various aspects of developing Node.js and choices for async control flow, debugging, continuous integration or finding packages. The results also tell Node developers major pain-point: debugging.

The survey was open for 35 days from 11 July until 15 August 2016. During this period, 1126 Node.js developers completed it. 55% of them have more than two years of Node.js experience, while 26% uses Node between one and two years. 20% works at a company that is publicly traded, 7% at a Fortune 500 enterprise.

Technologies used with Node.js

MongoDB became the go-to database

Node.js Survey - What databases are you using? MongoDB wins.

According to the results, MongoDB is clearly the go-to database for Node.js developers. Roughly ⅔ of our respondents claimed that they use MongoDB with their Node.js applications. It's also worth noticing that the popularity of Redis is massively increasing with the experience of Node engineers. This trend is also true in the case of PostgreSQL and ElasticSearch.

Node.js Monitoring and Debugging from the Experts of RisingStack

Build performant applications using Trace
Learn more

Node.js Survey - Database usage and developer experience

"The popularity of Redis and PostgreSQL is heavily increasing with developer XP" via @RisingStack #nodejs #survey

Click To Tweet


Redis leads as a caching solution, but a lot of developers still don’t do it

Node.js Survey - What do you use for caching? Redis wins.

Half of our respondents said that they are using Redis for caching, but a staggering 45% stated that they don’t use any. Cross referencing the answers with developer experience allows us to see that the popularity of Redis is quite high amongst long-time Node users, compared to engineers with less than one year of Node.js experience.

Node.js Survey - Caching usage and developer experience

The popularity of messaging systems is still low

According to our survey, 58% of Node.js developers don't use any messaging systems. This means that either developers are rarely using messaging in their microservices system, they use REST API-s or they don’t have a sophisticated system in place.

Node.js Survey - What messaging systems are you using? RabbitMQ wins.

Those who use Messaging systems answered that they mostly use RabbitMQ (24% of all respondents). If we only investigate the responses of people who use messaging systems, RabbitMQ beats the rest of the existing solutions by far.

Node.js Survey - Messaging system usage

Node.js apps are most likely running on AWS

According to our survey, 43% of Node.js developers use AWS for running their applications, but running an own datacenter is popular as well (34%) especially amongst enterprises (nearly 50% of them has own datacenters) - but this is no surprise.

Node.js Survey - Where do you run your Node.js apps? AWS.

"43% of Node.js developers use AWS for running their applications" via @RisingStack #nodejs #survey @awscloud

Click To Tweet

What’s interesting though is that Heroku and DigitalOcean are competing neck and neck to become the second biggest cloud platform for Node.js. According to our data, DigitalOcean is more popular with smaller companies (under 50) while Heroku stays strong as an enterprise solution as well.

Node.js Survey - Running apps and company size

Docker dominates in the Node community

Currently, Docker containers are the go-to solution for most of Node.js developers (47% of all respondents claimed to use it - which is 73% of all container tech users in the survey). Docker seems to be equally popular within all company sizes - but advanced developers appear to be using it much more (the ones with over one year’s of experience).

Node.js Survey - What container techs or VMs are you using? Docker.

"Docker containers are the go-to solution for most of Node.js developers." via @RisingStack #nodejs #survey @docker

Click To Tweet

64% of the respondents said that they use some container technology - which means that the popularity of containers rose since the last major Node.js survey from 45% with a significant 20% increase since January 2016.

Node.js Survey - Container techs and developer experience.

Amazon Container Service is the first choice for running containers

Node.js Survey - How do you run your containers? Amazon Container Service wins.

While Amazon Container Service leads as the choice of running containers with Node.js, it’s worth noting that Kubernetes is already on 25% according to our survey, and it seems to be popular especially with enterprise Node.js developers.

Node.js Survey - Running Amazon and Kubernetes with company size

Node.js development

Configuration files are being used more often than environmental variables

The majority of Node developers (59% vs. 38%) prefer config files over credentials. Only 29 respondents (3%) stated that they use both.

Node.js Survey - Environment variables or config files? Config files wins.

Using only configuration files suggests a possible security problem since it implies that credentials are stored in the repositories. If you have the credentials to productions systems in GitHub, you can quickly run into trouble with rogue developers.

Using Env vars is highly recommended for secrets - while developers can still use config files in general.

Promises lead with async control flow

In Node.js - most of the core libraries are working with callbacks. The results show that Node.js users are leaning towards using promises right now.

Node.js Survey - What do you use for async control flow? Promises wins.

Around half a year ago there was a pull-request in the core Node.js repository asking for async functions to return a native Promise. The answer for this was: “A Promises API doesn’t make sense for core right now because it's too early in the evolution of V8-based promises and their relationship to other ES* features. There is tiny interest within the TC in exploring this in the core in the short-term.”

Maybe it’s time to revisit the issue - since the demand is present.

Developers trust the console.log for debugging

Console.log is leading the race amongst other debugging solutions like the Node Inspector, the Built-in debugger and the debug module. Around ¾ of Node developers use it for finding errors in their applications - while much-sophisticated solutions are available as well.

Node.js Survey - How do you debug your applications? Using the console.log

"Around ¾ of Node developers use console.log to find errors in their applications." via @RisingStack #nodejs #survey

Click To Tweet

A closer look at the data lets us know that more experienced developers are leaning towards the Node Inspector and the Debug Module as well.

Node.js Survey - Debugging applications and developer experience

APM’s are still quite unpopular in the Node.js community

According to the responses in our survey, only ¼ of Node.js developers use APMs - application performance monitoring tools - to identify issues in their applications. Although, emerging trends in the dataset suggest that APM usage grows with company size and developer experience.

Node.js Survey - How do you identify issues in your app? Using logs.

SaaS CI’s still have a low market share in the Node community

According to the answers in our survey, using shell scripts is the most popular way of pushing code to staging or production environments - but Jenkins clearly wins among continuous delivery and integration platforms so far, and is becoming more popular as company size increases.

Node.js Survey - What do you use to push code or containers? Shell scripts win.

Node.js developers rarely update dependencies

Frequently updating dependencies is highly recommended with Node.js applications - since around 15% of npm packages carry a known vulnerability & 76% of Node shops use vulnerable dependencies according to a recent survey.

Node.js Survey - How often do you update dependencies? Less frequently than a month.

Updating dependencies less frequently than every week exposes applications to severe attacks all the time. According to our survey, 45% of Node.js developers update dependencies less frequently than a month, and 27% of them update dependencies month-by-month. Only 28% answered that they update dependencies at least every week.

These numbers correlate neither with company size nor with developer experience.

Node.js developers Google for their packages

According to our survey, the majority of developers use Google to find packages and decide which one of them they should use. Although the popularity of the npmjs.org/npms.io search platforms is 56% amongst our respondents, the data shows that it goes up to almost 70% for the demographic group of experienced (more than four years of Node development)! Preference increases with experience in this case.

Node.js Survey - How do you decide what package to pick? People mostly Google for them.

Junior Node.js developers don’t know what semantic versioning is

Although 71% of our respondents uses semantic versioning when publishing/consuming modules, this number should be higher in our opinion. Everyone should use semantic versioning since npm works with semver! Updating packages without using it can easily break Node.js applications.

Node.js Survey - Do you use semantic versioning? Mostly yes.

If we dig deeper in the dataset we can see that around half of the Node developers with less than a year experience don’t know what semver is or don’t use it - while advanced developers are embracing it on a much higher level.

Node.js teams introduce new tools and technologies very fast

According to our survey 35% of Node developers can introduce new tech/tools/product to their companies in a few days, and 29% in just a few weeks.

Node.js Survey - How much time is needed to introduce new technologies, tools or products to your company? A few weeks.

"35% of Node devs can introduce new tech/tools to their companies in a few days." via @RisingStack #nodejs #survey

Click To Tweet

If we investigate the data more thoroughly, a not-so-surprising pattern emerges which lets us know that the time needed to introduce new tech/tools is gradually increasing with the size of a company.

Debugging is the most severe pain-point for developing with Node.js

We also asked Node developers about what's their biggest pain points regarding development. The top answers were:

  • Debugging / Profiling / Performance Monitoring
  • Callbacks and Callback hell
  • Understanding Async programming
  • Dependency management
  • Lack of conventions/best practices
  • Structuring
  • Bad documentation
  • Finding the right packages

"The biggest pain point of Node.js development is debugging." via @RisingStack #nodejs #survey

Click To Tweet

Conclusion

Developing Node.js is still an interesting and ever-changing experience. We'd like to thank for the the engineers who took their time with answering to our questions, and we hope that the information presented in the article is valuable for the whole Node community.

The full dataset is going to be released and linked in this blogpost in a few days.


Node Hero - Debugging Node.js Applications

Node Hero - Debugging Node.js Applications

This article is the 10th part of the tutorial series called Node Hero - in these chapters, you can learn how to get started with Node.js and deliver software products using it.

In this tutorial, you are going to learn debugging your Node.js applications using the debug module, the built-in Node debugger and Chrome's developer tools.

Bugs, debugging

The term bug and debugging have been a part of engineering jargon for many decades. One of the first written mentions of bugs is as follows:

It has been just so in all of my inventions. The first step is an intuition, and comes with a burst, then difficulties arise — this thing gives out and [it is] then that "Bugs" — as such little faults and difficulties are called — show themselves and months of intense watching, study and labor are requisite before commercial success or failure is certainly reached.

Thomas Edison


Debugging Node.js Applications

One of the most frequently used approach to find issues in Node.js applications is the heavy usage of console.log for debugging.

"Console.log is efficient for debugging small snippets but we recommend better alternatives!” @RisingStack #nodejs

Click To Tweet

Let's take a look at them!

The debug module

Some of the most popular modules that you can require into your project come with the debug module. With this module, you can enable third-party modules to log to the standard output, stdout. To check whether a module is using it, take a look at the package.json file's dependency section.

To use the debug module, you have to set the DEBUG environment variable when starting your applications. You can also use the * character to wildcard names. The following line will print all the express related logs to the standard output.

DEBUG=express* node app.js  

The output will look like this:

Logfile made with the Node debug module

Node.js Monitoring and Debugging from the Experts of RisingStack

Build performant applications using Trace
Learn more

The Built-in Node.js Debugger

Node.js includes a full-featured out-of-process debugging utility accessible via a simple TCP-based protocol and built-in debugging client.

To start the built-in debugger you have to start your application this way:

node debug app.js  

Once you have done that, you will see something like this:

using the built-in node.js debugger

Basic Usage of the Node Debugger

To navigate this interface, you can use the following commands:

  • c => continue with code execution
  • n => execute this line and go to next line
  • s => step into this function
  • o => finish function execution and step out
  • repl => allows code to be evaluated remotely

You can add breakpoints to your applications by inserting the debugger statement into your codebase.

function add (a, b) {  
  debugger
  return a + b
}

var res = add('apple', 4)  


Watchers

It is possible to watch expression and variable values during debugging. On every breakpoint, each expression from the watchers list will be evaluated in the current context and displayed immediately before the breakpoint's source code listing.

To start using watchers, you have to define them for the expressions you want to watch. To do so, you have to do it this way:

watch('expression')  

To get a list of active watchers type watchers, to unwatch an expression use unwatch('expression').

Pro tip: you can switch running Node.js processes into debug mode by sending the SIGUSR1 command to them. After that you can connect the debugger with node debug -p <pid>.

"You can switch running #nodejs processes into debug mode by sending the SIGUSR1 command to them.” via @RisingStack

Click To Tweet

To understand the full capabilities of the built-in debugger, check out the official API docs: https://nodejs.org/api/debugger.html.

The Chrome Debugger

When you start debugging complex applications, something visual can help. Wouldn’t be great to use the familiar UI of the Chrome DevTools for debugging Node.js applications as well?

node.js chrome developer tools debugging

Good news, the Chrome debug protocol is already ported into a Node.js module and can be used to debug Node.js applications.

To start using it, you have to install node-inspector first:

npm install -g node-inspector  

Once you installed it, you can start debugging your applications by starting them this way:

node-debug index.js --debug-brk  

(the --debug-brk pauses the execution on the first line)

It will open up the Chrome Developer tools and you can start to debug your Node.js applications with it.

Download the whole Node Hero series as a single pdf

Next up

Debugging is not that hard after all, is it?

In the next chapter of Node Hero, you are going to learn how to secure your Node.js applications.

If you have any questions or recommendations for this topic, write them in the comments section.


Hunting a Ghost - Finding a Memory Leak in Node.js

Hunting a Ghost - Finding a Memory Leak in Node.js

Finding a Node.js memory leak can be quite challenging - recently we had our fair share of it.

One of our client's microservices started to produce the following memory usage:


Node.js memory leak in Trace

Memory usage grabbed with Trace by RisingStack - our Node.js Performance monitoring and debugging tool

You may spend quite a few days on things like this: profiling the application and looking for the root cause. In this post, I would like to summarize what tools you can use and how, so you can learn from it.

The TL;DR version

In our particular case the service was running on a small instance, with only 512MB of memory. As it turned out, the application didn't leak any memory, simply the GC didn't start collecting unreferenced objects.

Why was it happening? As a default, Node.js will try to use about 1.5GBs of memory, which has to be capped when running on systems with less memory. This is the expected behaviour as garbage collection is a very costly operation.

The solution for it was adding an extra parameter to the Node.js process:

node --max_old_space_size=400 server.js --production  

Still, if it is not this obvious, what are your options to find memory leaks?



Monitor your memory usage and find leaks with ease. - check out Trace by RisingStack!


Understanding V8's Memory Handling

Before diving into the technics that you can employ to find and fix memory leaks in Node.js applications, let's take a look at how memory is handled in V8.

Definitions
  • resident set size: is the portion of memory occupied by a process that is held in the RAM, this contains:
    • the code itself
    • the stack
    • the heap
  • stack: contains primitive types and references to objects
  • heap: stores reference types, like objects, strings or closures
  • shallow size of an object: the size of memory that is held by the object itself
  • retained size of an object: the size of the memory that is freed up once the object is deleted along with its' dependent objects
How The Garbage Collector Works

Garbage collection is the process of reclaiming the memory occupied by objects that are no longer in use by the application. Usually, memory allocation is cheap while it's expensive to collect when the memory pool is exhausted.

An object is a candidate for garbage collection when it is unreachable from the root node, so not referenced by the root object or any other active objects. Root objects can be global objects, DOM elements or local variables.

The heap has two main segments, the New Space and the Old Space. The New Space is where new allocations are happening; it is fast to collect garbage here and has a size of ~1-8MBs. Objects living in the New Space are called Young Generation. The Old Space where the objects that survived the collector in the New Space are promoted into - they are called the Old Generation. Allocation in the Old Space is fast, however collection is expensive so it is infrequently performed .

Why is garbage collection expensive? The V8 JavaScript engine employs a stop-the-world garbage collector mechanism. In practice, it means that the program stops execution while garbage collection is in progress.

Usually, ~20% of the Young Generation survives into the Old Generation. Collection in the Old Space will only commence once it is getting exhausted. To do so the V8 engine uses two different collection algorithms:

  • Scavenge collection, which is fast and runs on the Young Generation,
  • Mark-Sweep collection, which is slower and runs on the Old Generation.

For more information on how this works check out the A tour of V8: Garbage Collection article. For more information on general memory management, visit the Memory Management Reference.

Tools / Technics You Can Use to Find a Memory Leak in Node.js

The heapdump module

With the heapdump module, you can create a heap snapshot for later inspection. Adding it to your project is as easy as:

npm install heapdump --save  

Then in your entry point just add:

var heapdump = require('heapdump');  

Once you are done with it, you can start collecting heapdump with either using the $ kill -USR2 <pid> command or by calling:

heapdump.writeSnapshot(function(err, filename) {  
  console.log('dump written to', filename);
});

Once you have your snapshots, it's time to make sense of them. Make sure you capture multiple of them with some time difference so you can compare them.

Google Chrome DevTools

First you have to load your memory snapshots into the Chrome profiler. To do so, open up Chrome DevTools, go to profiles and Load your heap snapshots.

find a node.js memory leak with chrome load profiles

Once you loaded them it should be something like this:

chrome heap snapshot for finding a node.js memory leak

So far so good, but what can be seen exactly in this screenshot?

One of the most important things here to notice is the selected view: Comparison. This mode enables you to compare two (or more) heap snapshots taken at different times, so you can pinpoint exactly what objects were allocated and not freed up in the meantime.

The other important tab is the Retainers. It shows exactly why an object cannot be garbage collected, what is holding a reference to it. In this case the global variable called log is holding a reference to the object itself, preventing the garbage collector to free up space.

Low-Level Tools

mdb

The mdb utility is an extensible utility for low-level debugging and editing of the live operating system, operating system crash dumps, user processes, user process core dumps, and object files.

gcore

Generate a core dump of a running program with process ID pid.

Putting it together

To investigate dumps, first we have to create one. You can easily do so with:

gcore `pgrep node`  

After you have it, you can search for the all the JS Objects on the heap using:

> ::findjsobjects

Of course, you have to take successive core dumps so that you can compare different dumps.

Once you identified objects that look suspicious, you can analyze them using:

object_id::jsprint  

Now all you have to do is find the retainer of the object (the root).

object_id::findjsobjects -r  

This command will return with id of the retainer. Then you can use ::jsprint again to analyze the retainer.

For a detailed version check out Yunong Xiao's talk from Netflix on how to use it:

Recommended Reading

UPDATE: Read the story of how we found a memory leak in our blogging platform by comparing heapshots with Trace and Chrome's DevTools.

You have additional thoughts or insights on Node.js memory leaks? Share it in the comments.