Trace Node.js Monitoring
Need help with Node.js?
Learn more
Need Node.js support?
Learn more

project structure

Advanced Node.js Project Structure Tutorial - Node.js at Scale

Advanced Node.js Project Structure Tutorial - Node.js at Scale

Project structuring is an important topic because the way you bootstrap your application can determine the whole development experience throughout the life of the project.

In this Node.js project structure tutorial I’ll answer some of the most common questions we receive at RisingStack about structuring advanced Node applications, and help you with structuring a complex project.

These are the goals that we are aiming for:

  • Writing an application that is easy to scale and maintain.
  • The config is well separated from the business logic.
  • Our application can consist of multiple process types.

Node.js at Scale is a collection of articles focusing on the needs of companies with bigger Node.js installations and advanced Node developers. Chapters:

The Node.js Project Structure

Our example application is listening on Twitter tweets and tracks certain keywords. In case of a keyword match, the tweet will be sent to a RabbitMQ queue, which will be processed and saved to Redis. We will also have a REST API exposing the tweets we have saved.

You can take a look at the code on GitHub. The file structure for this project looks like the following:

.
|-- config
|   |-- components
|   |   |-- common.js
|   |   |-- logger.js
|   |   |-- rabbitmq.js
|   |   |-- redis.js
|   |   |-- server.js
|   |   `-- twitter.js
|   |-- index.js
|   |-- social-preprocessor-worker.js
|   |-- twitter-stream-worker.js
|   `-- web.js
|-- models
|   |-- redis
|   |   |-- index.js
|   |   `-- redis.js
|   |-- tortoise
|   |   |-- index.js
|   |   `-- tortoise.js
|   `-- twitter
|       |-- index.js
|       `-- twitter.js
|-- scripts
|-- test
|   `-- setup.js
|-- web
|   |-- middleware
|   |   |-- index.js
|   |   `-- parseQuery.js
|   |-- router
|   |   |-- api
|   |   |   |-- tweets
|   |   |   |   |-- get.js
|   |   |   |   |-- get.spec.js
|   |   |   |   `-- index.js
|   |   |   `-- index.js
|   |   `-- index.js
|   |-- index.js
|   `-- server.js
|-- worker
|   |-- social-preprocessor
|   |   |-- index.js
|   |   `-- worker.js
|   `-- twitter-stream
|       |-- index.js
|       `-- worker.js
|-- index.js
`-- package.json

In this example we have 3 processes:

  • twitter-stream-worker: The process is listening on Twitter for keywords and sends the tweets to a RabbitMQ queue.
  • social-preprocessor-worker: The process is listening on the RabbitMQ queue and saves the tweets to Redis and removes old ones.
  • web: The process is serving a REST API with a single endpoint: GET /api/v1/tweets?limit&offset.

We will get to what differentiates a web and a worker process, but let's start with the config.

Node.js Monitoring and Debugging from the Experts of RisingStack

Check your service dependencies in production using Trace
Learn more

How to handle different environments and configurations?

Load your deployment specific configurations from environment variables and never add them to the codebase as constants. These are the configurations that can vary between deployments and runtime environments, like CI, staging or production. Basically, you can have the same code running everywhere.

A good test for whether the config is correctly separated from the application internals is that the codebase could be made public at any moment. This means that you can be protected from accidentally leaking secrets or compromising credentials on version control.

Your config is correctly separated from the apps internals if the codebase could be made public at any moment.

Click To Tweet

The environment variables can be accessed via the process.env object. Keep in mind that all the values have a type of String, so you might need to use type conversions.

// config/config.js
'use strict'

// required environment variables
[
  'NODE_ENV',
  'PORT'
].forEach((name) => {
  if (!process.env[name]) {
    throw new Error(`Environment variable ${name} is missing`)
  }
})

const config = {  
  env: process.env.NODE_ENV,
  logger: {
    level: process.env.LOG_LEVEL || 'info',
    enabled: process.env.BOOLEAN ? process.env.BOOLEAN.toLowerCase() === 'true' : false
  },
  server: {
    port: Number(process.env.PORT)
  }
  // ...
}

module.exports = config  

Config validation

Validating environment variables is also a quite useful technique. It can help you catching configuration errors on startup before your application does anything else. You can read more about the benefits of early error detection of configurations by Adrian Colyer in this blog post.

This is how our improved config file looks like with schema validation using the joi validator:

// config/config.js
'use strict'

const joi = require('joi')

const envVarsSchema = joi.object({  
  NODE_ENV: joi.string()
    .allow(['development', 'production', 'test', 'provision'])
    .required(),
  PORT: joi.number()
    .required(),
  LOGGER_LEVEL: joi.string()
    .allow(['error', 'warn', 'info', 'verbose', 'debug', 'silly'])
    .default('info'),
  LOGGER_ENABLED: joi.boolean()
    .truthy('TRUE')
    .truthy('true')
    .falsy('FALSE')
    .falsy('false')
    .default(true)
}).unknown()
  .required()

const { error, value: envVars } = joi.validate(process.env, envVarsSchema)  
if (error) {  
  throw new Error(`Config validation error: ${error.message}`)
}

const config = {  
  env: envVars.NODE_ENV,
  isTest: envVars.NODE_ENV === 'test',
  isDevelopment: envVars.NODE_ENV === 'development',
  logger: {
    level: envVars.LOGGER_LEVEL,
    enabled: envVars.LOGGER_ENABLED
  },
  server: {
    port: envVars.PORT
  }
  // ...
}

module.exports = config  


Config splitting

Splitting the configuration by components can be a good solution to forego a single, growing config file.

// config/components/logger.js
'use strict'

const joi = require('joi')

const envVarsSchema = joi.object({  
  LOGGER_LEVEL: joi.string()
    .allow(['error', 'warn', 'info', 'verbose', 'debug', 'silly'])
    .default('info'),
  LOGGER_ENABLED: joi.boolean()
    .truthy('TRUE')
    .truthy('true')
    .falsy('FALSE')
    .falsy('false')
    .default(true)
}).unknown()
  .required()

const { error, value: envVars } = joi.validate(process.env, envVarsSchema)  
if (error) {  
  throw new Error(`Config validation error: ${error.message}`)
}

const config = {  
  logger: {
    level: envVars.LOGGER_LEVEL,
    enabled: envVars.LOGGER_ENABLED
  }
}

module.exports = config  

Then in the config.js file we only need to combine the components.

// config/config.js
'use strict'

const common = require('./components/common')  
const logger = require('./components/logger')  
const redis = require('./components/redis')  
const server = require('./components/server')

module.exports = Object.assign({}, common, logger, redis, server)  

You should never group your config together into "environment" specific files, like config/production.js for production. It doesn't scale well as your app expands into more deployments over time.

You should never group your config together into environment specific files. It doesn’t scale well! #nodejs

Click To Tweet

How to organize a multi-process application?

The process is the main building block of a modern application. An app can have multiple stateless processes, just like in our example. HTTP requests can be handled by a web process and long-running or scheduled background tasks by a worker. They are stateless, because any data that needs to be persisted is stored in a stateful database. For this reason, adding more concurrent processes are very simple. These processes can be independently scaled based on the load or other metrics.

In the previous section, we saw how to break down the config into components. This comes very handy when having different process types. Each type can have its own config only requiring the components it needs, without expecting unused environment variables.

In the config/index.js file:

// config/index.js
'use strict'

const processType = process.env.PROCESS_TYPE

let config  
try {  
  config = require(`./${processType}`)
} catch (ex) {
  if (ex.code === 'MODULE_NOT_FOUND') {
    throw new Error(`No config for process type: ${processType}`)
  }

  throw ex
}

module.exports = config  

In the root index.js file we start the process selected with the PROCESS_TYPE environment variable:

// index.js
'use strict'

const processType = process.env.PROCESS_TYPE

if (processType === 'web') {  
  require('./web')
} else if (processType === 'twitter-stream-worker') {
  require('./worker/twitter-stream')
} else if (processType === 'social-preprocessor-worker') {
  require('./worker/social-preprocessor')
} else {
  throw new Error(`${processType} is an unsupported process type. Use one of: 'web', 'twitter-stream-worker', 'social-preprocessor-worker'!`)
}

The nice thing about this is that we still got one application, but we have managed to split it into multiple, independent processes. Each of them can be started and scaled individually, without influencing the other parts. You can achieve this without sacrificing your DRY codebase, because parts of the code, like the models, can be shared between the different processes.

How to organize your test files?

Place your test files next to the tested modules using some kind of naming convention, like <module_name>.spec.js and <module_name>.e2e.spec.js. Your tests should live together with the tested modules, keeping them in sync. It would be really hard to find and maintain the tests and the corresponding functionality when the test files are completely separated from the business logic.

Place your test files next to the tested modules using some kind of naming convention, like module_name.spec.js

Click To Tweet

A separated /test folder can hold all the additional test setup and utilities not used by the application itself.

Where to put your build and script files?

We tend to create a /scripts folder where we put our bash and node scripts for database synchronization, front-end builds and so on. This folder separates them from your application code and prevents you from putting too many script files into the root directory. List them in your npm scripts for easier usage.

Download the whole building with Node.js series as a single pdf

Conclusion

I hope you enjoyed this article on project structuring. I highly recommend to check out our previous article on the subject, where we laid out the 5 fundamentals of Node.js project structuring.

If you have any questions, please let me know in the comments. In the next chapter of the Node.js at Scale series, we’re going to dive deep into JavaScript clean coding. See you next week!


Writing a JavaScript Framework - Project Structuring

Writing a JavaScript Framework - Project Structuring

In the last couple of months Bertalan Miklos, JavaScript engineer at RisingStack wrote a next generation client-side framework, called NX. In the Writing a JavaScript Framework series, Bertalan shares what he learned during the process:

In this chapter, I am going to explain how NX is structured, and how I solved its use case specific difficulties regarding extendibility, dependency injection and private variables.

The series includes the following chapters.

  1. Project structuring (current chapter)
  2. Execution timing
  3. Sandboxed code evaluation
  4. Data binding introduction
  5. Data Binding with ES6 Proxies
  6. Custom elements
  7. Client-side routing

Project Structuring

There is no structure that fits all projects, although there are some general guidelines. Those who are interested can check out our Node.js project structure tutorial from the Node Hero series.

An overview of the NX JavaScript Framework

NX aims to be an open-source community driven project, which is easy to extend and scales well.

  • It has all the features expected from a modern client-side framework.
  • It has no external dependencies, other than polyfills.
  • It consists around 3000 lines altogether.
  • No module is longer than 300 lines.
  • No feature module has more than 3 dependencies.

Its final dependency graph looks like this:

JavaScript Framework in 2016: The NX project structure

This structure provides a solution for some typical framework related difficulties.

  • Extendibility
  • Dependency injection
  • Private variables

Achieving Extendibility

Easy extendibility is a must for community driven projects. To achieve it, the project should have a small core and a predefined dependency handling system. The former ensures that it is understandable, while the latter ensures that it will stay that way.

In this section, I focus on having a small core.

The main feature expected from modern frameworks is the ability to create custom components and use them in the DOM. NX has the single component function as its core, and that does exactly this. It allows the user to configure and register a new component type.

component(config)  
  .register('comp-name')

The registered comp-name is a blank component type which can be instantiated inside the DOM as expected.

<comp-name></comp-name>  

The next step is to ensure that the components can be extended with new features. To keep both simplicity and extendibility, these new features should not pollute the core. This is where dependency injection comes handy.

Dependency Injection (DI) with Middlewares

If you are unfamiliar with dependency injection, I suggest you to read our article on the topic : Dependency Injection in Node.js.

Dependency injection is a design pattern in which one or more dependencies (or services) are injected, or passed by reference, into a dependent object.

DI removes hard burnt dependencies but introduces a new problem. The user has to know how to configure and inject all the dependencies. Most client-side frameworks have DI containers doing this instead of the user.

A Dependency Injection Container is an object that knows how to instantiate and configure objects.

Another approach is the middleware DI pattern, which is widely used on the server side (Express, Koa). The trick here is that all injectable dependencies (middlewares) have the same interface and can be injected the same way. In this case, no DI container is needed.

I went with this solution to keep simplicity. If you ever used Express the below code will be very familiar.

component()  
  .use(paint) // inject paint middleware
  .use(resize) // inject resize middleware
  .register('comp-name')

function paint (elem, state, next) {  
  // elem is the component instance, set it up or extend it here
  elem.style.color = 'red'
  // then call next to run the next middleware (resize)
  next()
}

function resize (elem, state, next) {  
  elem.style.width = '100 px'
  next()
}

Middlewares execute when a new component instance is attached to the DOM and typically extend the component instance with new features. Extending the same object by different libraries leads to name collisions. Exposing private variables deepens this problem and may cause accidental usage by others.

Having a small public API and hiding the rest is a good practice to avoid these.

Handling privacy

Privacy is handled by function scope in JavaScript. When cross-scope private variables are required, people tend to prefix them with _ to signal their private nature and expose them publicly. This prevents accidental usage but doesn't avoid name collisions. A better alternative is the ES6 Symbol primitive.

A symbol is a unique and immutable data type, that may be used as an identifier for object properties.

The below code demonstrates a symbol in action.

const color = Symbol()

// a middleware
function colorize (elem, state, next) {  
  elem[color] = 'red'
  next()
}

Now 'red' is only reachable by owning a reference to the color symbol (and the element). The privacy of 'red' can be controlled by exposing the color symbol to different extents. With a reasonable number of private variables, having a central symbol storage is an elegant solution.

// symbols module
exports.private = {  
  color: Symbol('color from colorize')
}
exports.public = {}  

And an index.js like below.

// main module
const symbols = require('./symbols')  
exports.symbols = symbols.public  

The storage is accessible inside the project for all modules, but the private part is not exposed to the outside. The public part can be used to expose low-level features to external developers. This prevents accidental usage since the developer has to explicitly require the needed symbol to use it. Moreover, symbol references can not collide like string names, so name collision is impossible.

The points below summarize the pattern for different scenarios.

1. Public variables

Use them normally.

function (elem, state, next) {  
  elem.publicText = 'Hello World!'
  next()
}

2. Private variables

Cross-scope variables, that are private to the project should have a symbol key added to the private symbol registry.

// symbols module
exports.private = {  
  text: Symbol('private text')
}
exports.public = {}  

And required from it when needed somewhere.

const private = require('symbols').private

function (elem, state, next) {  
  elem[private.text] = 'Hello World!'
  next()
}

3. Semi-private variables

Variables of the low level API should have a symbol key added to the public symbol registry.

// symbols module
exports.private = {  
  text: Symbol('private text')
}
exports.public = {  
  text: Symbol('exposed text')
}

And required from it when needed somewhere.

const exposed = require('symbols').public

function (elem, state, next) {  
  elem[exposed.text] = 'Hello World!'
  next()
}

Conclusion

If you are interested in the NX framework, please visit the home page. Adventurous readers can find the NX source code in this Github repository.

I hope you found this a good read, see you next time when I’ll discuss execution timing!

If you have any thoughts on the topic, share it in the comments.

Node Hero - Node.js Project Structure Tutorial

This is the 7th part of the tutorial series called Node Hero - in these chapters, you can learn how to get started with Node.js and deliver software products using it.

Most Node.js frameworks don't come with a fixed directory structure and it might be challenging to get it right from the beginning. In this tutorial, you will learn how to properly structure a Node.js project to avoid confusion when your applications start to grow.

UPDATE: We wrote another article about Node.js project structuring, which discusses advanced techniques as well.

The 5 fundamental rules of a Node.js Project Structure

There are a lot of possible ways to organize a Node.js project - and each of the known methods has their ups and downs. However, according to our experience, developers always want to achieve the same things: clean code and the possibility of adding new features with ease.

In the past years at RisingStack, we had a chance to build efficient Node applications in many sizes, and we gained numerous insights regarding the dos and donts of project structuring.

We have outlined five simple guiding rules which we enforce during Node.js development. If you manage to follow them, your projects will be fine:

Rule 1 - Organize your Files Around Features, Not Roles

Imagine, that you have the following directory structure:

// DON'T
.
├── controllers
|   ├── product.js
|   └── user.js
├── models
|   ├── product.js
|   └── user.js
├── views
|   ├── product.hbs
|   └── user.hbs

The problems with this approach are:

  • to understand how the product pages work, you have to open up three different directories, with lots of context switching,
  • you end up writing long paths when requiring modules: require('../../controllers/user.js')

"Rule 1: Organize your files around features, not roles!" via @risingstack

Click To Tweet

Instead of this, you can structure your Node.js applications around product features / pages / components. It makes understanding a lot easier:

// DO
.
├── product
|   ├── index.js
|   ├── product.js
|   └── product.hbs
├── user
|   ├── index.js
|   ├── user.js
|   └── user.hbs

Node.js Monitoring and Debugging from the Experts of RisingStack

Build performant applications using Trace
Learn more

Rule 2 - Don't Put Logic in index.js Files

Use these files only to export functionality, like:

// product/index.js
var product = require('./product')

module.exports = {  
  create: product.create
}

Rule 3 - Place Your Test Files Next to The Implementation

Tests are not just for checking whether a module produces the expected output, they also document your modules (you will learn more on testing in the upcoming chapters). Because of this, it is easier to understand if test files are placed next to the implementation.

"Rule 3: Place your test files next to the implementation." via @risingstack

Click To Tweet

Put your additional test files to a separate test folder to avoid confusion.

.
├── test
|   └── setup.spec.js
├── product
|   ├── index.js
|   ├── product.js
|   ├── product.spec.js
|   └── product.hbs
├── user
|   ├── index.js
|   ├── user.js
|   ├── user.spec.js
|   └── user.hbs

Rule 4 - Use a config Directory

To place your configuration files, use a config directory.

.
├── config
|   ├── index.js
|   └── server.js
├── product
|   ├── index.js
|   ├── product.js
|   ├── product.spec.js
|   └── product.hbs

Rule 5 - Put Your Long npm Scripts in a scripts Directory

Create a separate directory for your additional long scripts in package.json

.
├── scripts
|   ├── syncDb.sh
|   └── provision.sh
├── product
|   ├── index.js
|   ├── product.js
|   ├── product.spec.js
|   └── product.hbs

Download the whole Node Hero series as a single pdf

Next up

In the next chapter of Node Hero, you are going to learn how to authenticate users using Passport.js. Until the next chapter comes out, feel free to ask any questions you ran into!

UPDATE: We wrote another article about Node.js project structuring, which discusses advanced techniques as well.