Elixir in Production: What Is It Used For?

RisingStack's services:

Search

Node.js Experts

Learn more at risingstack.com

Sign up to our newsletter!

In this article:

We’ve already written a series of articles about why we think Elixir is a great choice, but don’t take our word for it – there are many success stories out there about using Elixir in production that not only prove that the language is mature enough to be a solid choice, but it can be even more effective than the usual frequently used languages and frameworks thanks to the features provided by BEAM and OTP.

From startups to established enterprises, our examples clearly outline Elixir’s strengths:

  • Scalability – Effortlessly handles sudden surges in traffic and data.
  • Fault Tolerance – Maintains stability and uptime even during system failures.
  • Cost Efficiency – Reduces infrastructure needs.

After understanding how companies like Discord integrated Elixir to handle real-time communication and how Pinterest gained significant cost savings with it, we hope you’ll find Elixir inspiring enough to take a look at.

Incredible Developer Productivity with Elixir at Remote.com

Remote.com is built on Elixir from the ground up, for good reasons. 

“It offers incredible developer productivity due to its intuitive and straight-forward syntax, its well-designed standard libraries, and its convenient level of abstraction.“ – according to Peter Ullrich, a Senior Elixir Engineer at Remote.

“We rarely have to reinvent the wheel,” says Peter, because Elixir provides them with the tools they need to improve their services while maintaining their desired productivity levels at the same time.

It’s no wonder why Elixir became a serious player in the webdev space:

  • Phoenix, its web framework is one of the best choices for writing webapps.
  • It’s concurrency-focused design allows effortless scalability.
  • The Erlang VM (BEAM) is built in a “let it crash” philosophy, thatencourages developers to build applications that can gracefully handle failures and self-recover.

When it comes to the ecosystem and particular use cases, Elixir proved to be a great choice for Remote.com.  For data processing they use Broadway, for multimedia processing Membrane. 

“Its recent ventures into machine learning (Nx) and collaborative development (Livebook) put Elixir on the map of many more industries and allowed it to be used for a whole range of new use-cases.”

“It Just Works” – Elixir by Accident at Multiverse 

Most companies on this list chose Elixir after careful consideration and because of its clear advantages. Multiverse, a UK-based Ed-Tech startup (with a $220M Series D behind them), is built with Elixir simply because when the company was founded, an external agency was contracted to build the initial platform. And they happened to be an Elixir shop.

However, while most companies seem to try to get rid of their initial platform by rebuilding it from the ground up, or just barely keeping it running with endless “quick fixes”, Multiverse considers itself very lucky to have Elixir as the foundation by this “lucky accident.”

Razvan, a Senior Engineering Manager at Multiverse, is confident that it will help Multiverse scale, and they – too – are in contact with the creators of the language and the frameworks they use. Although the community is small compared to other languages, it is very active and supportive. 

This is something we especially love about Elixir at RisingStack, too.

“I’m coming from a JavaScript background and I can say that the first thing that amazed me is that Elixir, well, just works.”

When Razvan joined the team with little to no experience, he was able to get their Elixir platform up and running in a couple of hours without any glitches.

  • The environment feels robust, cohesive and you don’t waste hours and hours trying to run it, to debug a cryptic error.
  • The Elixir docs are amazing. It is actually useful, to the point and with examples. 
  • The tooling is powerful and testing is a first class citizen of Elixir development. 
  • Functional programming is just awesome!

On top of that, the frameworks aren’t too bad either! 

“We’re currently using Elixir with Phoenix and LiveView, and our engineers are very happy with that, mostly because they don’t have to switch between languages all the time.” – said Razvan.

Less Servers, Same Performance at Pinterest

Pinterest estimates that it has saved about $2 million per year in server costs since it successfully adopted Elixir.

Security, scalability, and fault tolerance are all important aspects of a system, but we can’t ignore the economical side of keeping it running – we have to think about costs too. The engineering team at Pinterest tried to find a solution for all of the above, and Elixir provided it for them. 

Pinterest managed to replace 200 servers running Python with just 4 running on Elixir, while providing the same performance as before. Besides, maintenance became much easier as well.

Here are some highlights from the interview for those who are in a rush:

  • Pinterest chose Elixir because they were looking for a system that was easy for programmers to understand.
  • Elixir’s main strengths: friendly syntax, powerful metaprogramming features, and incorporation of the Actor model.
  • Besides the cost saving, the performance and reliability of the systems went up despite running on drastically less hardware.
  • In total, Pinterest reduced its server size by about 95% thanks to Elixir.
  • Despite running on less hardware, the response times dropped significantly, as did errors.

How Two Elixir Nodes Outperformed 20 Ruby Nodes by 83x

Veeps is a streaming service that also offers ticket-based online events where it’s not uncommon for fans to immediately jump on ticket sales once an event is announced.

While the previous Ruby on Rails system was able to service thousands of users, or even tens of thousands, the engineering team found it next to impossible to scale it further to allow more visits while maintaining performance at the same time.

Vincent Franco, the CTO, convinced management to make the switch to Elixir to future-proof the company’s tech stack. It took about 8 months to rewrite everything in Elixir and Phoenix, but it turned out that the effort was worth it.

Two Elixir nodes replaced 20 Ruby on Rails nodes, and those two can handle 83x more users than before. The project was managed by an external company due to a lack of internal experience with Elixir, but by the time it was finished, Veeps was able to establish an internal team that can build and expand on the newly built Elixir-based systems.

Elixir Powers Emerging Markets – Literally

Access to electricity is a given in developed countries, but still a rare commodity in other parts of the world. SparkMeter aims to change that with their grid-management solutions. They operate smart meters that communicate with a grid management unit which is connected to cloud servers.

This may not sound like an unsolvable challenge so far, but there are additional complexities. The servers and the grid management unit communicate via cellular network which is prone to failure, and the electricity powering the systems may also go down time-to-time. Fault-tolerance was crucial in circumstances like this, And Elixir together with the Nerves platform is the perfect choice for this situation. 

Nerves is an open-source platform that combines the BEAM virtual machine and Elixir ecosystem to easily build and deploy embedded systems for production. A highlight of this setup is that it can handle cases when parts of the system are down.

Finding capable engineers was also a problem, but after the project started with outside consultants, their in-house team was trained in Elixir to be able to maintain and further develop the system as a long-term solution which worked out really well.

In the new system, the grid management unit communicates with the meters via radio, using Rust for hardware control and Elixir Ports for data processing. Communication with cloud servers over 3G or Edge required a custom protocol to minimize bandwidth usage, crafted uniquely to fit their specific needs.

Additionally, their system includes a local web interface accessible via Wi-Fi, using Phoenix LiveView, and a cloud-based system that processes data through a custom TCP server and a Broadway pipeline, storing it in PostgreSQL. This robust setup allows SparkMeter to maintain high availability and manage its resources efficiently, despite the challenging environments it operates in. The team also managed to reduce the complexity of the previous architecture by replacing the old one that was using Ubuntu and Docker for the system level, Python/Celery and RabbitMQ for asynchronous processing, and Systemd for managing starting job processes with just Elixir and Nerves.

Multiplayer With Elixir: 10000 Players in the Same Session

X-Plane 11 is one of the best flight simulators in the world, aiming for unparalleled accuracy for even pilots to practice in a safe environment. It used to have a simple peer-to-peer solution for multiplayer, but the developers wanted to change that.

This proved to be a huge challenge: the team did not have experience in the subject, but they needed a solution that could support way more concurrent players than an average multiplayer game, and it needed to do that with as much accuracy as possible.

With a criteria like this, simply adding more servers was not an adequate solution so they excluded Ruby and Python among others. The top three choices were Rust, Go and Elixir: Elixir won because of its fault-tolerance capabilities and predictable latency. In addition to that, Elixir and Erlang have built-in support for parsing binary packets which made the implementation easier.

The whole project took only 6 months, and that included the time to learn Elixir because the lead developer had no prior experience with the language.

It’s so lightweight that the entire player base in North America is served by a single server running on 1 eight-core machine with 16GB of memory. 

The solution is open-source if you want to take a look.

How PepsiCo Uses Elixir

Many companies internally build their own tools for specific purposes, and PepsiCo is no exception. The Search Marketing and Sales Intelligence Platform teams handle immense amounts of data coming from their search partners that needed a support of a robust platform.



This complex data pipeline is managed by the Data Engineering team, which initially collects and stores the data in the Snowflake Data Cloud. An Elixir application then processes this data and routes it to PostgreSQL or Apache Druid, depending on its characteristics. A Phoenix application serves this processed data to internal teams and interacts with third-party APIs.

The team used Elixir to create a domain-specific language that translates business queries into data structures. It’s easy for them to extend it as they need and provides a solid foundation, even when connecting to several different third-party APIs.

David Antaramian, a Software Engineering Manager at PepsiCo praises the Erlang runtime and its standard library, particularly for managing large datasets without frequently accessing the database. He emphasizes the use of Erlang’s in-memory table storage, ETS, which allows them to efficiently store hundreds of thousands of rows. This capability is crucial for handling the massive amounts of data PepsiCo deals with.

The Elixir ecosystem effectively complements Erlang, supporting both front-end and server-side operations at PepsiCo. The company’s front-end, built in React, connects with the server using the Absinthe GraphQL toolkit atop the Phoenix web framework. For database interactions, the Ecto library manages communications with PostgreSQL. Additionally, the esaml and Samly libraries are used for authentication across PepsiCo’s network, demonstrating a practical application of tools from the Erlang and Elixir communities.

4 Billion Messages Every Day on Discord With Elixir

Discord is one of the most famous adopters of Elixir since they have been building the platform on it from day one. Beside a Python API in a monolith architecture, Discord has about 20 different services built with Elixir.

However, this choice was not without risk. In 2015, when Elixir v1.0 came out and Discord was founded, the team gambled and bet on Elixir, hoping the language would mature and evolve over time. Their bet paid off wonderfully, as seen from the growing user base.

Jake Heinz, Lead Software Engineer at Discord said about Elixir: “In terms of real-time communication, the Erlang VM is the best tool for the job. It is a very versatile runtime with excellent tooling and reasoning for building distributed systems”. 

Those 20+ services are powered by 400-500 Elixir servers, and amazingly only maintained by a handful of engineers.

The team uses Distributed Erlang, facilitated by etcd for service discovery and configuration, to create a partially meshed network rather than a fully connected one. This setup allows for efficient and scalable communication across Discord’s numerous services, including their audio and video platforms, which operate over 1000+ nodes. When needed, engineers from other teams can collaborate with the Chat Infrastructure Team operating these services and build on it with their assistance.

Discord also uses Rust to complement Elixir, with the help of the Rustler project to bridge the gap between the two languages by hooking a custom data structure built in Rust directly into the Elixir servers. The flexibility of it allows engineers to solve uptime-related problems often in just a few minutes

Although none of the engineers had prior experience with Elixir before joining Discord, they quickly pick up the pace and thanks to Erlang VM, even able to efficiently debug a live system if needed.

Conclusion

As seen from these case studies, most companies needed not only scalability but also ease of maintenance and future-proofing – Elixir was able to provide all of these, thus proving its maturity for an environment that is more than ready for use in production.

Our team also works with Elixir more and more often, utilizing it in situations where mighty JavaScript falls short.

If you need advice with Elixir, reach out to us for a chat!

Share this post

Twitter
Facebook
LinkedIn
Reddit

Node.js
Experts

Learn more at risingstack.com

Node.js Experts