NATS

NATS is a high-performant messaging system:

• High performant - meaning - it's fast
• Messaging system - meaning - it's meant to let distributed services communicate (you might know other messaging systems like Kafka, RabbitMQ or Pulsar)

NATS offers a whole lot of concepts and features that are too comprehensive to list here. We'll talk about the most important concepts and features that we use.

Why do we use it?

To build a scallable, event-driven application (MACH) we need a messaging system for our services to efficiently communicate with each other. This allows us to build a scalable and resilient system, providing a way for services to communicate without needing to know about each other.

Core NATS & JetStream

Core NATS

This is the foundational functionality in a NATS system. Everything operates on a publish-subscribe model using subject/topic-based addressing.

Publish-subscribe model allows services to communicate without needing to know about each other. It contains 3 parts:

• Publishers: These are services that publish messages (data) to a specific topic.
• Subscribers: These are services that want to receive messages (data) of topics they are interested in.
• Message broker: This is the messaging system (NATS) that acts as a middleman. It takes the message from the publisher(s) and distributes it to all the subscribers interested in that topic.

Topics (Subjects)

A topic is just a string of that form a name the publisher and subscriber can use to find each other. A subject cal be divided in subject hierarchies via dots.

Example:

• time.us
• time.us.east
• time.us.east.atlanta
• time.eu.east
• time.eu.east.warsaw

When subscribing to a topic, we can make use of wildcards to subscribe to multiple topics at once.

• Match a single token via *.
  • time.*.east will match time.us.east and time.eu.east.
• Match multiple tokens via >.
  • time.us.> will match time.us.east and time.us.east.atalanta.
  • > will match single subject!

NOTE

NATS can handle 10s of millions subjects efficiently, therefore, you can use fine-grained addressing for your business entities.

Jetstream

This is an enhancement to the Core NATS system to add persistence capabilities. When running a NATS instance you must specify whether Jetstream is enabled or not. This persistency layer enables some very cool things:

Streams

It's possible to keep messages stored on so called "streams". These streams are the base of Jetstream that allow for great solutions. Such a stream has some configuration that allows for endless solutions:

Option	Description
Subject	A list of subjects to store messages of (Wildcards are supported).
RetentionPolicy	Defines when messages in the stream can be automatically deleted. This can be LimitsPolicy (default) that deletes messages based on various limits (MaxMsgs, MaxBytes, ...), WorkQueuePolicy where messages are deleted when consumed once and InterestPolicy where messages are deleted when consumed by all the consumers interested.
DiscardPolicy	Only applies when the stream has at least one limit defined (MazMsgs, MaxBytes, ...). Can be either DiscardOld which removes the oldest message from the stream or DiscardNew which rejects new messages.
MaxMsgs	Maximum amount of messages stored in the stream. (Messages are discarded based on the Discard policy mentioned below).
MaxBytes	Maximum number of bytes stored in the stream. (Messages are discarded based on the Discard policy mentioned below).
MaxAge	Maximum age of any message in the stream before it's removed.

Consumers

A consumer is an interface for clients to consume a subset of messages stored on a stream and will keep track of which messages were delivered and acknowledged by clients. This can provide an at least once guarantee instead of NATS Core which provides an at most once guarantee.

To be clear:

• Streams are responsible for storing the published messages.
• Consumers are responsible for tracking the delivery and acknowledgements of messages.

There are some differences in consumers:

• Dispatch type
  • Push: Messages will be delivered to a specified subject a subscriber listens to
  • Pull: Batches of messages are requested by a subscriber on demand.

• Persistency

  • Durable: Consumer has persistent state, clients can resume reading messages from the point left off.
  • Ephemeral: Consumer will be automatically cleaned up after a period of inactivity.

Key/Value Store

Jetstream not only delivers great capabilities associated with "streaming", but also allows for some functionalities not found in messaging systems.

Such a feature is the Key/Value store, which allows to create "buckets" (which are streams under the hood) and use them to store values. It includes functionalities like:

• put: set a key to a value
• get: retrieve the value of a given key
• delete: clear any value of a given key
• Determine how long the store will keep the value for (TTL).
• Watch for changes happening for a key (similar to subscribing to a subject)
• Keep track of historical values (default the history of a bucket is set to 1)

INFO

NATS also offers an "Object Store" to do the same thing, but with files.

• Load balancing: Be able to load balance streams of messages between multiple instances without messages being handled duplicate.
• Guaranteed message delivery: Be sure no messages are unseen.
• Replay messages: View history, built up your current state by viewing the actions
• Key value storage

How do we use it?

First of all our messages are conform to the (CloudEvents - JSON event format)[https://cloud.google.com/eventarc/docs/workflows/cloudevents]. It's basically a common way to describe event data that looks like this:

Attribute	Description
data	The payload of the event data.
datacontenttype	The type of the data that has been passed. (application/json)
id	The unique identifier for the event.
source	The source of the event. (project-${process.env.NODE_ENV as string}/topics/${topic})
specversion	The version used for this event. (e.g. v1, v2)
type	The type of event. (e.g. chat-message.read, chat-message.new)
time	Event generation time in ISO-format.

Some use-cases that we use it for are listed below:

Realtime data distribution

When we want to accomplish real-time data updates (e.g. chat) we make use of a combination of NATS and Websockets.

We do this by publishing the data to a topic that the front-end must subscribe to. Subscribing to this topic is made accessible via a websocket, the websocket server is handling whether the user is authenticated and authorized to subscribe to the topic.

WARNING

The way you structure your topics is an important thing to consider! Take into account the wildcards.

INFO

In the future we would like to leave out Websockets completely and let front-end subscribe directly to NATS via a NATS clients. Currently Scaleway doesn't allow us to setup token authentication & authorization of consumers.

Caching

Instead of using Redis, we opted in many projects to use NATS KV storage instead. Mostly because using NATS KV storage is cheaper than a managed Redis and it offers more.

INFO

Apart from Key/Value Store, we haven't used Jetstream capabilities yet in any project, but it can come in handy!

How can I get started?

CLI

Want to test something without setting up code? The CLI has got you covered!

1. Install NATS CLI

For macOS:

brew tap nats-io/nats-tools
brew install nats-io/nats-tools/nats

2. Create NATS context

The NATS cli supports multiple named configurations called "context". We can create a configuration as following:

nats context save <name> --server <host> --creds <(optionally) path to credentials file>

# Example: nats context save local --server 127.0.0.1:4222
# Example: nats context save project-staging --server nats://nats.mnq.fr-par.scaleway.com:4222 --creds /Users/x/project/nats-staging.creds

3. Choose a NATS context

Choose the configuration context you want to use:

nats context select

4. Explore!

• Subscribe to topic:

nats sub <topic>

# Example: nats sub 'belgium.cars.1'
# Example: nats sub '*.cars.*'
# Example: nats sub '>'

• Publish message on topic:

nats pub <topic> <data>

# Example: nats pub 'topic' 'test'
# Example: nats pub 'belgium.cars.1' '{
#   "data": { "speed": 70, "engineSpeed": 3178, "gps": {"lon": 5.419690, "lat": 50.907780}},
#   "datacontenttype": "application/json; charset=utf-8",
#   "id": "MESSAGE_ID",
#   "source": "project-local/topics/belgium.cars.1",
#   "specversion": "1.0",
#   "type": "car.data",
#   "time": "2024-10-24T07:08:19Z"
# }'

TIP

You can open multiple terminals and see what's happening.

Node.js library

Package docs

Want to learn more?

• NATS documentation
• Synadia youtube channel
• Slack workspace