RabbitMQ vs. Memphis.dev

What Is RabbitMQ?

RabbitMQ is a lightweight and easy-to-deploy messaging queue for on-premises and cloud environments. It supports multiple messaging protocols. RabbitMQ can be deployed in distributed and federated configurations to meet high-scale, high-availability requirements.

What Is Memphis.dev?

Memphis is a next-generation message broker.

A simple, robust, and durable cloud-native message broker wrapped with an entire ecosystem that enables fast and reliable development of next-generation event-driven use cases.

Memphis.dev enables building next-generation applications that require large volumes of streamed and enriched data, modern protocols, zero ops, rapid development, extreme cost reduction, and a significantly lower amount of dev time for data-oriented developers and data engineers.

General

License

Both technologies are available under fully open-source licenses. Memphis also has a commercial-based distribution with added security, tiered storage, and more.

Components

Memphis uses MongoDB for GUI state management only and will be removed soon, making Memphis without any external dependency. Both Memphis and RabbitMQ achieve consensus by using RAFT.

Message Consumption Model

RabbitMQ uses a push-based approach synonymous with traditional messaging systems.

Memphis uses a pull-based architecture where consumers pull messages from the server, and long-polling ensures new messages are made available instantaneously.

Pull-based architectures are often preferable for high throughput workloads as they allow consumers to manage their flow control, fetching only what they need.

Storage Architecture

RabbitMQ uses index-based storage systems. These keep data in a tree structure to provide the fast access necessary for acknowledging individual messages.

Memphis uses a distributed commit log called streams (made by NATS Jetstream) as its storage layer, which can be written entirely on the broker’s (server) memory or disk. In addition, Memphis abstracts the offsets by default, so saving a record of the used offsets resides on Memphis and not on the client.

Messaging

*Available for Memphis cloud users.

Data Flow

RabbitMQ uses a distinct, bounded data flow. Messages are created and sent by the producer and received by the consumer. Memphis uses an unbounded data flow, with the key-value pairs continuously streaming to the assigned station.

Data Usage

RabbitMQ is best for transactional data, such as order formation, placement, and user requests. Memphis works great for transactional and operational data like process operations, auditing and logging statistics, and system activity.

Message Retention

RabbitMQ pushes messages to consumers. These messages are removed from the queue once they are processed and acknowledged. Memphis is a log. It uses continuous messages, which stay in the station (queue) until the retention period expires.

Topology

RabbitMQ uses the exchange queue topology — sending messages to an exchange where they are, in turn, routed to various queue bindings for the consumer’s use. Memphis employs the publish/subscribe topology, sending messages across the streams to the correct stations and then consumed by users in the different authorized groups.

Architecture Differences

When choosing between Memphis and RabbitMQ, the internal operations and fundamental design can be essential considerations.

The components of RabbitMQ’s Architecture consist of the following:

• Queue: It is in charge of keeping track of messages that have been received and may have configuration data that specifies what it can do with a message.
• Exchange: An exchange receives messages sent to RabbitMQ and determines where they should be forwarded. Exchanges define the routing strategies used for messages, most frequently by examining the data characteristics transmitted with the message or included inside its attributes.
• Producer: Produces messages and sends them to a broker server (publishes). A payload and a label are the two components of a message. The user’s desired data to convey is the payload. The label specifies who should receive a copy of the message and describes the payload.
• Consumer: It subscribes to a queue and is connected to a broker server.
• Broker: Applications can exchange information and communicate with one another through a broker.
• Binding: It tells an exchange which queues to distribute messages. Additionally, the binding will instruct the exchange to filter which messages it is permitted to add to a queue for specific exchange types.

Memphis architecture is designed using the following components:

• Replicas: One of the most crucial elements of Memphis is replication, which ensures that messages are published and consumed even when the broker encounters a problem.
• RAFT: It maintains data coordination between brokers, such as configuration, location, data, and status details.
• Producer: Producers push or publish messages to a Memphis station created on a Memphis broker. Producers can also send messages to a broker synchronously or asynchronously.
• Consumers: Individuals who subscribe to a Memphis station and pull messages from it. Memphis allows data to be stored in additional storage platforms used by programs for online transaction processing (OLTP) besides internal disks and memory.
• Broker: Acts as a Memphis server or broker. The number of streams for each message is defined in accordance with the order in which the broker stores the messages.

Scalability and Redundancy

RabbitMQ uses a round-robin queue to repeat messages. The messages are divided among the queues to boost throughput and balance the load. Additionally, it enables numerous consumers to read messages from various queues simultaneously.

In Memphis, scalability and redundancy are provided by stream objects. The stream is duplicated across numerous brokers. In the event that one of the brokers fails, the data can still be served by another broker.

If data is stored in only one broker, the dependence on that broker will grow, which is hazardous and increases the likelihood that it will fail. Additionally, distributing the streams will vastly improve throughput.

Multi-Region/Multi-Cloud

The highest availability capability in RabbitMQ is cluster and quorum queues, which enable cross-node replication.

Besides mirroring and striping stations across multiple brokers, that can ultimately span across AZs. Partners and Memphis cloud users can establish a Memphis super-cluster across regions in an active/passive topology.

Multi-Tenancy

RabbitMQ is a multi-tenant system: connections, exchanges, queues, bindings, user permissions, policies, and some other things belong to virtual hosts and logical groups of entities.

Memphis supports multi-tenancy using namespaces which offers a complete separation from connections, producers, consumers, security, dedicated dashboard, including node selection.

Queue Striping

All data/state required for operating a RabbitMQ broker is replicated across all nodes. An exception is message queues, which by default reside on one node, though they are visible and reachable from all nodes. To replicate queues across nodes in a cluster, use a queue type that supports replication. This topic is covered in the Quorum Queues guide.

Memphis replicates data across brokers based on defined policy (Replicas). Memphis station can also span and stripe data across brokers to achieve higher throughput and availability.

Features