AWS SQS in a Large Scale Application

In today’s article, I’m going to show you how we use AWS SQS in our Laravel application and how it helps us manage 1.6 billion operations each month.

In the image below, you can see our typical bill for a month of AWS SQS usage:

Before entering into the details of our system design, I would introduce the SQS technology, why you should consider a messaging system, and how to compare it with similar services.

What Is AWS SQS?

Amazon Simple Queue Service is a fully managed messages queueing system to enable communication between distributed software components or microservices.

Decoupling some parts of your system allows you to run and fail your software components independently. It also allows you to build easily scalable applications.

Message queues act as intermediaries:

What Is a Message?

This is an important question I want to answer to help developers who are approaching this technology for the first time.

Every message queuing system accepts a text as the content of a message. The most common way to make different systems exchange structured data is to use JSON objects.

JSON can be easily serialized as text when pushing a new message onto the queue and deserialized when it is consumed on the other side of the system.

The Laravel Way

Laravel implements its own format when exchanging messages with queue systems. Here is an example:

The JSON is automatically created by the internal queue system of the framework when you dispatch a job. It is somewhat hidden from the eyes of the developer.

The JSON message contains all data needed by Laravel to understand what job class should be executed and under what conditions (attempts, maxTries, etc.).

If your application doesn’t have a prepackaged solution like the one provided by Laravel, you can define your own format. The goal is to provide the consumer of the queue with all information to process the desired task.

How To Consume Messages from the Queue

You have to implement a system that continuously polls the queue waiting for a message to arrive.

Laravel provides a built-in “worker” that you can run with a simple command:

php artisan queue:work

This command runs a background process that asks for new messages. When a message is retrieved, the worker can run the job class included in the message.

You can see this process in the \Illuminate\Queue\Worker.php class:

while (true) {
	// Before reserving any jobs, we will make sure this queue is not paused and
	// if it is we will just pause this worker for a given amount of time and
	// make sure we do not need to kill this worker process off completely.
	if (! $this->daemonShouldRun($options, $connectionName, $queue)) {
		$status = $this->pauseWorker($options, $lastRestart);

		if (! is_null($status)) {
			return $this->stop($status, $options);
		}

		continue;
	}
	
	...

	// If the daemon should run (not in maintenance mode, etc.), then we can run
	// fire off this job for processing. Otherwise, we will need to sleep the
	// worker so no more jobs are processed until they should be processed.
	if ($job) {
		$jobsProcessed++;

		$this->runJob($job, $connectionName, $options);

		if ($options->rest > 0) {
			$this->sleep($options->rest);
		}
	} else {
		$this->sleep($options->sleep);
	}
	
	...
}

Why Should You Use AWS SQS?

Since my product is a developer tool, I discuss with other fellow developers almost every week about the internal design of their software. 

In most of the applications I’ve had the opportunity to analyze, a messaging system was introduced to enable asynchronous task execution. 

The same is for Inspector.

I think this is the most common use case for messaging systems today. Microservices or large distributed software, in general, are very niche use cases. Probably belonging more to enterprise environments.

What Is an Asynchronous Task?

Imagine your product offers the ability to import account information by uploading large files. The user picks the file from the local PC and clicks “Import.”

This operation can take a while to be completed. You have to open the file, read each line, convert it into the appropriate format, and store this information in your system.

It could be impossible to handle this process at the time of an HTTP request.

Thanks to a message queueing system, we can store the file on our servers and push a message to the “import” queue with the path where the file is. 

The next step is to make a background system that waits for these messages to handle the import process. Users don’t need to wait until the import is done. In the meantime, they can continue to use the product or leave it waiting to be notified when the import is completed:

How Inspector Uses AWS SQS

The most important challenge of Inspector is to be able to process large amounts of traffic, also guaranteeing good performance.

The endpoint where Inspector receives data from the monitored applications handles about 10 million HTTP requests per day. With this volume, the problem is that these data packets can’t be processed on the fly. 

At the beginning of our adventure, we used a self-managed Redis as a message queueing system. But, with increasing volumes, we have preferred to rely on a managed service.

The strategy is to use AWS SQS to be able to push these tasks to a queue, which will be consumed by another separate system in the background (the processing pipeline):

Ingestion nodes are implemented in Node.js. They are built as a very simple and extremely scalable script. The worker servers that consume the queue are, instead, a system component built on top of the Laravel framework.

We could have chosen any other technology to manage the message queue. At this stage of the company, SQS guarantees scalability without any complexity for us since it is a fully managed service.

Other solutions could also save us costs. We prefer to guarantee our customers that the platform works perfectly and avoid the risk of introducing weaknesses that could cause unexpected downtimes.

Conclusion

At this point, you should have a better idea of what AWS SQS is, how to use AWS SQS to scale your application, why you should use AWS SQS, and more. I hope you have taken away some valuable information from this article. If so, feel free to like and share.