Useful Tools for Local Development With AWS Services

Over the last 2.5 years, I've been working with AWS and a wide range of its services. During this time, I noticed that for most projects, it's useful to be able to test your application against AWS services without having to deploy or move your code into the cloud. There are several free solutions available for you to use depending on the services required by your project. In this post, I'll describe some of the tools that I use.

DynamoDB Local

At one of my previous projects, we made extensive use of the combination of DynamoDB and Elasticsearch for storing and querying data. The fact that DynamoDB is a managed database service with immense scale and performance benefits makes DynamoDB a great fit for high traffic applications.

As a user, it's quite simple to use as it's a key-value store. Most of the other AWS databases are managed instances of existing services. However, DynamoDB is an AWS specific service that you can't really download and install locally. Luckily, back in 2018, AWS introduced a simpler way to work with DynamoDB utilizing DynamoDB local, a dockerized version of DynamoDB. You can simply run as a docker container to develop and test against.

Running DynamoDB local is as simple as executing:

$ docker run -p 8000:8000 amazon/dynamodb-local 


Or, if it's part of a bigger set of dependencies, you could leverage docker-compose.

YAML
 




xxxxxxxxxx
1


 
1
version: '3'
2
 
           
3
services:
4
 dynamodb:
5
   image: amazon/dynamodb-local:latest
6
   hostname: dynamodb-local
7
   ports:
8
      - "8000:8000"



With that, it's a matter of running:

$ docker-compose up


And you should see something like:

Shell
 




xxxxxxxxxx
1


 
1
Starting dynamodb_dynamodb_1 ... done
2
Attaching to dynamodb_dynamodb_1
3
dynamodb_1  | Initializing DynamoDB Local with the following configuration:
4
dynamodb_1  | Port:     8000
5
dynamodb_1  | InMemory: true
6
dynamodb_1  | DbPath:   null
7
dynamodb_1  | SharedDb: false
8
dynamodb_1  | shouldDelayTransientStatuses:     false
9
dynamodb_1  | CorsParams:       *



With the AWS CLI, you can easily query for available tables:

$ aws dynamodb list-tables --endpoint-url http://localhost:8000 


Which should result in something like:

JSON
 




xxxxxxxxxx
1


 
1
{
2
    "TableNames": []
3
}



And of course, you can use the AWS SDK with your preferred language as well.

I hear you thinking: are there no limitations? Yes, of course, there are some limitations with using DynamoDB local compared to the managed service. For instance, parallel scans are not supported (they will happen sequentially). Most limitations are nicely outlined in the DynamoDB Developer Guide.

Amazon also provides docker images for other services as well, like AWS Step functions Local and OpenDistro for Elasticsearch. Be sure to check out the Amazon repo on DockerHub for other useful images.

LocalStack

When you're developing a simple service that only depends on DynamoDB, DynamoDB local is a good choice. However, once you start to leverage more and more services, it might be worthwhile to look for other options as not all services are available as single docker images.

When you're building services that are part of a microservices architecture, you're probably using other AWS services like SNS, SQS, and perhaps S3. This is where a tool like LocalStack can add a lot of value. So what is LocalStack?

LocalStack is a project open-sourced by Atlassian that provides an easy way to develop AWS cloud applications directly from your localhost. It spins up a testing environment on your local machine that provides almost the same feature parity and APIs as the real AWS cloud environment, minus the scaling and robustness, of course.

Your application to AWS via LocalStack

Localstack focuses primarily on providing a local AWS cloud environment that adheres to the AWS APIs and offers a free and pro version, which you can leverage depending on your requirements. In my experience, the free/community version offers a lot of value and supports a whole range of services.

You can install LocalStack via pip if you're familiar with python and its package system, or you can use it via docker(compose). On my Mac, I found that installing LocalStack as a python package was a bit of a hassle, so I always prefer to use it via docker-compose.

Using LocalStack with docker-compose is as simple as creating a docker-compose.yml file with the content:

YAML
 




xxxxxxxxxx
1
20


 
1
version: '3'
2
 
           
3
services:
4
 localstack:
5
   image: localstack/localstack:0.12.2
6
   ports:
7
     - "4566-4599:4566-4599"
8
     - "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}"
9
   environment:
10
     - SERVICES=${SERVICES- }
11
     - DEBUG=${DEBUG- }
12
     - DATA_DIR=${DATA_DIR- }
13
     - PORT_WEB_UI=${PORT_WEB_UI- }
14
     - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
15
     - KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
16
     - DOCKER_HOST=unix:///var/run/docker.sock
17
     - HOST_TMP_FOLDER=${TMPDIR}
18
   volumes:
19
     - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
20
      - "/var/run/docker.sock:/var/run/docker.sock"



If you're running on a Mac, be sure to prepend TMPDIR=/private$TMPDIR before running:

$ TMPDIR=/private$TMPDIR docker-compose up 


Afterward, you should see something similar to the following output:

Shell
 




xxxxxxxxxx
1
46


 
1
Recreating localstack_localstack_1 ... done
2
Attaching to localstack_localstack_1
3
localstack_1  | Waiting for all LocalStack services to be ready
4
localstack_1  | 2020-12-03 20:45:34,940 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
5
localstack_1  | 2020-12-03 20:45:34,943 INFO supervisord started with pid 13
6
localstack_1  | 2020-12-03 20:45:35,951 INFO spawned: 'dashboard' with pid 19
7
localstack_1  | 2020-12-03 20:45:35,953 INFO spawned: 'infra' with pid 20
8
localstack_1  | 2020-12-03 20:45:35,958 INFO success: dashboard entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
9
localstack_1  | 2020-12-03 20:45:35,958 INFO exited: dashboard (exit status 0; expected)
10
localstack_1  | (. .venv/bin/activate; exec bin/localstack start --host)
11
localstack_1  | Starting local dev environment. CTRL-C to quit.
12
localstack_1  | LocalStack version: 0.12.2
13
localstack_1  | 2020-12-03 20:45:37,470 INFO success: infra entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
14
localstack_1  | Waiting for all LocalStack services to be ready
15
localstack_1  | Starting edge router (https port 4566)...
16
localstack_1  | Starting mock ACM service on http port 4566 ...
17
localstack_1  | Starting mock API Gateway service on http port 4566 ...
18
localstack_1  | Starting mock CloudFormation service on http port 4566 ...
19
localstack_1  | Starting mock CloudWatch service on http port 4566 ...
20
localstack_1  | Starting mock DynamoDB service on http port 4566 ...
21
localstack_1  | Starting mock DynamoDB Streams service on http port 4566 ...
22
localstack_1  | Starting mock EC2 service on http port 4566 ...
23
localstack_1  | Starting mock ES service on http port 4566 ...
24
localstack_1  | Starting mock Firehose service on http port 4566 ...
25
localstack_1  | Starting mock IAM service on http port 4566 ...
26
localstack_1  | Starting mock STS service on http port 4566 ...
27
localstack_1  | Starting mock Kinesis service on http port 4566 ...
28
localstack_1  | Starting mock KMS service on http port 4566 ...
29
localstack_1  | [2020-12-03 20:45:43 +0000] [21] [INFO] Running on http://0.0.0.0:47689 (CTRL + C to quit)
30
localstack_1  | 2020-12-03T20:45:43:INFO:hypercorn.error: Running on http://0.0.0.0:47689 (CTRL + C to quit)
31
localstack_1  | [2020-12-03 20:45:43 +0000] [21] [INFO] Running on https://0.0.0.0:4566 (CTRL + C to quit)
32
localstack_1  | 2020-12-03T20:45:43:INFO:hypercorn.error: Running on https://0.0.0.0:4566 (CTRL + C to quit)
33
localstack_1  | Starting mock Lambda service on http port 4566 ...
34
localstack_1  | Starting mock CloudWatch Logs service on http port 4566 ...
35
localstack_1  | Starting mock Redshift service on http port 4566 ...
36
localstack_1  | Starting mock Route53 service on http port 4566 ...
37
localstack_1  | Starting mock S3 service on http port 4566 ...
38
localstack_1  | Starting mock Secrets Manager service on http port 4566 ...
39
localstack_1  | Starting mock SES service on http port 4566 ...
40
localstack_1  | Starting mock SNS service on http port 4566 ...
41
localstack_1  | Starting mock SQS service on http port 4566 ...
42
localstack_1  | Starting mock SSM service on http port 4566 ...
43
localstack_1  | Starting mock Cloudwatch Events service on http port 4566 ...
44
localstack_1  | Starting mock StepFunctions service on http port 4566 ...
45
localstack_1  | Waiting for all LocalStack services to be ready
46
localstack_1  | Ready.



As you can see, it starts a whole bunch of services out of the box. If you don't use all those services, you can also provide a list of services required when starting localstackby providing aSERVICESvariable like:

$ TMPDIR=/private$TMPDIR SERVICES=s3,sqs docker-compose up


Now you should see in the startup output that it only started S3 and SQS:

Shell
 




xxxxxxxxxx
1
23


 
1
Recreating localstack_localstack_1 ... done
2
Attaching to localstack_localstack_1
3
localstack_1  | Waiting for all LocalStack services to be ready
4
localstack_1  | 2020-12-03 20:58:06,180 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
5
localstack_1  | 2020-12-03 20:58:06,183 INFO supervisord started with pid 13
6
localstack_1  | 2020-12-03 20:58:07,187 INFO spawned: 'dashboard' with pid 19
7
localstack_1  | 2020-12-03 20:58:07,191 INFO spawned: 'infra' with pid 20
8
localstack_1  | 2020-12-03 20:58:07,195 INFO success: dashboard entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
9
localstack_1  | 2020-12-03 20:58:07,195 INFO exited: dashboard (exit status 0; expected)
10
localstack_1  | (. .venv/bin/activate; exec bin/localstack start --host)
11
localstack_1  | Starting local dev environment. CTRL-C to quit.
12
localstack_1  | LocalStack version: 0.12.2
13
localstack_1  | 2020-12-03 20:58:08,856 INFO success: infra entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
14
localstack_1  | Waiting for all LocalStack services to be ready
15
localstack_1  | Starting edge router (https port 4566)...
16
localstack_1  | Starting mock S3 service on http port 4566 ...
17
localstack_1  | [2020-12-03 20:58:12 +0000] [21] [INFO] Running on http://0.0.0.0:55251 (CTRL + C to quit)
18
localstack_1  | 2020-12-03T20:58:12:INFO:hypercorn.error: Running on http://0.0.0.0:55251 (CTRL + C to quit)
19
localstack_1  | [2020-12-03 20:58:13 +0000] [21] [INFO] Running on https://0.0.0.0:4566 (CTRL + C to quit)
20
localstack_1  | 2020-12-03T20:58:13:INFO:hypercorn.error: Running on https://0.0.0.0:4566 (CTRL + C to quit)
21
localstack_1  | Starting mock SQS service on http port 4566 ...
22
localstack_1  | Waiting for all LocalStack services to be ready
23
localstack_1  | Ready.



Update: I just learned that homebrew also supports installing LocalStack. I've not used it, so I can't say if it's any good, but it looks pretty simple :-)

$ brew install localstack


If you don't want to manually start LocalStack via docker-compose but want to start it, for instance, during your build/test phase, you can also leverage testcontainers and add alocalstackrule to your Unit test:

Java
 




xxxxxxxxxx
1


 
1
DockerImageName localstackImage = DockerImageName.parse("localstack/localstack:0.11.3");
2
 
           
3
@Rule
4
public LocalStackContainer localstack = new LocalStackContainer(localstackImage)
5
        .withServices(S3);



AWSLocal

If you're already using LocalStack, it's worthwhile to also install awslocal. It's a CLI that proxies the AWS CLI and adds the --endpoint-url http://localhost:4566/ after every command, so you don't have to.

You can install it by running:

$ pip install awscli-local


Commandeer

Localstack used to come with a Web UI, which is now marked as deprecated. As an alternative, I would recommend using Commandeer. It's a very useful tool and also supports working with LocalStack (next to a whole bunch of other services). It can give a nice overview of the services started with LocalStack and offers dashboards and UIs, for example, DynamoDB, in which you can explore data, create tables, etc.

Local Stack: API Gateway

If you know other useful tools that help you in your day to day work working with and developing on the AWS platform, feel free to leave a comment!

 

 

 

 

Top