Microservice: Async Rest Client to DynamoDB using Spring Boot

gowebing 2024-12-02

Overview

Starting from Spring framework 5.0 and Spring Boot 2.0, the framework provides support for asynchronous programming, so does AWS SDK starting with 2.0 version.

In this post, I will be exploring using asynchronous DynamoDB API and Spring Webflux by building a simple reactive REST application. Let's say we need to handle HTTP requests for retrieving or storing some Event (id:string, body: string). The event will be stored in DynamoDB.

It might be easier to simply look at the code on Github and follow it there.

Maven Dependencies

Let's start with Maven dependencies for WebFlux and DynamoDB asynchronous SDK.

      XML 
    
xxxxxxxxxx
 
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-webflux</artifactId>
        <version>2.2.4.RELEASE</version>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>dynamodb</artifactId>
        <version>2.10.76</version>
    </dependency>
</dependencies>

DynamoDB Stuff

Spring Configuration

A simple config were we set up a connection to DynamoDB. For test purposes, we need to specify dynamoEndpoint but for real application, we need to specify the AWS region. DynamoEndpoint will point to the local DynamoDB instance which we will start during tests.

      Java 
    
xxxxxxxxxx
 
@Configuration
public class AppConfig {
    @Value("${aws.accessKey}")
    String accessKey;
    @Value("${aws.secretKey}")
    String secretKey;
    @Value("${dynamodb.endpoint:}")
    String dynamoEndpoint;
    @Bean
    AwsBasicCredentials awsBasicCredentials(){
        return AwsBasicCredentials.create(accessKey, secretKey);
    }
    @Bean
    DynamoDbAsyncClient dynamoDbAsyncClient(
      AwsBasicCredentials awsBasicCredentials
    ){
        DynamoDbAsyncClientBuilder clientBuilder = DynamoDbAsyncClient.builder();
        clientBuilder
                .credentialsProvider(StaticCredentialsProvider.create(awsBasicCredentials));
                if(!dynamoEndpoint.isEmpty()){
                    clientBuilder.endpointOverride(URI.create(dynamoEndpoint));
                }
        return clientBuilder.build();
    }
}

Application.yaml with connection details.

      YAML 
    
xxxxxxxxxx
 
aws:
 accessKey: any
 secretKey: any
dynamodb:
 endpoint: http://localhost:8000/

Reactive DynamoDB Service

Unfortunately, async version of AWS SDK doesn't have support for DynamoDBMapper yet (you can track mapper's readiness here), so table creation, sending requests and parsing responses need to be done by "low level" API.

So we need to create DynamoDbService for handling:

Creation table if it does not exists
Saving and retrieving event

      Java 
    
 
 
       
     
 
       
 
      
 
       
 
      
 
      
 
     
 
           x 
          
 
           
 
          
 
           
 
          

               
           
@Service
public class DynamoDbService {
    public static final String TABLE_NAME = "events";
    public static final String ID_COLUMN = "id";
    public static final String BODY_COLUMN = "body";
    final DynamoDbAsyncClient client;
    @Autowired
    public DynamoDbService(DynamoDbAsyncClient client) {
        this.client = client;
    }
    //Creating table on startup if not exists
    @PostConstruct
    public void createTableIfNeeded() throws ExecutionException, InterruptedException {
        ListTablesRequest request = ListTablesRequest
                .builder()
                .exclusiveStartTableName(TABLE_NAME)
                .build();
        CompletableFuture<ListTablesResponse> listTableResponse = client.listTables(request);
        CompletableFuture<CreateTableResponse> createTableRequest = listTableResponse
                .thenCompose(response -> {
                    boolean tableExist = response
                            .tableNames()
                            .contains(TABLE_NAME);
                    
                    if (!tableExist) {
                        return createTable();
                    } else {
                        return CompletableFuture.completedFuture(null);
                    }
                });
        //Wait in synchronous manner for table creation
        createTableRequest.get();
    }
    public CompletableFuture<PutItemResponse> saveEvent(Event event) {
        Map<String, AttributeValue> item = new HashMap<>();
        item.put(ID_COLUMN, AttributeValue.builder().s(event.getUuid()).build());
        item.put(BODY_COLUMN, AttributeValue.builder().s(event.getBody()).build());
        PutItemRequest putItemRequest = PutItemRequest.builder()
                .tableName(TABLE_NAME)
                .item(item)
                .build();
        return client.putItem(putItemRequest);
    }
    public CompletableFuture<Event> getEvent(String id) {
        Map<String, AttributeValue> key = new HashMap<>();
        key.put(ID_COLUMN, AttributeValue.builder().s(id).build());
        GetItemRequest getRequest = GetItemRequest.builder()
                .tableName(TABLE_NAME)
                .key(key)
                .attributesToGet(BODY_COLUMN)
                .build();
        return client.getItem(getRequest).thenApply(item -> {
            if (!item.hasItem()) {
                return null;
            } else {
                Map<String, AttributeValue> itemAttr = item.item();
                String body = itemAttr.get(BODY_COLUMN).s();
                return new Event(id, body);
            }
        });
    }
    private CompletableFuture<CreateTableResponse> createTable() {
        KeySchemaElement keySchemaElement = KeySchemaElement
                .builder()
                .attributeName(ID_COLUMN)
                .keyType(KeyType.HASH)
                .build();
        AttributeDefinition attributeDefinition = AttributeDefinition
                .builder()
                .attributeName(ID_COLUMN)
                .attributeType(ScalarAttributeType.S)
                .build();
        
        CreateTableRequest request = CreateTableRequest.builder()
                .tableName(TABLE_NAME)
                .keySchema(keySchemaElement)
                .attributeDefinitions(attributeDefinition)
                .billingMode(BillingMode.PAY_PER_REQUEST)
                .build();
        return client.createTable(request);
    }
}
 
       
 
      

Reactive REST Controller

A simple controller with the GET method for retrieving events by id and POST method for saving events in DynamoDB. We can do it in two ways — implement it with annotations or get rid of annotations and do it in a functional way. There is no performance impact, in almost most cases it is absolutely based on individual preference what to use.

Annotated Controllers

      Java 
    
xxxxxxxxxx
 
@RestController
@RequestMapping("/event")
public class AnnotatedController {
    final DynamoDbService dynamoDbService;
    public AnnotatedController(DynamoDbService dynamoDbService) {
        this.dynamoDbService = dynamoDbService;
    }
    @GetMapping(value = "/{eventId}", produces = MediaType.APPLICATION_JSON_VALUE)
    public Mono<Event> getEvent(@PathVariable String eventId) {
        CompletableFuture<Event> eventFuture = dynamoDbService.getEvent(eventId);
        return Mono.fromCompletionStage(eventFuture);
    }
    @PostMapping(consumes = MediaType.APPLICATION_JSON_VALUE)
    public void saveEvent(@RequestBody Event event) {
        dynamoDbService.saveEvent(event);
    }
}

Functional Endpoints

This is a lightweight functional programming model in which functions are used to route and handle requests. Here I created EventHandler just for the simplicity of reading the code.

      Java 
    
xxxxxxxxxx
 
@Configuration
public class HttpRouter {
    @Bean
    public RouterFunction<ServerResponse> eventRouter(DynamoDbService dynamoDbService) {
        EventHandler eventHandler = new EventHandler(dynamoDbService);
        return RouterFunctions
                .route(GET("/eventfn/{id}")
                        .and(accept(APPLICATION_JSON)), eventHandler::getEvent)
                .andRoute(POST("/eventfn")
                        .and(accept(APPLICATION_JSON))
                        .and(contentType(APPLICATION_JSON)), eventHandler::saveEvent);
    }
    static class EventHandler {
        private final DynamoDbService dynamoDbService;
        public EventHandler(DynamoDbService dynamoDbService) {
            this.dynamoDbService = dynamoDbService;
        }
        Mono<ServerResponse> getEvent(ServerRequest request) {
            String eventId = request.pathVariable("id");
            CompletableFuture<Event> eventGetFuture = dynamoDbService.getEvent(eventId);
            Mono<Event> eventMono = Mono.fromFuture(eventGetFuture);
            return ServerResponse.ok().body(eventMono, Event.class);
        }
        Mono<ServerResponse> saveEvent(ServerRequest request) {
            Mono<Event> eventMono = request.bodyToMono(Event.class);
            eventMono.map(dynamoDbService::saveEvent);
            return ServerResponse.ok().build();
        }
    }
}

Spring DynamoDB Integration Test

Maven Dependencies

For running integration test with DynamoDB we need DynamoDBLocal, which is not really the DynamoDB, but SQLite with implemented DynamoDB interfaces on top of it.

      XML 
    
xxxxxxxxxx
 
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>DynamoDBLocal</artifactId>
    <version>1.12.0</version>
    <scope>test</scope>
</dependency>
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <version>2.10</version>
            <executions>
                <execution>
                    <id>copy</id>
                    <phase>test-compile</phase>
                    <goals>
                        <goal>copy-dependencies</goal>
                    </goals>
                    <configuration>
                        <includeScope>test</includeScope>
                        <includeTypes>so,dll,dylib</includeTypes>
                        <!--Keep an eye on output directory - it will be used for starting dynamodb-->
                        <outputDirectory>${project.basedir}/target/native-libs</outputDirectory>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
<repositories>
    <repository>
        <id>dynamodb-local-oregon</id>
        <name>DynamoDB Local Release Repository</name>
        <url>https://s3-us-west-2.amazonaws.com/dynamodb-local/release</url>
    </repository>
</repositories>

DynamoDB server

Now we need to start DynamoDB before the test run. I prefer to do it as JUnit Class Rule, but we can also do it as a spring bean.

      Java 
    
xxxxxxxxxx
 
public class LocalDynamoDbRule extends ExternalResource {
    protected DynamoDBProxyServer server;
    public LocalDynamoDbRule() {
        //here we set the path from "outputDirectory" of maven-dependency-plugin
        System.setProperty("sqlite4java.library.path", "target/native-libs");
    }
    @Override
    protected void before() throws Exception {
        this.server = ServerRunner
            .createServerFromCommandLineArgs(new String[]{"-inMemory", "-port", "8000"});
        server.start();
    }
    @Override
    protected void after() {
        this.stopUnchecked(server);
    }
    protected void stopUnchecked(DynamoDBProxyServer dynamoDbServer) {
        try {
            dynamoDbServer.stop();
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }
}

Running Test

Now we can create an integration test and test get event by id and save the event.

      Java 
    
xxxxxxxxxx
 
@RunWith(SpringRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class IntegrationTest {
    @ClassRule
    public static LocalDynamoDbRule dynamoDbRule = new LocalDynamoDbRule();
    @Autowired
    private WebTestClient webTestClient;
    @Test
    public void getEvent() {
        // Create a GET request to test an endpoint
        webTestClient
                .get().uri("/event/1")
                .accept(MediaType.APPLICATION_JSON)
                .exchange()
                // and use the dedicated DSL to test assertions against the response
                .expectStatus().isOk()
                .expectBody(String.class).isEqualTo(null);
    }
    @Test
    public void saveEvent() throws InterruptedException {
        Event event = new Event("10", "event");
        webTestClient
                .post().uri("/event/")
                .body(BodyInserters.fromValue(event))
                .exchange()
                .expectStatus().isOk();
        Thread.sleep(1500);
        webTestClient
                .get().uri("/event/10")
                .accept(MediaType.APPLICATION_JSON)
                .exchange()
                .expectStatus().isOk()
                .expectBody(Event.class).isEqualTo(event);
    }
}

Docker

Here we gonna prepare our application for running in docker, so it will be ready for deploying to AWS.

HINT: Starting from Java 10 you can specify how much memory JVM will use depending on container memory.-XX:MaxRAMPercentage=75.0 means JVM won't use more than 75% of a container memory.

Dockerfile

      Dockerfile 
    
xxxxxxxxxx
 
# Use our standard java12 baseimage
FROM openjdk:12-alpine
# Copy the artifact into the container
COPY target/dynamodb-spring-*-exec.jar /srv/service.jar
# Run the artifact and expose the default port
WORKDIR /srv
ENTRYPOINT [ "java", \
    "-XX:+UnlockExperimentalVMOptions", \
    "-XX:+ExitOnOutOfMemoryError", \
    "-XX:MaxRAMPercentage=75.0", \
    "-Djava.security.egd=file:/dev/./urandom", \
    "-jar", "service.jar", \
    "--spring.profiles.active=prod" ]
EXPOSE 8080

Building the docker container itself docker build -t spring-dynamo.

Also, let's see what was generated by docker image ls.

      Plain Text 
    
xxxxxxxxxx
 
REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE
spring-dynamo       latest              a974d880400e        About a minute ago   364MB
openjdk             12-alpine           0c68e7c5b7a0        12 months ago        339MB

Finally, our POC is ready!

Happy coding :)