Using Multiple Azure Storage Accounts From a Single Spring Boot App

gowebing

Spring projects in general are opinionated: 80-90% of use cases are handled "by default", and code is often much more concise than would be required otherwise due to Spring's preference of convention over configuration. These and other "opinions" can result in dramatically less code to write and maintain and as a result, more focused impact.

In the vast majority of cases where Azure Storage is used from an application, there is no compelling advantage to using more than a single Azure storage account. But there are edge cases, and having the ability to use multiple Azure Storage accounts from a single app - even if we might only need that capability around 10% of the time - could provide an incredibly useful extension of our storage superpowers.

This article is the result of a collaboration with Shi li Chen.

It's all About Resources

The Spring Framework defines the Resource interface and provides several implementations built upon Resource to facilitate developer access to low-level resources. In order to handle a particular kind of resource, two things are required:

A Spring application evaluates resources in question using one or more registered resolvers. When the type of resource is identified, the appropriate Resource implementation is used to access and/or manipulate the underlying resource.

If the implementations built into Spring Framework don't fulfill your use case, it's fairly straightforward to add support for additional types of resources by defining your own implementations of AbstractResource and ResourcePatternResolver interfaces.

This article will introduce the Spring Resource, review Spring Cloud Azure's implementation of Spring's Resource (especially with regard to Azure Storage Account considerations and limitations), and consider how to expand said implementation to address those edge cases in which it would be useful to access multiple Azure Storage Accounts from a single Spring Boot application.

Getting Resourceful

We've already mentioned that the Spring Framework defines several useful Resource implementations. As of this writing, the default types are:

As mentioned earlier, each resource will have a corresponding resource resolver.

Enabling your Spring Boot application to use a custom Resource requires the following actions:

NOTE: Your resolver must be added to the default resource loader's resolver set using the org.springframework.core.io.DefaultResourceLoader#addProtocolResolver method, but this code is present in AbstractAzureStorageProtocolResolver; extending that class to create your implementation accomplishes this on your behalf unless you choose to override its setResourceLoader method.

A ResourceLoader attempts to resolve each Resource by comparing its defined location/format with all registered protocol pattern resolvers until a non-null resource is returned. If no match is found, the Resource will be evaluated against Spring's built-in pattern resolvers.

Spring Resources in Spring Cloud Azure

Spring Cloud Azure provides two Spring resource and resource pattern resolver implementations. In this article, we only discuss the implementation of the Azure Storage Blob resource. You can examine the source code for Spring Cloud Azure Resources at Spring Cloud Azure and related documentation at Resource Handling.

NOTE: We use Spring Cloud Azure Starter Storage Blob version 4.2.0 for analysis and experiments.

Implementation of AbstractResource

The abstract implementation AzureStorageResource for Spring Cloud Azure primarily defines the format of the Azure storage resource protocol and accommodates the unique attributes of the Azure Storage Account service, e.g. the container name and file name. It is important to note that AzureStorageResource is decoupled from the Azure Storage SDK.

The Spring Framework interface WritableResource represents the underlying API we build upon to read from and write to the Azure Storage resource.

Java
  
abstract class AzureStorageResource extends AbstractResource implements WritableResource {

    private boolean isAzureStorageResource(@NonNull String location) {
        ......
    }

    String getContainerName(String location) {
        ......
    }

    String getContentType(String location) {
        ......
    }

    String getFilename(String location) {
        ......
    }

    abstract StorageType getStorageType();
}


The StorageBlobResource is Spring Cloud Azure Storage Blob's implementation of the abstract class AbstractResource.

We can see StorageBlobResource uses the BlobServiceClient from the Azure Storage Blob SDK to implement all abstract methods, relying on the service client to interact with the Azure Storage Blob service.

Java
  
public final class StorageBlobResource extends AzureStorageResource {

    private final BlobServiceClient blobServiceClient;
    private final BlobContainerClient blobContainerClient;
    private final BlockBlobClient blockBlobClient;

    public StorageBlobResource(BlobServiceClient blobServiceClient, String location, Boolean autoCreateFiles,
                               String snapshot, String versionId, String contentType) {
        ......

        this.blobContainerClient = blobServiceClient.getBlobContainerClient(getContainerName(location));
        BlobClient blobClient = blobContainerClient.getBlobClient(getFilename(location));
        this.blockBlobClient = blobClient.getBlockBlobClient();
    }

    @Override
    public OutputStream getOutputStream() throws IOException {
        try {
            ......
            return this.blockBlobClient.getBlobOutputStream(options);
        } catch (BlobStorageException e) {
            throw new IOException(MSG_FAIL_OPEN_OUTPUT, e);
        }
    }

    ......    

    @Override
    StorageType getStorageType() {
        return StorageType.BLOB;
    }
}


Implementation of ResourcePatternResolver

Spring Cloud Azure provides an abstract implementation AbstractAzureStorageProtocolResolver. This class incorporates general processing of the Azure storage resource protocol, exposes specific capabilities of the Azure Storage Account service, and adds the requisite logic to the default resource loader. Like AzureStorageResource, the AbstractAzureStorageProtocolResolver is also not coupled to the Azure Storage SDK.

Java
  
public abstract class AbstractAzureStorageProtocolResolver implements ProtocolResolver, ResourcePatternResolver,
    ResourceLoaderAware, BeanFactoryPostProcessor {

    protected final AntPathMatcher matcher = new AntPathMatcher();
    protected abstract StorageType getStorageType();
    protected abstract Resource getStorageResource(String location, Boolean autoCreate);
    protected ConfigurableListableBeanFactory beanFactory;
    protected abstract Stream<StorageContainerItem> listStorageContainers(String containerPrefix);
    protected abstract StorageContainerClient getStorageContainerClient(String name);

    @Override
    public void setResourceLoader(ResourceLoader resourceLoader) {
        if (resourceLoader instanceof DefaultResourceLoader) {
            ((DefaultResourceLoader) resourceLoader).addProtocolResolver(this);
        } else {
            LOGGER.warn("Custom Protocol using azure-{}:// prefix will not be enabled.", getStorageType().getType());
        }
    }

    @Override
    public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {
        this.beanFactory = beanFactory;
    }

    @Override
    public Resource resolve(String location, ResourceLoader resourceLoader) {
        if (AzureStorageUtils.isAzureStorageResource(location, getStorageType())) {
            return getResource(location);
        }

        return null;
    }

    @Override
    public Resource[] getResources(String pattern) throws IOException {
        Resource[] resources = null;

        if (AzureStorageUtils.isAzureStorageResource(pattern, getStorageType())) {
            if (matcher.isPattern(AzureStorageUtils.stripProtocol(pattern, getStorageType()))) {
                String containerPattern = AzureStorageUtils.getContainerName(pattern, getStorageType());
                String filePattern = AzureStorageUtils.getFilename(pattern, getStorageType());

                resources = resolveResources(containerPattern, filePattern);
            } else {
                return new Resource[] { getResource(pattern) };
            }
        }

        if (null == resources) {
            throw new IOException("Resources not found at " + pattern);
        }

        return resources;
    }

    @Override
    public Resource getResource(String location) {
        Resource resource = null;

        if (AzureStorageUtils.isAzureStorageResource(location, getStorageType())) {
            resource = getStorageResource(location, true);
        }
        if (null == resource) {
            throw new IllegalArgumentException("Resource not found at " + location);
        }

        return resource;
    }

    /**

     * Storage container item.

     */

    protected static class StorageContainerItem {
        private final String name;

        ......
    }

    protected static class StorageItem {
        private final String container;
        private final String name;
        private final StorageType storageType;

        ......
    }

    protected interface StorageContainerClient {

        ......
    }
}


The resource resolver AzureStorageBlobProtocolResolver is Spring Cloud Azure Storage Blob's implementation of ResourcePatternResolver. It encapsulates resources according to the location or storage item pattern based on BlobServiceClient and returns the associated StorageBlobResource.

Java
  
public final class AzureStorageBlobProtocolResolver extends AbstractAzureStorageProtocolResolver {

    private BlobServiceClient blobServiceClient;

    @Override
    protected StorageType getStorageType() {
        return StorageType.BLOB;
    }

    @Override
    protected Resource getStorageResource(String location, Boolean autoCreate) {
        return new StorageBlobResource(getBlobServiceClient(), location, autoCreate);
    }

    private BlobServiceClient getBlobServiceClient() {
        if (blobServiceClient == null) {
            blobServiceClient = beanFactory.getBean(BlobServiceClient.class);
        }

        return blobServiceClient;
    }
}


Opinions

As mentioned at the beginning of this post, the default capabilities fulfill the requirements admirably in the vast majority of circumstances. But in accordance with the Spring ethos, Spring Cloud Azure Starter Storage Blob was designed to seamlessly address 80-90% of use cases "out of the box", while still allowing for remaining (edge) cases with some extra effort.

As written, the storage blob resource supports multiple container operations using the same storage account. The salient point is that the blob paths under different containers can be properly resolved into StorageBlobResource objects. Combining the earlier code for StorageBlobResource, the blob resource must hold a blob service client, and if blobServiceClient.getBlobContainerClient(getContainerName(location)) successfully returns a BlobServiceClient, the blob resource can be resolved and retrieved.

The BlobServiceClient bean represents an Azure Storage Account in the Azure Storage Blob SDK, meaning that the current implementation does not support simultaneous availability using multiple Azure Storage Accounts.

Developing an Extended Version of Spring Cloud Azure Starter Storage Blob

For those rare cases in which it might be useful to simultaneously access multiple Azure Storage accounts from the same application, there is a way to make that happen. To demonstrate this capability, let's create a new library called spring-cloud-azure-starter-storage-blob-extend. The only external dependency for this new library is the existing spring-cloud-azure-starter-storage-blob.

Extend the Storage Blob Properties

While the primary goal is to support multiple storage accounts, a secondary design goal is to use a similar structure to AzureStorageBlobProperties in order to minimize the learning curve and to retain Spring Cloud Azure 4.0's out-of-the-box authentication features.

Java
  
public class ExtendAzureStorageBlobsProperties {

    public static final String PREFIX = "spring.cloud.azure.storage.blobs";
    private boolean enabled = true;
    private final List<AzureStorageBlobProperties> configurations = new ArrayList<>();

    public boolean isEnabled() {
        return enabled;
    }

    public void setEnabled(boolean enabled) {
        this.enabled = enabled;
    }

    public List<AzureStorageBlobProperties> getConfigurations() {
        return configurations;
    }
}

Dynamically Register Storage Blob Beans

Since there will be multiple Storage Account configurations, we must name the beans corresponding to each storage account. The cleanest approach is to simply use the account name as the bean name.

Now, let's dynamically register these beans with the Spring context.

Java
  
@Configuration(proxyBeanMethods = false)
@ConditionalOnProperty(value = { "spring.cloud.azure.storage.blobs.enabled"}, havingValue = "true")
public class ExtendStorageBlobsAutoConfiguration implements BeanDefinitionRegistryPostProcessor, EnvironmentAware {

    private Environment environment;
    public static final String EXTEND_STORAGE_BLOB_PROPERTIES_BEAN_NAME = "extendAzureStorageBlobsProperties";

    @Override
    public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {
        AzureGlobalProperties azureGlobalProperties =
            Binder.get(environment)
                  .bind(AzureGlobalProperties.PREFIX, AzureGlobalProperties.class)
                  .orElse(new AzureGlobalProperties());
      
        ExtendAzureStorageBlobsProperties blobsProperties =
            Binder.get(environment)
                  .bind(ExtendAzureStorageBlobsProperties.PREFIX, ExtendAzureStorageBlobsProperties.class)
                  .orElseThrow(() -> new IllegalArgumentException("Can not bind the azure storage blobs properties."));

        // merge properties
        for (AzureStorageBlobProperties azureStorageBlobProperties : blobsProperties.getConfigurations()) {
            AzureStorageBlobProperties transProperties = new AzureStorageBlobProperties();
            AzureGlobalPropertiesUtils.loadProperties(azureGlobalProperties, transProperties);

            copyAzureCommonPropertiesIgnoreTargetNull(transProperties, azureStorageBlobProperties);
        }

        DefaultListableBeanFactory factory = (DefaultListableBeanFactory) beanFactory;

        registryBeanExtendAzureStorageBlobsProperties(factory, blobsProperties);

        blobsProperties.getConfigurations().forEach(blobProperties -> registryBlobBeans(factory, blobProperties));
    }

    private void registryBeanExtendAzureStorageBlobsProperties(DefaultListableBeanFactory beanFactory,
                                                               ExtendAzureStorageBlobsProperties blobsProperties) {
        BeanDefinitionBuilder beanDefinitionBuilder = BeanDefinitionBuilder.genericBeanDefinition(ExtendAzureStorageBlobsProperties.class,
            () -> blobsProperties);

        AbstractBeanDefinition rawBeanDefinition = beanDefinitionBuilder.getRawBeanDefinition();

        beanFactory.registerBeanDefinition(EXTEND_STORAGE_BLOB_PROPERTIES_BEAN_NAME, rawBeanDefinition);
    }

    private void registryBlobBeans(DefaultListableBeanFactory beanFactory, AzureStorageBlobProperties blobProperties) {
        String accountName = getStorageAccountName(blobProperties);

        Assert.hasText(accountName, "accountName can not be null or empty.");

        registryBeanStaticConnectionStringProvider(beanFactory, blobProperties, accountName);

        registryBeanBlobServiceClientBuilderFactory(beanFactory, blobProperties, accountName);

        registryBeanBlobServiceClientBuilder(beanFactory, accountName);

        registryBeanBlobServiceClient(beanFactory, accountName);

        registryBeanBlobContainerClient(beanFactory, blobProperties, accountName);

        registryBeanBlobClient(beanFactory, blobProperties, accountName);
    }

    private void registryBeanBlobServiceClientBuilder(DefaultListableBeanFactory beanFactory,
                                                      String accountName) {
        BeanDefinitionBuilder beanDefinitionBuilder = BeanDefinitionBuilder.genericBeanDefinition(BlobServiceClientBuilder.class,
            () -> {
                BlobServiceClientBuilderFactory builderFactory =
                    beanFactory.getBean(accountName + BlobServiceClientBuilderFactory.class.getSimpleName(),
                        BlobServiceClientBuilderFactory.class);

                return builderFactory.build();
            });

        AbstractBeanDefinition rawBeanDefinition = beanDefinitionBuilder.getRawBeanDefinition();

        beanFactory.registerBeanDefinition(
            accountName + BlobServiceClientBuilder.class.getSimpleName(), rawBeanDefinition);
    }

    ......

    @Override
    public void setEnvironment(Environment environment) {
        this.environment = environment;
    }
}

Extend the AzureStorageBlobProtocolResolver

The next task is to make any container resolvable by the same resource pattern resolver. Specifying a storage blob resource location such as azure-blob-accountname://containername/test.txt, the resolver will use that to locate the appropriate BlobServiceClient bean by Azure Storage Account name and return the storage resource.

Java
  
public class ExtendAzureStorageBlobProtocolResolver extends ExtendAbstractAzureStorageProtocolResolver {

    private final Map<String, BlobServiceClient> blobServiceClientMap = new HashMap<>();

    @Override
    protected Resource getStorageResource(String location, Boolean autoCreate) {
        return new ExtendStorageBlobResource(getBlobServiceClient(location), location, autoCreate);
    }

    private BlobServiceClient getBlobServiceClient(String locationPrefix) {
        String storageAccount = ExtendAzureStorageUtils.getStorageAccountName(locationPrefix, getStorageType());

        Assert.notNull(storageAccount, "storageAccount can not be null.");

        String accountKey = storageAccount.toLowerCase(Locale.ROOT);

        if (blobServiceClientMap.containsKey(accountKey)) {
            return blobServiceClientMap.get(accountKey);
        }

        BlobServiceClient blobServiceClient = beanFactory.getBean(
            accountKey + BlobServiceClient.class.getSimpleName(), BlobServiceClient.class);

        Assert.notNull(blobServiceClient, "blobServiceClient can not be null.");

        blobServiceClientMap.put(accountKey, blobServiceClient);

        return blobServiceClient;
    }
}

Again, you need to add the bean ExtendAzureStorageBlobProtocolResolver to the Spring context.

Testing the Spring Cloud Azure Starter Storage Blob Extend

You can use the Spring Initializr to generate a Spring Boot 2.6.7 or greater project with Azure Storage support (or build on this storage blob sample if you prefer).

Add the extending starter dependency to the pom.xml file:

XML
  
<dependency>
  <groupId>com.azure.spring.extend</groupId>
  <artifactId>spring-cloud-azure-starter-storage-blob-extend</artifactId>
  <version>1.0-SNAPSHOT</version>
</dependency>

Delete the src/main/resources/application.properties file or add the following configuration file application-extend.yml, which enables multiple storage account usage:

application-extend.yml

YAML
  
spring:
  cloud:
    azure:
      storage:
        blob:
          enabled: false
        blobs:
          enabled: true
          configurations:
            - account-name: ${FIRST_ACCOUNT}
              container-name: ${FIRST_CONTAINER}
              account-key: ${ACCOUNT_KEY_OF_FIRST_ACCOUNT}
            - account-name: ${SECOND_ACCOUNT}
              container-name: ${SECOND_CONTAINER}
              account-key: ${ACCOUNT_KEY_OF_SECOND_ACCOUNT}

NOTE: You must provide values for the environment variables above (listed in all capital letters) with active Azure Storage Account resource information.

Add class com.azure.spring.extend.sample.storage.resource.extend.SampleDataInitializer with the following body:

Java
  
@Profile("extend")
@Component
public class SampleDataInitializer implements CommandLineRunner {

    final static Logger logger = LoggerFactory.getLogger(SampleDataInitializer.class);
    private final ConfigurableEnvironment env;
    private final ExtendAzureStorageBlobProtocolResolver resolver;
    private final ExtendAzureStorageBlobsProperties properties;

    public SampleDataInitializer(ConfigurableEnvironment env, ExtendAzureStorageBlobProtocolResolver resolver,
                                 ExtendAzureStorageBlobsProperties properties) {
        this.env = env;
        this.resolver = resolver;
        this.properties = properties;
    }

    /**
     * This is used to initialize some data for each Azure Storage Account Blob container.
     */
    @Override
    public void run(String... args) {
        properties.getConfigurations().forEach(this::writeDataByStorageAccount);
    }

    private void writeDataByStorageAccount(AzureStorageBlobProperties blobProperties) {
        String containerName = blobProperties.getContainerName();

        if (!StringUtils.hasText(containerName) || blobProperties.getAccountName() == null) {
            return;
        }

        String accountName = getStorageAccountName(blobProperties);

        logger.info("Begin to initialize the {} container of the {} account", containerName, accountName);

        long currentTimeMillis = System.currentTimeMillis();

        String fileName = "fileName-" + currentTimeMillis;

        String data = "data" + currentTimeMillis;

        Resource storageBlobResource = resolver.getResource("azure-blob-" + accountName + "://" + containerName +"/" + fileName + ".txt");

        try (OutputStream os = ((WritableResource) storageBlobResource).getOutputStream()) {
            os.write(data.getBytes());

            logger.info("Write data to container={}, fileName={}.txt", containerName, fileName);
        } catch (IOException e) {
            logger.error("Write data exception", e);
        }

        logger.info("End to initialize the {} container of the {} account", containerName, accountName);
    }
}

Run the sample with the following Maven command:

mvn clean spring-boot:run -Dspring-boot.run.profiles=extend

Finally, verify the expected outcome. Your console should display the following output:

  
c.a.s.e.s.s.r.e.SampleDataInitializer    : Begin to initialize the container first of the account firstaccount.
c.a.s.e.s.s.r.e.SampleDataInitializer    : Write data to container=first, fileName=fileName-1656641340271.txt
c.a.s.e.s.s.r.e.SampleDataInitializer    : End to initialize the container first of the account firstaccount.
c.a.s.e.s.s.r.e.SampleDataInitializer    : Begin to initialize the container second of the account secondaccount.
c.a.s.e.s.s.r.e.SampleDataInitializer    : Write data to container=second, fileName=fileName-1656641343572.txt
c.a.s.e.s.s.r.e.SampleDataInitializer    : End to initialize the container second of the account secondaccount.

All sample project code is published at the repository spring-cloud-azure-starter-storage-blob-extend-sample.

Within this extended application, it's still possible to revert to the original, single storage account usage of Spring Cloud Azure Starter Storage Blob by adding the following configuration file application-current.yml:

YAML
  
spring:
  cloud:
    azure:
      storage:
        blob:
          account-name: ${FIRST_ACCOUNT}
          container-name: ${FIRST_CONTAINER}
          account-key: ${ACCOUNT_KEY_OF_FIRST_ACCOUNT}
current:
  second-container: ${SECOND_CONTAINER}

NOTE: You must set or replace the listed environment variable assigned values with active Azure Storage Account resource information.

Run the sample with the following Maven command:

mvn clean spring-boot:run -Dspring-boot.run.profiles=current

To verify the correct operation using a single storage account, compare the terminal output with that listed below:

  
c.a.s.e.s.s.r.c.SampleDataInitializer    : StorageApplication data initialization of 'first-container' begin ...
c.a.s.e.s.s.r.c.SampleDataInitializer    : Write data to container=first-container, fileName=fileName1656641162614.txt
c.a.s.e.s.s.r.c.SampleDataInitializer    : StorageApplication data initialization of 'first-container' end ...
c.a.s.e.s.s.r.c.SampleDataInitializer    : StorageApplication data initialization of 'second-container' begin ...
c.a.s.e.s.s.r.c.SampleDataInitializer    : Write data to container=second-container, fileName=fileName1656641165411.txt
c.a.s.e.s.s.r.c.SampleDataInitializer    : StorageApplication data initialization of 'second-container' end ...

Conclusion

Implementing a specific resource type and corresponding pattern resolver is relatively simple, largely thanks to clear documentation, the many built-in implementations, and common usage within the Spring technology stack.

One point that warrants attention is the protocol definition for the resource, e.g. the Azure Storage Blob Resource. We must note whether we are using azure-blob:// or azure-blob-[account-name]:// and plan app capabilities accordingly. Additionally, since the identifier of a network resource must be uniquely identifiable, the latter location format may result in a much longer name and also exposes the name of the storage account. These tradeoffs need to be evaluated in light of requirements and risk profiles.

References and Useful Resources

Latest Spring Resources documentation

Extended starter for Spring Cloud Azure Starter Storage Blob and sample on GitHub

 

 

 

 

Top