Multi-Cloud Integration
Before we jump into the details, let’s answer the question: “What is cloud integration architecture?” Cloud integration architecture is the combination of the cloud components and several cloud providers that allows exchanging of data across several data centers.
For example, imagine that we have a large banking solution. Some banks don’t want to migrate their core and secured data. They want to keep it in their on-premises data centers. However, they can’t avoid using clouds. They migrate existing projects, new projects that are starting, and even the whole call centers and back office to the cloud. Building this complex architecture requires knowledge of the integration service and tools that the clouds provide.
Let’s see what components AWS, Google Cloud, and Azure provide.
Common Integration Services
Knowing general cloud integration best practices and patterns is crucial. However, knowing what exactly each cloud provider offers is also important. Here we will get into the top integration services that AWS, Google Cloud, and Azure provide. Also, we look at what benefits and drawbacks each service has.
AWS
AWS has several integration services that provide powerful features alongside simplicity.
SQS (Simple Queue Service)
SQS is a simple query service or distributed queuing system. It’s a queue that stores the messages that should be processed later. SQS is a pull-based service that allows the decoupling of cloud components and microservices.
Pros:
- Includes dead-letter queue: Automatically separates unprocessed messages and pushes them into the dead-letter queue
- The choice between a standard queue and FIFO
- Automatic duplication removal: SQS will automatically detect and remove duplicated messages.
Cons:
- Doesn’t support one-to-many message broadcasting
SWF (Simple Workflow Service)
SWF is a task coordination and workflow service. It allows the building of parallels and sequential workflows. It allows for building a reliable distributed system where the tasks are logical units or functions of your application. SWF is suitable for processing human-based operations like order workflow or procedural requests.
Pros:
- Reliability: Workflows and state tracking are reliable as they run at high-availability data centers
- Flexibility and logical separation: SWF allows architects and users to modify workflow elements easily. Also, SWF separates the control flow of your background job
Cons:
- A lot of bugs in the AWS Management Console and a lack of general support
- The setup process is difficult and takes a lot of time
- Lack of limit control tracking: It can lead to throttling issues
AWS Step Functions
AWS step functions allow us to create a serverless workflow or state machines; for example, to automate call center tasks. Step Functions is an orchestration function that orchestrates other Lambda functions.
Pros:
- Decreasing application complexity by decoupling workflow logic from business logic
Cons:
- Step Functions uses a specific state machine to define workflows, helpful only for the Step Functions service. This may lead to vendor lock-in.
Google Cloud (GC)
The GC has an integration ecosystem, an Integration-Platform-as-a-Service (iPaaS). It provides a set of tools and services to manage and connect applications. The Google Cloud iPaaS contains the following services:
Integration Designer
Integration designer is a no-code UI tool that provides the ability to build workflows that consist of triggers and tasks (see below). Tasks and triggers are connected with an edge and fork and join. This service looks pretty similar to SWF.Triggers
Triggers are an essential workflow element. Triggers are usually an entry point for the workflow and represent an event that initiates the task. For example, API trigger or pub/sub trigger.
Tasks
Tasks are units of work (or functions) that comes after triggers. One example is Data Mapping, where you can integrate one data model into another. Some predefined tasks include the Call Integration task, For Each Parallel task, For Each Loop task, Timer task, and many others.
Alongside many advantages, Google Cloud has some drawbacks:
- Lack of data centers compared to Azure and AWS
- It is a relatively new framework. Some features may be in preview (or in beta) and contain some bugs.
Azure
Azure offers an Azure integration service set. It contains a variety of services to set up strong integration between applications. Below are set of the most powerful integrations services Azure offers.
API Management
API Management is an API gateway service that integrates several APIs and connects Azure Functions, Logic Apps, and Service Bus.
Pros:
- API Management provides many features like authentication, authorization, cache responses, and fine-grain quotas and limits.
- Flexible customization through the API policies
Cons:
- The developer portal contains many usability issues and bugs.
- Scaling costs can be too high. API management scales automatically; however, there is no good transparency on scaling costs.
Logic Apps
Logic Apps is a serverless, low-code, workflow integration tool (like GC Integration designer and AWS SWF).
Pros:
- The service can connect SQL Server, Event Grid, Oracle, ServiceNow, and Office 365. We can select more than 200 connectors.
- Ability to host maps and schemas locally
Cons:
- Difficult to manage and maintain complex integration workflows
- Issues with user interface, especially with complex workflows
Service Bus
Service Bus is a service that implements the event bus pattern. It provides an event bus based on the queue and publish/subscribe integration principle when one component produces a message, and others consume it.
Pros:
- The service also supports translations, dead letters, multi-consumer topic-based logic, and other features.
- Powerful message routing and filtering capabilities
Cons:
- Slow performance of the HTTPs connection
- Option to monitor messages, queues, and topics is quite limited
- No integrated UI tool to debug messages, queries, and filters
Event Grid
With the Event Grid, you can join cloud resources that produce events (publishers) and resources that handle the events (subscribers). The event grid looks similar to AWS SNS. It is also based on the push approach. Event Grid listens to the events from one component and can trigger other when it reserves messages. For example, one application can send a message to the Azure queue and an email notification.
Pros:
- Enterprise integration: Supports almost all Azure event sources; e.g., Functions, Event/IoT Hubs, Logic Apps, Service Bus, SignalR, and many more
- Contains options to monitor and debug messages
- Includes automatic dead-letter queue
- Lower cost: A pay-per-event price model results in very low cost.
Cons:
- Transparency on how to store and enable dead-letter can be improved.
Azure Arc
Azure Arc is a service that allows us to connect services from different cloud providers and implement multi-cloud integrations. This service will as a main integration service banking multi-cloud application.
Pros:
- Supports services like SQL Server and Kubernetes
- Supports Windows and Linux operation systems
Cons:
- Sometimes, the agent connection can be unstable and hard to debug.
Conclusion
In the article, we’ve gone through the top components to build cloud integration. However, integration solutions don’t have to be enterprise-level: they can meddle or even be small. For example, we have application A which requires the exchange of data between application B. We can create integration middleware with Azure Durable Functions or AWS Step Functions. With these services, we can create workflows for data mapping, notification, and even some plain analytics.
