Architecture Decision for Choosing Right Digital Integration Patterns – API vs. Messaging vs. Event

IT environments and application portfolios are growing rapidly, and they are fundamentally hybrid in nature. Due to the growing need to connect heterogeneous applications on hybrid cloud, in various locations and by different service providers, there is a need for using robust and fit-for-purpose integration techniques. Direct APIs (e.g. Rest, gRPC), Messaging, and Events are the most common integration choices for digital applications. While making these important architectural choices, often organizations and practitioners take an extreme stand towards one or the other of these integration patterns. However, it is not the question of whether to use Direct API, Messaging, or Event, but it is more a question of when to use each of these integration Patterns.

Problem Statement

The three integration patterns (Direct API, Messaging, and Events) are important for modern applications. However, it is often a challenge for designers and developers to determine the integration option best suited for different scenarios and the implications of the choice of integration. In the section below, we provide a guiding framework to help make this decision better based on the context of the integration and organizational factors

A Closer Look at API, Messaging, and Event-Based Integrations

Direct API 

Direct Application Programming Interface (API), allows two heterogeneous applications to talk to each other. For example, each time we use an app on our mobile devices, the app is likely making several API calls to various digital services.

Direct APIs can be designed to be Blocking (Synchronous) or Non-Blocking (Asynchronous). Of these, Non-Blocking APIs are preferred to ensure resources are not blocked when the consumer is waiting for a response from provider. Non-blocking APIs also help create independently scalable integration model between API Consumers and API Provider

Strengths

• The operational complexity is lower for Direct APIs compared to the other two options
• For channel applications as well as Business partner application interactions, several integration standards have emerged over the last decade for different industries. These standards can be leveraged while using API based Integration

Weaknesses 

• While using APIs, it is important to consider scenarios when the API provider
  1.  is not available to serve exactly at that point in time when a request arrives (e.g. there are only limited instance of provider service, and all  are busy at that time)
  2. Provides extremely slow performance or is unable to scale up beyond a limit such scenarios need to be handled by API consumers, to deliver an acceptable response to the user.
• Resiliency patterns should be adopted at both the consumer and the provider. Patterns such as Circuit Breakers should be used on the consumer side, and patterns like pacing, throttling should be evaluated to address overloading on the provider side.

One can argue that Blocking APIs (synchronous APIs) are not highlighted here. However, any Direct API can be designed to be blocking or non-blocking calls, but that needs to be considered during design time with a strong preference to non-blocking APIs

Messaging

A Message is fundamentally an asynchronous mode of communication between two applications — it is an indirect invocation, such that two applications do not directly connect to each other. Thus, the Messaging technique decouples the consumer and provider, and removes the need of provider being available at exact same point in time as the consumer. It also addresses the scalability limitations of the provider. For example, your legacy system can take only 100 requests per second and it cannot dynamically scale, message-based integration will, at a minimum, ensure it does not crash even when it gets 200 requests (significantly higher) per second.

Strengths

• It decouples consumer and provider, removes the need of provider being available to act at exact same point in time, when the consumer is online
• Can be used to throttle or pace the requests – Consider scenario when consumer service is not able to respond as fast as request are produced.
• Provides a graceful mechanism to address provider availability and scalability limitations.
• Most message bus provides mechanisms for assured delivery

Caution

• Operational complexity is relatively higher than API as Messaging infrastructure needs to be managed and monitored. Most Messaging backbones are designed to handle very high volumes and keep latency to microseconds. However, if not configured correctly depending on the expected load, this can add significant latency.

Event

Event depicts the occurrence of business activity. Any number of interested and authorized parties (services) can subscribe to the event and take the appropriate action in response to the event. Events can occur as a continuous stream when business events continue to happen, like currency exchange rate change as an event. By taking advantage of this continuous stream, applications can react immediately and also reason about the future based on what happened in the past.

For example, most financial organizations have been leveraging events based integration, in the domain of high-value transactions so they can be used by multiple systems for analysis of fraud, anti money laundering, and compliance analysis.

Strengths

• A high degree of loose coupling between producer and consumer
• Strong enabler for parallel processing and distributed transactions – You should also explore usage of SAGA patterns for Distributed transaction management
• Easy-to-throttle or scale-out, keeping high performance, as well as managing optimum cost
• One-to-many interactions when multiple consumers are needed to process information from one provider
• Consumers of an event can continue to grow without any change on producer side or to event
• If properly designed Event analytics can be used to track and manage important Business KPIs

Weaknesses

• For distributed transactions that are managed with event-driven architectures, transaction integrity needs to be built, monitored and managed explicitly. This requires applying the concept of a global transaction identifier and correlation ID, idempotency, compensation, and addressing all possible error scenarios, for end-to-end flow of events in each transaction.
• Operational complexity to ensure Event Infrastructure is optimized for high performance is relatively high. This requires a comprehensive understanding of expected transactions, volumes and optimization parameters for event infrastructure, including careful planning and monitoring of the number of partitions for each topic, number of nodes, number of clusters, etc.

In addition to these integration modes, we must acknowledge that whole Industry is trying to move away from monolith applications into modular applications (if not microservices architecture) and distributed transaction management. With emergence of distributed transactions, need for event driven architecture and patterns for managing distributed transactions is gaining a lot of attention.

Event-driven solutions can also exploit serverless computing for zero operations, for details refer to our publication on "Emerging Patterns for Serverless Computing."

API vs. Messaging vs. Events – Comprehensive Decision Guide

Key Takeaways

• A Direct API is suitable for channel applications, external partners, compliance to Industry Integration standards (e.g. Open Banking) for partner integrations. It is also appropriate for simple transactions keeping operational complexity to a minimum. As a caution, to address availability issues, resiliency patterns such as circuit breakers and Bulkhead should be employed
• Messaging is suitable for integrating with applications that need to be decoupled, to ensure guaranteed delivery (even when the other application is not available, a message will be delivered when it is available)
• Loosely coupled nature of event-based design, with the ability to have events stored for on-demand consumptions by consumers, offers high resiliency as compared to tightly-coupled integration patterns.
• Events play a major role in managing distributed transactions with high resiliency, performance, and availability. This is a major advantage in microservices-based advanced architectures, particularly modernizing legacy monoliths.
• Although APIs would be high-performing compared to an exact scenario with event-based integration, for distributed transactions where end-to-end transaction is broken into smaller transactions, Event Orchestration can ensure parallel processing addressing performance requirement (consider this as one complex operation versus parallel processing of many simpler operations). For event-driven architectures and distributed transactions make sure you understand major design patterns.

Common Mistakes

• Do not assume that APIs are always blocking calls for consumers, they can be designed to be non-blocking
• Parallel processing for the consumer can only be achieved with Messaging/Events. With non-blocking APIs this can be achieved with APIs as well
• One integration mode is superior to others. Choose the right integration mode based on your requirements and criteria listed above. Many organizations define standards in extremes and by default bias towards one specific pattern without a decision tree.
• The interface definition is only required for API-based integration. Interface definition requires proper thinking irrespective of any integration mode, for continuous evolution, such that most changes are backward compatible, minimizing impact to consumers.
• Not exploiting some of the proven design patterns to address common issues and trying to reinvent the wheel. For example, patterns for distributed transaction patterns like SAGA, CQRS, etc.
• Ensuring sequence/ordering of events requires proper understanding of Event engines, do not assume events will always be sorted by default. Using the sequence guarantee from an Event Bus has implications on performance that must be evaluated.