Imagine this: You just moved away from home for the first time to go to college. Your mom calls. "How are you? Is anything going on?" she asks.

"I'm good. Nothing exciting is going on yet," you reply. It's good to hear from her.

Twenty minutes later, you get the same call: "How are you? Is anything going on?" There's nothing going on. You get the same call every twenty minutes for the rest of your life. What a nightmare!

Thankfully, this isn't your reality, but this is the reality for many REST APIs out there, constantly getting polled with the same request over and over again. It is a communication nightmare.

As web services become more integrated and dependent on sharing data between systems, this wasteful use of REST APIs becomes increasingly problematic. Thankfully, there are modern solutions for the polling problem.

Integrations

Integration has been a major theme for the Shoutbase team lately. We’re asking questions like “how can we make the tools that we already use work even better?” and “how can we automate tasks and streamline our workflow?” Integrating systems makes work on the frontend easier, but implementing integrations on the backend can get complicated real fast. So we’re also asking questions like “how can we make meaningful integrations without having to overhaul our system, make extensive changes, and add increasingly complicated code to maintain?” In other words, “how can we make integrations simple?”

I love simple. It’s important that the end-user has a straightforward experience. And as an engineer, I want my specs and code to be simple, too. Simplicity isn’t always possible; I was reminded of that while implementing OAuth2 in our last Shoutbase release. But simple is an ideal nonetheless.

While seeking a simple integration solution, we came across Zapier. Zapier is a platform for creating custom integrations called “Zaps.” I took a deep dive into Zapier (you can read more about that here), but one thing that really resonated with me was how they handled real-time communication between web services: REST Hooks. It was simple. I liked it. We added it to Shoutbase.

In the rest of this article I want to clarify what webhooks are, what REST Hooks is, why you might want to use Rest Hooks, how the pattern works, how we implemented it in Shoutbase, and how to address potential security concerns related to the software pattern of REST Hooks.

Webhooks

Key to understanding REST Hooks is understanding webhooks. Webhooks allow web apps to communicate with each other, sending automated messages to notify others that something new has occurred. In a typical API, the client-side application calls the server-side application and consumes data. Webhooks are often referred to as “reverse APIs” because they flip this relationship. Instead of the server waiting to hear back from the client, the server-side application will call the client-side application through an end-point URL provided by the client application. That end-point URL (supplied by the client and consumed by the server-side application) is a webhook. It’s that simple.

The beauty of a webhook flow is that it occurs in real time. As soon as the server has something it wants to report, it relays the information to the client via the webhook. The client doesn’t have to routinely poll the server, checking to see if there is new data. There are other ways to achieve this communication between services. For more on that, I think O'Reilly does a great job of clarifying the difference between webhooks, websockets, and server-sent events in this article. But I’m focusing on webhooks because that’s what REST Hooks uses.

What REST Hooks Is and How It Works

According to the website, “REST Hooks itself is not a specification, it is a collection of patterns that treat webhooks like subscriptions.” The REST Hook subscriptions are created, updated, and deleted using a REST API. The pattern is easy to understand and implement and advances the power of the traditional REST API. With REST Hooks, the REST API is able to communicate with other apps in real time, via webhooks, without a complicated setup. The REST Hooks pattern has four basic requirements.

1. Mechanism to store subscriptions
2. Mechanism to modify subscriptions via API
3. List of event types & implementation of events
4. Mechanism to send hooks

The data needed to store and manage the REST Hooks is simple. Each REST Hook record needs only three pieces of information: An event name or names you want to subscribe to, a subscription owner or account relationship, and a target url (webhook) to send payloads too. The record may also include a state, active or inactive. You could also add additional information to the record in your own custom implementation, but that’s all you need for the basic REST Hooks implementation.

Next you need basic CRUD capabilities to modify the subscriptions. There’s nothing novel here. All you need are REST endpoints to create, update, delete, and read subscriptions. As new subscriptions are needed, REST Hook subscriptions can be created. As end points change, we can update them. And when subscriptions are no longer relevant, the client can choose to delete them.

The most challenging part is perhaps defining and handling events. Thankfully, most modern APIs, Shoutbase included, already have an event handling system to tap into. If you’re new to the idea of event-driven architecture, you can read more about it here. Basically, you need to define a system that contains a list of events and event listeners, and then using those two pieces, you can transmit information about events as they occur.

In the case of REST Hooks, when the event listener picks up a new event, you need logic to find any matching REST Hook subscriptions in your database. When a matching REST Hook subscription, based on event and originating user, is found, you can send your notification or payload to the webhook defined in the matching REST Hook subscription.

The Diagram below depicts the process of creating webhook subscriptions and handling events using the REST Hooks pattern. Your implementation may appear slightly different depending on the setup of your API.

Why Use REST Hooks?

Now that we understand what REST Hooks is and how it works, let’s dive in to why it's so useful. The major argument for REST Hooks is really an argument against the alternatives. First, let’s talk about REST APIs. This popular paradigm powers the web, but has some serious limitations. One of the most common ways to get “updates” on data from REST APIs is through polling, or routinely making the same request in hopes that new data might be found (just like your mom calling every 20 minutes). There are two problems with polling. First, there are a lot of wasted calls to the server. According to a case study by Zapier, using traditional REST API polling, “Over a representative time period, Zapier polled for changes 30 million times but only took action on 460,000 of those polls.” For those doing the math, that is 1.5% efficient. Over 98% of the calls were a complete waste.

Beyond the inefficiency of making fruitless calls, polling is also far from real time. Realistically, polling only occurs every 10-15 minutes. Any more than that, and the margin of wasted API calls would be even greater. Polling is more or less a hack for receiving real-ish time data. On top of that, polling can be complicated. It requires a mechanism for managing and storing previous state, and comparing new state against previous state to find changes.

REST Hooks solves those problems, while also working within the popular REST paradigm. There’s no need for polling because webhooks simply wait for new notifications and receive updates on data as it originates. The communication is basically real-time. And there is no need to maintain previous state and make comparisons. The event encapsulates the data changes, and the changes are communicated directly to the webhook.

REST Hooks in Shoutbase

As I mentioned before, I recently implemented REST Hooks to Shoutbase. The current case is limited and it hasn’t been documented for wide developer usage yet, but I’m excited to open it up in the future.

Right now, we’re using REST Hooks within our Shoutbase Zapier integration. We created a REST Hook trigger called “Prepare Report.” When a user creates a Zap using the “Prepare Report” trigger, Zapier posts a REST Hook subscription to the Shoutbase API to create the new webhook subscription, containing a webhook “target URL” from Zapier. When the Zap is turned off or deleted, Zapier makes a request to the Shoutbase API to delete the subscription. The Shoutbase API can send data to Zapier using the webhook when certain events occur.

Thanks to the existing infrastructure of Shoutbase, implementing REST Hooks was fairly straightforward. I added new REST endpoints for creating, deleting, and modifying the REST Hook subscriptions. I added a new table in our PostgreSQL database for storing the subscriptions. The Shoutbase API is written in Golang, so I added a type for the REST Hook subscriptions, which I dubbed “WebhookSubscription.”

WebhookSubscriptions in Shoutbase look like this:

type WebhookSubscription struct {
    ID        string
    OwnerID   string
    EventName string
    TargetUrl string
    CreatedAt time.Time
    UpdatedAt time.Time
}

Shoutbase has a multilayer architecture, with distinct separation between the presentation layer (or resource layer, containing the endpoints), application layer (or API layer, containing the business logic), and the data layer (connecting to the database). So I added a new API for WebhookSubscriptions that contains all the logic required to fulfill the REST endpoint requests and communicate with the database. This API contains all the security gatekeeping for subscriptions.

The final piece of the puzzle is handling the events. Shoutbase already has a powerful event system that we use for communicating with our client app via WebSockets, managing events relevant to other integrations (like Stripe), and passing asynchronous notifications and events between different parts of the Shoutbase API. I was able to create a new WebhookSubscriptionsEvents API that taps into the existing event stream and handles specific events for REST Hook subscriptions. When an event occurs that matches a REST Hook subscription in our database, we simply post the event payload to the target URL in the subscription.

REST Hooks was right for Shoutbase because we had a lot of the right ingredients already. We had a database to store REST Hook subscriptions. We had an existing REST API. And we already had a way of submitting, listening to, and handling events. It also filled the need to communicate in real time with Zapier to make more meaningful integrations. In the future, we hope to open up our REST Hooks API in Shoutbase for developers to create custom integrations with webhooks.

Security

When data is passed from one system to another, security is always a concern. Before I conclude this discussion on REST Hooks, I want to clarify issues related to security because it is important to us at Shoutbase and Olio Apps. The REST Hooks documentation describes ways to implement the pattern with security in mind. I will list them here, as well as include my own suggestions.

1. Only accept new subscriptions and update requests from authorized sources. This may go without saying, but anonymous users should not be able to access or make changes to your API’s REST Hook subscriptions. You can use your existing API authentication to verify sources. Since the REST Hook subscriptions contain a user or account relationship, only the user who owns the subscription should be able to view or make changes to it.

2. Limit the events that are subscribable. If you have a system of events already in place, you may need to decide which ones are appropriate for subscriptions. Define which events can be subscribed to via REST Hook subscriptions and prevent subscriptions to unsubscribable, private events.

3. Never send sensitive information. This also goes without saying, but payloads containing private information should never be sent via REST Hooks. The docs recommend sending “skinny payloads,” or payloads containing little to no real data, but rather, a notification that an event has occurred. The client that receives the payload can then make a request back to the API for the new data. This ensures the client still has the proper authorization to access the data.

4. Be transparent about existing subscriptions to user accounts. The documentation for REST Hooks doesn’t mention this, but I think it’s important to note. Giving the power to the users to review and understand what subscriptions exist and where their data might be flowing will help ensure account security. If a user sees a subscription they don’t recognize or no longer desire, they can delete it. This is something I hope to implement in the Shoutbase App before we widen the scope of REST Hooks. We currently have something similar for managing OAuth authorizations in the Shoutbase UI.

By carefully considering and implementing the suggestions described above, REST Hooks can enhance your API without posing a risk to your users or your service.

Conclusion

Integrating with other systems is not without its fair share share of complications. In efforts to make things appear seamless to end-users, a lot of heavy lifting has to occur behind the scenes. I prefer solutions that have three qualities: efficiency, effectiveness, and ease of implementation. In this case, REST Hooks fit the bill. We can now use REST Hooks to pass real-time data from events to webhooks and maintain those subscriptions via our REST API.