Simon Willison’s Weblog

Making a Discord bot with PHP (via) Building bots for Discord used to require a long-running process that stayed connected, but a more recent change introduced slash commands via webhooks, making it much easier to write a bot that is backed by a simple request/response HTTP endpoint. Stuart Langridge explores how to build these in PHP here, but the same pattern in Python should be quite straight-forward. # 14th January 2024, 10:56 pm

Standard Webhooks 1.0.0 (via) A loose specification for implementing webhooks, put together by a technical steering committee that includes representatives from Zapier, Twilio and more.

These recommendations look great to me. Even if you don’t follow them precisely, this document is still worth reviewing any time you consider implementing webhooks—it covers a bunch of non-obvious challenges, such as responsible retry scheduling, thin-vs-thick hook payloads, authentication, custom HTTP headers and protecting against Server side request forgery attacks. # 8th December 2023, 4:16 am

Centrifuge: a reliable system for delivering billions of events per day (via) From 2018, a write-up from Segment explaining how they solved the problem of delivering webhooks from thousands of different producers to hundreds of potentially unreliable endpoints. They started with Kafka and ended up on a custom system written in Go against RDS MySQL that was specifically tuned to their write-heavy requirements. # 6th December 2021, 1:41 am

NGINX: Authentication Based on Subrequest Result (via) TIL about this neat feature of NGINX: you can use the auth_request directive to cause NGINX to make an HTTP subrequest to a separate authentication server for each incoming HTTP request. The authentication server can see the cookies on the incoming request and tell NGINX if it should fulfill the parent request (via a 2xx status code) or if it should be denied (by returning a 401 or 403). This means you can run NGINX as an authenticating proxy in front of any HTTP application and roll your own custom authentication code as a simple webhook-recieving endpoint. # 4th October 2019, 3:36 pm

Should You Build a Webhooks API? (via) We had to solve for pretty much all of these issues when we built Eventbrite’s webhooks—this article would have saved us a lot of time! # 2nd October 2017, 5:01 am

Hookbox (via) For most web projects, I believe implementing any real-time comet features on a separate stack from the rest of the application makes sense—keep using Rails, Django or PHP for the bulk of the application logic, and offload any WebSocket or Comet requests to a separate stack built on top of something like Node.js, Twisted, EventMachine or Jetty. Hookbox is the best example of that philosophy I’ve yet seen—it’s a Comet server that makes WebHook requests back to your regular application stack to check if a user has permission to publish or subscribe to a given channel. “The key insight is that all application development with hookbox happens either in JavaScript or in the native language of the web application itself”. # 29th July 2010, 9:48 am

webhook-relay. Another of my experiments with Node.js: webhook-relay is a self-contained queue and webhook request sending agent. Your application can POST to it specifying a webhook alert to be sent off, and webhook-relay will place that request in an in-memory queue and send it on its own time, avoiding the need for your main application server to block until the outgoing request has been processed. # 19th March 2010, 10:17 am

nginx_http_push_module. More clever design with webhooks—here’s an nginx module that provides a comet endpoint URL which will hang until a back end process POSTs to another URL on the same server. This makes it much easier to build asynchronous comet apps using regular synchronous web frameworks such as Django, PHP and Rails. # 17th October 2009, 4:48 pm

Cloudvox. A brand new startup offering “API-driven phone calls” with a beautifully simple webhooks based API. # 8th October 2009, 11:31 pm

PubSubHubbub for Google Alerts. “Think of it as a search API that tells *you* when it finds new results.” # 23rd September 2009, 9:30 pm

cloud-crowd. New parallel processing worker/job queue system with a strikingly elegant architecture. The central server is an HTTP server that manages job requests, which are farmed out to a number of node HTTP servers which fork off worker processes to do the work. All communication is webhook-style JSON, and the servers are implemented in Sinatra and Thin using a tiny amount of code. The web-based monitoring interface is simply beautiful, using canvas to display graphs showing the system’s overall activity. # 21st September 2009, 11:09 pm

PostBin. Handy debugging tool for webhooks—create a TinyURL-style URL, then see a log of any POST requests made to that address. # 21st September 2009, 11:03 pm

RSSCloud Vs. PubSubHubbub: Why The Fat Pings Win. A PubSubHubbub advocate explains the differences between the two proposals: most importantly, PubSubHubbub includes the actual new content with the “fat ping” whereas RSSCloud just notifies you that you should poll the RSS feed, leading to a potential thundering herd. I’m still hoping one of those specs will detail a way in which they can be used for scalable regular WebHook-style notifications without any feed infrastructure at all. # 10th September 2009, 3:49 pm

Scriptlets—Quick web scripts (via) From the prolific Jeff Lindsay, a pastebin-style tool for short server-side scripts written in Python, JavaScript or PHP that executes them within a Google App Engine powered sandbox. The Java code that implements the service is available on GitHub. # 13th August 2009, 1:51 pm

The Pushbutton Web: Realtime Becomes Real. Anil Dash is excited by the potential for PubSubHubBub and Webhooks to make near-real-time scalable event publishing accessible to regular web developers. So am I. # 24th July 2009, 6:30 pm

Webhooks behind the firewall with Reverse HTTP. Hookout is a Ruby / rack adapter that lets you serve a web application from behind a firewall, by binding to a Reverse HTTP proxy running on the internet (such as the free one provided by reversehttp.net). Useful for far more than just webhooks, this means you can easily expose any Ruby web service to the outside world. An implementation of this as a general purpose proxy server would make it useful for applications written in any language. # 22nd July 2009, 1:46 pm

PubSub-over-Webhooks with RabbitHub. RabbitMQ, the Erlang-powered AMQP message queue, is growing an HTTP interface based on webhooks and PubSubHubBub. # 1st July 2009, 8:22 pm

pubsubhubbub. From Brad Fitzpatrick, a simple but clever way of using web hooks (HTTP callbacks) to inform subscribers that an Atom feed has updated in almost real-time—solving the constant polling problem and making it easier for small sites to offer publish-subscribe APIs. Any Atom feed can delegate subscriber updates to a “hub” server. An example hub server implementation is provided running on App Engine. # 20th April 2009, 6:49 pm

Web Hooks and the Programmable World of Tomorrow. Tour de force presentation on Web Hooks by Jeff Lindsay. Tons of really good ideas—provided your application isn’t Flickr sized, there’s a good chance you could implement web hooks pretty cheaply and unleash a huge flurry of creativity from your users. GitHub makes a great case study here. # 16th February 2009, 9 pm

Post-Commit Web Hooks for Google Code Project Hosting (via) I really, really like web hooks (which I’ve been calling “callback APIs”, but it looks like “web hooks” is the term that’s sticking). I’m interested in their scaling challenges—I’ve heard XMPP advocates argue that a web hook style model simply won’t scale for really large sites. # 4th February 2009, 10:22 am