7 ms·

Automatically sending Webmentions from a static website

75 points by chrislo 6y ago

dane-pgp 6y ago
It still amazes me that people have managed to create a viable system of decentralized comments across independent, statically-generated websites. Perhaps I should heed the (humorous) warning by Ben Werdmuller here, though: https://aaronparecki.com/2013/05/21/4/xkcd#mentions https://aaronparecki.com/2013/05/21/4/xkcd#mentions
- dane-pgp 6y ago
  The page I linked to was mostly intended as a nice example of Webmentions being used in the wild, but I also thought it made a nice point about the risks of centralized systems versus decentralized ones. Examples of the latter would be spam (as mentioned elsewhere above), and coordination problems when trying to upgrade the standard in potentially backwards-incompatible ways. Hopefully the continued adoption of Webmention will prove that these risks are manageable.
xwdv 6y ago
I’ve never even heard of Webmentions. Should I have?
- alexmingoia 6y ago
  Webmentions are the modern standard that evolved out of trackbacks and pingbacks. If you’ve written a blog post that references or links to another blog post, Webmention is the standard way to notify them. An example use case for Webmentions is blog comments. A blog may publish a post which is a reply to another blog, then send a Webmention to notify them. The author of the blog post being replied to could choose to display the comment below their post.
leephillips 6y ago
I like the idea, but I wonder if I should bother: in other words, are people actually using this? Is this the new pingback, because that seemed like a good idea, too, but it never got off the ground.
- simonw 6y ago
  I helped create pingback [https://simonwillison.net/tags/pingback/ https://simonwillison.net/tags/pingback/], which eventually made it into WordPress and hence achieved pretty widespread distribution. My concern here is spam. With hindsight both Pingback and Trackbacks (which Pingback was a response to) essentially became APIs fit spammers to add junk to your site. Can webmention avoid the sane fate?
  - necrotic_comp 6y ago
    I would think that because most of the sites are static, they are going to be doing some moderation before re-rendering and posting the linkbacks. The README says that one of the steps is verification [https://github.com/converspace/webmention/blob/master/README.md#verification https://github.com/converspace/webmention/blob/master/README...], but it's a little disappointing that it's essentially an "exercise left for the reader".
    - avian 6y ago
      I think Webmention is viable at the moment just because of the low adoption across the web. Just like with pingbacks and trackbacks, any kind of manual moderation becomes infeasible when valid posts get swarmed under thousands of automated spam posts. The typical verification step (check if the originating site contains a valid link) is trivial to work around. One attempt to address that is Vouch [1], but I believe it's largely untested in the wild. [1] https://indieweb.org/Vouch https://indieweb.org/Vouch
    - simonw 6y ago
      "The receiver SHOULD perform a HTTP GET request on source to confirm that it actually links to target" We did that in pingback too. It turned out to be trivial for spammers to circumvent.
      - necrotic_comp 6y ago
        That's a bummer. Did you find out any other more robust ways to filter out spam ?
ngcc_hk 6y ago
I am more interest in the action flow then the webmention
superkuh 6y ago
I implemented webmentions for my static site back around the end of 2019. My solution for receiving on my static site was to just log POSTs and review them with my eyes. Then if someone has actually sent a webmention (as opposed to pingback spam) I just use curl to respond. There's no need to have anything automated at all. ref: "A static, manual system for receiving webmentions (and pingback) with nginx" http://superkuh.com/blog/2020-01-10-1.html http://superkuh.com/blog/2020-01-10-1.html
- toomim 6y ago
  That's what OP originally did as well.
- Hedja 6y ago
  Just to correct your blog post, the Webmentions spec says in this scenario where you're just logging, you "MUST" return a 202 Accepted. I went through a similar phase (logsink with 204) which I corrected recently. https://www.w3.org/TR/2017/REC-webmention-20170112/#h-receiving-webmentions https://www.w3.org/TR/2017/REC-webmention-20170112/#h-receiv... I was wondering if I should've sent this comment as a webmention instead but I guess old habits die hard.
abdullahkhalids 6y ago
Why do you need an always on service for this? Just send the webmentions whenever you compile your static website - that's the only time content actually changes. On Pelican, you can do it using this plugin https://chezsoi.org/lucas/blog/pelican-pingback-and-webmentions.html https://chezsoi.org/lucas/blog/pelican-pingback-and-webmenti...
- Vinnl 6y ago
  Make sure to only do it after you've deployed it - part of the WebMention process is that the mentioned page actually checks the page that sends the mention for the given link. (Also annoying if the link is added client-side.)
  - abdullahkhalids 6y ago
    To be fair [emphasis mine] > Upon receipt of a POST request containing the source and target parameters, the receiver SHOULD verify the parameters (see Request Verification below) and then SHOULD queue and process the request asynchronously, to prevent DoS attacks. https://www.w3.org/TR/2017/REC-webmention-20170112/ https://www.w3.org/TR/2017/REC-webmention-20170112/ Though, probably, this won't happen in practice. So better to be careful.
    - Vinnl 6y ago
      Even if done asynchronously, if the link is not present the moment you send the WebMention, it can still happen between sending it and deploying it. But you're right, if the two are close together and a receiver is actually processing them asynchronously, it'll likely go well.
makeboss 6y ago
This is a great idea, wonder how I've missed this.
qubyte 6y ago
I also use Remy Sharp’s code, but as a library. I have netlify call this glitch [1] I wrote on successful deploy. It compares the new sitemap.txt with one it has cached and scans new URLs which match a pattern for mentions to dispatch. 1: https://glitch.com/edit/#!/lean-send-webmentions https://glitch.com/edit/#!/lean-send-webmentions

75 points by chrislo 6y ago

It still amazes me that people have managed to create a viable system of decentralized comments across independent, statically-generated websites. Perhaps I should heed the (humorous) warning by Ben Werdmuller here, though: https://aaronparecki.com/2013/05/21/4/xkcd#mentions https://aaronparecki.com/2013/05/21/4/xkcd#mentions

dane-pgp 6y ago
The page I linked to was mostly intended as a nice example of Webmentions being used in the wild, but I also thought it made a nice point about the risks of centralized systems versus decentralized ones. Examples of the latter would be spam (as mentioned elsewhere above), and coordination problems when trying to upgrade the standard in potentially backwards-incompatible ways. Hopefully the continued adoption of Webmention will prove that these risks are manageable.

xwdv 6y ago

I’ve never even heard of Webmentions. Should I have?

alexmingoia 6y ago
Webmentions are the modern standard that evolved out of trackbacks and pingbacks. If you’ve written a blog post that references or links to another blog post, Webmention is the standard way to notify them. An example use case for Webmentions is blog comments. A blog may publish a post which is a reply to another blog, then send a Webmention to notify them. The author of the blog post being replied to could choose to display the comment below their post.

leephillips 6y ago

I like the idea, but I wonder if I should bother: in other words, are people actually using this? Is this the new pingback, because that seemed like a good idea, too, but it never got off the ground.

simonw 6y ago
I helped create pingback [https://simonwillison.net/tags/pingback/ https://simonwillison.net/tags/pingback/], which eventually made it into WordPress and hence achieved pretty widespread distribution. My concern here is spam. With hindsight both Pingback and Trackbacks (which Pingback was a response to) essentially became APIs fit spammers to add junk to your site. Can webmention avoid the sane fate?
- necrotic_comp 6y ago
  I would think that because most of the sites are static, they are going to be doing some moderation before re-rendering and posting the linkbacks. The README says that one of the steps is verification [https://github.com/converspace/webmention/blob/master/README.md#verification https://github.com/converspace/webmention/blob/master/README...], but it's a little disappointing that it's essentially an "exercise left for the reader".
  - avian 6y ago
    I think Webmention is viable at the moment just because of the low adoption across the web. Just like with pingbacks and trackbacks, any kind of manual moderation becomes infeasible when valid posts get swarmed under thousands of automated spam posts. The typical verification step (check if the originating site contains a valid link) is trivial to work around. One attempt to address that is Vouch [1], but I believe it's largely untested in the wild. [1] https://indieweb.org/Vouch https://indieweb.org/Vouch
  - simonw 6y ago
    "The receiver SHOULD perform a HTTP GET request on source to confirm that it actually links to target" We did that in pingback too. It turned out to be trivial for spammers to circumvent.
    - necrotic_comp 6y ago
      That's a bummer. Did you find out any other more robust ways to filter out spam ?

ngcc_hk 6y ago

I am more interest in the action flow then the webmention

superkuh 6y ago

I implemented webmentions for my static site back around the end of 2019. My solution for receiving on my static site was to just log POSTs and review them with my eyes. Then if someone has actually sent a webmention (as opposed to pingback spam) I just use curl to respond. There's no need to have anything automated at all. ref: "A static, manual system for receiving webmentions (and pingback) with nginx" http://superkuh.com/blog/2020-01-10-1.html http://superkuh.com/blog/2020-01-10-1.html

toomim 6y ago
That's what OP originally did as well.
Hedja 6y ago
Just to correct your blog post, the Webmentions spec says in this scenario where you're just logging, you "MUST" return a 202 Accepted. I went through a similar phase (logsink with 204) which I corrected recently. https://www.w3.org/TR/2017/REC-webmention-20170112/#h-receiving-webmentions https://www.w3.org/TR/2017/REC-webmention-20170112/#h-receiv... I was wondering if I should've sent this comment as a webmention instead but I guess old habits die hard.

abdullahkhalids 6y ago

Why do you need an always on service for this? Just send the webmentions whenever you compile your static website - that's the only time content actually changes. On Pelican, you can do it using this plugin https://chezsoi.org/lucas/blog/pelican-pingback-and-webmentions.html https://chezsoi.org/lucas/blog/pelican-pingback-and-webmenti...

Vinnl 6y ago
Make sure to only do it after you've deployed it - part of the WebMention process is that the mentioned page actually checks the page that sends the mention for the given link. (Also annoying if the link is added client-side.)
- abdullahkhalids 6y ago
  To be fair [emphasis mine] > Upon receipt of a POST request containing the source and target parameters, the receiver SHOULD verify the parameters (see Request Verification below) and then SHOULD queue and process the request asynchronously, to prevent DoS attacks. https://www.w3.org/TR/2017/REC-webmention-20170112/ https://www.w3.org/TR/2017/REC-webmention-20170112/ Though, probably, this won't happen in practice. So better to be careful.
  - Vinnl 6y ago
    Even if done asynchronously, if the link is not present the moment you send the WebMention, it can still happen between sending it and deploying it. But you're right, if the two are close together and a receiver is actually processing them asynchronously, it'll likely go well.

makeboss 6y ago

This is a great idea, wonder how I've missed this.

qubyte 6y ago

I also use Remy Sharp’s code, but as a library. I have netlify call this glitch [1] I wrote on successful deploy. It compares the new sitemap.txt with one it has cached and scans new URLs which match a pattern for mentions to dispatch. 1: https://glitch.com/edit/#!/lean-send-webmentions https://glitch.com/edit/#!/lean-send-webmentions