15 ms·

On Extensions, Userscripts, and Archivers

48 points by saxamaphone69 14y ago

protosstype 14y ago
We must construct additional pylons.
devfagfag 14y ago
i would think the answer is fairly obvious. 4chan needs an official API that allows for http connects to remain open. this way extension and 3rd party application developers have something to write software around that doesn't impede the ability to serve standard page GET requests to web users.
- franzus 14y ago
  HTTP is the ultimate API. Everything else is designed for censorship of 3rd party tools. (See twitter.)
rdl 14y ago
The main problem seems to be revenue. With 4chan, you could probably do a unique form of advertising -- companies pay ~$100/cpm to NOT have their products or brand featured on any 4chan pages. :)
- deleted 14y ago
  [deleted]
- ramblerman 14y ago
  Lol, In all seriousness though I'm pretty sure they could still find regular customers without too much trouble. Not to perpetuate any stereotypes but fleshlight / masculine enlargement products and so forth would be happy to advertise there.
  - phpnode 14y ago
    I'd imagine that kind of adult product would perform pretty badly on 4chan. The audience is definitely not in buying mode when they're there, and it's not like most people are going there explicitly for porn, so they don't have the kind of "motivation" that drives that kind of purchase. Probably things like cool / novelty items would do better there, things that appeal to teenagers with not too much money.
    - rdl 14y ago
      I think a lot of people just use ad blockers.
      - Paul_S 14y ago
        I've not seen an ad on the Internet since 2006. Apparently I'm evil and I'm bringing down the economy. If that means fewer jobs in advertising I count that as a win.
        trentmb 14y ago
        Adblock Plus for FireFox has an option to whitelist non-intrusive ads.
duskwuff 14y ago
Stack Overflow has solved a similar problem using WebSockets -- most of their pages will automatically update (e.g, to display the presence of edits, new answers, and added comments) immediately using a socket.
- jQueryIsAwesome 14y ago
  Yeah, totally, and some time _after_ he implements that he should micro-ban anyone making too many requests for the same pages.
nohbody 14y ago
HTTP 304, Not Modified? There's no accounting for an inconsiderate user base. This has always been a big problem for 4chan. I remember several instances where the site was effectively taken offline for days by a JavaScript virus embedded in PNG images that required the user to follow directions in the image in order to spread.
- redthrowaway 14y ago
  That wasn't /b/tards being inconsiderate, just mind-numbingly retarded. There's been a few JS worms like that, and it's almost always complete morons spreading them.
  - nohbody 14y ago
    The point being that if 4chan users are (on the whole, anyway) too stupid to not spread JS viruses, asking them to not poll the servers so much is certainly a lost cause. Getting the extension authors on board is probably the best option, because there's only a few of them, and they are not stupid. Stupid users don't have the skills to write broken extensions for themselves. The only alternative would be (as others have said) to add auto-refresh JS to every page and thus bypass the need for extensions entirely. If the number of people who currently use these extensions is small enough, however, this could just as easily drive their traffic up because now every user on the site gets the same functionality without doing anything.
    - te_chris 14y ago
      If moot charges by CPM that could solve his cash problem....
drivebyacct2 14y ago
Use WebSockets. Use Pubsubhubbub. Outside of that and reducing the minimum poll time, there aren't a lot of ways of solving this problem.
soup10 14y ago
Meh, I'm always super unimpressed when simple text based websites have trouble scaling. Everything that's highly requested should be available in memory, and it should be trivial to spit it out instantly. I'm not a scaling wizard, but I'd guess 99/100 times the reason CRUD apps have problems scaling is because they are over-engineered, and there is a tendency to solve scaling issues by adding another layer of complexity instead of optimizing the root application.
- unimpressive 14y ago
  4chan is an image board. None of the content is there long enough to be considered "Highly requested.". And like half the posts have jpeg's and png's attached. Besides, how does caching solve the "My bandwidth bills are killing my wallet!" issue?
  - soup10 14y ago
    Here's how I would scale 4chan. All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). These are all the high bandwidth items, all thats left is the text/html, which isn't that much work. I'd argue that all of the text could be served from 1 nice box if you wanted to(multiple boxes make it more complicated but not that much more). Send the post to each box, add it to a table/indexes in memory and write the post to disk and backups for recovery purposes. Then either update and cache every page the new post affects, or mark the pages dirty and update and cache them the next time they are requested. Done, all pages served out of memory super fast, what am I missing? As far as bandwidth bills, most browsers observe cache settings and won't re-download what it has already downloaded. His complaint is about getting hit too hard serving the html/text not the images.
    - true_religion 14y ago
      They already use CloudFlare. > In the past 30 days, CloudFlare has proxied 1,331,004,996 page views from 4chan
redthrowaway 14y ago
Image preloading and autogif are fantastic features, so getting rid of them would be a net negative. Instead, why not have them be controlled by some client side js, so that images/gifs are only preloaded for the next 20 or so posts from what the user is currently reading? If the user stops scrolling, stop preloading. Preload the final 5 or so images to cover skip-to-bottom, but the vast majority of the preloaded images will not be viewed, and so should not be loaded.
- saxamaphone69 14y ago
  Most users' arguments to that are they open a thread, click preload images/expand images, walk away, and so when the thread is dead/they come back, they have all the posts, and the full resolution images (not thumbnails). And the image expanding is done through userscripts/client side.
MichaelGG 14y ago
"despite having a bunch of fast drives in RAID 10 and 64 GB of RAM, our image server simply cannot keep up. " Wait, so he only has a single image server?
- gcb 14y ago
  It's not like a site that forgets it's contents in 2min or less needs more than that.
  - TazeTSchnitzel 14y ago
    It's not really 2 minutes, it's far more than that (a thread I watched lasted >1 day), but a LOT of content drops off the site every second.
gcb 14y ago
The key take away is: if you don't look at YOUR server logs, but relay on 3rd party solutions, you are missing the picture. By large. That Google analytics "guess" was abysmal. Client side logging is fine to complement server side.
dm8 14y ago
Thats interesting. And 4chan had only one guy (moot) managing all of this makes it even more impressive. By the way, some of the posters commenting on 4chan degrading morality. This is an anon forum and its just a reflection of our society! The Guardian summarized 4chan as "lunatic, juvenile... brilliant, ridiculous and alarming". Perhaps the best description for 4chan. And remember 4chan gave us LolCats, Memes, Rage faces etc. P.S. By the way, why did mods kill the other thread?
- sneak 14y ago
  > And remember 4chan gave us LolCats, Memes, Rage faces etc. 4chan gave us a voice. We are now legion. Shared culture, sure - but it showed us there was an "us" to share that culture among. Its real effects are much, much larger.

48 points by saxamaphone69 14y ago

We must construct additional pylons.

i would think the answer is fairly obvious. 4chan needs an official API that allows for http connects to remain open. this way extension and 3rd party application developers have something to write software around that doesn't impede the ability to serve standard page GET requests to web users.

franzus 14y ago
HTTP is the ultimate API. Everything else is designed for censorship of 3rd party tools. (See twitter.)

rdl 14y ago

The main problem seems to be revenue. With 4chan, you could probably do a unique form of advertising -- companies pay ~$100/cpm to NOT have their products or brand featured on any 4chan pages. :)

deleted 14y ago
[deleted]
ramblerman 14y ago
Lol, In all seriousness though I'm pretty sure they could still find regular customers without too much trouble. Not to perpetuate any stereotypes but fleshlight / masculine enlargement products and so forth would be happy to advertise there.
- phpnode 14y ago
  I'd imagine that kind of adult product would perform pretty badly on 4chan. The audience is definitely not in buying mode when they're there, and it's not like most people are going there explicitly for porn, so they don't have the kind of "motivation" that drives that kind of purchase. Probably things like cool / novelty items would do better there, things that appeal to teenagers with not too much money.
  - rdl 14y ago
    I think a lot of people just use ad blockers.
    - Paul_S 14y ago
      I've not seen an ad on the Internet since 2006. Apparently I'm evil and I'm bringing down the economy. If that means fewer jobs in advertising I count that as a win.
      - trentmb 14y ago
        Adblock Plus for FireFox has an option to whitelist non-intrusive ads.

duskwuff 14y ago

Stack Overflow has solved a similar problem using WebSockets -- most of their pages will automatically update (e.g, to display the presence of edits, new answers, and added comments) immediately using a socket.

jQueryIsAwesome 14y ago
Yeah, totally, and some time _after_ he implements that he should micro-ban anyone making too many requests for the same pages.

nohbody 14y ago

HTTP 304, Not Modified? There's no accounting for an inconsiderate user base. This has always been a big problem for 4chan. I remember several instances where the site was effectively taken offline for days by a JavaScript virus embedded in PNG images that required the user to follow directions in the image in order to spread.

redthrowaway 14y ago
That wasn't /b/tards being inconsiderate, just mind-numbingly retarded. There's been a few JS worms like that, and it's almost always complete morons spreading them.
- nohbody 14y ago
  The point being that if 4chan users are (on the whole, anyway) too stupid to not spread JS viruses, asking them to not poll the servers so much is certainly a lost cause. Getting the extension authors on board is probably the best option, because there's only a few of them, and they are not stupid. Stupid users don't have the skills to write broken extensions for themselves. The only alternative would be (as others have said) to add auto-refresh JS to every page and thus bypass the need for extensions entirely. If the number of people who currently use these extensions is small enough, however, this could just as easily drive their traffic up because now every user on the site gets the same functionality without doing anything.
  - te_chris 14y ago
    If moot charges by CPM that could solve his cash problem....

drivebyacct2 14y ago

Use WebSockets. Use Pubsubhubbub. Outside of that and reducing the minimum poll time, there aren't a lot of ways of solving this problem.

soup10 14y ago

Meh, I'm always super unimpressed when simple text based websites have trouble scaling. Everything that's highly requested should be available in memory, and it should be trivial to spit it out instantly. I'm not a scaling wizard, but I'd guess 99/100 times the reason CRUD apps have problems scaling is because they are over-engineered, and there is a tendency to solve scaling issues by adding another layer of complexity instead of optimizing the root application.

unimpressive 14y ago
4chan is an image board. None of the content is there long enough to be considered "Highly requested.". And like half the posts have jpeg's and png's attached. Besides, how does caching solve the "My bandwidth bills are killing my wallet!" issue?
- soup10 14y ago
  Here's how I would scale 4chan. All static items served from s3/cloudfront. Posted images pushed to s3/cloudfront. (Or wherever filehosting is cheapest). These are all the high bandwidth items, all thats left is the text/html, which isn't that much work. I'd argue that all of the text could be served from 1 nice box if you wanted to(multiple boxes make it more complicated but not that much more). Send the post to each box, add it to a table/indexes in memory and write the post to disk and backups for recovery purposes. Then either update and cache every page the new post affects, or mark the pages dirty and update and cache them the next time they are requested. Done, all pages served out of memory super fast, what am I missing? As far as bandwidth bills, most browsers observe cache settings and won't re-download what it has already downloaded. His complaint is about getting hit too hard serving the html/text not the images.
  - true_religion 14y ago
    They already use CloudFlare. > In the past 30 days, CloudFlare has proxied 1,331,004,996 page views from 4chan

redthrowaway 14y ago

Image preloading and autogif are fantastic features, so getting rid of them would be a net negative. Instead, why not have them be controlled by some client side js, so that images/gifs are only preloaded for the next 20 or so posts from what the user is currently reading? If the user stops scrolling, stop preloading. Preload the final 5 or so images to cover skip-to-bottom, but the vast majority of the preloaded images will not be viewed, and so should not be loaded.

saxamaphone69 14y ago
Most users' arguments to that are they open a thread, click preload images/expand images, walk away, and so when the thread is dead/they come back, they have all the posts, and the full resolution images (not thumbnails). And the image expanding is done through userscripts/client side.

MichaelGG 14y ago

"despite having a bunch of fast drives in RAID 10 and 64 GB of RAM, our image server simply cannot keep up. " Wait, so he only has a single image server?

gcb 14y ago
It's not like a site that forgets it's contents in 2min or less needs more than that.
- TazeTSchnitzel 14y ago
  It's not really 2 minutes, it's far more than that (a thread I watched lasted >1 day), but a LOT of content drops off the site every second.

gcb 14y ago

The key take away is: if you don't look at YOUR server logs, but relay on 3rd party solutions, you are missing the picture. By large. That Google analytics "guess" was abysmal. Client side logging is fine to complement server side.

dm8 14y ago

Thats interesting. And 4chan had only one guy (moot) managing all of this makes it even more impressive. By the way, some of the posters commenting on 4chan degrading morality. This is an anon forum and its just a reflection of our society! The Guardian summarized 4chan as "lunatic, juvenile... brilliant, ridiculous and alarming". Perhaps the best description for 4chan. And remember 4chan gave us LolCats, Memes, Rage faces etc. P.S. By the way, why did mods kill the other thread?

sneak 14y ago
> And remember 4chan gave us LolCats, Memes, Rage faces etc. 4chan gave us a voice. We are now legion. Shared culture, sure - but it showed us there was an "us" to share that culture among. Its real effects are much, much larger.