6 ms·

yep, i work for a major university and our canvas instance is down. this is really, really bad. edit: here's the list of impacted universities (unsure if the

yep, i work for a major university and our canvas instance is down. this is really, really bad. edit: here's the list of impacted universities (unsure if they all have their canvas instances offline, but i'd be surprised if not): http://91.215.85.103/pay_or_leak/instructure_affected_schools_list.txt http://91.215.85.103/pay_or_leak/instructure_affected_school...

mrsvanwinkle 4mo ago
[dead]
12_throw_away 4mo ago
tbh this has me wondering if canvas "instances" are actually as isolated and segregated from each other as they're supposed to be.
- wky 4mo ago
  It's possible that Instructure's servers got compromised: dig canvas.ucdavis.edu [...] ;; ANSWER SECTION: canvas.ucdavis.edu. 1974 IN CNAME ucdavis-vanity.instructure.com. ucdavis-vanity.instructure.com. 60 IN A 18.173.121.125 ucdavis-vanity.instructure.com. 60 IN A 18.173.121.103 ucdavis-vanity.instructure.com. 60 IN A 18.173.121.15 ucdavis-vanity.instructure.com. 60 IN A 18.173.121.18 dig canvas.duke.edu ;; ANSWER SECTION: canvas.duke.edu. 300 IN CNAME duke-vanity.instructure.com. duke-vanity.instructure.com. 60 IN A 18.173.121.125 duke-vanity.instructure.com. 60 IN A 18.173.121.18 duke-vanity.instructure.com. 60 IN A 18.173.121.103 duke-vanity.instructure.com. 60 IN A 18.173.121.15
  - mrsvanwinkle 4mo ago
    that's what the screenshot says. They rooted Instructure servers.
- javawizard 4mo ago
  Define "as they're supposed to be". Back when I worked for Instructure ~10 years ago, Canvas was effectively a single, giant, monolithic multitenant app with one instance backed by several thousand app servers and ~100 separate Postgres database clusters that any app server could talk to. Schools were grouped onto pools of app severs and Postgres database clusters more or less according to locality and cluster availability. I want to say a handful of the largest schools got their own clusters, but I'm not certain, and at any rate their clusters could certainly all talk to each other. It was actually kind of neat from a technical perspective: any Rails model across the entire Canvas world could have a "foreign key" pointing to any other Rails model anywhere else. Among other things, this allowed for users who could administer multiple Canvas organizations, even if those organizations resided on different Postgres clusters. https://github.com/instructure/switchman https://github.com/instructure/switchman is their gem that made that all work. (I put "foreign key" in quotes because the whole thing was implemented in software, not with actual database FKs, for obvious reasons.) --- Of course, the massive downside to that sort of thing is that if you manage to pop one Canvas app server, you have the keys to the kingdom. I wonder if they'll sharpen the edges between clusters in response to this... --- (Disclaimer: I left Instructure back in 2017; much could have changed since then, and my memory could be faulty about the specifics. Caveat emptor.)
- SamuelAdams 4mo ago
  It depends on what you pay for. If you need FedRamp or IL4+ compliance you are likely on dedicated infrastructure. Everyone else uses multi tenancy.
starkrights 4mo ago
The source txtfile has since either been dos'd or deleted (at least it was when I tried to access) Someone dumped the content into a google doc on reddit[1] if anyone's interested. [1]: https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sTsluR_0XQ81Z0v_Lrd0g/edit?usp=sharing https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sT...
- rigrassm 4mo ago
  > The source txtfile has since either been dos'd or deleted (at least it was when I tried to access) > Someone dumped the content into a google doc on reddit[1] if anyone's interested. > [1]: https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sT https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sT... Thanks for linking this. Ended up finding my kids school district on the list unfortunately.
Cider9986 4mo ago
Here's an archive https://archive.is/eB2hE https://archive.is/eB2hE
GaryBluto 4mo ago
https://web.archive.org/web/20260507042014fw_/http://91.215.85.103/pay_or_leak/instructure_affected_schools_list.txt https://web.archive.org/web/20260507042014fw_/http://91.215....

mrsvanwinkle 4mo ago

[dead]

12_throw_away 4mo ago

tbh this has me wondering if canvas "instances" are actually as isolated and segregated from each other as they're supposed to be.

wky 4mo ago
It's possible that Instructure's servers got compromised: dig canvas.ucdavis.edu [...] ;; ANSWER SECTION: canvas.ucdavis.edu. 1974 IN CNAME ucdavis-vanity.instructure.com. ucdavis-vanity.instructure.com. 60 IN A 18.173.121.125 ucdavis-vanity.instructure.com. 60 IN A 18.173.121.103 ucdavis-vanity.instructure.com. 60 IN A 18.173.121.15 ucdavis-vanity.instructure.com. 60 IN A 18.173.121.18 dig canvas.duke.edu ;; ANSWER SECTION: canvas.duke.edu. 300 IN CNAME duke-vanity.instructure.com. duke-vanity.instructure.com. 60 IN A 18.173.121.125 duke-vanity.instructure.com. 60 IN A 18.173.121.18 duke-vanity.instructure.com. 60 IN A 18.173.121.103 duke-vanity.instructure.com. 60 IN A 18.173.121.15
- mrsvanwinkle 4mo ago
  that's what the screenshot says. They rooted Instructure servers.
javawizard 4mo ago
Define "as they're supposed to be". Back when I worked for Instructure ~10 years ago, Canvas was effectively a single, giant, monolithic multitenant app with one instance backed by several thousand app servers and ~100 separate Postgres database clusters that any app server could talk to. Schools were grouped onto pools of app severs and Postgres database clusters more or less according to locality and cluster availability. I want to say a handful of the largest schools got their own clusters, but I'm not certain, and at any rate their clusters could certainly all talk to each other. It was actually kind of neat from a technical perspective: any Rails model across the entire Canvas world could have a "foreign key" pointing to any other Rails model anywhere else. Among other things, this allowed for users who could administer multiple Canvas organizations, even if those organizations resided on different Postgres clusters. https://github.com/instructure/switchman https://github.com/instructure/switchman is their gem that made that all work. (I put "foreign key" in quotes because the whole thing was implemented in software, not with actual database FKs, for obvious reasons.) --- Of course, the massive downside to that sort of thing is that if you manage to pop one Canvas app server, you have the keys to the kingdom. I wonder if they'll sharpen the edges between clusters in response to this... --- (Disclaimer: I left Instructure back in 2017; much could have changed since then, and my memory could be faulty about the specifics. Caveat emptor.)
SamuelAdams 4mo ago
It depends on what you pay for. If you need FedRamp or IL4+ compliance you are likely on dedicated infrastructure. Everyone else uses multi tenancy.

starkrights 4mo ago

The source txtfile has since either been dos'd or deleted (at least it was when I tried to access) Someone dumped the content into a google doc on reddit[1] if anyone's interested. [1]: https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sTsluR_0XQ81Z0v_Lrd0g/edit?usp=sharing https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sT...

rigrassm 4mo ago
> The source txtfile has since either been dos'd or deleted (at least it was when I tried to access) > Someone dumped the content into a google doc on reddit[1] if anyone's interested. > [1]: https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sT https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sT... Thanks for linking this. Ended up finding my kids school district on the list unfortunately.

Cider9986 4mo ago

Here's an archive https://archive.is/eB2hE https://archive.is/eB2hE

GaryBluto 4mo ago

https://web.archive.org/web/20260507042014fw_/http://91.215.85.103/pay_or_leak/instructure_affected_schools_list.txt https://web.archive.org/web/20260507042014fw_/http://91.215....