You are viewing a single comment's thread.

view the rest of the comments →

karenrussell63 ago

tdlr; Outage seems intentional to me, a result of bad cloudflare config and not ddos.

Voats server remained up and running, we know because both the preview site and the API were working the entire time. Only Cloudflare's service was affected. If it happens again, an easy way to know if it's a server issue or Cloudflare is to visit CF's debug page: https://voat.co/cdn-cgi/trace and if it loads then the server is online.

A 503 error is defined as:

Server is overloaded or having maintenance issues and unable to handle the request at this time.

Cloudflare further describes it has:

Note not all web servers will serve this response, some will simply refuse or drop the connection. If a drop happens with communication from the origin to Cloudflare this will generate a 522 response.

Cloudflare will serve a 503 under the following circumstances:

  • A 503 is returned from your origin
  • I’m Under Attack mode is enabled
  • Always Online feature was triggered

On another cloudflare support page they explain it as:

A 503 Service Unavailable message with no cloudflare in the message means you need to contact your hosting provider for assistance (it generally means your host is rate limiting requests to your site).

On the other hand, a 503 Service Temporarily Unavailable error message with "cloudflare" in it means you are hitting a connection limit in a Cloudflare datacenter. Please contact Cloudflare support with the following information:

I have not personally used cloudflare but what I could find about connection limits had to do with cloudflare's rate limiting.

Rate Limiting provides the ability to configure thresholds, define responses, and gain valuable insights into specific URLs of websites, applications, or API endpoints. It adds granular HTTP/HTTPS traffic control to complement Cloudflare’s DDoS protection and Web Application Firewall (WAF) solutions. Cloudflare charges based on “good” requests i.e requests that match a rule you have created and are allowed to origin servers. This also reduces bandwidth costs by eliminating unpredictable traffic spikes or attacks

Atko states:

I isolated the issue and did what I could to bring the services back online

I didn't notice a "cloudflare" message with the 503 error, which suggest the hosting company is rate limiting requests. This could be explained by DDOS attacks or bandwidth issues, however the fact that the preview site and API were live make me doubt it had anything to do with voats hosting co. Along with the fact that Atko "did what he could to bring the services back" suggests that it was some sort of configuration issue with cloudflare. Atko wouldn't technically need access to voats server so long as he had access to voats cloudflare account.

If it was a cloudfare setting that needed to be fixed, we can narrow it down to the following:

1. A 503 is returned from your origin  
2. I’m Under Attack mode is enabled  
3. Always Online feature was triggered   
4. Rate Limiting-routing rules to allow requests through to the server  

I've already explained why I don't think it's the first one, I don't believe it to be #3 either.
This leaves #2 & #4, both would have required PuttItOut or someone else to either enable "under attack mode" or mess with the existing routing rules causing everyone to no longer have access. (cloudflare thinks all requests are malicious and refuses access.)

Obviously it could be something else that I'm just not aware of but with everything taken together and Atko's "we may be back", "did what i could", it seems like Atko was hesitant and uncertain that he fixed to issue completely. Maybe because he was surprised to see that it was a bad setting that was previously working.

I'm only speculating and not claiming to know that it was intentional, but I'll be honest it does seem intentional. I appreciate Atko troubleshooting for us, but with a big outage like this we are owed an explanation, especially since PuttItOut is MIA.

@Vindicator @Atko @PeaceSeeker @kevdude

Vindicator ago

Thanks for this contribution, KR63.