submitted 5.8 years ago by PuttItOut

Edit 8

We just were made aware of a security issue on Voat concerning the saving of anonymous content. As a security measure I took the website down immediately in order to patch it. Patch is now in place and the security issue has been resolved.

Edit 7

I have a bottleneck detection routine running currently and when it thinks we are getting backed up, it will flip on the maintenance page, take a debug dump of the necessary processes, wait until the existing queue is reduced to expected levels, and then reenable the UI/API traffic. Based on what I've noticed this process never runs over 30 seconds. If you experience this, just wait for one goat minute, then click back in. Told you it would be fun!

Edit 6

I will be pushing up new versions of the website as we build them that attempt to address the issues we are experiencing. When a new version is pushed, the site will go offline temporarily and you will see ugly errors and/or slow or hanging responses. Isn't this fun? I'm all smiles over here. /s

Edit 5

The thought crossed my mind about how I'd explain the issue Voat is experiencing and I came up with "Everything is fine until it isn't." I laughed.

On a technical note, Voat is experiencing a thread starvation issue (randomly, not based load, likely old code somewhere). When this happens, queues build up, the process doesn't recover, and requires manual intervention. Also known as "baby sitting."

I'm going to keep my eye on things as best I can. Until I get it fixed, it's going to be fun.

Other than this rather annoying situation, the update went smoother than expected, so at least I have that going for me.

Edit 4

With software you often just trade one problem for another. Old code wouldn't use enough resources, this code uses too much. It is funny.

Making a few changes, let's see if we can correct this issue.

Edit 3

Just noticed very high memory consumption. Had to restart the service. Obviously we have an issue somewhere.

Edit 2

I've noticed we have some hanging (slow responses) in a few areas: Messages inbox, comment replies, loading more comments. Seems to be intermittent but we still have some work to do on this version.

Edit

Main site has now been updated. Had to purge the entire cache so will take a while to rebuild it. I will continue monitoring the site and hopefully things continue working as they should.

Just a quick note that we will be incrementally updating Voat over the next few days (don't forget to know your password). We are doing this piecemeal, one area at a time because of the scope of this update. If we don’t experience any major issues we will continue moving forward.

Voat may very well experience hiccups or downtime during this transition.

Wish us luck and see you Goats on the other side!

Thank you to every Goat who has helped test on the preview site. Couldn't do it without you guys. Much love.

471 comments

You are viewing a single comment's thread.

view the rest of the comments →

FridayJones 5.8 years ago

I notice that any comments more than three levels deep do not have any numbers indicating how many upvotes or downvotes they have received

link

PuttItOut 5.8 years ago

There is a reason for this madness.

link

FridayJones 5.8 years ago

Occam's Razor would lead me to the conclusion that the reason for the madness is that's gone mad, but something about the way you said it makes me think you might have a less tautological reason behind that. Do you care to share the reason here?

link

PuttItOut 5.8 years ago

I like to be abstract so I can see who puts in the basic effort required for critical thinking.

Hints: The reality with this change is that it isn't a change at all. Once you figure out why then this will make sense. Nothing of value is hidden.

link

FridayJones 5.8 years ago

I think I have fingered out a reasonable guess as to the answer: The deeper-nested comments are more recent, and the new code conceals the vote counts for a specified period of time.

link

speedisavirus 5.7 years ago

I would wager money it's due to how they manage votes. I remember it not being particularly scale friendly when it was last explained to me. Reddit does fuzzy votes that are eventually somewhat right so they are close but mostly not accurate which is fine when scaling to large vote counts. Voat does literal voat counts so updating the votes at scale can create concurrency load at scale as well as load retrieving data

link

FridayJones 5.7 years ago

That kind of sounds like a bass ackwards way of keeping track of and displaying vote tallies then. Instead of keeping the vote counts in tables indexed to user accounts, why not index the vote count tables to the comment numbers? That way you don't get to peer through a user's history as an Admin to see what their voting history is, but you get an accurate and up to date tally of all up and down votes retrieved at the same time as the comment itself, without any internal bandwidth or processing.

Or just get rid of voting entirely, and let the comments stand for themselves. But then the low value commenters might drown out the high value commenters.

I'm glad I don't have to make these decisions.

link

speedisavirus 5.7 years ago

They can do this sort of scheme but you really need a queuing system in the middle to process it over time when things are just happening too fast. I could be getting parts of the description wrong but having worked on a m that measured traffic in millions of interactions per day I know how we made it work. Not cheap but it worked.

link

PuttItOut 5.7 years ago

Close, but the variable isn't time.

link

FridayJones 5.7 years ago

Are you assuming that the deeper nested a comment is, the less useful the voting mechanism is at weighting it? That seems like a reasonable assumption. How come the arrows are still showing on those comments?