Slow site

**Chille** · 12-18-2011

Ok to stop the conspiracys im gonna use my power of thread killing and just post here now

you owe me one :3

**darky89** · 12-18-2011

Originally Posted by Chille

Ok to stop the conspiracys im gonna use my power of thread killing and just post here now

you owe me one :3

I'm posting just to spite you.

**Mr.Bishop** · 12-18-2011

OK, seriously... you didn't read what I have been posting.

Originally Posted by Bligmerk

This situation with this site slowing down to almost crashing intermittently has been going on for years, at least 4 that I can remember. I remember some of the early ones and they were explainable. There was the launch of the PS3 and this site was absolutely jammed with people trying to get pre-orders, where it was best to camp out, which stores had stock left. Then during E3, it would jam up. It would again jam up for TGS.

The difference here was those times could be associated with specific events. Now, with the slowdowns this past year, there are multiple reasons being given by the people running this place and they have no consistency and don't make rational sense if all the explanations are viewed at the same time. The slowdown that started this thread sure looked like a DDoS because the usual number of active connections was going above 5000 when it had normally settled down to well below 1000, usually not getting over 500.

The explanation that it was a stress test doesn't make any sense. Anybody that has hung around here long enough knows this site can't handle more than about 2000 to 3000 active connections. This has been known and obvious for many years, why keep doing stress tests to prove the same thing over and over without actually having changed the infrastructure?

There is way, way, way more traffic than just in the forums first of all. Remember that - PSU's homepage and news program is what generates about 2/3 of the traffic to the site, so the amount of connections to the forums is irrelevant compared to what we have to handle as a whole. Looking at the forum stats to start is leading you completely astray.

Second on this point is that the issue is Bytemark hosting. They're a cool little botique hosting company which PSU has been on from the beginning. They're not big guys, though they try really hard and are really nice to us. However, there is issues with their hardware. We've been optimizing in and out the servers that they host for us... but there is something wrong with the load balancer. TBH we just can not for the life of us figure it out. So your assessment that the site crashes under a certain amount of load is correct... its on physical boxes at a smaller host. So past a certain point, blammo it chokes out. When we change the infrastructure this will completely mitigate this issue. We're going to move to a cloud setup with elastic hosting so we can handle any spikes without breaking a sweat. It will be awesome, and this will finally once and for all solve the issue that has been plaguing the site since its inception.

Originally Posted by Bligmerk

Further up this thread, Mr. Bishop said it *definitely* is not bots, they are absolutely sure. In fact, the statement at the time was that a "fortuitous set of circumstances" was resulting in the higher traffic. And at the time, the registered user numbers were clicking steadily upward. Suddenly, the active numbers dropped back to the new normal, and the registered user numbers started clicking back down, obviously bogus accounts being deactivated.

Our content is being posted on heavily trafficked aggregation sites in prominent spots. See example in earlier post for an exact source of the traffic. This is happening at scheduled times because it is part of a syndication partnership that the owners of PSU have developed with their colleagues at these websites.

This is likely causing semi-interested users to quickly register, and then for whatever reason they are losing interest and leaving. That is common for this type of an arrangement, you can grab people's attention quickly with a cool piece of PS3-related content, but they don't necessarily stay around for the community. Totally normal stats is that something like 90-95% of the traffic to a community website just reads a bit of content and leaves.

Originally Posted by Bligmerk

Now, we have even another explanation that "redirects" are occurring from some ambiguous site periodically at night. Now wait a second. That is how DDoS works, using redirects to pummel a site into crashing. And a DDoS isn't necessarily a full scale DDoS, it may be a stress test on its own to find out how many active connections it takes to bring down the site. The whole point is to quantify and catalog vulnerabilities, to launch a full attack when the time comes.

Redirects is an improper terminology. Sorry, Cuguy and I had been talking and I mentioned briefly what my suspicion was for the traffic surges (after I had started my investigation) and he used the word re-directs. It is not re-directs, it is content syndication partnerships. Again, see above example, and previous lengthy explanations.

Also, that is absolutely NOT how a DDoS attack works. You are completely, totally, utterly incorrect. The KEY to a DDoS attack is that the traffic comes from many sources.

DISTRIBUTED
DENIAL
of
SERVICE

Distributed is what makes a DDoS attack successful. Computers or servers which have a tiny piece of malware on them make them part of a Bot net. This bot net is spread as widely across the globe as you possibly can manage, and then you assault a site from everywhere. If a DDoS attack came from one site re-directing traffic, you'd block that IP and it would be done. We could do that in about 9 seconds of work from the forum admin panel.

Stress testing works in much the same way. You can get services which do this, and you do it to see the load on the server. We'll do this after the new setup is in place.

Stress testing to find vulnerabilities.... well you're mixing too many ideas here. DDoS isn't elegant... you have a big botnet, you point it at a site and it either goes down or not. If you want it to go down, you make a bigger botnet. You don't get any information from secretly peppering a site with some re-directs... you just find out that the site needs a new fucking web host... not one that has load balancers which choke under load.

What i think you're REALLY saying is that there is someone who targets PSU.com as a hacker. THIS is true... and we have caught on. I will not hide the fact that there is someone that targets this place, whether an old member of the community or a current one. Someone out there has a hard-on for hacking PSU and making our lives miserable. This has caused us to have to implement some extremely strict security measures, and there are more in place every day as we fight the battle with this revenge hacker. We've caught plenty of exploratory missions this person has tried to take into the code, and have all but neutralized his efforts. Picking away at weak parts of the code is definitely an excellent hacking technique. This is very possible and is super annoying to deal with. We're good at that too though, so I'm absolutely confident that now that we've really audited the code we are in an extremely well defended position.

DDoSing isn't hacking. Hackers may create botnets to DDoS with by hacking... but launching a DDoS attack is not hacking. Its a very primitive type of attack, and actually almost impossible to defend against. Usually they don't launch it against a site, they launch it against the whole host and just bring down your service provider.

Originally Posted by Bligmerk

It doesn't matter to me what the real reason is, just that every time it happens, it is a disincentive for active interest in the site.

This is absolutely true, and for this I'm extremely sorry. We thought that we would be able to fix the hosting setup at Bytemark, but it is just past the capacity that they can offer us. The server move we're about to do will end this issue once and for all - and I PROMISE you... we will not have slowdowns or crashes due to traffic load.

Please, post your theories if you have 'em. I'll spend all day explaining each and every one of them why they aren't true!

**darky89** · 12-18-2011

Maybe Chuck Norris has it in for us.

**Chille** · 12-18-2011

Originally Posted by darky89

Maybe Chuck Norris has it in for us.

Chuck Noriss never attacks, Chuck Noriss always kills.

**Bligmerk** · 12-19-2011

Originally Posted by Mr.Bishop

OK, seriously... you didn't read what I have been posting.

all day explaining each and every one of them why they aren't true!

Yes, I have. All those I listed were reasons you and/or Cuguy have given, except for the DDoS. And you end with one of the worst wrap ups because you are saying you have a negative answer for everything else and won't confirm what it is, you are saying you know what it is not but you don't know exactly what it is. I know what DDoS is. The thing with these zombie bots are they are like zombies, not quite without life like a bot but not really alive either. These zombie bots are going to be on multiple ISP's, multiple MACS, multiple accounts --in conjunction with multiple redirects-- coming from everywhere to one location. There is no way on your end to find out where the web center is.

Anyway, it is just my comment on this condition, and you say "with PSU from the beginning", do you mean PS3Land also? The odd thing about these high traffic times are they seem to be happening around the same time, at night here, so it doesn't affect me much. Also, they tend to gradually increase in periodicity until it gets really bad and members start bringing it up here. Then something gets cleared out for awhile and the site starts responding more smoothly, at least for awhile.

You are losing members from these essential outages. The registered is now down to 1800 from the 2400 it climbed to during the outages, which started this thread and another contradiction in itself. Registered user numbers increased at the exact same time a lot of regular members couldn't even connect. These are just observations and it is your board.

**Mr.Bishop** · 12-19-2011

Originally Posted by Bligmerk

Yes, I have. All those I listed were reasons you and/or Cuguy have given, except for the DDoS. And you end with one of the worst wrap ups because you are saying you have a negative answer for everything else and won't confirm what it is, you are saying you know what it is not but you don't know exactly what it is. I know what DDoS is. The thing with these zombie bots are they are like zombies, not quite without life like a bot but not really alive either. These zombie bots are going to be on multiple ISP's, multiple MACS, multiple accounts --in conjunction with multiple redirects-- coming from everywhere to one location. There is no way on your end to find out where the web center is.

Anyway, it is just my comment on this condition, and you say "with PSU from the beginning", do you mean PS3Land also? The odd thing about these high traffic times are they seem to be happening around the same time, at night here, so it doesn't affect me much. Also, they tend to gradually increase in periodicity until it gets really bad and members start bringing it up here. Then something gets cleared out for awhile and the site starts responding more smoothly, at least for awhile.

You are losing members from these essential outages. The registered is now down to 1800 from the 2400 it climbed to during the outages, which started this thread and another contradiction in itself. Registered user numbers increased at the exact same time a lot of regular members couldn't even connect. These are just observations and it is your board.

I'm sorry for not mentioning it, but what I have said repeatedly is both myself and my development team have dealt with many true DDoS Attacks. I don't know why you don't think I"m confirming where it is... I did many times. We have content syndication partnerships around the web. I have compiled a list, checked them, confirmed traffic numbers... I've spent literally like a week getting all of this checked and checked again.

There is no DDoS attack. None. Zero, zip, zilch, nada. I absolutely know how to spot one, and I absolutely know where the traffic is coming from, and I know that it is legit.

So now, please can we end this talk of DDoS attacks? It is absurd that it keeps going on. I don't even know why you would think we would hide that from you. Telling you it is a DDoS would make our lives easier.

Please, for the love of god... let us get on with performing a server upgrade, which will FIX the outage issue. Believe me, more than even you guys I don't want to see downtime on the site. It makes my skin crawl! I do believe that the PSland sites were also on Bytemark, though admittedly I am not 100% sure.

We're going to move the site to a cloud host, which has elastic properties around the VMware boxes we'll put it on. That way we can burst as high as we need to and not have any outages. We just have to adjust the site a bit to make it fit with the hosting structure, get the new VM boxes spun up and site moved. We're doing it all carefully and meticulously to address this issue once and for all.

Please trust us here - this is not a DDoS attack. This is good stuff for PSU and we are simply experiencing growing pains from massive traffic growth. Its awesome!

**Bligmerk** · 12-20-2011

Originally Posted by Mr.Bishop

I'm sorry for not mentioning it, but what I have said repeatedly is both myself and my development team have dealt with many true DDoS Attacks. I don't know why you don't think I"m confirming where it is... I did many times. We have content syndication partnerships around the web. I have compiled a list, checked them, confirmed traffic numbers... I've spent literally like a week getting all of this checked and checked again.

There is no DDoS attack. None. Zero, zip, zilch, nada. I absolutely know how to spot one, and I absolutely know where the traffic is coming from, and I know that it is legit.

Please trust us here - this is not a DDoS attack. This is good stuff for PSU and we are simply experiencing growing pains from massive traffic growth. Its awesome!

I appreciate the response and the matter is really not that big of a deal to me, but here we are again. Fine, it is definitely 100% not a DDoS "test" attack.

But then you say this, "We have content syndication partnerships around the web. I have compiled a list, checked them, confirmed traffic numbers... I've spent literally like a week getting all of this checked and checked again."

So, when you have this active connection suddenly go over 5,000 during the night so that some active members are unable to make a connection is a good thing? And you have confirmed it? And that eventually this gets worse and worse until the site appears to be down especially during trade shows is "good stuff"? And that after enough of these experiences an active member decides it is not worth it and goes inactive is also "growth"? Alrighty, then.

**Agriel** · 12-20-2011

Originally Posted by Bligmerk

I appreciate the response and the matter is really not that big of a deal to me, but here we are again. Fine, it is definitely 100% not a DDoS "test" attack.

But then you say this, "We have content syndication partnerships around the web. I have compiled a list, checked them, confirmed traffic numbers... I've spent literally like a week getting all of this checked and checked again."

So, when you have this active connection suddenly go over 5,000 during the night so that some active members are unable to make a connection is a good thing? And you have confirmed it? And that eventually this gets worse and worse until the site appears to be down especially during trade shows is "good stuff"? And that after enough of these experiences an active member decides it is not worth it and goes inactive is also "growth"? Alrighty, then.

honestly man, do you have problems reading or something, he has stated like a million times that they are right now in the process of doing a major upgrade to the servers, and that the new servers are elastic and can handle any type of traffic that the internet can throw at it. How fucking hard is it to read and understand that, yet here we are reading the last part of your post as if you have a problem with selective reading or something.

**Mr.Bishop** · 12-20-2011

Originally Posted by Bligmerk

I appreciate the response and the matter is really not that big of a deal to me, but here we are again. Fine, it is definitely 100% not a DDoS "test" attack.

But then you say this, "We have content syndication partnerships around the web. I have compiled a list, checked them, confirmed traffic numbers... I've spent literally like a week getting all of this checked and checked again."

So, when you have this active connection suddenly go over 5,000 during the night so that some active members are unable to make a connection is a good thing? And you have confirmed it? And that eventually this gets worse and worse until the site appears to be down especially during trade shows is "good stuff"? And that after enough of these experiences an active member decides it is not worth it and goes inactive is also "growth"? Alrighty, then.

Basically growth is a good thing... but before it is a really good thing it is a bad thing for the user experience. Due to the nature of physical servers, they have maximum loads. When you exceed these, you can only add MOAR hardware to a point. Eventually the code to server relationship breaks down, and you start to have crashes you can't stop. This is the point we're at.

Growth = good.

Crashing = bad.

Growth + crashing - solution = bad.

Growth + crashing + solution = growing and successful business.

Can't get to the last thing without the first though. Eventually in a growth path for a site you experiences load past what your current setup is and you have to fix it. Sure this site should have done that a long, long, long time ago... but there's a lot of things the previous owners should have done. We had one major crash issue plague us, and this second one has prompted us to get a solution.

I'm not arguing that crashes are good... but when they're a sign of growth - if you can handle them they're fantastic. They absolutely suck to begin with, but when you upgrade your system and serve gazillions of pages like a champ, you monetize much better and have a much more solid business.

Dig?

**F34R** · 12-20-2011

I'd say:

Growth + crashing + solution = still bad, but not as bad.
Growth + crashing + solution + implementation = good.

**GhostFox61** · 12-20-2011

Just curious, are we getting the expected 5-10% of these 5,000 unique visitors every night converted into new members?

It doesn't feel like we have gained thousands of new members from this during the past few weeks, but I may be wrong.

**Mr.Bishop** · 12-21-2011

Originally Posted by F34R

I'd say:

Growth + crashing + solution = still bad, but not as bad.
Growth + crashing + solution + implementation = good.

Now on that point sir, I stand 100% corrected. A solution not implemented is not a solution. Its just a plan.

**F34R** · 12-21-2011

Originally Posted by GhostFox61

Just curious, are we getting the expected 5-10% of these 5,000 unique visitors every night converted into new members?

It doesn't feel like we have gained thousands of new members from this during the past few weeks, but I may be wrong.

I'll go ahead and answer that with a resounding 'NO'. Who wants to join when the site keeps crashing? Who COULD join when the times they are visiting, the site keeps crashing?

**Mr.Bishop** · 12-21-2011

Originally Posted by F34R

I'll go ahead and answer that with a resounding 'NO'. Who wants to join when the site keeps crashing? Who COULD join when the times they are visiting, the site keeps crashing?

This is definitely true as well. Henceforth why we have it as an absolute #1 priority to get the server move done and done right.

It would be wonderful to retain some people! These types of visitors are notoriously not good for turning into regular users, but at least a few wouldn't be too much to ask for.

But again F34R you are very correct when you say that having such a poor experience for them leads to no problem reading a few news articles, but a very bad community experience.

**F34R** · 12-21-2011

After that is done, the site should see some decent growth with a syndication program like that.

**Mr.Bishop** · 12-21-2011

Originally Posted by F34R

After that is done, the site should see some decent growth with a syndication program like that.

Lets hope so. This place is rad.... and if we can ever get past dealing with legacy nightmares, we might be able to really make a dent in growing this community.

Thanks for talking this out with me BTW. I really appreciate you - and everyone here - being willing to go back and forth with me. I really want to get this place into a shape that we all can be proud of it.. and I know that isn't always going to be an easy task or an easy discussion to have.

We are a community and a team in the end. Stick with us, and I promise you that eventually we WILL prevail!