Unread postby Kalah » 26 Jan 2007, 02:27
Studio Commentary: "Today, we bring you a story of investigative journalism, as our reporter-in-the-field tries to find the true cause of the recent problems of Celestial Heavens."
Reporter: "Yes, most of you have probably noticed the server problems of late, giving you the message "This Account Has Exceeded Its CPU Quota" instead of the lovely header, the PotD and the news of the latest patches. This has happened before, and disables all Celestial traffic, in and out, making it impossible for anyone to even visit the site – but this time it seems the problem is greater than before. We begin our search for the cause of the problem with its first occurrences months ago. Mr. Admin, Sir, when did this first happen?"
Angelspit: "A quick search revealed we first experienced this back in April last year."
Reporter: "That long ago? And still it persist?"
Angelspit: "No, it's back, not still here."
Reporter: "What's going on?"
Angelspit: "At first we thought maybe it was ThunderTitan flooding the boards, but then we found out that the downtime has in fact caused him to go into withdrawals and he had to be committed to Wonderland for a week."
Reporter: "Right. Can you tell me... what did you do when it first happened?"
Angelspit: "First, we did a bit of fire fighting, just to cope. Removed some of the more CPU consuming features on the main site etc. Then we had one of our in-house wizards start doing some re-coding to make the scripts more efficient."
Reporter: "What did he do, exactly?"
Angelspit: "I think you'd better ask him about that."
Reporter: "Ah, yes, trade secrets, perhaps?"
Angelspit: "No, it's just that I didn't understand nuthin' of what he said."
Reporter: "Riiiight... Well, I'm pretty sure I can handle myself – so, Mr. Protecyon...
Protecyon: "I'm busy, whaddayawant!?"
Reporter: "I was just wondering about this CPU problem... now, when it first occurred..."
Protecyon: "I heard ya. The problems were first happening due to inefficient scripts and the increased traffic. Then the problem was compounded by us providing additional features, most notably the forums."
Reporter: "So why didn't you just make the scripts more efficient?"
Protecyon: "a'ight, wannabe. Lemme tell ya. The reason that the optimization would take so long is because the majority of the inefficiencies were due to the constant polling of the database. Every time a user refreshed a page, a complete reload of all the information is required from the database. This is unnecessary because some of the information may still be valid, and it is a waste of resources to request information that is still valid. Optimally what would happen is that only the information that has changed will be requested."
Reporter: "Well, I understood that much. So tell me about what you were doing to rectify the problem."
Protecyon: "Well, we did a lot of things. First, a little optimisation. Then we removed and changed some of the more CPU consuming features, then we did some major work on Behemoth's Lair and the other sites we were hosting – that had some really inefficient scripts running, and were more popular than anticipated. We also started working closer together with the host. That fixed the problems."
Reporter: "Then what happened?"
Protecyon: "Heroes 5 hit the shelves."
Reporter: "Ah."
Protecyon: "Within days our traffic doubled. At the most we had 12.000 hits a day and some 70 users online at once just in the forums."
Reporter: "Ye-haa. But the server held up nicely, didn't it? Why did the problems re-surface a few months later?"
Angelspit: "We found out that the Bastion site was on our server under the new WoG site and they had an awful page that required up to 30 seconds of CPU usage – the average for a typical CH page is about 0.2 seconds."
Reporter: "Which explains why it was fixed again soon after, right?"
Protecyon: "Yea, I fixed the damn thing. Now lemme get back to work."
Reporter: "Hang on, hang on... The thing returned a month later."
Angelspit: "That was just me making backups. A good idea, as we've seen in the official forums lately."
Reporter: "Yeah, but you know, the reason I'm doing this story is because it is happening again – I mean right now."
Angelspit: "Yeah."
Reporter: "I understand you got a note from the host early this year?"
Angelspit: "Yeah, here it is."
Reporter: "Well, let's see... months of research... system performance techniques... I/O tuning... changes to our servers... yada-yada-yada... decrease server load of between
100-300%! Sounds impressive. When are they putting this into effect?"
Angelspit: "They already have."
Reporter: "Yikes. So when are you changing hosts?"
BlueHost: "Hey! We'll have no talking about that around here!"
Reporter: "Why I'll talk about whatever I want! This is a free forum, a free world, and I'm a member of the world's free press and... YAOW!"
Kalah: "Hey, man, is that a cattle prod?"
BlueHost: "Yeah, it's great for pokin' the little pests... wanna try it out?"
Kalah: "Yeah, I've been thinking of upgrading from my cricket bat..."
Reporter: "YAOW! Stop that, will ya?"
BlueHost: "You gonna stop with the questions?"
Reporter: "I'll be very, very good."
BlueHost: "Good boy."
Reporter: "Kalah, now that I have you here..."
Kalah: "Bye."
Reporter: "Eh... erhm... ah, right, the Quota thing – Angelspit, could you tell me..."
Angelspit: "Look, I already told you all I know, now go away."
Reporter: "But will you consider changing... I mean, when does the deal with the current host expire?"
Angelspit: "Why?"
Reporter: "Well, I thought that I could ask you whether you'll be renewing..."
BlueHost: "Hey! Are you at it again!?"
Reporter: "YOWZA!"
BlueHost: "I told you not to talk about these things."
Reporter: "Okay, okay... just tell me, when can I ask about it?"
Angelspit: "Medio 2007."
Reporter: "Thanks. So, what happens now?"
Protecyon: "We'll keep working on it. We still have some more things to try."
Studio Commentary: "I guess that's all we'll get out of the host mob for now folks... YOW!! I mean... the nice host and scripting guys... ehrm. Next time, by request of another member, we're bringing the member interviews back – starting with ScarlettP. See you then."
In War: Resolution, In Defeat: Defiance, In Victory: Magnanimity, In Peace: Goodwill.