Jump to content

Photo

RE: Emergency Maintenance (and protest?)


  • This topic is locked This topic is locked
18 replies to this topic

    RawR

  • Grumpy
  • Administrators
  • 3,553 posts
  • LocationHere of course!
Um... I'll post more details later.

Tuesday roughly 3:30AM UTC - Server taken offline for emergency maintenance.
Tuesday 12:00PM UTC - Server Restored.

------------------- EDIT --------------------
Here's the delayed explanation (including why it was delayed). It gets pretty technical though.

Since around Jan 20th, we've been under a peculiar traffic. We could call it a "ddos" attack, but the situation is a bit odd to call it that. But the attacks were too small (relatively) to actually affect the server to go into denial, maybe slow down a bit (Be in mind, our server handles thousands of people, serving over 5000 pages per minute, which translates to 10K~100K files served per minute. That's out legitimate traffic. Small attacks doesn't really compare against the actual traffic). Also as a tidbit of a knowledge, a high traffic sites always get dos attacks from some stupid kid that thinks it's cool to (d)dos a site (then gets blocked by the server (usually automatically) and concludes the unreachable server "down"...). That... along with millions of random bots throughout the net that just goes around looking for exploits in servers. I think that part is an unforunate reality of operating a server. Back on topic, I had chosen to deal with the issue with a bit of leisure as it wasn't making a big effect (a mistake).

Jan 23rd comes a bit of a nightmare. The biggest culprit (I won't name because I believe it was not actually intentional because I know who it is) who was effectively ddosing out website got pretty large. They were requesting pretty much every single page in Batoto's existance at great speeds and in an... "ugly" way. Combined with other abusive usages that was found, it went past the tipping point making our server go into denial of service. This wouldn't have been that big of a deal normally, just reboot, block the problematic traffic and continue. But... I can't even connect to the server anymore. -_-;;

I call the datacenter to ask my server to be manually rebooted. A machine says leave a message, they're temporarily out of office. -_-;;
I call the biggest culprit's headquarters, have them forward me to their tech department. A machine says leave a message. They clearly went home already. -_-;;
Well, I leave a message (a ticket actually) with the datacenter to reboot and a message to the culprit saying they're a freaking idiot with lot of servers.
Few minutes later, datacenter tells me they're rebooting. ^_^
Few minutes later, they say it refuses to reboot. -_-;; So, they're doing a hard reboot (power off, power on. Don't do this kids!! This is bad for your computer's health).
A hard boot later, our server was online, I was busy at blocking off problematic traffic of quite a few sources. I may have blocked some users as well by accident... lol (They'll see my email then anyway). But, all the ones I banned were requesting like a 1000 page per minute, so I'm sure they're not real people... I can't read 1000 pages of comics in a minute, can you?
But our server was keep on going into denial. I didn't really get why until I found out that our RAID had failed after the hard reboot. This isn't all that likely to happen, but it happened to, not one, but two of our raid drives. -_-;; You can read more about what RAID is on wikipedia, but in short, it's like combining multiple hard drives to create a mega hard drive. Faster, bigger, awesomer. Awesomer is the most important part. So, without RAID functioning properly, it was trying to serve everyone (it's surprising how quickly people come back... they must be pressing F5 every minute instead of just stop procrastinating and doing homework) from few disk (instead of all) while rebuilding raid at the same time. Our disk just couldn't handle it (much of cache is also gone after a reboot, so everything comes from disk now). So, the choice was... let it rebuild and leave server on making a slow ass donkey site or let it rebuild in peace and at the fastest speed possible, turning off everything that'll use the disk. RAID rebuild took about 7~8hrs. Once one of the raid rebuild was complete, I was finally able to see "how much time left". Fine, set alarm to x minutes later, went to sleep. Woke up at 7am (my time) turned stuff back on (actually my sys admin did that. I was few minutes behind him. Does he not sleep??). Kept site monitor on monitor with monitoring tool to make sure it's running fine. Then chose to get some snooze because I only had like an hour of sleep at this point. The completely not lazy me woke up at 2:30PM and here we are. Seems like it's still running fine. yay!

For some who might be wondering... don't you need to go to work? Meh, no one's gonna miss me because I'm gone for a day or two. This is the power of working in the tech industry!! Get your stuff done on time and no one gives a damn! :D

--------------- Part 2 - The Protest ----------------
The above was inevitable, so I chose to make the best of the situation. While I like to keep this site neutral as possible on many grounds, I felt this was the best choice. Batoto did not have any form of official protest against SOPA/PIPA. While they're utterly terrible bills, I did not feel the importance of taking this site down due to 1. the bother, I'm already loosing sleep. 2. felt it was already well in the knowledge of many people and 3. did not feel that it was my place to say it. The case of ACTA however a bit a different. The site was down anyway, not that many people know that it's on the brink of being passed and on the last point, I can be convinced. lol

The Internet censorship is a great deal. This site wouldn't exist if these bills were pushed to the full force which has not yet happened (but as we see with Megauploads, we can see how US would have used ACTA to keep a surveillance on MU owners without any court issued warrant). This site has far too much freedom of speech (to the level of what we wish) to really coexist with heavy surveillance and censorship. The Internet is the most creative forefront in the world today. Vast amount of software, media, and etc are being made and developed so fast and I believe that's largely due to licenses like Creative Commons, GNU, MIT, etc which allows sharing. The knowledge and the works are able to quickly build on top of each other creating a result much better than any single person could have created. This site houses hundreds, maybe thousands of programs made by other people who allowed it to be shared for free. I couldn't have made this site from scratch by myself, there's just too many things. And this is why I believe freedom is important for the sake of the growth of our future. Restricting this creativity would do nothing more than keep obsolete business models continue in their paths of inefficiency and restrict the growth of the Internet.

If you want to see the protest page, actually our Error 500 page is still availabe by just typing it out.
http://www.batoto.net/50x.html

Posted 24 January 2012 - 12:21 PM


    Russet Potato

  • Sieg
  • Members
  • 492 posts
  • LocationŠumeru.
Post the details now! I'm curiousssss.

Not sure if one is allowed to post here, but it would probably be disabled if not! Oh, well.

Posted 24 January 2012 - 01:12 PM


    Baked Potato

  • Locationin the tower!
Protest, no just joking :P

@Vaygr, He was rebuilding the raid. so one harddisk was either broke or it was a software bug. of course he could've added a harddisk for extra speed or capacity(doubt capacity was the reason). perhaps that was the reason why the website went slow lately.

Edited by Laurens D, 24 January 2012 - 01:50 PM.

Posted 24 January 2012 - 01:49 PM

Posted Image

This is not my fault!

You could have bought my cookies.


    Fried Potato

well I guess that's the reason for the blue batoto images on the comic images under the 'highest rated comics' and 'recently added comics' list as seen from a manga page like http://www.batoto.net/comic/_/comics/air-gear-r9 ?

Posted 24 January 2012 - 03:27 PM

Posted Image
Favorite Manga: Record of Fallen Vampire, Hayate the Combat Butler, Kekkaishi
Main reason: Akabara Strauss and plot/story flow, Tennousu Athena arc/Katsura Hinagiku, Kidoin Nura and story flow

    A Kitty In A Penguin Suit

  • LocationLurking nearby...
maybe that's also the reason why links keep sending me to the wrong place, pages only load what seems to be unicode [no expert] and stuff. things are getting better, so i know those are just temporary. :) gambatte~

Posted 24 January 2012 - 04:22 PM

Catch my story; the Neverender Series

Batoto's Rules Repository | Rules, Guidelines and FAQs

Rules and Regulations | How to get help | Frequently Asked Questions

If all else fails, PM me

And use the REPORT button if you see

any content that may violate site policy.

And D.F.T.B.A.!


    Russet Potato

  • Sieg
  • Members
  • 492 posts
  • LocationŠumeru.

Protest, no just joking :P

@Vaygr, He was rebuilding the raid. so one harddisk was either broke or it was a software bug. of course he could've added a harddisk for extra speed or capacity(doubt capacity was the reason). perhaps that was the reason why the website went slow lately.


Ah, yeah. I was more curious about the "protest?" part.

Posted 24 January 2012 - 05:09 PM


    Baked Potato

  • LocationWonderwall
Ah, that explains what happened yesterday.

Edited by Katsuri, 24 January 2012 - 07:28 PM.

Posted 24 January 2012 - 07:28 PM

sig-1.png


    RawR

  • Grumpy
  • Administrators
  • 3,553 posts
  • LocationHere of course!
Details posted. Got a bit long though...

Posted 24 January 2012 - 08:59 PM


    Russet Potato

  • hados9
  • Contributor
  • 352 posts
  • LocationIn line for Moderator

I can't read 1000 pages of comics in a minute, can you?


Shhh but i can read 1000 manga pages a minute but it depends on your definition of reading :P
and i hate ppl who think just cause they can do little ddos attacks lets do it to everyone -.- drives me up the wall
and well nice work in getting it fixed and all working again with minimal down time and a good cover story for the down time :P
and yes the tech industry is a lovely place for not going to work as you can do your work at home if need be and its true no one gives a shit if you come in aslong as your works up-to-date or finished :D

Posted 24 January 2012 - 09:34 PM

Dedicated Uploader- Hados9

Posted Image


Posted Image


    Fried Potato

  • Locationin your mind
Thanks for the explanation :).

Posted 24 January 2012 - 10:03 PM

"Don't ask what the world needs. Ask what makes you come alive, and go do it. Because what the world needs is people who have come alive." Howard Thurman

"Learn from yesterday, live for today, hope for tomorrow." Albert Einstein

Visit my deviantart


    Couch Potato

  • LocationBaseball Field
naw I see why. Thanks for informing us. :]

Posted 25 January 2012 - 12:51 AM

Salute to the Goddess of Little Lambs, bakero!!! ~ 1b4b0fc2-386f-4674-ad7b-5ae3c8642336_zps

Spoiler

    Baked Potato

  • Locationin the tower!
I see, about:
But, all the ones I banned were requesting like a 1000 page per minute

Well, when a css, image, js files load each are counted as 1 request so 1000 per minute is not hard. Or perhaps you got attacked with a 'slowloris' attack.

About the protest, my bad :P thought you ment a protest for the maintance :P because the topic was not finished yet :P

Posted 25 January 2012 - 03:58 AM

Posted Image

This is not my fault!

You could have bought my cookies.


    RawR

  • Grumpy
  • Administrators
  • 3,553 posts
  • LocationHere of course!

I see, about:

But, all the ones I banned were requesting like a 1000 page per minute

Well, when a css, image, js files load each are counted as 1 request so 1000 per minute is not hard. Or perhaps you got attacked with a 'slowloris' attack.

About the protest, my bad :P thought you ment a protest for the maintance :P because the topic was not finished yet :P

No, I meant 1000 html pages.

Posted 25 January 2012 - 05:01 AM


    Russet Potato

  • Sieg
  • Members
  • 492 posts
  • LocationŠumeru.
Oh. So, that's why the site is randomly superslow and non-functional for me now and then.


Interesting read. Thanks, Grumpy.

Posted 25 January 2012 - 06:49 AM


    Baked Potato

  • Locationin the tower!

Oh. So, that's why the site is randomly superslow and non-functional for me now and then.


Interesting read. Thanks, Grumpy.


Always on bad moments :P, for example when you are reading the ending of a manga. After the maintance it seem to work great now. well, for me atleast

No, I meant 1000 html pages.

I see, then doubt that it's a number that you can view in 1 minute :P

Edited by Laurens D, 25 January 2012 - 12:06 PM.

Posted 25 January 2012 - 12:02 PM

Posted Image

This is not my fault!

You could have bought my cookies.


    Baked Potato

  • hero
  • Contributor
  • 1,381 posts
thanks Grumpy, for telling us what was going on with the site.
I hope everyone already knew about ACTA...

Posted 25 January 2012 - 01:14 PM


    Potato Spud

  • LocationNew Zealand
Thanks for the explanation.

Posted 26 January 2012 - 09:12 AM


    Potato Spud

What I'm really worried about ACTA is not just its content, but the manner in which it is being dealt with. The countries involved are trying to pass it without letting the general populace know what is really going on. I say this because I know one of the countries that is involved with this, South Korea, is almost keeping it a secret that they have agreed to this ACTA thing. There is no coverage by the Korean media on this issue that I know of, and nobody really talks about it, which to me seems very troubling. I don't know if any other countries that has agreed to ACTA is also keeping this under wraps, and if so I would really like to know about it so I can actually tell how bad the situation is.

Posted 11 February 2012 - 05:53 AM

-One that cleanses God and exorcises the Devil-



Posted Image


"This isn't a shoujo manga, you know"

-Kamijou Touma-


Spoiler


    Potato Sprout

  • LocationSydney
I can read very very fast(skim and scan) but 1000page in a minute is impossible. I am a newbie and thank you for all your hard work to fixed everything and I know about this web. Thank you

Posted 13 February 2012 - 01:47 AM