When a server asks for help!

So it’s been a couple of days since the last post but I’ve been occupied with Nsteins WCM, customer support and server maintenance.

This post has to do with server maintenance and the proper diagnosis of problems.

About a month ago one of our most important customers called arguing that their web page was “down”. I’m no longer working full time for that customer but because I’m basically the server admin the ball was thrown to me.

Not all the site was down, it was a subdomain wich was aquired to a third party because they did such a great job with a project. So the company I work for was given the task to improve and mantain the site. Since the guy that created the site was proficient with Javascript and not that good with PHP he decided to code the whole site in Javascript and deal with the database through simple PHP scripts (right! AJAX).

He even coded a javascript obfuscator! He’s a man in his old 40’s so I was really impressed.

Anyway there’s this tiny minimum html code and then a HUGE obfuscated Javascript code.

Since the former programmer wasn’t really a programmer he coded the stuff to work forgeting about efficiency, security and errors management. Being the lack of errors management the issue that made me go bananas for like a week.

The site was displaying just HTML code, anything related to databases was blank. No error messages, not a clue of what was happening. So my coleagues asked me to take a look at the server and so I did.

First thing I checked were the files then the permissions. After that I went to php.ini and enabled error messages. Then I got this error saying that the script weren’t able to connect to the database, cool! this looked like an easy task!

Typed top, view the server status and realized MySQL was down but trying to start. Why wasn’t it starting?

I manually tried to start MySQL and got an error that after a couple of minutes I realized means the server HD is full. It was a 70 Gb HD so what was growing so fast?

Went to mysql data folder and realized a table was using 17Gb, ok still I’m missing lots of Gb so i started to look for very big files playing with find:

find / -type f -size +1000000k -exec ls -lh {} \;

I got a huge list and saw a couple of access log files related to apache. All of them adding a total of approx 12Gb. So I went and deleted them and left one so Apache could use it.

Everything went fine for a couple of weeks and then it happened again and I did the same thing I did before: delete the files and restart MySQL.

A week ago happened the same and also yesterday so I thought the database was growing really fast and indeed it was taking all the disk’s free space so I went ando told that to my boss and made a petition to get another disk up ASAP.

Just today the server went down again! I deleted the files and freed 2 Gb, a couple of hours later it went down again! WTF! something weird is happening! the database couldn’t grow so fast so I took a dive in the server searching again for long files and ther it was! a HUGE log file (25Gb) called mysqld.log…..

So stupid me read the last 50 lines and became aware that a table was damaged and needed to be repaired… the whole 25 Gb were the same message!!!! To make it better the damaged table size was 17Gb… 😮

repair table [table];

After a couple of hours I got a message indicating the table was repaired, reduced the log file size to cero and gave me a hug, everything was ok. Later that day just checked the logs again and surprise! they were empty. The thing is that the damaged table didn’t stored any onfo for about a month…. Next time the first thing I’ll check will be logs.

If a server asks for help this is how it’ll talk to me. Rookie no more hahahaha

Tags: , , , ,

Comments are closed.