Archive for September, 2009

When a server asks for help!

Saturday, September 26th, 2009

So it’s been a couple of days since the last post but I’ve been occupied with Nsteins WCM, customer support and server maintenance.

This post has to do with server maintenance and the proper diagnosis of problems.

About a month ago one of our most important customers called arguing that their web page was “down”. I’m no longer working full time for that customer but because I’m basically the server admin the ball was thrown to me.

Not all the site was down, it was a subdomain wich was aquired to a third party because they did such a great job with a project. So the company I work for was given the task to improve and mantain the site. Since the guy that created the site was proficient with Javascript and not that good with PHP he decided to code the whole site in Javascript and deal with the database through simple PHP scripts (right! AJAX).

He even coded a javascript obfuscator! He’s a man in his old 40’s so I was really impressed.

Anyway there’s this tiny minimum html code and then a HUGE obfuscated Javascript code.

Since the former programmer wasn’t really a programmer he coded the stuff to work forgeting about efficiency, security and errors management. Being the lack of errors management the issue that made me go bananas for like a week.

The site was displaying just HTML code, anything related to databases was blank. No error messages, not a clue of what was happening. So my coleagues asked me to take a look at the server and so I did.

First thing I checked were the files then the permissions. After that I went to php.ini and enabled error messages. Then I got this error saying that the script weren’t able to connect to the database, cool! this looked like an easy task!

Typed top, view the server status and realized MySQL was down but trying to start. Why wasn’t it starting?

I manually tried to start MySQL and got an error that after a couple of minutes I realized means the server HD is full. It was a 70 Gb HD so what was growing so fast?

Went to mysql data folder and realized a table was using 17Gb, ok still I’m missing lots of Gb so i started to look for very big files playing with find:

find / -type f -size +1000000k -exec ls -lh {} \;

I got a huge list and saw a couple of access log files related to apache. All of them adding a total of approx 12Gb. So I went and deleted them and left one so Apache could use it.

Everything went fine for a couple of weeks and then it happened again and I did the same thing I did before: delete the files and restart MySQL.

A week ago happened the same and also yesterday so I thought the database was growing really fast and indeed it was taking all the disk’s free space so I went ando told that to my boss and made a petition to get another disk up ASAP.

Just today the server went down again! I deleted the files and freed 2 Gb, a couple of hours later it went down again! WTF! something weird is happening! the database couldn’t grow so fast so I took a dive in the server searching again for long files and ther it was! a HUGE log file (25Gb) called mysqld.log…..

So stupid me read the last 50 lines and became aware that a table was damaged and needed to be repaired… the whole 25 Gb were the same message!!!! To make it better the damaged table size was 17Gb… 😮

repair table [table];

After a couple of hours I got a message indicating the table was repaired, reduced the log file size to cero and gave me a hug, everything was ok. Later that day just checked the logs again and surprise! they were empty. The thing is that the damaged table didn’t stored any onfo for about a month…. Next time the first thing I’ll check will be logs.

If a server asks for help this is how it’ll talk to me. Rookie no more hahahaha

ASP and PHP running under Apache on Linux

Wednesday, September 16th, 2009

So for years I’ve been developing on PHP, I’m even certified and certainly it’s a great language but unfortunately ASP programmers are way better paid so I decided to start learning ASP. I do have windows installed on my lap but my server is a CentOs with Apache hosted on my home.

Since I have quite a lot of things hosted there installing IIS on a Windows and set it to answer on another port wasn’t my desire so I wento to google for a couple of hours and after I typed “Apache ASP” I found this project called Mono. Very interesting but the main thing is that through a couple of installations and onfigurations you may get your linux/apache server running asp pages, even .Net Wow!

What did I do? First I installed the mono module for apache which is called mod_mono.so after that I only added the module to my apache with the following lines:

LoadModule mono_module /etc/httpd/modules/mod_mono.so

Then installed the following with yum: XSP and mono-devel (mono-devel will also instal mono-web and mono-winforms).

Then after installing XSP I created a Virtual host on apache to host asp apps.

The Virtual Host entry looks something like this:

<VirtualHost *:80>

ServerName asp.localdomain

ServerAlias *.asp.localdomain

DocumentRoot /var/www/html/localdomain/asp

MonoAutoApplication disabled

AddHandler mono .asp .aspx .ascx .asax .ashx .config .cs .asmx .axd

MonoApplications “/:/var/www/html/localdomain/asp”

</VirtualHost>

After this I copied the files under /usr/lib/xsp/test to the folder I specified as the DocumentRoot

Then I restarted apache and typed on a browser: http://asp.localdomain/index.aspx

And Bingo!!!! It’s aliiiive!

Oh don’t forget to read mono documentation here and here!

Managing yum repositories (Dag)

Saturday, September 5th, 2009

So I got a memcached project working fine on my dev env. I showed it to my boss and told him all about the advantages of having memcached working on a production server so he gave me green light to implement it.

Since we have a dedicated hosting package (Codero, former Aplus) I thought It’ll be easier to ask our hosting provider to install memcached on our servers… they mailed us back saying that it would be considered advanced support and we’ll be charged 99dlls if we wanted to install it… of course we said no. We had ssh access so we were able to do it and spare the money 😀

Since on my dev env I just did yum installs I had the idea it’ll be the same on prod server, WRONG!

Th thing is that they have the default yum repositories and memcached was found on Dag repo so I needed to add it to get memcached. Looked for info over the web and found this web with the proper instructions.

I’m on a CentOS server so I’ll put here a brief description of what I did.

1. go to /etc/yum.repos.d/

2. create a file called dag.repo with this content:

[dag]

name=Dag RPM Repository  for Red Hat Enterprise Linux

baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag

gpgcheck=1

enabled=1

3. download the GPG key to your server and import it with the following command:

rpm –import RPM-GPG-KEY.dag.txt

4. congratulate yourself!

[dag]
name=Dag RPM Repository for Red Hat Enterprise Linux
baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag
gpgcheck=1
enabled=1

Now I was able to install the memcached server and the memcached php module:

yum install memcached

yum install php-pecl-memcache

Then I just restarted httpd and done!

Hope this is helpful.

osCommerce installation problems solved!

Thursday, September 3rd, 2009

I decided to install oscommerce to sell stuff.

Created a vhost, created the database, downloaded oscommerce, copied the files to the proper directory, assigned the right privileges and typed the proper url, up to here everything is really easy, looks smooth.

So here comes an installation screen. It looks like an easy process right?

First you type in the database connection info after you click ok you get a nice message saying that the database was created successfully!, the next steps are trivial, it’s a couple of normal settings. On the last step you can see 2 buttons that take you to the catalog or to the administration tool.

So I clicked catalog…. boom! nothing! just a nasty error page with something like this:

Warning: require(includes/languages/.php) [function.require]: yadda yadda on line 288

Ok, so I tried the administration tool… crap! same screen! something wrong is happening!

Being a PHP programmer I was like OK fine let’s correct the code, so I opened my eclipse and started debugging… after a couple of hours of adding flags, debuging, linking code logic and more I started thinking that it might be a DB problem so I went to the database console and check the language table WOW! it’s empty! ok, lt’s see configuration… empty!

Ok, I don’t think oscommerce delivers an empty database right? so after looking at the code I ended up on line 57 of the file install/rpc.php and there’s the sql source file.

Next thing I did was directly import the file to mysql…. ok 😀

Did a couple of queries to see if the languages and configuration tables were empty and they were not so the last step was to refresh the catalog and administration tools and guess what? They both work!!!!

So I’m writing this post to help anyone that gets the same errors.

I’ll correct the code and let oscommerce know something is wrong with their code.

Memcached, working!!!!

Thursday, September 3rd, 2009

Hard work, a couple of hours of investigation and a few modified lines of code later I got a project working with memcached and let me tell you it does work!!!

I was very happy when I saw my database being released from a lot of processes.

What I love the most is the options memcached gives you to choose how much time a cached object will remain active in the server!!! for very uncommon queries I set the cache to remain on the server for 15 min while for the other more persistent queries I decided to give them no limit.

I’m really pleased! just sent an email to my boss so he can check this out and it’s 2 o’clock in the morning so job done and now it’s time to go to sleep.

Memcached

Wednesday, September 2nd, 2009

Wow!

I’m in charge of a site that uses a lot of resources, thousands of images, thousands of db rows and literally thousands of visitors each day. We are currently getting approximately 60 to 70 thousand visitors but the thing is that by the time we get to 60 thousand the server starts to slow down. The require for images, queries and number of petitions per second made us think of a redesign of the site.

First step was to load the balance between 2 servers for http request and for database queries. Each server will have its own database and web code. Over those servers we put a load balancer hardware.

Second step was clean the database, eliminate duplicated rows and reindex fields. Throw away those fields that were not used and improve queries.

Third step was cleaning up the code, delete lines of unused code and repurpose a couple of functions. We also set a template engine (Smarty).

Now it seems the site is working fine server speed improved but still the database queries were too slow… what was left to do? Memcache!

First install memcache on the server. In my case I’m running CentOS so I just had to do:

yum install memcached

Then install php’s memcache support:

yum install php-pecl-memcache

Restart httpd and then type the following command on your server’s console

memcached -d -u apache -m 512 -p 11211

What this does is that it starts the memcached daemon with apache user privileges usin 512 Mb of RAM memory and using port 11211.

Since you’ll be needing port 11211 don’t forget to check your iptables config so you may be able to use it.

That’s it! everything is ready, the only thing left is modify your php’s code and you’re on your way.

I haven’t tested it but to be honest I really think it will save us lot’s of resources from the server. I’ll post the result in a few days.