Hey guys,

Mike referred me here and said you should be able to help me out.

I run a very large (500k+ impressions/month) website that runs a typical LAMP setup.

Recently we started to see a lot of high load (50+ at peak hours) so we started trying to find out what was causing the problem. We bought a new server and moved the db onto it. The db server never sees load above 1, meanwhile the apache server is still getting slammed.

Relevant info:
Apache server: apache 1.3.27, php 4.3.9, mysql 4.0.23 and redhat 7.3
DB server: mysql 4.0.23, fedora RC3

Site is almost 100% CSS, all text based db, Smarty templating system, and adodb w/ cache'ing enabled.

apache compile options are as follows:
Code:


All default modules plus mod_php4 and mod_gzip are loaded.

I was seeing some seg fault errors on the apache server, so I decided to move everything over to the db server just to test it out. I have not seen any seg fault errors on the db server, but have still seen numerous errors from apache and php such as:

[Tue Mar 15 18:24:52 2005] [warn] child process 32229 did not exit, sending another SIGHUP

I've tried to tweak the config to close out child processes, but have not had much luck.

Right now I'm at a loss. I've read up on tweaking apache and mysql for memory efficiency, but can't seem to get anything out of it. The mysql tweaking helped, but any apache tweaks I do have a minimal effect, if any.

Lastly, the servers are both pretty beefy, Dell Powedge, both with 2 gigs of ram, SCSI drives, and PERC RAID controllers.

Any help is appreciated. If you need any more info, please ask and I'll provide it.
Back to top

I'm not really sure, what is the machine running at in terms of processor and memory useage, if you sit watching top run in the terminal for example what's it at during peak and slow times, what % of processor and memory is apache using?

You want to be sure you don't have a hardware problem first I think, like a mechanic told me once when asking about a 'new' battery we'd put in the car that died: did you test it?

If there's bad memory in there that would cause major problems.

Why did you use Apache 1.3 instead of 2.0, 2.0 is supposed to handle high loads better. Plus it's a lot more pleasant to work with.

Redhat 7.3 is getting pretty old too, have you thought about using something running the 2.6 kernel, or freebsd? Have you checked the obvious, like are you getting slammed by 'connect' requests for instance, without realizing it? I had a tiny site that suddenly almost stopped working because some machine out there started shooting connect, not get, requests at it constantly, took a while to figure that one out, easy to see by reading the logfiles.

Are you getting 50 simultaneous visitors at once? If I remember my apache stuff right, apache 1.3 starts a new instance of itself for every request, or something like that, whereas 2.0 does it differently, can't remember the exact mechanics of it, but it's much better for higher loads.
Back to top

I agree about the memory, which is why I moved everything over to the db server, to _somewhat_ rule out a hardware problem. It's obviously not 100%, but it certainly is a relevant data point.

As for memory usage. Swap was killing us, but the tweaking got it down considerably.

I'm not sure what total percentage apache is using, but it typically dominates the process list. The memory usage seems to be around 0.3% per httdp process and the cpu usage is right around 10% (give or take 2).
Back to top

Couldn't find all that much, some places seem to suggest that can be mismatched configuration files, here's one you might want to take a look at, deals with the slapper worm.

Why are you sticking with redhat 7.3? And apache 1.3?

It seems odd that swap was a problem if you have 2 gigabytes of ram installed if each process is only taking .3% of ram, 50 concurrent processes would just be 15% of memory unless my math is off.
Back to top

Thanks for the replies jeff. I am on my way to work, but will answer some of your questions and then get back to you in detail later this afternoon.

I apologize, I said 500k per month, I meant 500k per day (minor typo, right?).

The db server is running Fedora RC3 with the 2.6 kernel. The apache server is still running 2.4 on 7.3. I'd like to upgrade it.

I am not running apache 2.0 because, well, no reason really. Have always run 1.3.

I will dig through the logs later today and get back to you.
Back to top

LOL, that's a slight difference, I was trying to figure out how you could be having issues with something like 8 users a second.

No wonder you're having problems. I read a bit on Apache2 and PHP, some good points raised all around. More here and here. So you may be right in not wanting to move to 2.0, although if you use the prefork apache 2 that avoids the threading issues discussed.

First question: is the server dual processor? If not, I don't think some of these issues will be resolvable.

Second question, how many http requests does each page create? If you have images you might think of moving all images to an image server, which doesn't have to be very high end since all it will do is send images. It seems to me that if you have this type of load on the main box you want it to do as little as humanly possible in terms of requests per page. For example, if you have 2 stylesheets, make them one, etc.

By my math, you're delivering 5.75 pages per second, probably with a peak of 10-15. Does the 500k refer to pages or hits? If it's pages, then you'd have to multiply that number by the number of components required to build the page.

Then on the php end of things, how are things being loaded? Are the includes/requires optimized to make sure each library only loads once per request? PHP can be heavily optimized to radically drop server load, but it takes work, I don't know how your programming is designed, but that's a major issue, not as major as the mysql queries being optimized, but if you have setup a standalone mysql server that should have helped as you noted, but it still doesn't mean that the actual queries are optimized.

I've never done a site this large, but some issues are pretty standard.

If you don't have dual processors, you are sure to have problems.

By the way, Apache is developed on BSD from what I read, so that's another thing to think about, FreeBSD is rock solid, but you'd need somebody who's up on working with that type of system. You can install most of the Linux components on freebsd by the way, but I suspect you probably don't want to go that route.
Back to top

LOL, that's a slight difference, I was trying to figure out how you could be having issues with something like .2 users a second.

No wonder you're having problems. I read a bit on Apache2 and PHP, some good points raised all around. More here and here. So you may be right in not wanting to move to 2.0, although if you use the prefork apache 2 that avoids the threading issues discussed.

First question: is the server dual processor? If not, I don't think some of these issues will be resolvable.

Second question, how many http requests does each page create? If you have images, js, and css library files, you might think of moving those to a separate server, which doesn't have to be very high end since all it will do is send static files. It seems to me that if you have this type of load on the main box you want it to do as little as humanly possible in terms of requests per page. For example, if you have 2 stylesheets, make them one, etc.

By my math, you're delivering 5.75 pages per second, probably with a peak of 10-15. Does the 500k refer to pages or hits? If it's pages, then you'd have to multiply that number by the number of components required to build the page.

Then on the php end of things, how are things being loaded? Are the includes/requires optimized to make sure each library only loads once per request? PHP can be heavily optimized to radically drop server load, but it takes work, I don't know how your programming is designed, but that's a major issue, not as major as the mysql queries being optimized, but if you have setup a standalone mysql server that should have helped as you noted, but it still doesn't mean that the actual queries are optimized.

I've never done a site this large, but some issues are pretty standard.

If you don't have dual processors, you are sure to have problems.

By the way, Apache is developed on BSD from what I read, so that's another thing to think about, FreeBSD is rock solid, but you'd need somebody who's up on working with that type of system. You can install most of the Linux components on freebsd by the way, but I suspect you probably don't want to go that route.
Back to top

The apache server is dual 1ghz P3 Xeons.
The db server is currently single processor with a dual processor board.



I'd have to ask our php developer, I don't to any of the coding at all. There are next to 0 images on the site, as I mentioned above, it is almost 100% CSS.



Pages.



Again, I will have to ask our dev guy. He isn't around today, but I will refer him here and see if he can't answer your questions.

Thanks again for the help Jeff. I will have our dev guy provide you with more info as soon as possible.
Back to top

If you are getting 500k pages per day, and on each of those pages, you have let's say 2 CSS library files and 1 javascript file, that would come to 2 million requests per day to the server. If you can get the css and js files to another server, again, it doesn't have to be that great, then you've cut your http requests down by 75%, which is a a pretty substantial reduction in server load. It also depends on how many packets it has to send out, ie how big the css, js, and html pages are. If I understand this correctly that is.

1 gigaherz isn't that fast nowadays, it's ok though. I'd be looking at the processor and memory useage a lot, there's many more advanced tests you can do, I can't think of them off hand, but there's lots of ways to figure out where a bottle neck is.


Put another processor in it, that will make mysql happy, and the os happy.
Back to top

Hola, I run the site along with BA. Thanks all for giving such input -- we're desperately trying to get a few things stable.

Thank you for bringing up the issue about the includes being only required once per request -- it's something I actually half-assed overlooked. I'll have to do some changes, and we'll see if that helps out any.

Also, would moving the style sheets, and such, to another server help us? We currently do have two style sheets -- they can be easily merged. I'll also have to try this during the week.

The MySQL queries are more or less optimized -- the ones that are hit most atleast. There are maybe some spots that can be modified quite a bit, but these are areas that don't get much 'attention' from our users.
Back to top