Articles » Administrator » HTTP Accelerator in front of Fundanemt

Author

Christian Jørgensen studies Computer Science at DIKU and partner in GMTA.

Website
E-mail

A caching HTTP-accelerator in front of your Fundanemt using Squid

by Christian Jørgensen, May 2005

In this note I will explain how I did the configuration of a Squid proxy to run in front of Fundanemt. When you reach a certain amount of pages in your Fundanemt installation things sometimes slow down. In this particular installation there is roughly 150 pages in two languages (a total of 300 pages) and a request takes about a second to complete. On a high-traffic site, a second is very long time, and could probably result in a bottleneck.

I was fortunate enough to have a server for this website alone, but it can be done with multiple websites too.

Modifications to Fundanemt

In order to cache anything, we need to add some headers to the HTTP response headers provided by Fundanemt. This i did last in the constructor for the Html-class:


<?php

 $lu = $this->lastUpdate();

 header('Last-Modified: '.gmdate('D, d M Y H:i:s \G\M\T', $lu['timestamp']));

 header('Expires:       '.gmdate('D, d M Y H:i:s \G\M\T', time() + 3600));

 header('Date:          '.gmdate('D, d M Y H:i:s \G\M\T', time()));

?>

You need to decide how long time your content can live without a refresh from the database. In this particular case, I have chosen an hour (3600 seconds).

The Squid proxy

Then we need to install the Squid proxy. Using Debian GNU/Linux that's an easy task to accomplish:

apt-get install squid

Afterwards we need to tune the default configuration. I changed the following lines:

http_port 80
httpd_accel_host localhost
httpd_accel_port 81
httpd_accel_single_host on
httpd_accel_with_proxy on
httpd_accel_uses_host_header on

I commented out all the http_access lines and added my own (replace example.com with your domain):

acl okdomains dstdomain example.com
http_access deny !okdomains
http_access allow all

Then (if you want to) you can tune the cache settings (how much RAM/disk usage - max/min object sizes). Look for the cache_* settings.

Apache setup

Finally we need to move apache from port 80 to port 81. This is done by changing the Port directive in the apache httpd.conf:

Port 81

Now restart you services (again - this is Debian GNU/Linux syntax):

/etc/init.d/apache restart
/etc/init.d/squid restart

Then you should be up and running. Watch /var/log/squid/access.log for more information.

Prefetching content

You would probably like to prefetch your content in order for your clients not to experience the one second delay of generating the content. This can be achieved by using wget to "crawl" the site at a given interval (set up a cronjob for your preference and again, replace example.com with your domain):

cd /tmp
mkdir remove_me/
cd remove_me
nice -19 wget -r --cache=off -q http://www.example.com/
cd ..
rm -rf remove_me/

The result

Here is the output from ApacheBench running from a host connected with 100 mbit to the server running Squid (for the non-technical readers - I am making 250K requests with 1000 concurrent connections):

# ab -c 1000 -n 250000 http://example.com/page/
  This is ApacheBench, Version 1.3d <$Revision: 1.73 $> apache-1.3
  Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
  Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/
 
 Benchmarking example.com (be patient)
 Completed 25000 requests
 Completed 50000 requests
 Completed 75000 requests
 Completed 100000 requests
 Completed 125000 requests
 Completed 150000 requests
 Completed 175000 requests
 Completed 200000 requests
 Completed 225000 requests
 Finished 250000 requests
 Server Software:        Apache/1.3.33
 Server Hostname:        example.com
 Server Port:            80
 
 Document Path:          /page/
 Document Length:        5196 bytes
 
 Concurrency Level:      1000
 Time taken for tests:   134.120 seconds
 Complete requests:      250000
 Failed requests:        10
  (Connect: 0, Length: 10, Exceptions: 0)
 Broken pipe errors:     0
 Total transferred:      1397325017 bytes
 HTML transferred:       1299021031 bytes
 Requests per second:    1864.00 [#/sec] (mean)
 Time per request:       536.48 [ms] (mean)
 Time per request:       0.54 [ms] (mean, across all concurrent requests)
 Transfer rate:          10418.47 [Kbytes/sec] received
 
 Connnection Times (ms)
               min  mean[+/-sd] median   max
 Connect:        0   301  849.7     87 21131
 Processing:    38   225 1005.5    130 67681
 Waiting:       10   224 1005.5    129 67680
 Total:         38   526 1335.3    223 67714
 
 Percentage of the requests served within a certain time (ms)
   50%    223
   66%    260
   75%    298
   80%    326
   90%    587
   95%   3198
   98%   3319
   99%   3648
  100%  67714 (last request)

It is important to note that it was the network (100 mbit) that made the bottleneck now.

FundaUsers