Category Archives: Benchmarks

Track performance by user?

Thanks to the awesome folks at SOASTA we’re now using their mPulse system instead of our own boomerang install. This gives us 2 major wins. First, we’re including the tracking code in the non-blocking asynchronous iframe method, which gives the best possible performance at this point. Second, we can actually see the data. Previously, we just weren’t getting visibility into our boomerang data. We had the data, but weren’t using it, which was a total waste.

Looking at our stats today, mPulse tracks the median page load time. I was looking at the data and thinking, I wonder what it looks like per user. For example, I wonder if users with faster connections typically hit more pages. If they do, that means our median average user load time is actually lower than our median page load time.

Take two users, Alice and Bob. Alice is on her desktop in London with a 100Mbps line. She visits 8 pages. The average load time for her 8 pageviews is 1.2s. Bob is on his iPad over 3G in Alabama. (We’re in the UK, so London is closer!) Bob visits 4 pages. The average load time for the 4 pages is 2.3s. Now our arithmetic mean is somewhere in between the two, but our median, in this case, is one of Alice’s pageviews.

What would be really interesting, is to group pageviews by users. To count up all the Alices and Bobs, and then calculate the median (and 95th, 99th percentile) on their averages. That actually tells me, 50% of our users saw a page load of <1.4s, 95% <8s.

Having said all that, the data might actually look very similar to what we’re currently seeing. I’ll try to dig out some of our archived boomerang data, do some analysis, and post an update once I have more info.


Benchmarking Rackspace dedicated vs cloud

People keep telling me that Magento performs better on dedicated hardware. I haven’t been able to find any numbers to support this, but I’ve heard it so often it’s either a very popular myth, or it’s true.

Now that our Rackspace dedicated box is online, I’m trying some benchmarks toput some numbers against the comparison. I wanted to test different operating systems and server sizes, so I booted 10 servers, one of each Ubuntu 10.04 LTS and RHEL 5.5 at 0.5, 1, 2, 4, 8 GB.

In order to automate the setup, I ran the following command on the Ubuntu boxes:

mkdir .ssh && chmod 700 .ssh && echo ssh-rsa <<snip>> > .ssh/authorized_keys && chmod 600 .ssh/authorized_keys && locale-gen en_GB.UTF-8 && update-locale LANG=en_GB.UTF-8 && apt-get update && apt-get --yes dist-upgrade && apt-get --yes install sysbench screen && reboot now && exit

Then on the RHEL boxes, something vaguely similar:

mkdir .ssh && chmod 700 .ssh && echo ssh-rsa <<snip>> > .ssh/authorized_keys && chmod 600 .ssh/authorized_keys && yum -y update && rpm -Uvh && yum -y install screen sysbench && reboot && exit

To automate the actual testing, I created a script and uploaded it to each of the servers. It’s a simple nested for loop to run each test 3 times.


for threads in 1 4 8 16 32 64
	for r in 1 2 3
		sysbench --num-threads=$threads --test=cpu run > sysbench_cpu_${threads}_threads_$(date +%Y-%m-%d_%H-%M-%S).log
		sleep 30

	for r in 1 2 3
		sysbench --num-threads=$threads --test=memory run > sysbench_memory_${threads}_threads_$(date +%Y-%m-%d_%H-%M-%S).log
		sleep 30

	sysbench --num-threads=$threads --test=fileio --file-test-mode=rndrw prepare

	for r in 1 2 3
		sysbench --num-threads=$threads --test=fileio --file-test-mode=rndrw run > sysbench_fileiorndrw_${threads}_threads_$(date +%Y-%m-%d_%H-%M-%S).log
		sleep 30

Then I connected to each server, uploaded the script (actually copied / pasted into vim, seemed quicker) chmod +x and ran it. The scripts are running now on 10 machines…


I started writing this post about 2 months ago and haven’t yet published it. The bottom line was that memory comparisons were roughly even between virtual and physical environments. However, disk IO was hugely variable on the virtual hardware, at the top end it was comparable to the dedicated hardware, at the bottom end about 10% of that.

I didn’t notice any difference between operating systems, but I didn’t look for it very hard either. The results for the cloud servers were all over the place, while the dedicated box was very consistent.

My takeaway result was that disk is unpredictable in the cloud. If you’d like to see the actual results to make a more detailed analysis, let me know in the comments and I’ll dig out the numbers. For now I’m going to finally publish this! 🙂

CloudFlare slowed down our site

I deployed CloudFlare onto one of our sites today. I wanted to see hands on exactly how it works. I ran a simple benchmark. I ran 3 test 4 times each. The three tests were all from the same location, at 3 different network speeds. I ran the test 4 times, once in each configuration. I had CloudFlare enabled and disabled, and static assets coming from 3 domains or 1 domain. Here’s the results:

In every case, with CloudFlare was slower than without. Every single comparison, I ran the numbers. Some comparisons were very close, 0.02 or 0.03 seconds in it. But CloudFlare did not come out ahead one single time.

It could be a network issue. Maybe the test server ( / Gloucester) is very close to our server and far from the CloudFlare servers. But even so, I’d have expected some performance gain from all the “magic” CloudFlare is supposed to do.

I’ll update with further tests. I’m also going to email CloudFlare and ask for their comments. I’ll post anything salient here.

[ Update: I’ve published some further test results here. ]