Thanks to the awesome folks at SOASTA we’re now using their mPulse system instead of our own boomerang install. This gives us 2 major wins. First, we’re including the tracking code in the non-blocking asynchronous iframe method, which gives the best possible performance at this point. Second, we can actually see the data. Previously, we just weren’t getting visibility into our boomerang data. We had the data, but weren’t using it, which was a total waste.
Looking at our stats today, mPulse tracks the median page load time. I was looking at the data and thinking, I wonder what it looks like per user. For example, I wonder if users with faster connections typically hit more pages. If they do, that means our median average user load time is actually lower than our median page load time.
Take two users, Alice and Bob. Alice is on her desktop in London with a 100Mbps line. She visits 8 pages. The average load time for her 8 pageviews is 1.2s. Bob is on his iPad over 3G in Alabama. (We’re in the UK, so London is closer!) Bob visits 4 pages. The average load time for the 4 pages is 2.3s. Now our arithmetic mean is somewhere in between the two, but our median, in this case, is one of Alice’s pageviews.
What would be really interesting, is to group pageviews by users. To count up all the Alices and Bobs, and then calculate the median (and 95th, 99th percentile) on their averages. That actually tells me, 50% of our users saw a page load of <1.4s, 95% <8s.
Having said all that, the data might actually look very similar to what we’re currently seeing. I’ll try to dig out some of our archived boomerang data, do some analysis, and post an update once I have more info.
Today we’ve produced our first useful graphs from the 770k boomerang data points we have collected. This is one of the graphs we produced, and I’ll post it here only because it’s the first one I personally produced. At last, after 4 months we’re actually seeing data.
What does it tell us? That page load time is not very uniform. Next step, linear regression comparing page load time against user’s available bandwidth.
Following my last post about CloudFlare, I ran some further benchmarks in response to the feedback from their team. Here’s the summary:
- This only happens on some user agents on some operating systems, CloudFlare will not give me a list of user agents.
- On browsers where this is enabled, we see a marked improvement, where it’s not, we see no gain or a small loss.
I ran two sets of tests (using only browsers where RocketLoader is enabled).
We probably won’t implement CloudFlare across all our sites. I might still experiment on one of our higher traffic sites now that we’re running boomerang and gathering real user data to compare. However, the black box nature of CloudFlare fundamentally leaves me feeling uneasy.
The product appears to be in beta, which wasn’t clear when we signed up. I thought it was a polished product ready for production use. But much of the support chat is about RocketLoader being in beta, no list of user agents at this time, etc.
Bottom line, CloudFlare hasn’t done what I expected. I’ll test mod_pagespeed and we’ll probably go with that, pending any major roadblocks.