Perl Template Roundup October 2010: FAQ

This section attempts to anticipate some likely questions.

If you find that you have a query that isn't answered below, please ask and I'll get an answer to you and update the FAQ with it.

How was the data for these benchmarks gathered?

You probably want to read the methodology section if you want the details, but the short answer is with the Template::Benchmark module.

Why isn't template engine X represented?

All benchmarking is done with the Tempate::Benchmark module, so the most likely answer is that there isn't a plugin for that template engine.

If there is a plugin available, it may be that it wasn't available at the time that the benchmarks were generated.

There may be several reasons why there is no plugin available, firstly Template::Benchmark can only benchmark Perl template engines, so if the template engine isn't available in Perl, it won't be benchmarked.

If the template engine is entangled with a web-application framework, then it's possible that Template::Benchmark cannot isolate just the templating bits.

If the template engine has an unusual paradigm beyond just that of a general-purpose text templating system, it may not be possible for Template::Benchmark to benchmark it in a manner both consistent and fair to it and the other engines.

It may just be that the author of Template::Benchmark (that'd be me) hasn't heard of the template system in question and would be only too glad to add a plugin, or to accept one as a patch.

Why do some template engines show no performance drop-off?

Some of the benchmarked template features are highly optimizable, and those template engines that do optimize those situations will often show highly similar benchmark speeds across all repeats values.

This is quite normal and the results are accurate.

What do the performance numbers represent?

The numbers in the benchmarks are the number of times per second the benchmark operation could be performed.

While they do have a concrete meaning, I've chosen to leave off units because it's not really very important or useful to know in isolation that Template Engine X can perform 15,432 operations per second for a particular set of data on my particular hardware.

What is important is that the numbers are comparable to each other, which they are, not what their absolute values are.

If you really need to know in absolute terms how many templates per second you are going to get from a given template engine, you really need to benchmark it yourself, with your own data, on your own hardware.

What sort of error margin do the results have?

It appears 10% or below from casual observation, however given that it takes over 4 days to run the full set of benchmarks, I've not had time to perform an accurate analysis of the variation.

I'm quite, however unscientifically, confident of the 10% though. How confident you choose to be in my confidence is of course up to you...

What rounding is applied to the numbers?

All performance benchmarks are rounded to three significant figures before further processing, this makes the numbers somewhat more presentable and is still more precise than the accuracy of the benchmarks themselves.

Why haven't you got a graph of X vs Y?

I've chosen what I feel to be the most useful set of graphs for most people's use.

There's at least six axes of data that could be plotted, and potentially more if you do some more specific collation and analysis, it's possible to create well over forty thousand charts if you permute through those axes, even with performance fixed as one axis.

I'm also slightly constrained in what I can convince the Google Charts API to produce.

Why do you use Google Charts rather than ...?

It mostly did what I wanted, and more importantly I don't have to host the large number of chart images on my site - I simply don't have the bandwidth or quota for that.

Why aren't those quartiles balanced?

The quartiles totals on the statistics page aren't balanced because the underlying charts that the quartiles are calculated from simply don't have enough entries to provide an even distribution.

If you have a chart with 3 entries, you're going to struggle to have an entry in each quartile: at least one of them is going to be empty. If you have a chart with 5 entries, you're (most likely) going to have a quartile with one more than the others.

These differences soon add up across the 9000 entries counted in the grand summary.

As if that wasn't enough, in the case of a tied ranking between two or more entries, all those entries will end up dumped in the same quartile, which causes an even more "lumpy" distribution.

So, don't expect an even distribution in the totals lines, this is perfectly normal although it seems counter-intuitive at first glance.

© 2009-2013 Sam Graham, unless otherwise noted. All rights reserved.