Benchmarks
From VipsWiki
VIPS SMP benchmark
This benchmark is useful for testing the VIPS SMP system, for comparing host systems SMP implementations, and for testing for performance regressions or improvements between versions of VIPS. We have a separate set of benchmarks comparing VIPS to other image processing systems on the Speed and Memory Use page.
VIPS (from version 7.11.12) includes a benchmark for testing SMP systems. This benchmark is adapted from the system used to generate images for The National Gallery's Print on demand service. We have a couple of presentations about the background to POD available as well.
Images from a 10k by 10k studio digital camera are colour processed, resized, cropped and sharpened. You can see the exact sequence of operations the benchmark performs in the source code. This thing was originally processing images off a remote server over a 100 MBit network. No attempt was made to make it quick (there was no point); you could make im_benchmark a lot faster very easily if that was your aim.
There's a small shell script in vips-7.x/benchmark which runs im_benchmark on a test image with varying numbers of threads and reports the times. After building and installing VIPS you can type:
cd vips-7.x cd benchmark ./benchmarkn.sh
And see results for your system.
Results summary
| Processor | Clock (GHz) | CPUs | Time (s) | Speedup |
|---|---|---|---|---|
| Xeon X5600 | 2.8 | 4c | 1.08 | 13 x |
| Itanium2 | ? | 64 | 1.1 (est.) | 39.4 x |
| Opteron 8220 (64 bit) | 3.0 | 8 | 1.96 | 7.63 x |
| Dual quad-core intel (64 bit) | 3.0 | 8 | 2.8 | 7 x |
| Core 2 Extreme Quad (32 bit) | 2.66 | 4 | 3.69 | 3.78 x |
| Opteron 850 (HP server) | 2.4 | 4 | 4.25 | 3.7 x |
| Opteron 254 (HP workstation) | 2.7 | 2 | 6.4 | 1.9 x |
| P4 Xeon (64 bit) | 3.6 | 2 (4 ht) | 7 | 2.4 x |
| Core Duo (iMac) | 2.0 | 2 | 11.5 | 1.85 x |
| P4 Xeon (32 bit) | 3.0 | 2 (4 ht) | 19.7 | 1.6 x |
| PM (HP laptop) | 1.8 | 1 | 31.8 | -- |
| P4 (Dell desktop) | 2.4 | 1 | 36.6 | -- |
| EeePC atom/ssd | 1.6 | 1 (2 ht) | 41.5 | 1.6 x |
Time is real time (wall clock time) in seconds, Speedup is (real-1-cpu-time / real-max-cpu-time).
Results in detail
The results we've collected. Please paste more here.
For each one we've noted uname -a, gcc --version and vips --version.
We configured VIPS with no extra optimisation options, ie. everything just has the default -O2.
2 x Xeon X5600 (64bit), 2.8GHz running Ubuntu 9.04 server
=== 2 x Opteron 254 (64 bit), 2.7 GHz === IM_CONCURRENCY=1 13.25user 0.36system 0:13.97elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k 368inputs+126936outputs (3major+34118minor)pagefaults 0swaps 13.15user 0.39system 0:13.98elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126936outputs (0major+34121minor)pagefaults 0swaps IM_CONCURRENCY=2 13.31user 0.31system 0:07.02elapsed 193%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126944outputs (0major+29047minor)pagefaults 0swaps 9.50user 0.19system 0:04.99elapsed 194%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126936outputs (0major+38221minor)pagefaults 0swaps IM_CONCURRENCY=3 10.38user 0.29system 0:03.69elapsed 288%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126936outputs (0major+29355minor)pagefaults 0swaps 8.76user 0.22system 0:03.18elapsed 282%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126936outputs (0major+29056minor)pagefaults 0swaps IM_CONCURRENCY=4 9.40user 0.32system 0:02.48elapsed 390%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126936outputs (0major+28875minor)pagefaults 0swaps 13.51user 0.21system 0:03.55elapsed 385%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126936outputs (0major+28957minor)pagefaults 0swaps IM_CONCURRENCY=5 8.68user 0.19system 0:01.86elapsed 475%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126936outputs (0major+28349minor)pagefaults 0swaps 7.71user 0.20system 0:01.74elapsed 454%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126936outputs (0major+36092minor)pagefaults 0swaps 120.151 IM_CONCURRENCY=6 9.10user 0.13system 0:01.75elapsed 526%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126936outputs (0major+33039minor)pagefaults 0swaps 9.81user 0.16system 0:01.79elapsed 556%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126936outputs (0major+31385minor)pagefaults 0swaps IM_CONCURRENCY=7 10.69user 0.26system 0:01.72elapsed 636%CPU (0avgtext+0avgdata 0maxresident)k 8inputs+126936outputs (0major+33137minor)pagefaults 0swaps 8.64user 0.24system 0:01.38elapsed 643%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126936outputs (0major+34416minor)pagefaults 0swaps IM_CONCURRENCY=8 10.25user 0.26system 0:01.56elapsed 671%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126936outputs (0major+30208minor)pagefaults 0swaps 9.55user 0.30system 0:01.39elapsed 707%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126936outputs (0major+32512minor)pagefaults 0swaps IM_CONCURRENCY=9 8.74user 0.26system 0:01.08elapsed 831%CPU (0avgtext+0avgdata 0maxresident)k 8inputs+126936outputs (0major+34851minor)pagefaults 0swaps 8.55user 0.17system 0:01.14elapsed 758%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+126944outputs (0major+32756minor)pagefaults 0swaps
Linux mm-jcupitt2 2.6.20-15-generic #2 SMP Sun Apr 15 06:17:24 UTC 2007 x86_64 GNU/Linux gcc (GCC) 4.1.2 (Ubuntu 4.1.2-0ubuntu4) vips-7.12.1-Mon May 14 17:21:05 BST 2007 building test image ... tile=13 test image is 3770 by 5746 pixels starting benchmark ... chain=1 IM_CONCURRENCY=1 time -p vips im_benchmarkn temp.v temp2.v 1 real 12.70 user 12.21 sys 0.45 real 12.57 user 12.14 sys 0.40 vips im_avg temp2.v 120.134 IM_CONCURRENCY=2 time -p vips im_benchmarkn temp.v temp2.v 1 real 6.39 user 12.15 sys 0.44 real 6.38 user 12.15 sys 0.42 vips im_avg temp2.v 120.134
Pentium M (32 bit), 1.8 GHz
Linux banana 2.6.17-11-386 #2 Thu Feb 1 19:50:13 UTC 2007 i686 GNU/Linux gcc (GCC) 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5) vips-7.11.20-Tue Feb 13 13:47:53 GMT 2007 building test image ... tile=13 test image is 3770 by 5746 pixels starting benchmark ... chain=1 IM_CONCURRENCY=1 time -p vips im_benchmarkn temp.v temp2.v 1 real 31.83 user 31.41 sys 0.41 real 31.91 user 31.52 sys 0.37 vips im_avg temp2.v 120.134
Core Duo (32 bit), 2 GHz
Darwin pineapple.Belkin 9.4.0 Darwin Kernel Version 9.4.0: Mon Jun 9 19:30:53 PDT 2008; root:xnu1228.5.20~1/RELEASE_I386 i386 i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5465) vips-7.16.0-Thu Sep 4 11:43:34 BST 2008 building test image ... tile=13 test image is 3770 by 5746 pixels starting benchmark ... chain=1 IM_CONCURRENCY=1 time -p vips im_benchmarkn temp.v temp2.v 1 real 21.35 user 20.09 sys 1.34 real 21.37 user 20.09 sys 1.35 vips im_avg temp2.v 120.134 IM_CONCURRENCY=2 time -p vips im_benchmarkn temp.v temp2.v 1 real 11.62 user 20.76 sys 1.80 real 11.67 user 20.76 sys 1.86 vips im_avg temp2.v 120.134
4 x Opteron 850 (64 bit), 2.4 GHz
Linux roundtable 2.6.15-27-amd64-generic #1 SMP PREEMPT Fri Dec 8 17:50:54 UTC 2006 x86_64 GNU/Linux gcc (GCC) 4.0.3 (Ubuntu 4.0.3-1ubuntu5) vips-7.11.20-Mon Feb 12 18:05:51 GMT 2007 building test image ... tile=13 test image is 3770 by 5746 pixels starting benchmark ... chain=1 IM_CONCURRENCY=1 time -p vips im_benchmarkn temp.v temp2.v 1 real 16.19 user 15.48 sys 0.59 real 15.81 user 15.36 sys 0.52 vips im_avg temp2.v 120.134 IM_CONCURRENCY=2 time -p vips im_benchmarkn temp.v temp2.v 1 real 8.19 user 15.77 sys 0.47 real 8.33 user 15.95 sys 0.49 vips im_avg temp2.v 120.134 IM_CONCURRENCY=3 time -p vips im_benchmarkn temp.v temp2.v 1 real 6.18 user 15.82 sys 0.46 real 6.04 user 15.95 sys 0.53 vips im_avg temp2.v 120.134 IM_CONCURRENCY=4 time -p vips im_benchmarkn temp.v temp2.v 1 real 4.35 user 16.11 sys 0.55 real 4.25 user 15.86 sys 0.56 vips im_avg temp2.v 120.134
2 x Xeon (32 bit), 3 GHz
2.6.9-42.0.3.ELsmp gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-3) vips-7.11.12-Fri Oct 6 13:15:22 BST 2006 IM_CONCURRENCY=1 time vips im_benchmark temp.v temp2.v real 0m35.270s user 0m34.366s sys 0m0.934s IM_CONCURRENCY=2 time vips im_benchmark temp.v temp2.v real 0m21.914s user 0m41.269s sys 0m1.681s IM_CONCURRENCY=3 time vips im_benchmark temp.v temp2.v real 0m20.598s user 0m57.306s sys 0m2.765s IM_CONCURRENCY=4 time vips im_benchmark temp.v temp2.v real 0m19.781s user 1m11.393s sys 0m4.246s
2 x Xeon (64 bit), 3.6 GHz
Linux turner 2.6.17-10-generic #2 SMP Tue Dec 5 21:16:35 UTC 2006 x86_64 GNU/Linux gcc (GCC) 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5) vips-7.11.18-Mon Dec 18 18:19:27 GMT 2006 building test image ... tile=13 test image is 3770 by 5746 pixels starting benchmark ... chain=1 IM_CONCURRENCY=1 time -p vips im_benchmarkn temp.v temp2.v 1 real 17.60 user 16.58 sys 0.65 real 17.12 user 16.63 sys 0.59 vips im_avg temp2.v 120.134 IM_CONCURRENCY=2 time -p vips im_benchmarkn temp.v temp2.v 1 real 9.01 user 17.18 sys 0.78 real 8.99 user 17.12 sys 0.76 vips im_avg temp2.v 120.134 IM_CONCURRENCY=3 time -p vips im_benchmarkn temp.v temp2.v 1 real 7.78 user 22.02 sys 0.83 real 7.79 user 21.99 sys 1.00 vips im_avg temp2.v 120.134 IM_CONCURRENCY=4 time -p vips im_benchmarkn temp.v temp2.v 1 real 7.03 user 25.74 sys 1.16 real 7.02 user 25.60 sys 1.25 vips im_avg temp2.v 120.134
1 x P4, 2.4 GHz
MINGW32_NT-5.1 MM-DDAVIES1 1.0.10(0.46/3/2) 2004-03-15 07:17 i686 unknown gcc.exe (GCC) 3.4.2 (mingw-special) vips-7.11.17-Wed Nov 29 12:01:14 GMTST 2006 building test image ... tile=13 test image is 3770 by 5746 pixels starting benchmark ... chain=1 IM_CONCURRENCY=1 time -p vips im_benchmarkn temp.v temp2.v 1 real 36.59 user 0.01 sys 0.01 real 36.68 user 0.01 sys 0.01 vips im_avg temp2.v 120.072
Intel Core 2 Extreme Quad Core (QX6700), 2.66 GHz
A quick benchmark (11x11 unsharp mark of a 10kx10k image) shows:
1 Thread 166s 2 threads 82s 3 threads 55s 4 threads 42s
ie a linear speed-up
Linux degas.ecs.soton.ac.uk 2.6.19-1.2911.fc6 #1 SMP Sat Feb 10 15:51:47 EST 2007 i686 i686 i386 GNU/Linux gcc (GCC) 4.1.1 20070105 (Red Hat 4.1.1-51) vips-7.11.20-Fri Mar 2 12:47:29 GMT 2007 building test image ... tile=13 test image is 3770 by 5746 pixels starting benchmark ... chain=1 IM_CONCURRENCY=1 time -p vips im_benchmarkn temp.v temp2.v 1 real 15.73 user 14.70 sys 0.30 real 13.96 user 13.86 sys 0.27 vips im_avg temp2.v 120.134 IM_CONCURRENCY=2 time -p vips im_benchmarkn temp.v temp2.v 1 real 7.15 user 14.02 sys 0.23 real 7.12 user 13.96 sys 0.29 vips im_avg temp2.v 120.134 IM_CONCURRENCY=3 time -p vips im_benchmarkn temp.v temp2.v 1 real 4.77 user 13.98 sys 0.26 real 4.78 user 13.97 sys 0.25 vips im_avg temp2.v 120.134 IM_CONCURRENCY=4 time -p vips im_benchmarkn temp.v temp2.v 1 real 4.28 user 13.65 sys 0.27 real 3.69 user 14.06 sys 0.28 vips im_avg temp2.v 120.134
8 x Opteron 8220 (64 bit), 3.0 GHz
Linux raphael 2.6.22-10-generic #1 SMP Wed Aug 22 07:42:05 GMT 2007 x86_64 GNU/Linux gcc (GCC) 4.1.3 20070825 (prerelease) (Ubuntu 4.1.2-15ubuntu3) vips-7.12.4-Fri Aug 31 12:02:06 BST 2007 building test image ... tile=13 test image is 3770 by 5746 pixels starting benchmark ... chain=1 IM_CONCURRENCY=1 time -p vips im_benchmarkn temp.v temp2.v 1 real 15.04 user 14.64 sys 0.62 real 15.22 user 14.83 sys 0.72 vips im_avg temp2.v 120.134 IM_CONCURRENCY=2 time -p vips im_benchmarkn temp.v temp2.v 1 real 7.44 user 14.29 sys 0.61 real 7.01 user 13.36 sys 0.44 vips im_avg temp2.v 120.134 IM_CONCURRENCY=3 time -p vips im_benchmarkn temp.v temp2.v 1 real 4.58 user 13.29 sys 0.44 real 4.92 user 14.22 sys 0.44 vips im_avg temp2.v 120.134 IM_CONCURRENCY=4 time -p vips im_benchmarkn temp.v temp2.v 1 real 3.65 user 13.90 sys 0.60 real 3.93 user 14.59 sys 0.51 vips im_avg temp2.v 120.134 IM_CONCURRENCY=5 time -p vips im_benchmarkn temp.v temp2.v 1 real 2.98 user 14.25 sys 0.40 real 2.79 user 13.28 sys 0.38 vips im_avg temp2.v 120.134 IM_CONCURRENCY=6 time -p vips im_benchmarkn temp.v temp2.v 1 real 2.45 user 13.95 sys 0.42 real 2.32 user 13.07 sys 0.46 vips im_avg temp2.v 120.134 IM_CONCURRENCY=7 time -p vips im_benchmarkn temp.v temp2.v 1 real 12.57 user 13.43 sys 0.50 real 3.06 user 17.55 sys 0.58 vips im_avg temp2.v 120.134 IM_CONCURRENCY=8 time -p vips im_benchmarkn temp.v temp2.v 1 real 1.97 user 14.00 sys 0.44 real 2.16 user 15.31 sys 0.58 vips im_avg temp2.v 120.134
Asus Eee PC 1000 atom n270 1.6GHz SSD
Linux km-bigee 2.6.27-7-eeepc #1 SMP Fri Oct 31 11:36:36 MDT 2008 i686 GNU/Linux IM_CONCURRENCY=1 time -p vips im_benchmarkn temp.v temp2.v 1 real 69.22 user 67.38 sys 1.03 real 70.64 user 67.76 sys 1.04 IM_CONCURRENCY=2 time -p vips im_benchmarkn temp.v temp2.v 1 real 42.09 user 76.66 sys 1.12 real 41.52 user 76.45 sys 1.10
Dual quad-core Intel E5320 (64-bit), 3 GHz
Intel Xeon E5320 x 2 so 8 cores.
vips-7.11.20-Wed Nov 28 11:39:32 GMT 2007 building test image ... tile=13 test image is 3770 by 5746 pixels starting benchmark ... chain=1 IM_CONCURRENCY=1 time -p vips im_benchmarkn temp.v temp2.v 1 real 19.89 user 19.43 sys 0.33 real 20.41 user 19.45 sys 0.35 vips im_avg temp2.v 120.134 IM_CONCURRENCY=2 time -p vips im_benchmarkn temp.v temp2.v 1 real 10.01 user 19.61 sys 0.43 real 10.39 user 19.58 sys 0.41 vips im_avg temp2.v 120.134 IM_CONCURRENCY=3 time -p vips im_benchmarkn temp.v temp2.v 1 real 6.81 user 19.78 sys 0.38 real 6.78 user 19.80 sys 0.37 vips im_avg temp2.v 120.134 IM_CONCURRENCY=4 time -p vips im_benchmarkn temp.v temp2.v 1 real 5.17 user 19.87 sys 0.41 real 6.82 user 19.62 sys 0.37 vips im_avg temp2.v 120.134 IM_CONCURRENCY=5 time -p vips im_benchmarkn temp.v temp2.v 1 real 4.31 user 19.96 sys 0.46 real 5.16 user 19.79 sys 0.39 vips im_avg temp2.v 120.134 IM_CONCURRENCY=6 time -p vips im_benchmarkn temp.v temp2.v 1 real 3.48 user 19.83 sys 0.43 real 7.03 user 19.94 sys 0.40 vips im_avg temp2.v 120.134 IM_CONCURRENCY=7 time -p vips im_benchmarkn temp.v temp2.v 1 real 3.76 user 19.98 sys 0.44 real 3.68 user 19.86 sys 0.41 vips im_avg temp2.v 120.134 IM_CONCURRENCY=8 time -p vips im_benchmarkn temp.v temp2.v 1 real 2.84 user 20.06 sys 0.43 real 4.76 user 20.07 sys 0.48 vips im_avg temp2.v 120.134 IM_CONCURRENCY=9 time -p vips im_benchmarkn temp.v temp2.v 1 real 4.80 user 19.97 sys 0.47 real 2.79 user 20.04 sys 0.46 vips im_avg temp2.v 120.134
SGI Origin2000 supercomputer
VIPS 7.11.20 has also been run on a 64-CPU supercomputer (an SGI Origin2000) at Princeton. The results are:
| CPUs | Run time (s) | Speed up |
|---|---|---|
| 1 | 651.85 | 1 |
| 2 | 335.9 | 1.94 |
| 4 | 170.07 | 3.83 |
| 8 | 86.06 | 7.57 |
| 16 | 44.56 | 14.63 |
| 32 | 24.06 | 27.09 |
| 64 | 16.54 | 39.41 |
So about a 40 x speedup for 64 CPUs.
If you graph these numbers you get:
So it's pretty much linear up to about 30 CPUs (with a 27x speedup). The image being processed is 1.3GB so perhaps we are starting to see IO bandwidth limits.
