Calibration of the Sun Wireless Toolkit

Background
In my last post, I explained how to calibrate the Sun Wireless Toolkit emulator against a given MIDlet market share.

In this post I show provide simple calibration results from the Sun Wireless Tookit 2.2.

Results
The plot shows how the JBenchmark 2 score changes with VM execution speed for an emulated heap of 1MB.


The first thing to note is that, for an emulated speed of 1000 bytecode instructions/second, the JB2 score is only 120. This means that the toolkit can only provide accurate emulation of approximately 45% of the MIDP 2 terminals on the market today.

Next Steps
The graph, above, needs a whole lot more data points to be useful. The results are taken from a single PCs. It is not clear to me whether results differ on different PCs.

Thanks to Doug Wright at uiActive who actually ran the tests - my PC is still rather sick.

Wireless Toolkit Calibration for Real World Market Share

Applied use of Benchmark Results
From my previous posts here and here, it is possible to use a database of MIDP benchmark results to examine how market share varies according to the MIDlet heap memory and performance requirements.

In this post, I explain how to apply this information in order to calibrate the Sun Wireless Tookit against a particular handset market share.

What Problem Are We Solving?
In my experience, developers spend most of their time on a desktop environment, using an IDE like Eclipse with the Eclipse ME plugin to create and debug a MIDlet. Once the basic development and testing is complete, the MIDlet is transferred to a handset for testing.

Issues arise because of the gulf between the emulator performance and real world handset performance. Calibrating the emulator so that it performs in a similar way to the target handset can help to identify and eliminate performance related problems earlier in the development cycle. Simply put, developers become aware of problems during the development stage, rather than at a later test phase.

Sun Wireless Toolkit Support
The Sun Wireless Toolkit provides all of the tools required to adjust the emulator performance. The user can change:

1. VM speed emulation;
2. graphics latency,
   
3. network throughput; and,
4. heap size.
   

Heap is simple enough. However, for performance, there is no simple mapping between the number of bytecodes/millisecond and real world handset performance. Until now, that is.

Calibration How To
Calibration of the emulator is achieved as follows:

1. Download the JBenchmark 2 MIDlet JAD and JAR files from here; or, point your emulator at http://wap.jbenchmark.com/JBenchmark2.jad.
   
2. Decide the MIDlet market share you require.
3. Determine a heap memory and JB2 score (perhaps using the estimation tools I provided here).
4. Set the heap memory using the Wireless toolkit.
5. Check the 'Enable VM Speed Emulation' box.
6. Set the slider to 500.
7. Run the JB2 MIDlet.
8. Watch Mona Lisa, the little jumping guy, the animated 3D knight, and those those MIDP forms that nobody uses.
9. When the benchmark finishes look at the JB2 score.
10. If your KB2 score is greater than your required score, set your emulator slider to 250, if less than your required score, set it to 750.
    
11. Run the MIDlet again and repeat using a binary chop until you have an emulation speed that matches your market share.

The following screenshots are from my N70 showing the start and results page.

Before anyone asks, I used my N70 because my PC CPU fan just parted company with the CPU to which it should be bonded. As a result, I'm using the home MacBook to write this article and I can't actually get the screenshots from the emulator.

What Can Go Wrong?
The Sun Wireless Toolkit only enables VM emulation in the range 100-1000. This range is somewhat arbitrary and seems to have no grounding in real world handset performance. Thus, you may well find that the 1000 mark is too slow for your target handset JB2 score.

Next Steps
Once I get that CPU re-attached, and assuming there is no other long-term damage to my PC, I will update this post with some real mappings between JB2 and emulator bytecode/second scores. I'll then try and get someone else to do the same thing in the hope that there is a simple, globally applicable relationship between JB2 score and emulator settings (wouldn't that be nice!)

I keep saying that I am going to post market share results based on handset brand, and I will. What I want to do is embed an applet in this blog so readers can select any handset brand etc and generate their own chart. I just have to read how to embed an applet in a blog page...

Visualising MIDP Performance Change

Background
Previously, I posted a density plot that demonstrates how MIDlet market share changes as a function of handset heap memory and performance (as measured by the JBenchmark 2 score).

In this post I provide a visualisation showing how market share has changed over time.

Visualisation
The video, below, shows how performance is improving over time.



The video shows how application market share is affected by heap and performance. Each frame is based on one year's worth of data, up to, and including the month shown. For this reason, there is a large amount of noise in the early frames as there were very few MIDP 2 handsets in 2004.

To obtain a very large market share (90+%) of handsets released in 2007, you have to target pretty much the same performance criteria as you did in 2004. In the mid range, handset performance can be seen to be steadily improving.

What Next?
Next, I will run the numbers for Nokia, Motorola, Samsung and Sony Ericsson handsets.