Wireless Toolkit Calibration for Real World Market Share

Applied use of Benchmark Results
From my previous posts here and here, it is possible to use a database of MIDP benchmark results to examine how market share varies according to the MIDlet heap memory and performance requirements.

In this post, I explain how to apply this information in order to calibrate the Sun Wireless Tookit against a particular handset market share.

What Problem Are We Solving?
In my experience, developers spend most of their time on a desktop environment, using an IDE like Eclipse with the Eclipse ME plugin to create and debug a MIDlet. Once the basic development and testing is complete, the MIDlet is transferred to a handset for testing.

Issues arise because of the gulf between the emulator performance and real world handset performance. Calibrating the emulator so that it performs in a similar way to the target handset can help to identify and eliminate performance related problems earlier in the development cycle. Simply put, developers become aware of problems during the development stage, rather than at a later test phase.

Sun Wireless Toolkit Support
The Sun Wireless Toolkit provides all of the tools required to adjust the emulator performance. The user can change:

1. VM speed emulation;
2. graphics latency,
   
3. network throughput; and,
4. heap size.
   

Heap is simple enough. However, for performance, there is no simple mapping between the number of bytecodes/millisecond and real world handset performance. Until now, that is.

Calibration How To
Calibration of the emulator is achieved as follows:

1. Download the JBenchmark 2 MIDlet JAD and JAR files from here; or, point your emulator at http://wap.jbenchmark.com/JBenchmark2.jad.
   
2. Decide the MIDlet market share you require.
3. Determine a heap memory and JB2 score (perhaps using the estimation tools I provided here).
4. Set the heap memory using the Wireless toolkit.
5. Check the 'Enable VM Speed Emulation' box.
6. Set the slider to 500.
7. Run the JB2 MIDlet.
8. Watch Mona Lisa, the little jumping guy, the animated 3D knight, and those those MIDP forms that nobody uses.
9. When the benchmark finishes look at the JB2 score.
10. If your KB2 score is greater than your required score, set your emulator slider to 250, if less than your required score, set it to 750.
    
11. Run the MIDlet again and repeat using a binary chop until you have an emulation speed that matches your market share.

The following screenshots are from my N70 showing the start and results page.

Before anyone asks, I used my N70 because my PC CPU fan just parted company with the CPU to which it should be bonded. As a result, I'm using the home MacBook to write this article and I can't actually get the screenshots from the emulator.

What Can Go Wrong?
The Sun Wireless Toolkit only enables VM emulation in the range 100-1000. This range is somewhat arbitrary and seems to have no grounding in real world handset performance. Thus, you may well find that the 1000 mark is too slow for your target handset JB2 score.

Next Steps
Once I get that CPU re-attached, and assuming there is no other long-term damage to my PC, I will update this post with some real mappings between JB2 and emulator bytecode/second scores. I'll then try and get someone else to do the same thing in the hope that there is a simple, globally applicable relationship between JB2 score and emulator settings (wouldn't that be nice!)

I keep saying that I am going to post market share results based on handset brand, and I will. What I want to do is embed an applet in this blog so readers can select any handset brand etc and generate their own chart. I just have to read how to embed an applet in a blog page...