Calibration of the Sun Wireless Toolkit

Background
In my last post, I explained how to calibrate the Sun Wireless Toolkit emulator against a given MIDlet market share.

In this post I show provide simple calibration results from the Sun Wireless Tookit 2.2.

Results
The plot shows how the JBenchmark 2 score changes with VM execution speed for an emulated heap of 1MB.


The first thing to note is that, for an emulated speed of 1000 bytecode instructions/second, the JB2 score is only 120. This means that the toolkit can only provide accurate emulation of approximately 45% of the MIDP 2 terminals on the market today.

Next Steps
The graph, above, needs a whole lot more data points to be useful. The results are taken from a single PCs. It is not clear to me whether results differ on different PCs.

Thanks to Doug Wright at uiActive who actually ran the tests - my PC is still rather sick.

Visualising MIDP Performance Change

Background
Previously, I posted a density plot that demonstrates how MIDlet market share changes as a function of handset heap memory and performance (as measured by the JBenchmark 2 score).

In this post I provide a visualisation showing how market share has changed over time.

Visualisation
The video, below, shows how performance is improving over time.



The video shows how application market share is affected by heap and performance. Each frame is based on one year's worth of data, up to, and including the month shown. For this reason, there is a large amount of noise in the early frames as there were very few MIDP 2 handsets in 2004.

To obtain a very large market share (90+%) of handsets released in 2007, you have to target pretty much the same performance criteria as you did in 2004. In the mid range, handset performance can be seen to be steadily improving.

What Next?
Next, I will run the numbers for Nokia, Motorola, Samsung and Sony Ericsson handsets.

MIDP 2 Performance Frequency Distribution

Background
Previously, I looked at performance percentiles for MIDP 2 handsets over time and said I would provide an analysis of performance by handset vendor. Instead, I will stay at a higher level and look at some more descriptive statistics regarding MIDP 2 performance (JB2 scores) instead over the period January 2004 through July 2007.

The Analysis
The figure below shows the frequency distribution and cumulative frequency distribution of 609 handset JB2 scores.

The cumulative frequency curve approximates the percentile plot I presented in a prior post.

The distribution of JB2 scores has a strong negative skew (γ₁ = 1.07795). That is, more data is concentrated in the left tail than would be expected in a normally distributed data set. The distribution is also 'peaky' with kurtosis (γ₂ = 0.43981). This tells us that, as developers, we should be looking at the median rather than the arithmetic mean when making a decision regarding market share for our MIDlet.

In the JB2 data set, the arithmetic mean of 173.15 JB2 score is significantly greater than the median of 130. If we had naively targeted a score of 173 and expected a 50% market share, we would actually achieve a market share of approximately 35%. (Bear in mind also that the original Motorola RAZR has a JB2 score of only 49.)

Ranking of Release Date and Performance
In order to better visualise just how poor the relationship between release date and MIDP 2 performance is, the rank of the handset release date is plotted against the rank of the corresponding JB2 score. Ranks are normalised in the range 0 through 1.

This plot is another simple way of showing that there is no relationship between handset performance as measured by the JB2 benchmark and release date. The data set has a Kendall tau rank correlation coefficient of τ = 0.196 indicating a poor correlation.

Summary
What this and my previous posts tell us is that as developers we cannot build our mobile application with an expectation that handset performance is doubling every year. If we assume that the 609 handsets with a JBenchmark 2 score that were used in this analysis are representative of the entire population of MIDP 2 terminals, then we must continue to target low performance terminals.

As I said previously, this analysis is simplistic because it does not take into account market share, memory and other considerations. I will publish an analysis of these factors in coming weeks.

MIDP Performance over Time

Handset Performance
In my last post, I examined the gross relationship between MIDP performance over time by plotting JBenchmark 2 scores against handset release date as well as plotting percentile scores. The results are interesting for the fact that, at the low end, there has been very little change in performance over the past three and a half years to July 2007.

In this post I delve further into changes in MIDP 2 performance over time.

Performance Percentiles by Year
Performance percentiles are an extremely useful tool through which developers can target a particular handset market share and identify type model handsets for testing. The plot, below, shows performance percentiles by year. It was generated from the same JBenchmark2 data set of 609 handsets from January 2004 through July 2007 as my previous analysis.

This plot reinforces the fact that there has been little change in performance the low end. The bottom 20% of handsets released in 2007 have no better performance than the bottom 20% did in 2004.

Performance in the Mid-Range
Note the flattening of the percentile plots from 2004 - which is close to exponential - to 2007 - which is much closer to a linear relationship. This flattening is good news for developers as it means that mainstream handsets have seen a significant improvement in performance as compared to the top-end terminals.

What this Does Not Say
The information presented thus will enable managers to clearly define a handset market share based on either a percentile, which gives a performance specification, or a handset terminal, which gives a score through which market share can be derived.

The market share derived in such a way will always be biased unless weighted by vendor market share, or rather, market share of particular terminals. Terminal sales numbers are notoriously difficult to obtain for a given market (iPhone aside). However, vendor market share figures are published and will be used in my next post to look at what performance to expect from Nokia, Motorola, Samsung and Sony Ericsson terminals (the top 4 in Australia, where I live).

Targeted Development for Mass Market MIDP

Where To Start?
The problem of J2ME MIDP fragmentation is well publicised and I don’t intend to bore you by re-hashing the same old complaints. Instead, I want to present an analysis that will assist any developers who wish to create a mass-market application for J2ME MIDP terminals.

Say you want to create a MIDlet that will run on at least half of the MIDP terminals in the world. Where do you start?

You need to know two things:

1. What is the Performance (CPU/JVM) of the terminal at the 50th percentile of all MIDP terminals; and,
   
2. What is the minimum heap memory for any terminal with a speed of greater than or equal to the Performance identified in Step 1.
   

Of course you can turn these questions around the other way.

Analysis
Whereas heap memory for MIDP handsets can be gleaned from manufacturer web sites, performance cannot. For this reason, I will use a MIDP Benchmark result instead.

In this analysis, I use JBenchmark 2 (JB2) scores as a proxy for performance. I chose JB2 as there is a large database of published results that are freely available and also because over the years I have found it to be a good indicator of overall performance (which is, after all, what we want). In a later post, I intend to examine the various MIDP benchmarks to see whether they measure the same thing.

Unless otherwise stated, the results used in this analysis are drawn from a set of 609 JB2 scores that encompass MIDP 2 handsets released between February 2004 and June 2007 inclusive. Note that I use the JB2 First Test date as a proxy for handset release date in the following analysis. This means that my results are only as good as the folks who tested real handsets.

Change in MIDP Performance over Time
The figure below shows the progression in JB2 scores over time for the 609 MIDP 2 handsets terminals.
Interesting eh? What this shows is that the fastest terminals have become faster; slow terminals are still being released into the market in large numbers; and, that there has been no significant increase in performance at the low end since at least 2004.

The following figure shows the JB2 score percentiles.

The interesting fact about this graph is again, the low end. Roughly 30% of all MIDP handsets ever released have a JB2 score of less than 50. This is about the speed of the original Motorola RAZR (anyone who has worked with this handset will know that it is very slow indeed). 2007 handsets with a JB2 score of less than 50 include:

• Nokia 2610 and Nokia 6103
  
• Motorola SLVR L9 and Motorola L2
• LG KG200
• Samsung SGH-Z1oo, Samsung SGH-X160
  

Some of these are modern, mass market terminals. By the way, Samsung handsets occupy 18 of the bottom 20 handset slots with JB2 scores as low as 8.

Using the figure above, it is possible to answer the question:

If I wish to support 50% of all of the MIDP terminals ever released, what is the minimum performance my application must target?

Using 50% as an example, you can read from the graph that your MIDlet must work on a handset with a JB2 score of 130 or higher. If you wish to target 70%, the JB2 score falls to 57.

Type Model Handsets for Development and Testing
Given that you know your handset performance, you can use the JB2 Scores database to identify a Type Model handset for development and acceptance testing.

Again, using the JB2 score of 130 (50%), your developers could use the following handsets:

• Nokia 8800 (JB2 129)
• Samsung SPH-A800 (JB2 130)
• Motorola i605 (JB2 116)
  
• Sony Ericsson Z1010 (JB2 130)
• LG F2250 (JB2 117)
  

Using Type Models like this can eliminate a great deal of development pain as your developers know exactly what they are dealing with and that no terminal of a lesser capability will ever be supported.

You can identify Type Models yourself for a given percentile using the graph above and the JBechmark results database.

Simplistic?
Basing a specification on the results of these figures is rather simplistic of course. This is because the JB2 performance figures represent the entire range of MIDP 2 handsets from 2004 onwards; they do not represent the current state of the market.

That said, it is a starting point from which I will build a more complete picture in the posts that follow. You can expect to see from me:

• Handset memory analysis.
• Performance by handset vendor.
• Weighted analysis by vendor and handset market share (i.e. the the real answer).

Caveat Emptor
In this analysis two major assumptions are made:

1. The JBenchmark 2 score is representative of the handset under test's real performance characteristics.
2. The subset of MIDP 2 handsets with JBechmark 2 scores is representative of the entire universe of MIDP 2 handsets.

Finally, whereas I have done everything I can to ensure the accuracy of the results presented, there may be errors in the analysis and you should not rely on them without independently checking their veracity.