The MIDP Market Share-o-Matic Estimator

Where Were We?
Previously, I looked at MIDP 2 performance and heap memory and provided tools to estimate market share based off either of these measures. I also pointed out problems with any market share estimate based on just one of these measures.

In this post, I provide market share estimates based on both heap and performance criteria.

The Definitive MIDP 2 Market Share Estimator
And here it is...

Using the graph is simple. The X axis gives the heap memory (in this plot, I only show heap to 2.5 MB as it is below this figure where the rate in change in market share is greatest). The Y axis gives the JBenchmark 2 score (a measure of handset performance). The Z axis gives the market share percentile. Percentiles contour lines are plotted for convenience with the contour labels visible to the right of of the Y axis. The small grey dots on the graph show the data points from which the surface was interpolated.

Using the plot you can:

• Determine the addressable market for an existing application;
• Provide specifications for a MIDlet based on a required market share; or,
• Marvel at just how low the specifications are for a MIDlet that is really mass market.

Notable Features
This graph brings into sharp contrast just how important coding for speed and memory efficiency is in the MIDP world if you really want to address a large market share. At the low end, a small increase in MIDlet heap usage causes a significant loss in market share.

The graph is also notable for exactly the opposite reason.

Suppose you agree up front with your developer to live with a target of 768 KB heap and a 100 JB2 score (i.e, you want a market share of 50%). Then, say your developer then tells you that the completed MIDlet actually requires 1MB heap. The graph shows us that this 33% memory overrun will only cost 5% market share. Then, your developer tells you that, ahem, the finished MIDlet only runs on handsets with a JB2 score of 150!

A disaster? Not really. The graph shows us that the combined effect of the 33% memory and 50% performance overrun is a mere 10% reduction in market share.

Another way of saying this is that the rate in change in market share is greatest at low specification handsets. Given what we already know about the bottom 20% of the market (performance and heap), this should not come as a surprise.

Closing Thoughts
This analysis is useful, but needs extending to show similar graphs on a year by year basis. The analysis also needs to be performed for each of the major handset brands. And that is what I hope to provide next time.

A Trip Down MIDP Memory Lane

Changes in MIDP 2 Memory over Time
In this post, I will examine how heap memory has changed in MIDP 2 devices over time. I will also examine the link between heap memory and handset performance as measured by the JBenchmark 2 benchmark. Finally, I will look at why, as developers, we should be concerned with dynamic heap memory in MIDP terminals.

So We Get More Memory in our Handsets, Right?
The following figure shows the maximum heap memory for 609 MIDP 2 terminals with JBenchmark 2 scores over the period January 2004 through July 2007. Note that the memory axis has a log scale in the range 100 KB through 150,000 KB.
As with the performance plots I presented in an earlier post, this graph goes some way to dispelling the myth that all terminals have more memory in 2007 than they did in 2004. In fact, a large number of terminals are still being released into the market with a maximum heap of less than 1 MB. Indeed, it is astonishing that at least two 2007 vintage terminals report as little as 180 KB heap.

MIDP 2 Heap Memory Percentiles
Once again, from a developer perspective, heap memory percentiles are far more valuable than a scatter plot.
Again, our project manager can use this graph to provide his developer team with a handset terminal matching a required market share for the MIDlet under development. For example, there may be a requirement that the MIDlet address 60% of the MIDP 2 handsets released to July 2007, giving the developers 1 MB of heap at a minimum.

Indeed, a manager might extend this to provide a performance specification based on the JB2 percentile plots I presented previously (the JB2 target for a 60% market share is approximately 100).

Unfortunately, life is not that easy...

The Relationship between Performance and Heap Memory in MIDP Terminals
Our project manager, above, is assuming that performance and memory are ranked equally on MIDP 2 terminals.

The plot, below, shows that (unfortunately) there is no relationship between heap memory and performance as measured by the JBenchmark 2 score.
Rather annoyingly, what this says is if a you choose a 60% market share based off heap memory percentiles, your developers can salivate over a nice juicy 1 MB heap, but they then find that they must target a JB2 score of 38!

Unfortunately, things get even worse.

Dynamic Heap in MIDP 2
Many MIDP 2 terminals implement a memory sharing arrangement called dynamic or adaptive heap. This is where the heap is used for other applications as well as the JVM. What it means for a developer is that a clean handset that is used solely for test and development, may have a heap memory that is not representative of the same handset in the field.

The good news is that the folks at JBenchmark record the minimum and maximum heap seen from a handset under test, and these are from a variety of terminals 'in the wild'. Thus, the range in JBenchmark heap reports is likely to be a good indicator of real world terminal conditions. Maximum and minimum heap from 609 JBenchmark 2 results are shown, below.

Of the 609 handsets used in this study, 252 JBenchmark results (41%) show a difference between the high and low heap memory reported by a handset under test. So, as a developer, if you target a handset with 1 MB heap, and your MIDlet is optimised so that it uses every last byte of this space, you can expect no less than 40% of your terminal market to fail in the field as other applications eat into the precious heap via their shared memory architecture.

In some cases, the heap delta is very significant (the plotted point falls well below the 1:1 line). Of course, this will only be an issue when working with lower end terminals; losing 2 MB of a terminal with a maximum of 8 MB is likely to be far less important to the success of a MIDlet than losing 200 KB from a terminal with maximum of 1 MB heap. Dynamic heap is then an issue developers should be concerned about.

Where to From Here?
In my next post I will finally answer some questions concerning market share by presenting a density plot showing the market share than can be expected from a heap / performance pair.

MIDP 2 Performance Frequency Distribution

Background
Previously, I looked at performance percentiles for MIDP 2 handsets over time and said I would provide an analysis of performance by handset vendor. Instead, I will stay at a higher level and look at some more descriptive statistics regarding MIDP 2 performance (JB2 scores) instead over the period January 2004 through July 2007.

The Analysis
The figure below shows the frequency distribution and cumulative frequency distribution of 609 handset JB2 scores.

The cumulative frequency curve approximates the percentile plot I presented in a prior post.

The distribution of JB2 scores has a strong negative skew (γ₁ = 1.07795). That is, more data is concentrated in the left tail than would be expected in a normally distributed data set. The distribution is also 'peaky' with kurtosis (γ₂ = 0.43981). This tells us that, as developers, we should be looking at the median rather than the arithmetic mean when making a decision regarding market share for our MIDlet.

In the JB2 data set, the arithmetic mean of 173.15 JB2 score is significantly greater than the median of 130. If we had naively targeted a score of 173 and expected a 50% market share, we would actually achieve a market share of approximately 35%. (Bear in mind also that the original Motorola RAZR has a JB2 score of only 49.)

Ranking of Release Date and Performance
In order to better visualise just how poor the relationship between release date and MIDP 2 performance is, the rank of the handset release date is plotted against the rank of the corresponding JB2 score. Ranks are normalised in the range 0 through 1.

This plot is another simple way of showing that there is no relationship between handset performance as measured by the JB2 benchmark and release date. The data set has a Kendall tau rank correlation coefficient of τ = 0.196 indicating a poor correlation.

Summary
What this and my previous posts tell us is that as developers we cannot build our mobile application with an expectation that handset performance is doubling every year. If we assume that the 609 handsets with a JBenchmark 2 score that were used in this analysis are representative of the entire population of MIDP 2 terminals, then we must continue to target low performance terminals.

As I said previously, this analysis is simplistic because it does not take into account market share, memory and other considerations. I will publish an analysis of these factors in coming weeks.