Android Market Takes A Page from Amazon - Fails

For our regular readers, this post is quite different from normal, but it is something we have decided we can not ignore and that needs more attention from everyone who is a fan of Android, or Google in general.

Just recently (7/12/2011), Google started to release an update to the Android Market for devices 2.2 and above (a vast majority of devices).  This update includes a complete redesign and a lot of great new features that could help everyone participating in the market.  However, there is one change that is very negative all around for consumers and developers/publishers that will offset any of those positives in my opinion, but has not been discussed much.  This change is in how they are now choosing to try and "rank" user reviews of apps, and in many cases only show 20 of such reviews that they are deeming "worthy".

In theory, trying to find the most relevant and meaningful reviews on something is a great idea.  This is in fact something Google does really well at with it's PageRank and searching of the web on Google.com  It is an idea that is very much needed to help filter out the "spam" and make reviews meaningful in any App Store (sorry Apple, that is a generic term).  Amazon does this quite well with their regular products (to be determined on their own "Appstore"), and now Google is trying it out on their Android Market on devices and on the web version of the market as well (it has been on the web version for a little while now).

So, what is the problem then?  It is a great idea, they are trying to do it, and that is all good, right?  The problem is that their attempt is currently a clear failure and I can demonstrate this not only with the free version of our game Cribbage Pro, but more pointedly with Google's own self published apps as well.  They are making the reviews system irrelevant, unreliable and at best completely useless to the consumer and at worst a huge negative hit for many developers.

Let's take Google's "Gmail" app available on Android.  This app is clearly one of the best implementations of Gmail on any mobile device, and arguable the best email app for mobile devices period.  Every app has some haters, so it is would be expected there are a few negative reviews as well.  To be fair, if you have the full list of reviews on your older Android Market still and take a look, you will indeed find some bad reviews and people complaining about features they want and giving a lower rating as a result.  However, the app still has an overall 4.5 star rating with nearly 113,000 reviews as of this writing.  It is also one of the top rated in it's category (number 3 in the "Communication" free category).  It has a very good ratio of positive star ratings to negative star ratings as well, which is all to be expected for a great app.  So what do the reviews say?  Well, that tells a very different story.  The first three reviews are always what is shown by default for each app without clicking "More" or "Read all reviews" (which does not let you read all reviews, more on that later), so let's start in.

The first review shown for Gmail as of this writing is 3 out of 5 stars by "Adam" on June 21, 2011 (nearly a month ago).  As that star rating would indicate, it is a mediocre review, but actually it is not a review at all.  It is instead instructions on how to fix a problem that he was having with the app to help other users who may have a similar problem.  Good on you "Adam" for trying to help your fellow man, but this is not a review that will help you decide if this is a quality app or not in nearly all cases, and it really can't be argued to be the most relevant review in that regard either.  Let's go on to the second review shown and see if that will help us out.

The second review for Gmail is a 1 star out of 5 by "mark" on Jun 4, 2011 (getting even older now).  He seems to have had a problem with an update released around that time and decided to ding the app with a poor rating in response.  This kind of thing happens quite often in the Android Market, and is a fair review, but does the app still have this issue?  Did anyone else around the same time complain of similar issues?  Sorry, there is absolutely no way to know here.  It is a 1 star negative review, with no context and no way of knowing if it is still a problem or if more recent reviews may have stated it as resolved.   Again, a useless and likely outdated review that is very negative for what as stated earlier is nearly universally accepted as a great app.  We move on to the third and final review shown on the main app page.

The third review for Gmail is also a 1 star out of 5 review, this time by "Erion" on March 28, 2011.  Yes, you read that correctly.  March 28th and today it is July 14th.  Much older and more dated, and also a negative review of a great app.  He is complaining of a missing feature to download attachments, which seems to be a reasonable complaint.  So, is that still a problem nearly 4 months later?  Who knows, it might be, it might not be.  There is again no context of other reviews, no recent reviews showing this and no revision history by date that one could even try and use to divine that understanding.  In fact, this could just be a user error and there may have been a response from another user just moments later saying so but you have no way of knowing that.

We will stop there, as things only get worse and you can view what Google says is all their reviews by clicking the "Read all reviews" for yourself.  The next review as shown today is from way back in October of 2010 believe it or not and also negative, and the trend continues on for a while.  Interestingly, Google Gmail gives itself 4 pages of 10 reviews each that you can look at for it's app, where as nearly every other app will max out at 20 (two pages of 10; like their own published Google Search app is limited to).

So without even using my own app as an example (feel free to take a look, you will see the same trends and some very old reviews), or the many others who are impacted, we see that this new "ranking" of reviews has a few critical issues.  I'll outline them here for clarity.

• Reviews Are Old & Not Tied to a Release
• The most glaring issue that will first hit you is that the reviews being shown are almost all very old reviews in the context of how many reviews this app is getting every single day.  All apps we looked at that are impacted by this problem had this very same issue.  Since these reviews are not tied to a release of the app in any way (say like Apple does), and have no other reviews around them from the same time, we have no context in which to decide if this is helpful or not.
• Reviews Are Heavily Negative
• This is a great app, with many great reviews.  I can't link to those here because the web version now only shows these old reviews, but if you have an older Market on your device you can easily go there and see for yourself.  How can an app with such high star ratings overall, and such quantity of great reviews have such an overwhelming amount of negative reviews being selected for showing in the market?
• "Read all reviews" Does Not Let You Read All Reviews
• The option to "Read all reviews" is misleading at best.  In fact, it just shows more of a filtered and ranked list that Google has chosen to show you using it's magical black box algorithm (Google seems to like this idea of ranking things using complex formulas).  There is no way to see all the current reviews, no list showing the most recent or tied to this version, and no way to rank up or down as helpful or not helpful those same reviews that have been filtered by Google.  So if you don't find the current list helpful and you "down vote" them as "Unhelpful", you will have a very hard time finding a positive one to "up vote" as "Helpful", and good luck finding a recent one at all.

In the old market these things were much less of an issue as all reviews were always shown from the dawn of time and sorted by most recent first.  You had a natural context from other reviewers around the same time.  If someone mentioned an update causing a problem for them, you could find others possibly later saying the problem was fixed, or responding to the comment with clarifications or help.  People could leave reviews with helpful hints, or responses to other reviewers who claimed to be having issues, but in reality possibly failed to follow instructions correctly (say for an app that needs root, or a widget that can't be moved to SD card, etc.)

Please Google, let's admit there is a problem here and fix it.  I realize some apps don't have this problem for whatever reason the secret black box ranking is choosing better reviews (or at least not yet), but for those where it exists it really needs to be fixed.  In fact, some apps that have very poor overall ratings, now are showing only positive reviews.  So this problem is cutting both ways.  Your changes here are hurting us, your developers and your customers, and now that you are rolling this out to devices world wide this problem is about to explode.

If you have been hit by this same issue, or you disagree and want to tell us why we are just a bunch of whiny developers, let us know in the comments.  We would love to hear from you.