Just recently (7/12/2011), Google started to release an update to the Android Market for devices 2.2 and above (a vast majority of devices). This update includes a complete redesign and a lot of great new features that could help everyone participating in the market. However, there is one change that is very negative all around for consumers and developers/publishers that will offset any of those positives in my opinion, but has not been discussed much. This change is in how they are now choosing to try and "rank" user reviews of apps, and in many cases only show 20 of such reviews that they are deeming "worthy".
In theory, trying to find the most relevant and meaningful reviews on something is a great idea. This is in fact something Google does really well at with it's PageRank and searching of the web on Google.com It is an idea that is very much needed to help filter out the "spam" and make reviews meaningful in any App Store (sorry Apple, that is a generic term). Amazon does this quite well with their regular products (to be determined on their own "Appstore"), and now Google is trying it out on their Android Market on devices and on the web version of the market as well (it has been on the web version for a little while now).
So, what is the problem then? It is a great idea, they are trying to do it, and that is all good, right? The problem is that their attempt is currently a clear failure and I can demonstrate this not only with the free version of our game Cribbage Pro, but more pointedly with Google's own self published apps as well. They are making the reviews system irrelevant, unreliable and at best completely useless to the consumer and at worst a huge negative hit for many developers.
Let's take Google's "Gmail" app available on Android. This app is clearly one of the best implementations of Gmail on any mobile device, and arguable the best email app for mobile devices period. Every app has some haters, so it is would be expected there are a few negative reviews as well. To be fair, if you have the full list of reviews on your older Android Market still and take a look, you will indeed find some bad reviews and people complaining about features they want and giving a lower rating as a result. However, the app still has an overall 4.5 star rating with nearly 113,000 reviews as of this writing. It is also one of the top rated in it's category (number 3 in the "Communication" free category). It has a very good ratio of positive star ratings to negative star ratings as well, which is all to be expected for a great app. So what do the reviews say? Well, that tells a very different story. The first three reviews are always what is shown by default for each app without clicking "More" or "Read all reviews" (which does not let you read all reviews, more on that later), so let's start in.
We will stop there, as things only get worse and you can view what Google says is all their reviews by clicking the "Read all reviews" for yourself. The next review as shown today is from way back in October of 2010 believe it or not and also negative, and the trend continues on for a while. Interestingly, Google Gmail gives itself 4 pages of 10 reviews each that you can look at for it's app, where as nearly every other app will max out at 20 (two pages of 10; like their own published Google Search app is limited to).
So without even using my own app as an example (feel free to take a look, you will see the same trends and some very old reviews), or the many others who are impacted, we see that this new "ranking" of reviews has a few critical issues. I'll outline them here for clarity.
- Reviews Are Old & Not Tied to a Release
- The most glaring issue that will first hit you is that the reviews being shown are almost all very old reviews in the context of how many reviews this app is getting every single day. All apps we looked at that are impacted by this problem had this very same issue. Since these reviews are not tied to a release of the app in any way (say like Apple does), and have no other reviews around them from the same time, we have no context in which to decide if this is helpful or not.
- Reviews Are Heavily Negative
- This is a great app, with many great reviews. I can't link to those here because the web version now only shows these old reviews, but if you have an older Market on your device you can easily go there and see for yourself. How can an app with such high star ratings overall, and such quantity of great reviews have such an overwhelming amount of negative reviews being selected for showing in the market?
- "Read all reviews" Does Not Let You Read All Reviews
- The option to "Read all reviews" is misleading at best. In fact, it just shows more of a filtered and ranked list that Google has chosen to show you using it's magical black box algorithm (Google seems to like this idea of ranking things using complex formulas). There is no way to see all the current reviews, no list showing the most recent or tied to this version, and no way to rank up or down as helpful or not helpful those same reviews that have been filtered by Google. So if you don't find the current list helpful and you "down vote" them as "Unhelpful", you will have a very hard time finding a positive one to "up vote" as "Helpful", and good luck finding a recent one at all.
In the old market these things were much less of an issue as all reviews were always shown from the dawn of time and sorted by most recent first. You had a natural context from other reviewers around the same time. If someone mentioned an update causing a problem for them, you could find others possibly later saying the problem was fixed, or responding to the comment with clarifications or help. People could leave reviews with helpful hints, or responses to other reviewers who claimed to be having issues, but in reality possibly failed to follow instructions correctly (say for an app that needs root, or a widget that can't be moved to SD card, etc.)
Please Google, let's admit there is a problem here and fix it. I realize some apps don't have this problem for whatever reason the secret black box ranking is choosing better reviews (or at least not yet), but for those where it exists it really needs to be fixed. In fact, some apps that have very poor overall ratings, now are showing only positive reviews. So this problem is cutting both ways. Your changes here are hurting us, your developers and your customers, and now that you are rolling this out to devices world wide this problem is about to explode.
If you have been hit by this same issue, or you disagree and want to tell us why we are just a bunch of whiny developers, let us know in the comments. We would love to hear from you.