Content rating behind the firewall. Presentation to the SIKM group April 2011

1. Content rating behind the firewall April2011 Presented for SIKM by David Thomas, Deloitte

3. KX is built on a heavily customized version of SharePoint 2003

6. Stan Garfield posed the question “Has anyone had positive experiences with 1-5 star content rating mechanisms inside a firewall?”January 2010. Here are selection of responses (thank you to SIKM members):“I think that 5-star rating systems are ideal for apples to apples comparisons. Most knowledge objects (and of course people) cannot be compared in this manner” “I think there is an added complication in that inside the firewall it might also be important to know who is doing the rating. The CEO's rating might carry a little more weight than the janitor's” “With process documents, I'd like the idea of ratings because the purpose of the document is clear. For other documents, the case is muddier. “rating a book or toaster is very different from rating a specific piece of content. Products purchased on Amazon tend to have more common use cases” On a deployed CR system: ”At first, pretty much no one rated. We suspect this was for several of the reasons that you pose in your document but also because it was unclear what they were being asked to rate - the quality of the writing, whether or not they agreed with the author, whether or not they thought highly of the author, or whether they liked the quality of the document. In an effort to encourage participation, the sponsors clarified the intent of the ratings”

9. Type 1: ‘Favorites and Flags’ Design: Single value rating scheme - usually positive

10. Type 2: ‘This or That’ Design: Positive/Negative value: Yes/No, Like/Dislike, Up/Down

11. Type 3: Rating Schemes Design: Traditional 1-X rating scheme (1-5 and 1-10 are common)

13. Scale could be an issue (will there be enough people rating enough content to be meaningful)

14. There is a lag between the time a knowledge asset is accessed and the time a rating can fairly be made. The user may also no longer be logged into the repository at the time the rating could be applied

15. When someone watches a short video they watch and can rate quickly in most cases as the rating mechanism is often easy and convenient. They have consumed the media asset and are positioned to make a judgment on it. The value of a document is not known until after it has been downloaded and readand that can take time.

16. We could experience cultural resistance when trying to implement content rating

17. Anonymity

18. Lack of desire to rate content as poor will likely be evident

20. “a J-curve is considered less-than ideal for several reasons: The average aggregate scores all clump together between 4.5 to 4.7 and therefore they all display as 4- or 5-stars and are not-so-useful for visually sorting between options. Also, this sort of curve begs the question: Why use a 5-point scale at all? Wouldn't you get the same effect with a simpler thumbs-up/down scale, or maybe even just a super-simple favoritepattern?”

21. “If a user sees an object that isn't rated, but they like, they may also rate and/or review, usually giving 5-stars - otherwise why bother - so that others may share in their discovery. People don't think that mediocre objects are worth the bother of seeking out and creating internet ratings”

22. “There is one ratings curve not shown here, the U-curve, where 1 and 5 stars are disproportionately selected”

24. “The biggest difference is most likely that Autos Custom users were rating each other's content.The other sites had users evaluating static, unchanging or feed-based content in which they don't have a vested interest”

25. “Looking more closely at how Autos Custom ratings worked and the content was being evaluated showed why 1-stars were given out so often: users were providing feedback to other users in order to get them to change their behavior. Specifically, you would get one star if you 1) Didn't upload a picture of your ride, or 2) uploaded a dealer stock photo of your ride”

27. Deciding on a content rating design Based on the W distribution example, we asked some questions to determine whether a 1-5 rating scheme would work and we could get the desired W. Ultimately, we decided to custom develop a 1-5 Rating Scheme (Type 3). There were other drivers identified on that drove this decision.

29. Simple, familiar model to rate published content

30. Granularity – ability to get and average score and promote / remove content as needed

31. Alignment with SharePoint 2010 (future platform)  reduced disruption for the user when we moved

32. Resource constraints meant some deferral of some functionalityfor future releases. For the first release:

33. Single classification of knowledge asset - published content. (Other types would later follow).

34. Rating occurred on the content record only

38. Additional rating capabilities were deployed for qualifications in October/November – potentially raised awareness around rating in general.

40. Average of around 270 new raters each month although heavily skewed by the first 2 months.

43. What did we learn from the experience?

45. There are no real benchmarks for ‘success’. Project set target was 1-3% of viewed content would be rated. Noted if 1-2% of our MUVs rate content that equates to 2500 total ratings a year.(YouTube Rating from 0.1 – 0.5% of viewersis common (sign in to rate has impact?))

46. Value for knowledge assets can be situational – “one mans trash is another mans treasure”. Without comments system it is difficult to understand why something is rated a certain score.

47. Feel that our users are pre-disposed to rate a lot of content 3-4. They get the concept of best in class. We have firm wide methodologies that are broadly used – they would equate that with best in class/ 5 star.

48. Experienced excessive rating and of course, self rating.

49. Still some level of fear that if they rate something a 1, then the document author will find out – to the extent that we put that in the FAQs for the system to address this.

51. A model for removal/archive of ‘low quality content’. Business rule definition is still outstanding around this. Basic idea: If there is compelling evidence (multiple ratings) look to possibly retire the content earlier.

52. Additional marketing and promotions, sponsorship. Incentivization will drive a temporary increase in rating activity, but is not sustainable.

53. Demo rating as part of onboarding materials for new hires – set it as an expectation to rate assets that are used. This should be part of a broader initiative about the value of a knowledge sharing culture.

54. Authored or contributed content appears on your profile. Add rating of that content as well.

56. Rating is built into common platforms that we use for document management, collaboration and knowledge sharing. In theory if rating is pervasive, and if users see it all the time, they *may* use it more.

57. User behavior was generally consistentGenerally more ratings were applied by junior – mid level users (time, familiarity with the Portal) Common for users to rate items a ‘4’ or a ‘5’ We saw long term usage is around 1% of page views and 2% of total content in the content store rated in any one month Strong impact when the process is incentivized (not that surprising) Concern about the time it would take to get meaningful ratings on a lot of the content on the Portal Outliers are likely ‘power raters’ although might not be self-directed!

58. You-tube’s position on rating drove a change for them… ‘“Seems like when it comes to ratings it's pretty much all or nothing.Great videos prompt action; anything less prompts indifferenceThus, the ratings system is primarily being used as a seal of approval, not as an editorial indicator of what the community thinks out a video. Rating a video joins favoriting and sharing as a way to tell the world that this is something you love”. (22/9/09 YouTube blog) Questions? Feel free to contact davidthomas4@deloitte.com