Measuring On Demand Learning Bersin and Safari Books OnlineReport

Research Bulletin | 2011
BERSIN & ASSOCIATES

June 10, 2011 Volume 6, Issue 24

Measuring On-Demand
Learning: Safari Books
Online and the Value of
Digital Libraries

About the Author Introduction: Making an Impact Where
Performance Happens

Up until a few years ago, supporting on-demand learning was
not something learning and development (L&D) functions had to
worry about too much. L&D traditionally was more accustomed to
designing, developing and delivering content. However, the modern
Janet Clarey, learning organization is less concerned with formal learning – and
Senior Analyst
is more concerned with “where” performance happens and making
sure that employees have the resources available to do their work1.
As a result, new roles for L&D as “enablers and facilitators” of
learning have emerged. This is especially important because first-line
managers and supervisors know they need to keep their employees
informed on industry advances and trends, but they simply cannot
do that with constant change and the expectations of employees to
have everything at their fingertips.

The growing recognition of the importance of on-demand and
the increasing speed at which the human capital needs of the
organization are evolving, along with the changing needs and
BERSIN & ASSOCIATES, LLC
180 GRAND AVENUE
SUITE 320
OAKLAND, CA 94612
1
For more information, The High-Impact Learning Organization: WhatWorks® in
(510) 251-4400
the Management, Governance and Operations of Modern Corporate Training, Bersin &
INFO@BERSIN.COM Associates / Josh Bersin, May 2008. Available to research members at www.bersin.com/
WWW.BERSIN.COM library or for purchase at www.bersin.com/highimpact.

THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY.
BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

expectations for learning on the part of employees, bring a new
KEY POINT
challenge for L&D. How do you measure the value of on-demand
Informal learning learning and more specifically an on-demand digital library? This
research bulletin will help to answer that.
is learning that is
accidental, ad- This report will cover:
hoc, unplanned
and which
happens without
the discipline
of instructional
design.

The Increasing Role of On-Demand Learning

We categorize approaches to learning as “formal” and “informal.”
Informal learning is learning that is accidental, ad-hoc, unplanned
and which happens without the discipline of instructional design. It
has typically referred to learning that training professionals did not
necessarily pay attention to, but knew was occurring.

We categorize informal learning into three broad types – on-demand
learning, social learning and embedded learning. To help you to
make sense of how on-demand learning fits as a corporate learning
approach, we have developed the Bersin & Associates Enterprise
Learning Framework2. (See Figure 1.)

6114 LA SALLE AVENUE
SUITE 417
OAKLAND, CA 94611
2
(510) 654-8500 For more information, The Enterprise Learning Framework: A Modern Approach to
INFO@BERSIN.COM Corporate Training, Bersin & Associates / Josh Bersin and David Mallon, April 2009. Available
to research members at www.bersin.com/library.
WWW.BERSIN.COM

Measuring On-Demand Learning
Janet Clarey | Page 2

BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

Figure 1: Bersin & Associates Enterprise Learning Framework®

Enterprise Learning Framework®
Leadership Career Customer Project and Customer
Learning Development Development
Onboarding
Service Process Education
Systems

Programs Management
Technical
Compliance Sales Product Channel
IT Skills
Organization, Governance and Management

Professional Knowledge Training

Audiences Jobs Roles Competencies Proficiencies Preferences Demographics Geographies Business Problems
& Problems

Informal

Learning Architecture
Formal
Approaches

Instructor Led Training
Virtual Classroom
On-Demand Social Embedded
Games E-Learning Wikis, Blogs, Forums Performance Support
Simulations Search Expert Directories Feedback
Testing and Evaluation Books, Articles Social Networks Rotational Assignments
E-Learning Videos Communities of Practice After Action Reviews
Podcasts Conferences and Colloquium Quality Circles
Learning /Knowledge Portals Coaching and Mentoring Development Planning

Performance Information Content Change Measurement and
Consulting Architecture Development Management Evaluation
Disciplines Instructional Knowledge Program Community Business
Designs Management Management Management Intelligence

LMS, LCMS Content Rich Performance Reporting and
Tools Learning Portals Development
Collaboration
Media
Assessment
Support Analytics

& Technology Talent Mgt.
Systems
Content
Management
Social
Networking
Mobile
Search,
Tagging
Virtual
Classroom
Measurement
Systems

Executive Development Performance and Innovation Employee
Support Planning Talent Management Programs Feedback
Culture Learning integrated Knowledge Customer Mentoring and Learning from
With Business Planning Sharing Listening Knowledge Sharing Mistakes

Source: Bersin & Associates, 2009.

On-demand learning, the subject of this research bulletin, is considered
a learner-driven activity that employees can use when and as needed.
There are many forms of on-demand learning, such as videos, podcasts,
knowledge portals and on-demand digital libraries.

SUITE 417
OAKLAND, CA 94611
(510) 654-8500
INFO@BERSIN.COM
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

Figure 2: Bersin & Associates Informal Learning Approaches

E-‐Learning Wikis, Blogs, Forums Performance Support
Search Expert Directories Feedback
Books, Articles Social Networks Rotational Assignments
Videos Communities of Practice After Action Reviews
Podcasts Conferences and Colloquium Quality Circles
Learning /Knowledge Portals Coaching and Mentoring Development Planning


The On-Demand Digital Library for On-Demand Learning

Digital asset libraries are one way to support the increasing speed at
KEY POINT
which the human capital needs of the organization change. It also

Digital asset supports the changing needs and expectations for learning on the
part of employees. This is especially important because we are moving
libraries are one
toward learning that is 70 percent on the job, 20 percent collaborative
way to support the
and 10 percent formal3.
increasing speed at
which the human What follows is an example of how CDW is using an on-demand
digital library to meet the immediate needs of its organization and its
capital needs of
employees’ expectations.
the organization
change.

SUITE 417
OAKLAND, CA 94611
(510) 654-8500
3
INFO@BERSIN.COM This idea has multiple sources. It is commonly attributed to the L&D model according to
the Princeton University Learning Process.
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

Case in Point: Using an On-Demand Digital Library to
Facilitate Needs of a High-Technology Workforce at CDW

CDW has a population of 600 engineers who primarily design,
implement and support IT solutions for its customers. It is of the
utmost importance that its engineers stay abreast of emerging
technologies. However, getting engineers to a physical
classroom for training on a regular basis is just not feasible.
CDW’s workforce is highly distributed with field engineers and
other workers in remote offices. This, along with the challenge
of finding available time for training, means the company
had to find new ways to keep the workforce out in front of
technology advances.

One way in which CDW is meeting the technical training needs
KEY POINT
of its engineers is by providing them with access to the newest

One way in which
technology-related books online. This not only keeps them
current on emerging technologies, but also supports their need
CDW is meeting
for certification, reference and filling skills gaps.
the technical
training needs of At many organizations, licenses for various on-demand digital
its engineers is by libraries sit unused. CDW, on the other hand, gets the most
from its on-demand digital library because the company actively
providing them
manages the resource. In the two years that the company has
with access to the
been using the Safari Books Online library, CDW learners have
newest technology-
been requesting more licenses than the company has. This is a
related books good “problem.”
online.
CDW credits excellent facilitation of the on-demand digital
library by its organizational capability department as a factor in
its success.

For example, organizational capacity not only sources the right
on-demand resources, but it also has been instrumental in the
organization of book clubs. In effect a community of
SUITE 417
OAKLAND, CA 94611
(510) 654-8500
INFO@BERSIN.COM
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES
Case in Point: Using an On-Demand Digital Library (cont’d)

BEST PRACTICE
practice4, the book clubs are popular because they support
Book clubs different learner preferences, as well as the collaboration and

are popular at communication among members. Employees read new, relevant
books and meet weekly to discuss them. A facilitator (often
CDW because
a team lead) leads the face-to-face community and provides
they support
an available conference bridge for those who cannot join
different learner
the session.
preferences,
Of course not everyone likes reading books online. While some
as well as the
enjoy the flexibility and easy access afforded online, others find
collaboration and
reading online difficult. The community of practice also provides
communication
a traditional way in which to develop professionally, face to
among members. face, while collaborating and networking with co-workers.
A facilitator The community also provides structure and support – needed
leads the face-to- elements for learning transfer.
face community
In addition to on-demand digital books, CDW uses on-demand
and provides
CBT “nuggets,” which are accessed via the company’s learning
an available management system (LMS). The video “nuggets” of knowledge
conference bridge supplement other training and certifications for internal IT
for those who groups in various roles.
cannot join the
How CDW Measures the Value of Its On-Demand Digital Library
session.
The company measures the value of its on-demand digital
library in several ways. First, the company uses traditional
measures, such as average time spent and total usage by its
employees. The company compares the cost associated with
actually buying a book versus accessing it online. CDW also looks
at how many hours have been saved, as compared with offering
similar content in a formal setting. The company compares its
training budget over the previous year, looking at the amount

SUITE 417 4
A “community of practice” (or “CoP”) is often defined as a group of people who share
OAKLAND, CA 94611 an interest or concern about a common topic, and who deepen their knowledge in this area
(510) 654-8500 through ongoing interaction and relationship-building within their group. While communities
INFO@BERSIN.COM often come into being spontaneously, they nonetheless require nurturing if they are to
become valuable to the members and remain viable over the course of their evolution.
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

BEST PRACTICE
set aside for books. CDW then takes an average of that number
CDW has a “use and compares costs with the online resources used. Finally, it

it or lose it” uses employee surveys to measure end-user satisfaction.

policy in place for CDW has a “use it or lose it” policy in place for its on-demand
its on-demand digital library, which encourages people to use their licenses
digital library, and weeds out people who will not. While the company
wants to encourage the use of the on-demand digital library,
which encourages
it realizes that resources are finite and strives to maximize
people to use their
the organization’s investment by only giving licenses to
licenses and weeds
employees who will use them. CDW monitors usage using an
out people who administrative portal.
will not.
Initial Benefits

One of the greatest benefits to CDW is the ability to provide
a service that enables engineers to stay on top of current
technology and trends. For instance, its engineers like “rough
cuts” – books that are viewable even before publication.

Not surprisingly, CDW is now able to target a much larger
audience, rather than relying on a physical library. Its engineers
like to access the resources directly from a website. The ability
to download books, so that they can be read on an e-reader or
iPad application, is well-received by employees.

Another benefit is improved time management for engineers.
Access to on-demand resources keeps engineers from wasting
valuable time searching for credible information.

CDW also likes these resources because they provide less reliance
on, and yet another option over, costly and time-consuming
formal training.

Advice for Other Companies
CDW suggests that other companies considering an on-demand
digital library should start small, rather than with an enterprise
SUITE 417
OAKLAND, CA 94611 license model. Taking this approach, the company believes,
(510) 654-8500 allowed it to evaluate the resource and determine ROI.
INFO@BERSIN.COM
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

The company cautions others from just turning the resource
“on” and walking away. CDW has had that happen in the past
with a different on-demand digital library. When the company
introduced this resource to employees, it included a scavenger
hunt to incent the use of the resource. As a result, the company
experienced a bit of a groundswell with more people than
expected asking for licenses.

Where Does CDW Go from Here?

One of its goals is to make the online digital library part of
a standardized program by tying resources to roles and / or
courses. The company also hopes to add some more structure
around recommendations. Right now, someone may talk

recommendations come from co-workers, book reviews or word
of mouth. So, the company is seeking to formalize this resource
as part of its overall strategy.

Why Use On-Demand Digital Libraries?

The use of on-demand digital libraries is growing. Part of the reason
KEY POINT
is the ability of online books and digital assets, in general, to make

The use of on- knowledge sources available to employees’ on-demand – especially new
releases of books. These resources can be downloaded and read (or
demand digital
listened to) instantly on a desktop computer or on a number of mobile
libraries is
devices – something physical libraries cannot fully support.
growing – in part
because these Digital libraries can also save time and improve productivity. Knowledge
and information workers spend too much time searching for answers
resources can be
in a sea of unstructured information from non-vetted resources. Even
downloaded and
when they do find an answer, accuracy is often not at an expert level.
read & ASSOCIATES, LLC
BERSIN(or listened This “lost-time looking” is a hidden cost of both lost productivity and
to) instantly on a AVENUE
6114 LA SALLE
the inability to pinpoint time-critical issues.
desktop computer SUITE 417
OAKLAND, CA 94611
or on a number of
(510) 654-8500
mobile devices.
INFO@BERSIN.COM
WWW.BERSIN.COM


BERSIN & ASSOCIATES

Enabling the Knowledge Worker
BERSIN & ASSOCIATES

Figure 3: FindingrightRight Information atrightRight Timein the best Right Format
Finding the the information at the the time and and in the format….

What do you believe holds back knowledge workers in your organization?

Overwhelming volume of information
makes it difficult to notice and keep 68%
The problem is track of useful information.
“context” not
“content” Lack of effective tools (such as
search) makes it difficult to find the 34%
most useful information.

Frequent change of information
Reuse and makes it difficult to find the most 32%
current information.
standards
badly Inconsistency of information formats
needed or sources makes it difficult to use 23%
and comprehend new information.

Dynamic nature of job roles makes it
difficult to find sufficiently targeted or 16%
relevant information.

Job roles or conditions make it difficult
12%
to access sources of information.

0% 10% 20% 30% 40% 50% 60% 70% 80%


Copyright © 2009 Bersin & Associates. All rights reserved. Page 14
When it comes to finding the right on-demand digital resources,
lagging organizations may find their employees wondering, “How
is it that a 10-minutes search session with Google can often result in
more substantive learning than searching our intranet?” No one wants
employees spending a large part of their workday simply looking for
the information they need to do their job or making them wait for it.

SUITE 417
OAKLAND, CA 94611
(510) 654-8500
INFO@BERSIN.COM
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

Figure 4: Advantages for Using Digital Asset Libraries

Knowledge sources Ability to pinpoint time-‐critical
accessible on-‐demand through issues resulting in more
multiple channels substantive learning

Digital Asset
Libraries

Less time and fewer Authoritative, relevant,
opportunities lost in up-‐to-‐date information
searching for answers from experts


This loss of productivity impacts executives, as well as front-line
knowledge workers. When we asked senior leaders about their own
habits in searching for information, they said staying informed is costly
and time-consuming. While they felt that they were informed, these
senior leaders admitted that they devoted a significant amount of time
to the process – about four hours per week. Most were “too busy” to
read business books to find “authoritative” information.

When they did read, more than one-half did so while traveling or
while at home.5 Digital books by experts in their fields allowed them to
BERSIN & ASSOCIATES, LLC read authoritative information and choose the amount of information
SUITE 417
5
OAKLAND, CA 94611 For more information, How Executives Stay Informed: A Study of Resources Used and
Time Spent in Locating Critical Business Information, Bersin & Associates / Josh Bersin
(510) 654-8500
November 2005. Available to library members at www.bersin.com.
INFO@BERSIN.COM
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

they needed, whether it was several pages, a chapter or an entire
book. High-value, expert-written materials offered in a well-organized
reference format were viewed as a valuable business tool.


Measurement has always been a profound challenge for L&D.
Organizations struggle to measure the effectiveness and efficiency of the
training itself, the overall training function, the business value of training
in general, and training’s alignment and contribution to overall business
goals. Our research tells us that measurement is consistently rated as an
area that senior and midlevel training managers want to improve.6

Measuring the approaches to learning that fall under the umbrella of
the term “informal learning” (which encompasses everything from on-
the-job performance support to communities of practice) creates yet
another measurement challenge. For instance, how do you go about
measuring the effectiveness and efficiency of reading a chapter in
a book?

Our Impact Measurement Framework7, adapted for informal learning
environments, uses our Business Impact Model for Training8 to identify
specific measures that can be used to monitor and evaluate the causal
elements which drive business impact. (See Figure 5.)

6
For more information, High-Impact Learning Measurement: Best Practices, Models, and
Business-Driven Solution for the Measurement and Evaluation of Corporate Training, Bersin
& Associates / Josh Bersin November 2006. Available to research members at www.bersin.
com/library or for purchase at www.bersin.com/measurement.
7
For more information, The Impact Measurement Framework: An Update for the
Measurement of Informal Learning, Bersin & Associates / Josh Bersin, May 3, 2011. Available
SUITE 417 to research members at www.bersin.com/library.
OAKLAND, CA 94611
8
(510) 654-8500 For more information, High-Impact Learning Measurement: Best Practices, Models, and
Business-Driven Solution for the Measurement and Evaluation of Corporate Training, Bersin
INFO@BERSIN.COM
& Associates / Josh Bersin November 2006.
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

Figure 5: Impact Measurement Framework – Adapted for Informal Learning

Satisfaction Learning

Adoption Utility Efficiency Alignment Attainment
Did you reach the How well do the How efficient and How well were How well did you meet
programs solve
desired audience? cost-effective was program business specifically defined
the workforce’s
Did they complete particular problems? How did it compare priorities defined? client (business user
or comply as desired? How well did it align to other similar How well did business or customer) objectives?
Who did not comply to the specific job programs or units buy in on These may be revenue,
and why? related problems competitive programs? the value of this time to market,
and issues? How well did it use program relative to compliance,
Would learners the learner’s time ? other investments? time to complete, etc.
recommend this
Program program to
Design their peers?

Individual Performance Organizational Performance
General business measures or HR measures which
Indicators asked of learners and managers to gauge
are already captured in the organization (ie. Engagement,
performance improvements. Specific operational measures
retention). Special surveys can be used to determine
identified in the performance consulting process.
indicators using the “wisdom of crowds.”


For measuring on-demand learning, we like to use the word “utility” to
KEY POINT
indicate the “usefulness” of on-demand digital assets to the individual
Copyright © 2011 Bersin & Associates. All rights reserved. Page 1

The Bersin &
“learning” and his / her workgroups. It is an indicator9 of performance
which is easier to measure than the performance itself. Questions that
Associates Impact
are used to measure utility might include the following.
Measurement
Framework
also includes efficiency and effectiveness of your current job?

contribution,
feedback and your current job?
activity as measures
of learning for
digital assets. We also like to measure “attainment” – how well did people meet
specifically defined client (business user or customer) objectives? For
SUITE 417 example, a senior software engineer at Nuance University relates his use
OAKLAND, CA 94611 of the on-demand digital library to delivering a product to customers.
(510) 654-8500
INFO@BERSIN.COM 9
An “indicator” is an individual’s assessment of a metric, rather than the metric itself.
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

“Without this [resource], we would probably
not be able to ship Dragon Naturally
Speaking 11.5 on time. I relied heavily on our digital
content library to bring me up to speed. We keep
referring to this project as having a tight timeframe,
but our online library as made it possible for me to
actually work fast enough in a new OS, in a new
sublanguage, to meet the deadlines.”

Specifically adapted for informal learning, our Impact Measurement
Framework also includes contribution, feedback and activity as
measures of learning for digital assets. One organization utilizing such
measures is Unisys.

Figure 6: Informal Learning Measures

Contribution Feedback Activity

How well are people
What is the How many people
commenting on and
volume, frequency are accessing the
rating comment
and depth of system and at what
in a fair way?
contribution made? frequency?

Is feedback being
What percent of experts What groups have
read and used by
are contributing greatest access and
authors?
in what areas? which are not

using it yet?
Is feedback being
How regular and
ranked and analyzed
current is the How widely
by the organization?
response to comments distributed is

and questions? the activity in the
Are authors and others
program?
using feedback
How rapidly are
well?
people keeping How well are higher-‐

up with changing level people
BERSIN & ASSOCIATES, LLC Can we “rate”
needs? contributing?
6114 LA SALLE AVENUE authors well?
SUITE 417
OAKLAND, CA 94611
(510) 654-8500
INFO@BERSIN.COM
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

Case in Point: Unisys University

A worldwide information technology company, Unisys provides
KEY POINT
a portfolio of IT services, software and technology that solves

While Unisys
critical problems for clients. Due to the nature of its business,
it is very important that employees keep up with leading-edge
University ﬁnds
technology. To support that business need, Unisys University
that measuring
uses, among other learning solutions, a variety of on-demand
the value of on-
learning resources from several providers, including the Safari
demand learning Books Online library.
resources is hard to
Unisys University finds that measuring the value of on-demand
pin down, despite
learning resources is hard to pin down. How do you determine
this challenge, the
whether or not a person who read a book has improved his / her
company has come performance? Despite this challenge, Unisys has come up with a
up with a variety of variety of measures.
measures.
Unisys University uses surveys to flesh out any changes in
employee productivity or time-savings after it utilizes an on-
demand learning resource over some period of time. Although
time periods vary, the survey is often scheduled to be completed
prior to a program review meeting. A high and low metric is
used (based on users’ salaries) and a range is determined (based
on employees’ responses to the survey and usage figures). It
is not a perfect measure, due to the difference in roles and
responsibilities of the employees participating – and this is why
the company uses a range and other methods of measurement.

The company also poses open-ended questions to employees,
such as, “Give us an example of a time that someone else
noticed your improved skills.” Offering opened-ended questions
helps Unisys to learn of problems about which it might not have
otherwise been aware. For example, the company found that its
BERSIN & ASSOCIATES, LLC global audience had different technology issues based on how
6114 LA SALLE AVENUE and where audience members worked.
SUITE 417
OAKLAND, CA 94611 The measurement process is time-consuming, so Unisys
(510) 654-8500 University works with its solution providers to administer and
INFO@BERSIN.COM
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES
Case in Point: Unisys University (cont’d)

help them analyze the surveys and responses. By having vendors
do as much of the high-level analysis as possible, the University
gains broader comparisons and benchmarks, since vendor
analysis often includes data from other clients which have used
similar surveys.

Unisys University also solicits ideas from its employees about
KEY POINT
how the company can make the use of on-demand learning

By having vendors resources more valuable. Other than determining the
improvements in productivity associated with an online learning
do as much of the
resource, the most important feedback taken from the surveys
high-level survey
is the employee input about what would cause them to use the
analysis as possible,
resource more. (See Figure 7.)
the University
The company has learned some lessons about managing content
gains broader
and solution providers. For years, Unisys University maintained
comparisons and
an enterprise agreement with a vendor and realized it was
benchmarks, since
only using about 15 percent of the licenses for which it was
vendor analysis
paying. While the provider had a good digital library of content,
often includes data it was not a good fit for the employees. A huge library with
from other clients “something for everyone” is not really effective if it does not
which have used reflect targeted skills.
similar surveys.
Unisys University is currently offering enterprise access to some
on-demand resources, such as its online digital library of books.
The ability to download chapters or portions of books was
especially important to Unisys employees, and saves a great deal
of time over traditional e-learning content.

Where on-demand resources fit within the total learning
environment varies over time, partially due to changes in
corporate initiatives and partially due to contract requirements.
Employees have a single access point to all learning content
BERSIN & ASSOCIATES, LLC through an internal portal. However, some content resides
6114 LA SALLE AVENUE within an individual’s learning folder and other content is
SUITE 417
accessed by linking directly to a solution provider’s website. Easy
OAKLAND, CA 94611
access to the resources is critical.
(510) 654-8500
INFO@BERSIN.COM
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES
Case in Point: Unisys University (cont’d)

Unisys University does tie most of its on-demand resources
to some formal learning programs. While it does very little
instructor-led training (ILT), the University does use its on-
demand learning resources as pre-work for both face-to-face
training and self-paced e-learning.

It has built L&D guides that provide a listing of all of the
learning assets that support an employee’s role (e.g., project
management), a corporate initiative or program, or a skills set /
certification. These guides include e-learning, white papers,
books, abstracts and other learning resources.

The director of operations and finance at Unisys University
BEST PRACTICE
offered some advice to organizations that want to provide
employees with on-demand learning resources, such as
Unisys University
books online.
has built L&D
guides that If you’re just getting started, don’t buy an enterprise license.
provide a listing of It’s going to take some time to get some traction and you
will be spending more than you need during that time.
all of the learning
assets that support
an employee’s company. It is easy to find a provider with a lot of content,
role, a corporate but not so easy to find the right one.

initiative or
program, or a skills more or less licenses? Do you need multiple partners?
set / certiﬁcation.

that doesn’t change about the specific content needs of any
organization is that those needs are always changing.

Using multiple providers has worked out very well for Unisys
University. The content meets its needs and keeps solution
BERSIN & ASSOCIATES, LLC providers on their toes.
SUITE 417
OAKLAND, CA 94611
(510) 654-8500
INFO@BERSIN.COM
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

Figure 7: An Example of Unisys University’s Survey Questions

1. Would you recommend the content to a colleague?

2. How would you rate the training resources?

3. Please explain why you rated the training resource as you did.

4. What was the main reason why you accessed the (xxx) training resources identified above?

5. Is the learning content relevant to skills you use on your job?

6. Primarily, what skill were you seeking to develop?

7. How often does your job require you to use this skill?

8. Have you had the opportunity to use what you learned on your job?

9. As a result of the training, how much of a skill increase would you estimate you experienced?

Do you feel that your productivity has increased as a result? If so, how many hours would you estimate you
10.
save on a weekly basis?

11. Is the skill you were seeking to improve required for you to advance within the company?

12. If you did not have access to the (xxx) resources listed above, what alternatives would you have considered?

13. How much do you estimate the alternate option would have cost?

14. How often so you use the resources listed above?
Source: Unisys University, 2011.

Other Thoughts on Measurement

We can take some guidance from Johnson, Trabelis & Tin, who wrote,
The Successful Virtual Library: Partnership and Collaboration, in
establishing a guide for workplace learning. They said,

“… an emphasis on technology and information
without a focus on human components (such as
service and interaction) are the most important
determinants of the success of the virtual library
BERSIN & ASSOCIATES, LLC encouraging people to work together, develop their
6114 LA SALLE AVENUE skills, and form strong and trusting relationships.10”
SUITE 417
OAKLAND, CA 94611
(510) 654-8500 10
Source: “Library support for online learners: E-resources, e-services and human factors,”
INFO@BERSIN.COM In Theory and Practice of Online Learning, Johnson, K., Trabelsi, H. & Tin, T., Editors Terry
Anderson & Faith Elloumi, Athabasca University, 2004.
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

Whether or not there is focus on human components is an elusive
measurement, but it is important to keep in mind. As these
researchers continued,

“This approach considers the library as
an active partner of the learning community,
helping learners to become ‘information literates’
by integrating information literacy skills into the
curriculum. For example, the library can help learners
to evaluate critically the authority and authenticity
of the resources they find, and to enhance their
critical thinking skills. The library can also offer
support to learners, and can mentor their work by
offering one-to-one communication and interaction,
and by achieving a deeper level of
understanding of what learners need.11”
KEY POINT
Replace the word “library” with “L&D organization” and the message
is the same, which starts to introduce the concept of “metalearning,” or
An early decision
being aware of and taking control of one’s own learning, an important
was made to
aspect of learning today at Nuance University.
establish Nuance
University as an
online university, Case in Point: Nuance University – Metalearning and
with all content
Trending Analysis

and resources Nuance is a speech and language expert technology firm.
delivered through The company has 6,000 employees on all continents, is
a self-directed headquartered in Burlington, Massachusetts and has achieved
online platform 40 percent growth over six years. Nuance University focuses
on alignment.
that allows for
a cost-effective In 2007, there was no established technology learning program
solution and at Nuance. The company was growing very quickly at this point
consistent delivery
(via many acquisitions), leading to an environment with a

across the globe. AVENUE
6114 LA SALLE tremendous amount of change, intense timeframes for delivery
SUITE 417
OAKLAND, CA 94611
(510) 654-8500 11
Source: “Library support for online learners: E-resources, e-services and human factors,”
INFO@BERSIN.COM In Theory and Practice of Online Learning, Johnson, K., Trabelsi, H. & Tin, T., Editors Terry
WWW.BERSIN.COM Anderson & Faith Elloumi, Athabasca University, 2004.


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES
Case in Point: Nuance University (cont’d)

and complex, varied technical environments. Based on these
factors (and others), the employees had started to demand
a development program. The pace of its engineers’ work is
fast, requiring engineers to get up to speed on a variety of
technologies quickly. The company’s technology is very complex

may be required to work on multiple projects with vast
technology differences and they are expected to adapt.

An early decision was made to establish Nuance University as
BEST PRACTICE
an online university, with all content and resources delivered
through a self-directed online platform that allows for a cost-
To determine the
effective solution and consistent delivery across the globe.
best way to handle
Outside and / or instructor-led training is only available as an
the technical
exception if business needs require something unavailable
learning needs of through Nuance University. A small development team also
the organization, creates some proprietary resources, as business needs require.
as a ﬁrst step
To determine the best way to handle the technical learning
Nuance University
needs of the organization, as a first step Nuance University
put together a put together a steering committee to address the business
steering committee needs and develop a strategy. While working with the
to address the steering committee, as well as evaluating available content
business needs and and resources, it was clear early on that an informal learning
develop a strategy. resource would be of greater value than a formal learning
program. The feeling was that formal learning tools required
too much effort to transfer the knowledge from the learning
experience to the job. An informal tool would allow the
“heads-down” engineers to easily transfer from their work to
a learning resource and back – allowing for better integration
and workflow.

To test the chosen tool, the company initially started with a
small pilot group made up of 200 R&D people. The pilot group
SUITE 417 was given licenses to an on-demand digital content library. Not
OAKLAND, CA 94611 long after that, word of mouth about the value of the resources
(510) 654-8500 “incited a riot.” Clearly, the employees had found a valuable
INFO@BERSIN.COM
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

resource. The company’s technology resources included Safari
Books Online and the IEEE Digital Library, a niche provider of
speech science and technology. Neither are the cheapest, but
each are a great fit for the company and the employees loved it.

Nuance University uses its digital content library as both a
business tool and a learning tool. In fact, as a department, it
considers itself more of an internal consultant, rather than a
traditional learning and development team.

The measuring mechanism is unique in that it includes the
KEY POINT
measurement of business trends, as well as a very direct effort
to align learning content and resource usage with business
As the metrics
programs and priorities. For example, the company looks at
have matured,
trending around things like Android and other “hits” during
Nuance University
a particular development process. This helps the University to
has found that
manage the resources and understand how they are used within
quantitative the business.
measures of
As the metrics have matured, Nuance University has found that
the on-demand
quantitative measures (like usage) of the on-demand digital
digital library do
library do not hold much value for their executives on their
not hold much own, so it has focused heavily on including qualitative analysis
value for their in reporting to provide the much-needed context around
executives, so it learning numbers. Much of this effort is achieved through the
has focused heavily use of a learner survey that is sent to employees some period of

on including time after they have accessed a particular learning or reference
resource. The focus is almost entirely on the open-ended
qualitative analysis
comments in the survey, as well as on the employee comments
in reporting to
that can be worked into actual success (and other) stories. The
provide the much-
University reaches out to those employees and builds out a
needed context series of internal case studies that can be used for a variety of
around learning purposes (e.g., marketing and business-case development).
numbers. SALLE AVENUE
6114 LA
Looking ahead, Nuance University is thinking about how to
SUITE 417
further entrench its resources across the organization, to:
OAKLAND, CA 94611
(510) 654-8500
INFO@BERSIN.COM
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

Be available in all “natural places” in which employees look

Help employees to collaborate on projects around these

Help learners personalize their learning experience, so that

Allow learners to add their own knowledge and insights to
the body of knowledge accessed by other employees.

Safari Books Online: A Unique Approach to Driving
Value from a Digital Library

One provider that is worth a look is Safari Books Online, an on-demand
KEY POINT
digital library provider which offers an integrated source for more
The breadth and than 15,850 technology, digital media and business titles from various
publishers, including more than 1,200 video training tutorials.
depth of Safari’s
content goes well Safari Books Online, an organization headquartered in Sebastopol,
beyond books. Its California, is a joint venture between O’Reilly Media, Inc. and Pearson
content in video Education. The company’s offerings include access to the collections

form is extensive
of O’Reilly Media, Addison-Wesley, Prentice Hall, Peachpit Press, New
Riders, Sams, Que, Exam Cram, Cisco Press, Adobe Press and Adobe
and is one of the
Developer Library, as well as hundreds of additional titles from
reasons why this
John Wiley & Sons, Apress, Manning, Microsoft Press, McGraw Hill,
company stands
Oracle Press, IBM Press, Syngress, Morgan Kaufmann, Focal Press,
out in the on- Total Training, VTC, Class on Demand and Skill-Pill. Safari supplies
demand digital on-demand digital libraries to each of the companies profiled in this
library space. research bulletin.

The company’s depth and breadth in offerings is illustrated by
BERSIN & ASSOCIATES, LLC the number of content categories, including digital curricula for
SUITE 417
OAKLAND, CA 94611
(510) 654-8500 resources in video and text formats. (See Figure 8.)
INFO@BERSIN.COM
WWW.BERSIN.COM


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

Safari Books Online offer products for three different needs:

For the enterprise, its products are Safari Business Library (all business
books), Safari Bookshelf (up to 10 books per month), Safari Tech Library
(technology books only) and Safari Premium Library (all content,
including books and videos).

The breadth and depth of Safari’s content goes well beyond books. Its
content in video form is extensive and is one of the reasons why this
company stands out in the on-demand digital library space.

Another reason Safari stands out is the availability of interactive tools
on its platform. With these interactive tools, corporate users are able
to manage, organize and internalize the content. This is especially
important today because organizations increasingly need fingertip
knowledge. No business can avoid the need to operate better, smarter
and faster.

Figure 8: Safari Books Online Products and Subscriptions

SUITE 417
OAKLAND, CA 94611
(510) 654-8500
INFO@BERSIN.COM
WWW.BERSIN.COM Source: Safari Books Online, 2011.


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

Safari Books Online (Release 6.0) has many capabilities including:
KEY POINT

Through the
availability of
interactive tools on
Safari’s platform,
corporate users are
able to manage,
organize and
internalize the
content.

Some of the more unique capabilities that differentiate Safari Books
Online include:

12
-type capabilities, such as mashups13 and “share this” social

12
“Web 2.0” refers to a second generation of web-based communities and hosted services
(such as social-networking sites, wikis, folksonomies, weblogs / blogs, social bookmarking,
podcasts, RSS feeds, social software, web application programming interfaces / APIs, and
online web services), that aim to facilitate creativity, collaboration and sharing between
BERSIN & ASSOCIATES, LLC users. Although the term suggests a new version of the World Wide Web, it does not refer
to an update to any technical specifications, but to changes in the ways in which software
developers and end-users use the web.
SUITE 417
OAKLAND, CA 94611 13
In technology, a “mashup” is a web application that combines data from more than one
(510) 654-8500 source into a single integrated tool, thereby creating a new and distinct web service that was
not originally provided by either source. Content used in mashups is typically sourced from
INFO@BERSIN.COM
WWW.BERSIN.COM include web feeds (e.g., RSS / really simple syndication), web services and screen-scraping.


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

Conclusion

Measuring the approaches to learning that fall under the umbrella of
KEY POINT
the term “informal learning” creates a new measurement challenge

Supporting on- for organizations. Many companies (such as those featured in this
bulletin) are taking an innovative approach to measuring the value of
demand learning
informal learning.
is a function of
modern learning Supporting on-demand learning is a function of modern learning
organizations that organizations that are concerned with going to the point at which
performance happens and making resources available to learners. This
are concerned
has created a new role for L&D – that of the “enabler and facilitator”
with going to the
of all types of performance support, including the on-demand learning
point at which
content from digital libraries. Specifically adapted for informal learning,
performance our Impact Measurement Model14 takes into account contribution,
happens and feedback and activity as measures of learning for digital assets.
making resources
The increasing speed at which the human capital needs of the
available to
organization are evolving, as well as the changing needs and
learners.
expectations for learning on the part of employees, customers and
partners suggests on-demand digital libraries can fill an important
need. As a result, many companies are taking an innovative approach to
measuring the value of informal learning.

If your organization is ready to begin the transition to a modern
learning organization and is looking to leverage informal learning (and
mobile learning) in the enterprise, a simple starting point is providing
on-demand learning support through a digital library. The use of on-
BERSIN & ASSOCIATES, LLC demand digital content libraries is growing and we expect that trend
6114 LA SALLE AVENUE to continue.
SUITE 417
OAKLAND, CA 94611
(510) 654-8500
INFO@BERSIN.COM 14
For more information, The Impact Measurement Framework: An Update for the
WWW.BERSIN.COM Measurement of Informal Learning, Bersin & Associates / Josh Bersin, May 3, 2011.


BERSIN & ASSOCIATES
BERSIN & ASSOCIATES

The Bersin & Associates
Membership Program
This document is part of the Bersin & Associates Research Library. Our
research is provided exclusively to organizational members of the Bersin
& Associates Research Program. Member organizations have access to
the largest library of learning and talent management related research
available. In addition, members also receive a variety of products
and services to enable talent-related transformation within their
organizations, including:
Research – Access to an extensive selection of research reports,
such as methodologies, process models and frameworks, and

Benchmarking – These services cover a wide spectrum of HR and

Tools – Comprehensive tools for HR and L&D professionals,
including tools for benchmarking, vendor and system selection,
program design, program implementation, change management

Analyst Support – Via telephone or email, our advisory services are

Strategic Advisory Services – Expert support for

Member Roundtables® – A place where you can connect with
other peers and industry leaders to discuss and learn about the

IMPACT® Conference: The Business Of Talent – Attendance at
BERSIN & ASSOCIATES, LLC special sessions of our annual, best-practices IMPACT® conference.
Workshops – Bersin & Associates analysts and advisors conduct
SUITE 417
onsite workshops on a wide range of topics to educate, inform and
OAKLAND, CA 94611
(510) 654-8500
inspire HR and L&D professionals and leaders.
INFO@BERSIN.COM For more information about our membership program, please visit us
WWW.BERSIN.COM at www.bersin.com/membership.


Measuring On Demand Learning Bersin and Safari Books OnlineReport

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (16)

Similaire à Measuring On Demand Learning Bersin and Safari Books OnlineReport

Similaire à Measuring On Demand Learning Bersin and Safari Books OnlineReport (20)

Dernier

Dernier (20)

Measuring On Demand Learning Bersin and Safari Books OnlineReport