This document discusses a rubric-based approach for operationalizing and assessing the entrepreneurial mindset. It describes the Kern Entrepreneurship Education Network (KEEN), which focuses on cultivating the entrepreneurial mindset at private universities. The authors developed rubrics to evaluate student work and activities for how effectively they develop aspects of the entrepreneurial mindset. They provide examples of artifacts scored with the rubrics and describe a faculty training protocol for using the rubrics. The goal is to empirically assess the effectiveness of efforts to develop the entrepreneurial mindset at various levels, from activities to courses to programs. This will help improve entrepreneurship education programs.

1. A Rubric-based Approach
J-D Yoder for Operationalizing the
Professor and Chair
Mechanical Engineering Entrepreneurial Mindset:
Rob Kleine
Associate Professor of Marketing An exploratory
Pharmaceutical Business Program
Coordinator investigation

2. We’ve got rubrics, and we’re here
to help you.

3. Ohio Northern University
• Approximately 3500 students among five
colleges
• College of Engineering is about 450
students across 5 majors
• Largely traditional, residential students

4. Kern Entrepreneurship Education
Network (KEEN)
• A program of the Kern Family
Foundation
• Started in 2005
• Private universities with ABET
accreditation
• Focus on the entrepreneurial
mindset, not entrepreneurship

5. The KEEN Network
Baylor University (TX) Santa Clara University (CA)
Boston University (MA) Union College (NY)
Bucknell University (NY) University of Dayton (OH)
Gonzaga University (WA) University of Detroit-Mercy (MI)
Kettering University (MI) University of Evansville (IN)
Lawrence Technological University (MI) Villanova University (PA)
Mercer University (GA) Western New England University (MA)
Milwaukee School of Engineering (WI) Widener University (PA)
Ohio Northern University (OH) Worcester Polytechnic Institute (MA)
St. Louis University (MO)

6. The Entrepreneurial Mindset (KEEN)
1. Effectively collaborate in a team setting
2. Apply critical & creative thinking to ambiguous
problems
3. Construct & effectively communicate a customer-
appropriate value proposition
4. Persist through and learn from failure
5. Effectively manage projects through appropriate
commercialization or final delivery process
6. Demonstrate voluntary social responsibility
7. Relate personal liberties and free enterprise to
entrepreneurship

7. Motivation
• Many schools in the network have created
engineering-mindset-related activities.
• How do we know which ones work?
• How can we leverage “best in class”
activities?
• Note – don’t want it to be relative to student
feelings about the activity, but on the
outcomes.

8. Can you answer these questions?
• How effectively is my PROGRAM
cultivating the entrepreneurial mindset
in our students?
• How effective is each ELEMENT in my
program at cultivating intended aspects
of the entrepreneurial mindset
(whatever the unit: minor, course,
project, in-class activity, etc.)

9. Our effort:
• Develop a common set of
evaluation tools.
• Rubric-based.
• Evaluate student work.

10. A Rubric-based
Approach for
Operationalizing
and Assessing
the
Entrepreneurial
Mindset
Source: Kleine & Yoder (2011), JEEN

11. The process: Activity
Reflection Student
Revision? Artifact
Data
Rubrics
Analysis

12. Exploratory Investigations Utilizing the
Rubric-Based Approach
• Faculty training protocol
• Faculty using and testing the
rubrics

13. A faculty training protocol for rubric-based
assessment Preparation (Kleine & Reid 2012)
1. A student learning outcome of interest
2. A Rubric
3. A student activity that generates
student work (artifacts) relevant to the
learning outcome
4. At least two examples of student work
generated by the activity:
– High proficiency
– Low proficiency

14. Materials distributed to each
participant
1. The rubric that will be applied.
2. A score sheet for each artifact to be
evaluated.
3. Details about the activity (assignment)
that generated the student work.
4. Student work examples (artifacts) that
illustrate high and low proficiency (each
on a separate page.

15. Example Artifact Score Sheet
Artifact Score Sheet
(Please Refer to the Effective Collaboration Rubric)
Does Not Meet Meets Not
Expectations (1) Developing (2) Expectations (3) Proficient (4) Applicable
Contributes to
Team Meetings
Facilitates the
Contributions of
Team Members
Fosters
Constructive
Team Climate
Responds to
Conflict

16. Activity: A Reflection Paper
A one-page “Team reflection” should be written
to describe the results of your “Team
evaluation” rubric. This memo should allow
you to reflect on whether your team has been
effective, especially considering qualities of
effective teams we have discussed. You may
also include any concerns about team dynamics
(these concerns will not be shared with your
teammates without your consent).

17. Sample Artifact Low Proficiency (Student JJ)
Team LL has had a rough start. After having six team meetings the group
has failed to have a hundred percent attendance to a single meeting (AA has
missed 4, DD 3, RR 1, and JJ 0.) Without the whole group present, our
decision making and brainstorming has been continuously pushed back.
When we do have meetings they are somewhat productive, however, for the
following meeting we must repeat what was said previously to ensure
everyone is on the same page. Our team meetings usually last one
hour, during which I do most of the talking while everyone else just agrees
with me. Being the team leader I understand that it is my job to run the
meetings and to keep everyone on track, but it becomes a nuisance when
no one else has any suggestions and must be told to help.
The poor attendance is a result of poor communication skills. Now that
email addresses and phone numbers were exchanged I hope that
communicating will no longer be a problem.

18. Sample Artifact Low Proficiency (Student JJ)
After having six team meetings … When we do have meetings
they are somewhat productive, …
… Our team meetings … during which I do most of the talking
while everyone else just agrees with me. Being the team leader I
understand that it is my job to run the meetings and to keep
everyone on track, but it becomes a nuisance when no one else
has any suggestions and must be told to help.
The poor attendance is a result of poor communication skills.
Now that email addresses and phone numbers were exchanged I
hope that communicating will no longer be a problem.

19. Sample Artifact High Proficiency (Student MM)
PW Inc. is made up of four members: NN, KK, MM, and SS. The team has worked well together and all
the members get along with each other very well. All the individuals on the Team have contributed to the
project equally so far. This is very beneficial for the team because all the assignments are getting done in a
reasonable amount of time. The culture identity was divided up so that each person had to research certain
information. As a team everything has been evaluated and everyone has come to a consensus on team
assignments so that no individual’s opinion is less important than another. Each person has pulled his or her
weight on assignments and/or a task assigned to them, which shows that everyone has respect for each other
and wants to succeed. The key characteristic of the PW Inc. team is our ability to effectively communicate. The
team has a lot of meetings to insure that everything can get done in a timely manner, so communication is key
and so far it has been excellent. In addition to communication, attendance at each team meeting is valuable to
getting things accomplished in a timely manner which the group has done a great job of so far. The team
doesn’t waste time on unimportant things, but rather focuses on the duties that need to get done, such as
deciding the company name, working on the cultural identity, and creating the company.
One aspect I feel like the team has not done well is in group dynamics. I believe that some group
members don’t always participate in discussions because he or she doesn’t want to be wrong or cause a
distraction. My feeling is we are mature enough to not get offended if someone else has a better idea or an
opinion that we don’t necessarily agree with at the time.
Working together and communication are the two main keys to a successful team and so far I believe
that PW Inc. is building for success. We have followed these two keys and have accomplished all the tasks
assigned to the group.

20. Sample Artifact High Proficiency (Student MM)
As a team everything has been evaluated and everyone has come to a consensus … Each
person has pulled his or her weight on assignments and/or a task assigned to them, which
shows that everyone has respect for each other and wants to succeed. The key
characteristic of the PW Inc. team is our ability to effectively communicate.
… communication is key and so far it has been excellent.
… The team doesn’t waste time on unimportant things, but rather focuses on the duties
that need to get done, …
… the team has not done well is in group dynamics. I believe that some group members
don’t always participate in discussions because he or she doesn’t want to be wrong or cause
a distraction. My feeling is we are mature enough to not get offended if someone else has a
better idea or an opinion that we don’t necessarily agree with at the time.
Working together and communication are the two main keys to a successful team … I
believe that PW Inc. is building for success.

21. Example Artifact Score Sheet
Artifact Score Sheet
(Please Refer to the Effective Collaboration Rubric)
Does Not Meet Meets Not
Expectations (1) Developing (2) Expectations (3) Proficient (4) Applicable
Contributes to
Team Meetings
Facilitates the
Contributions of
Team Members
Fosters
Constructive
Team Climate
Responds to
Conflict

22. Distribution of Ratings on the “Fosters Constructive
Team Climate” row of the “Effective Collaboration in a
Team Setting” rubric
12
10
8
6
4
2
0
Does Not Meet Developing Meets Proficient Not Applicable
Expectations Expectations
JJ MM
NOTE: n=23. The distributions are statistically distinct (Chi-squared=27.5, df=4, p<.001).

23. “Effective Collaboration in a Team Setting” Aggregate
Ratings
40
35
30
25
20
15
10
5
0
Does Not Meet Developing Meets Proficient Not Applicable
Expectations Expectations
JJ MM

24. Faculty Using and Testing the Rubrics
• ONU Faculty in Colleges of
Engineering and Business
• Faculty at other KEEN institutions

25. Rubric Testing Protocol
Apply the
Identify a class rubric(s) to one or Provide feedback
Pick a KEEN
activity you do more examples of on how we can
learning outcome
already (or create student work improve the
(or two)
a new one) generated by the rubric(s)
activity

26. Work in progress
• Intend to use in additional courses at
ONU.
• Expand to other KEEN schools.
• All schools are welcome to participate.
• Use feedback to modify rubrics as
needed.
• Inventory entrepreneurial mindset
activities by learning outcome

27. Mapping Activities into Entrepreneurial
Mindset Student Outcomes
Contact:
Feb. 2012 KEEN Student Outcomes (KSO) ___________________________
1 2 3 4 5 6 7
Effectively
Construct & manage projects
effectively through
Apply critical & communicate a appropriate Relate personal
Effectively creative thinking customer- commercialization Demonstrate liberties and free
collaborate in a to ambiguous appropriate Persist through and or final delivery voluntary social enterprise to
team setting problems value proposition learn from failure process responsibility entrepreneurship
For-Credit
Courses
Provide course
number and
title
Co-curricular
Activities
Extra-curricular
Activities

28. Opportunities
• Data Aggregation
• Integrating with ABET
• Integrating Entrepreneurial Mindset and
General Education assessment systems

29. Conclusion
Our goal is to answer the question empirically:
How effective are our efforts to cultivate the
Entrepreneurial Mindset in our students at
each of the following levels:
• KEEN network
• Individual KEEN school
• Program
• Course
• Assignment/activity

30. Thank You!
Questions?

31. Misc debris follows

32. Connect General Education and the
Entrepreneurial Mindset
ONU Gen Ed Learning
Outcomes KEEN Learning Outcomes
1. Effective communication 1. Effectively collaborate in a team
setting
2. Critical and creative thinking 2. Apply critical & creative thinking to
3. Scientific and quantitative literacy ambiguous problems
4. An understanding of diverse cultures 3. Construct & effectively communicate
and their effects on human interaction a customer-appropriate value
proposition
5. Integration of concepts across 4. Persist through and learn from failure.
disciplines 5. Effectively manage projects through
6. Informed and ethical responses to appropriate commercialization or final
personal, civic, and global needs delivery process
6. Demonstrate voluntary social
7. Informed responses to aesthetics in responsibility
art or nature 7. Relate personal liberties and free
enterprise to entrepreneurship

34. Motivating vision
• Cultivate the entrepreneurial
mindset in every ONU student.
• Support those students interested
in becoming entrepreneurs

35. Entrepreneurial Mindset for All
Supporting Future
Entrepreneurs
Zach Ferres
Chris Corrado

36. Entrepreneurship Education at
ONU Circa 2004

37. Entrepreneurship education at
ONU Circa 2004

38. A Multi-Pronged Approach to Engaging
the Entrepreneurial Mindset
Faculty
Curriculum
Development
Entrepreneurial
Mindset Touch
Points across the
Four Years
Extra/Co-Curricular
Assessment
Activities

39. Highlights: Student Engagement
Student competitions
• Polar Innovation Exchange
• Elsewhere
• IIT Idea Chase
• Innovation Chase at LTU
Polar Innovation Exchange Competitions

40. Highlights: Speakers that Inspire
the Entire Campus Community
Paul Polak author of Out of Poverty
and co-founder International
Development Enterprises
Bob Peterson, Pixar Studios

41. Entrepreneurship education at
ONU Circa 2004

42. Thanks
• The Deans
• Dr. Baker and Dr. DiBiasio
• A variety of faculty, especially Dan
Ferguson and Tammy Schakett
• Fantastic students
• KEEN and NCIIA
• External allies