Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

ALASI15 Writing Analytics Workshop

Australian Learning Analytics Summer Institute 2015: Writing Analytics workshop

  • Identifiez-vous pour voir les commentaires

ALASI15 Writing Analytics Workshop

  1. 1. cic.uts.edu.au Writing Analytics workshop Simon Buckingham Shum, Simon Knight, Andrew Gibson, Philippa Ryan Australian Learning Analytics Summer Institute 2015
  2. 2. You  have  1000  students  who  would  like   detailed  feedback  on  their  reports.   2 Now.  Please.   (PS:  It’s  1.30am,  deadline  9am)   DraFs,  not  submiGed  assignments  
  3. 3. How  could  this  be  possible?   How  do  we  design  and  validate     wriKng  analyKcs?   What  issues  arise?   3
  4. 4. TEXT  ANALYTICS  IS  WELL  OUT  OF  THE  LAB.   THE  WRITING  ANALYTICS  PRODUCT  SPACE  IS  GROWING…   4 http://turnitin.com/en_us/features/turnitin-scoring-engine http://www.pearsonassessments.com/products/100000681/writing-coach.html
  5. 5. Session plan 5 1.30  Welcome  and  overview  of  session  (SBS)     1.35  What  are  the  biggest  challenges  you  face  in  your  students’  wriKng?  (talk  in  pairs)   1.40  Quick  feedback   1.45  What’s  out  there?  Some  products  and  free  tools,  and  issues  they  raise  (SK)   2.00  Introducing  reflecKve  wriKng  +  GoingOK  demo  (AG)   2.15  How  AWA  handles  reflecKve  wriKng  +  demo  (SBS)   2.30  How  AWA  handles  analyKcal  wriKng  +  demo  (SBS)   2.45  An  academic’s  experience  of  co-­‐designing  AWA  (PR)   2.55  Comments?   3.00  Hands-­‐on  with  GoingOK  or  AWA  or  other  free  web  tools   3.25  Final  comments   3.30  End  
  6. 6. WHAT  ARE  THE  BIGGEST   CHALLENGES  YOU  FACE  IN  STUDENT     ACADEMIC  WRITING?   6
  7. 7. WHAT’S  OUT  THERE  ALREADY   PROVIDING  AUTOMATED   ASSESSMENT  OF   ACADEMIC  WRITING?   7
  8. 8. REFLECTIVE     ACADEMIC  WRITING   ANALYTICS   8
  9. 9. DO  STUDENTS  WRITE  REFLECTIVELY  IN  YOUR  SUBJECT/ DISCIPLINE?   uts.edu.au   “Wri&ng  brings  together  past,    present  &  future.  When  we  write  we    call  on  the  past  and  an&cipate  the    future  even  as  we  inhabit  the    present;  at  the  same  &me,  we    engage  in  an  act  that  is  both  deeply    intellectual  and  overtly  physical”     Yagelski  p.192  2012.      
  10. 10. REFLECTIVE  WRITING:   uts.edu.au Provides   opportuniKes  for   a  deeper   approach  to   learning   Looks  back  on   the  moment  and   looks  forward   Encourages  shiFs   in  perspecKve   Is  personal     SubjecKve     An  individual   interpretaKon  of   experiences       Asks  students  to:   Comment  on   what  they  DON’T   know   Express   uncertainty   Convey  changes   in  understanding  
  11. 11. A  VERY  DIFFERENT  GENRE   uts.edu.au
  12. 12. REFLECTIVE  WRITING  POSES  SOME  PROBLEMS:   Challenging  to  teach:   Different  interpretaKons  of   what  reflecKve  wriKng  is/ what  it  looks  like     uts.edu.au Challenging  to  assess:   How  do  you  idenKfy  a  deep  or  a   superficial  reflecKon?  What  are  you   marking  the  students  on?  Can  some   students  be  advantaged/disadvantaged?  
  13. 13. A  RUBRIC:  LANGUAGE  &  DISCOURSE  FEATURES  &  TYPE  OF   INFORMATION     Context  of  the  event  that  triggers  the   reflecKon  (why,  when,  where,  who,  how   much,  what):  the  more  detail  the  beGer,   as  long  as  the  event  is  non-­‐trivial   Specific  informaKon  about  the  event;   specific  examples  of  acKons  ,  ideas,   learning  experiences   Change  between  assumpKons  of  learning   and/or  prior  knowledge  and  actual   performance     uts.edu.au Reference  to  the  past:   Kme  markers  and  use   of  past  tense  (when  I   started;  before  my   internship);  shiF   between  habitual  past   tense  (I  used  to)  and   the  present  or  the   recent  past  (since  then  I   have  …)  
  14. 14. FROM  INFORMAL  RUBRIC  TO  FORMAL  RHETORICAL  PATTERN   14
  15. 15. 15 FROM  INFORMAL  RUBRIC  TO  FORMAL  RHETORICAL  PATTERN  
  16. 16. AWA:  REFLECTIVE  ACADEMIC  WRITING   16
  17. 17. COMPARISON  OF  HUMAN  AND  MACHINE  ANNOTATION   17 human machine This counts as a “True Positive”
  18. 18. COMPARISON  OF  HUMAN  AND  MACHINE   18 human highlighting automated highlighting
  19. 19. 19 AWA:  REFLECTIVE  ACADEMIC  WRITING  
  20. 20. ANALYTICAL   ACADEMIC  WRITING   ANALYTICS   20
  21. 21. TYPICAL  ANALYTICAL/CRITICAL  RHETORICAL  MOVES   BACKGROUND  KNOWLEDGE   Recent  studies  indicate  …   …  the  previously  proposed  …   …  is  universally  accepted  ...             NOVELTY   ...  new  insights  provide  direct  evidence  ...   ...  we  suggest  a  new  ...  approach  ...   ...  results  define  a  novel  role  ...               OPEN  QUESTION/MISSING  INFORMATION   …  liGle  is  known  …   …  role  …  has  been  elusive   Current  data  is  insufficient  …   TREND   ...  emerging  as  a  promising  approach     Our  understanding  ...  has  grown   exponenKally  ...   ...  growing  recogniKon  of  the  importance  ...   CONTRASTING  IDEAS   …  unorthodox  view  resolves  …   paradoxes  …   In  contrast  with  previous   hypotheses  ...   ...  inconsistent  with  past   findings  ...   SIGNIFICANCE   studies  ...  have  provided  important   advances   Knowledge  ...  is  crucial  for  ...  understanding   valuable  informaKon  ...  from  studies   SURPRISE   We  have  recently  observed  ...  surprisingly   We  have  idenKfied  ...  unusual   The  recent  discovery  ...  suggests  intriguing  roles   SIGNALLING  AUTHOR  INTENT   The  goal  of  this  study  ...   Here,  we  show  ...   Altogether,  our  results  ...  indicate   21  
  22. 22. AWA: ACADEMIC WRITING ANALYTICS TOOL 22
  23. 23. 23 AWA: ACADEMIC WRITING ANALYTICS TOOL
  24. 24. AWA: ACADEMIC WRITING ANALYTICS TOOL 24 Highlighted sentences are colour- coded according to their broad type Sentences with Function Keys have more precise functions (e.g. Novelty)
  25. 25. AWA:  ANALYTICAL  ACADEMIC  WRITING   25 Roll over sentences with Fkeys for a popup reminding you of their meaning
  26. 26. AWA:  ANALYTICAL  ACADEMIC  WRITING  (UTS  CIVIL  LAW)   26
  27. 27. ENGAGING  THE  ACADEMIC  IN   CO-­‐DESIGNING  AWA   27 1.  Trigger  curiosity  through  CIC   briefings   2.  Submit  collaboraKon   proposal  to  CIC   3.  Agree  Kmelines   4.  Think  aloud  user  interface   walkthrough   5.  Preliminary  analyses   6.  SystemaKc  analyses   7.  Refine  parser   8.  Design  student  pilot   9.  Gather  student  feedback  
  28. 28. 28 STRUCTURED  ANNOTATION   FROM  THE  CIVIL  LAW   ACADEMIC  TO  TUNE  THE   PARSER  FOR  HER  DISCIPLINE     (classified  comments  in  word)  
  29. 29. 29 STARTING  FROM  AWA  OUTPUT.  ..  
  30. 30. 30 STRUCTURED  ANNOTATION  FROM  THE  CIVIL  LAW   ACADEMIC  TO  TUNE  THE  PARSER  FOR  HER  DISCIPLINE  
  31. 31. ENGAGING  THE  ACADEMIC  IN   CO-­‐DESIGNING  AWA   31 1.  Trigger  curiosity  through  CIC   briefings   2.  Submit  collaboraKon   proposal  to  CIC   3.  Agree  Kmelines   4.  Think  aloud  user  interface   walkthrough   5.  Preliminary  analyses   6.  SystemaKc  analyses   7.  Refine  parser   8.  Design  student  pilot   9.  Gather  student  feedback  
  32. 32. 32 UTS  CIVIL  LAW  STUDENT  VIEWS   “takes  the  emo&on  out  of  having  your  work  scru&nized”  respondent  12;  “it  was   not  embarrassing  in  the  way  that  it  can  be  when  a  tutor  or  marker  gives   feedback”  student  7  reflec1on  notes     “I  definitely  found  it  useful.  It  also  made  me  realise  that  I  tend  to  use  bold,   certain  language  in  making  my  point  towards  the  end  of  each  paragraph  rather   than  up  front  at  the  beginning  (when  introducing  my  point).”  Respondent  5   “I  realise  now  what  descrip&ve  wri&ng  is  -­‐  the  soKware  had  quite  a  bit  to  say   about  my  lack  of  jus&fica&on  -­‐  also  true  -­‐  pressed  for  &me  and  difficult   circumstances  have  caused  this  for  me  in  this  instance  -­‐  good  to  see  it  sampled.”   Respondent  9  
  33. 33. 33 UTS  CIVIL  LAW  STUDENT  VIEWS   it  is  possible  to  make  a  clearly  stated  point  in  an  academic  way  without  using  one  of  the   markers  …saying  that  if  a  paper  does  not  use  specified  'signposts'  suggests  that  the  wri&ng   is  not  clear  and  ‘academic’  (see  ’&ps’  on  the  results  page),  constricts  wri&ng  style.  I  think  it   is  possible  to  be  clear  about  your  posi&on  without  explicitly  saying  'my  posi&on  is…’.   respondent  11   “found  that  the  tool  was  limited  in  its  ability  to  pick  up  on  summary  sentences.  It  was  able   to  detect  phrases  such  as  ‘ul&mately,  this  essay  will  conclude,’  or  ‘in  conclusion,’  but  the   use  of  adverbs  such  as  ‘thus,’  and  ‘evidently,’  in  conclusive  statements  failed  to  be   recognized.”…   “Another  limita&on  is  that  certain  sentences,  which  were  recounts  or  mere  descrip&ons   were  deemed  important,  whilst  more  substan&ve  parts  of  the  essay  containing  arguments   and  original  voice  failed  to  be  detected.”    
  34. 34. ONGOING  WORK   From  highligh&ng  to  ac&onable  reports   •  How  to  bridge  the  gap  between  the  current  ability  to  highlight  sentences,  and   capability  to  generate  a  meaningful,  ac&onable  report     “Does  this  highligh&ng  mean  it’s  good?”   •  Without  geYng  into  automated  grading,  students  (and  educators)  are  s&ll  keen   to  know  if  there  are  signature  paZerns  that  are  proxies  for  quality  –  a  research   ques&on  we  are  now  inves&ga&ng     “Algorithmic  Accountability”   •  Cri&cal  to  forge  trust  between  the  stakeholders:  Educators,  Analysts,  Students     34
  35. 35. WRITING  ANALYTICS   HANDS-­‐ON   Wri&ng  Tools  tour:  hGp://utscic.edu.au/events   UTS  —  AWA:  hGp://bit.ly/utsawa     password:  alasi15   QUT  —  GoingOK:  hGp://alasi.nlytx.io  

×