Detailing how we did a major patron data cleanup Presented at ELUNA 2009

1. Taking Your Customers to the Cleaners: Historical Patron Data Cleanup and Routine Purge Preparation Roy Zimmer Western Michigan University

2. About 5 or 6 years ago… No more SSN switch to using WIN WIN is our Western Identification Number

3. About 5 or 6 years ago… No more SSN switch to using WIN Banner WIN is our Western Identification Number

4. About 5 or 6 years ago… No more SSN switch to using WIN Banner New campus ID cards WIN is our Western Identification Number

5. A few less years ago… Rewrote the patron update process to use Banner

6. A few less years ago… Rewrote the patron update process to use Banner Started thinking about not being SSN-based

7. 2007-2008 The WIN had become available in the data feeds for our patron update. Needed to change Institution ID interim step: arbitrary 14-digits -> WIN final step: WIN -> Bronco NetID Patron update was switched from being SSN-based to WIN-based. BroncoNetID is our single signon ID

8. Summer 2008 – What we started with Have data for about 74,000 patrons. About 183,000 barcodes (less than half are active!).

9. Summer 2008 – What we started with Have data for about 74,000 patrons. About 183,000 barcodes (less than half are active!). Several thousand duplicate records, one with SSN, one with WIN (in the SSAN field) The older duplicate record typically had charges, amounts owed, etc.

10. 2008: August – October Most of my time was spent on the cleanup… Dali

11. Patron duplicate detector – LB4020 foreign students various errors Sample follows… August

12. (WINs & SSNs above are not real) Sample output used one day

13. Our first run came up with 3489 duplicate patron records.

14. We created a program that used the LB4020 report as input to identify patron records that we wanted to alter – call it LB4020fix. These records needed to be extracted from Voyager for modification and re-import. Modify me with LB4020fix

15. Voyager has a patron extract utility, but it doesn’t extract all relevant data for a patron. We’d started using our own – patronsif.pl - years ago.

16. Voyager has a patron extract utility, but it doesn’t extract all relevant data for a patron. We’d started using our own – patronsif.pl - years ago. Voyager extract (Pptrnextr) Up to 3 patron-barcode + group combinations Similarly limited number of addresses WMU extract (patronsif.pl) Unlimited patron-barcode + group combinations Unlimited number of addresses + - + - -> +

17. Voyager has a patron extract utility, but it doesn’t extract all relevant data for a patron. We’d started using our own – patronsif.pl - years ago. For the patron cleanup we incorporated patronsif.pl into LB4020fix. Patron notes field problem: CR+LF stored if user pressed the RETURN key creates unwanted extra lines within a record drop_crlf utility replaces “CR+LF” with “space+space”

18. LB4020fix reads the duplicate report (LB4020) and extracts patron sif format data for the duplicate records. SIF-A new WIN-based records BroncoNetID in InstitutionID change expiredate to 1981.01.01 SIF-B old SSN-based records change InstitutionID to current BroncoNetID SIF-C new WIN-based records have the current update, expire, and purge dates and BroncoNetID The heart of the cleanup process

19. SIF-A new WIN-based records BroncoNetID in InstitutionID change expiredate to 1981.01.01 SIF-B old SSN-based records change InstitutionID to current BroncoNetID SIF-C new WIN-based records have the current update, expire, and purge dates and BroncoNetID update, key on SSN purge on expiredate 1982.01.01 [remove new records] 1 LB4020fix reads the duplicate report (LB4020) and extracts patron sif format data for the duplicate records. The heart of the cleanup process

20. SIF-A new WIN-based records BroncoNetID in InstitutionID change expiredate to 1981.01.01 SIF-B old SSN-based records change InstitutionID to current BroncoNetID SIF-C new WIN-based records have the current update, expire, and purge dates and BroncoNetID update, key on SSN purge on expiredate 1982.01.01 [remove new records] update, key on SSN [prep old records to be “new”] 1 2 LB4020fix reads the duplicate report (LB4020) and extracts patron sif format data for the duplicate records. The heart of the cleanup process

21. SIF-A new WIN-based records BroncoNetID in InstitutionID change expiredate to 1981.01.01 SIF-B old SSN-based records change InstitutionID to current BroncoNetID SIF-C new WIN-based records have the current update, expire, and purge dates and BroncoNetID update, key on SSN purge on expiredate 1982.01.01 [remove new records] update, key on SSN [prep old records to be “new”] update, key on InstID [unify old records with new data] 1 2 3 LB4020fix reads the duplicate report (LB4020) and extracts patron sif format data for the duplicate records. The heart of the cleanup process

22. SIF-A new WIN-based records have current BroncoNetID change expiredate to 1981.01.01 SIF-B old SSN-based records change InstitutionID to current BroncoNetID SIF-C new WIN-based records have the current update, expire, and purge dates and BroncoNetID update, key on SSN purge on expiredate 1982.01.01 [remove new records] update, key on SSN [prep old records to be “new”] update, key on InstID [unify old records with new data] 1 2 3 LB4020fix reads the duplicate report (LB4020) and extracts patron sif format data for the duplicate records. The heart of the cleanup process This clean-up process, with variations, was repeated many times. Details omitted here for the sake of brevity (and sanity).

23. Several things went awry along the way. Not all records could be matched up with a WIN or SSN (as reported by LB4020), so those had to be handled by assigning temporary SSNs, WINs, and/or Institution IDs.

24. Several things went awry along the way. Not all records could be matched up with a WIN or SSN (as reported by LB4020), so those had to be handled by assigning temporary SSNs, WINs, and/or Institution IDs. At another point, the interim records used in the process weren’t deleted during a purge. Those had to be detected, reassigned an older expiration date (1971.01.01), and carefully purged before proceeding.

25. We now had 1081 duplicate patron records.

26. We added the expiration date to the duplicate detector, LB4020. Now we could see that all the SSN-based records were expired, or about to be.

27. We added the expiration date to the duplicate detector, LB4020. Now we could see that all the SSN-based records were expired, or about to be. At this time we discovered that new WIN-based records were coming in as duplicates to SSN-based records that were typically set to expire 2008.09.08.

28. We added the expiration date to the duplicate detector, LB4020. Now we could see that all the SSN-based records were expired, or about to be. At this time we discovered that new WIN-based records were coming in as duplicates to SSN-based records that were typically set to expire 2008.09.08. This had to change!

29. We added the expiration date to the duplicate detector, LB4020. Now we could see that all the SSN-based records were expired, or about to be. At this time we discovered that new WIN-based records were coming in as duplicates to SSN-based records that were typically set to expire 2008.09.08. This had to change! And the semester was about to start…

30. Yes, we did avert disaster. But we had more problems. Early September…

31. Yes, we did avert disaster. But we had more problems. The duplicate detection report, which had grown to 60 pages, was now down to 1. The next day it had grown to 3 pages. Early September…

32. Yes, we did avert disaster. But we had more problems. The duplicate detection report, which had grown to 60 pages, was now down to 1. The next day it had grown to 3 pages. Some records not having all fields populated on the LB4020 duplicate detector caused problems. Also had to fix duplicate records where the SSAN field was null. Early September…

33. We removed several hundred obsolete records that had neither WIN nor SSN. Discovered records that had no Institution ID – yet another problem. Mid September…

34. We removed several hundred obsolete records that had neither WIN nor SSN. Discovered records that had no Institution ID – yet another problem. We are now down to 1 SSN-based record. Mid September… This person had our assigned WIN being the same as the SSN. Not supposed to happen! Identified 15 more such instances and submitted them to I.T. for correction.

35. Found some more SSN-based records – don’t know why they still existed – and converted them to being WIN-based. October… Flipped the “switch” so that we no longer get SSNs for our patron update.

36. Still had records from our NOTIS era – pre Summer 1998 Purged them if they: did not have life-time borrowing privileges did not have an SSN recorded did have an Institution ID Legacy data

37. Trouble ahead… 3M SelfCheck

38. Trouble ahead… Multiple Active Barcodes will NOT work with SelfCheck! 3M SelfCheck

39. 3M SelfCheck requires 1 active barcode per patron. We had 11058 patrons with multiple active barcodes.

40. 3M SelfCheck requires 1 active barcode per patron. We had 11058 patrons with multiple active barcodes. Wrote a program to whittle that down. Got them reduced to 300, but the next day, it was up to 1777!

41. 3M SelfCheck requires 1 active barcode per patron. We had 11058 patrons with multiple active barcodes. Wrote a program to whittle that down. Got them reduced to 300, but the next day, it was up to 1777! Under control now, with patrononeactive.pl, running Monday – Friday. This keeps only the most current active barcode for a patron.

42. 3M SelfCheck requires 1 active barcode per patron. We had 11058 patrons with multiple active barcodes. Wrote a program to whittle that down. Got them reduced to 300, but the next day, it was up to 1777! Under control now, with patrononeactive.pl, running Monday – Friday. This keeps only the most current active barcode for a patron. Forgot about those patron records without an Institution ID. Had 882 of them. Fixed them.

43. We looked at records created before 2008, those that had no SSN but did have an Institution ID. Extracted these records, modified them: expiredate = createdate purgedate = expiredate + 4 years Reimported these records. They should disappear with future annual patron purges. An eye towards the future…

44. We still had 11,696 records with no SSN (nor WIN). We expect most of these to be routinely purged in the future, leaving us with 456. What we ended with

45. We still had 11,696 records with no SSN (nor WIN). We expect most of these to be routinely purged in the future, leaving us with 456. When we started, we had about 250,000 patron records. We now have about 68,000. Duplicate records are routinely dealt with. We filter out all but the single most current active barcode for a patron. We will have annual patron purges. What we ended with

46. Know what you’re starting with. Keep your goal in mind. Figure out a good solution. Be flexible. Be ready for mistakes. Watch out for new/current data undoing your changes. Know when you’re done. Worthwhile points…

47. patronsif.pl drop_crlf lb4020.pl lb4020fix.pl patrononeactive.pl patrononactive.ksh Contact me if you would like to get any of the above. Resources

48. patronsif.pl as listed, gets patron data and puts it in patron SIF format. institution ID based. gets all patron+barcode groupings. (not site-specific) drop_crlf shell script that contains this line: perl -pi -e's// /g' $1 replaces CR+LF combination with two spaces. (this is useful anytime you use patronsif.pl) Some details on the resources…

49. lb4020.pl detects duplicate patron records. shows: name, expired (Y/N), SSAN, expire date, modify date, institution ID WMU-specific: indicates whether SSN or WIN in SSAN. modification required for your institution. lb4020fix.pl control structure around patronsif.pl code that uses lb4020.pl output as starting point for the fixing process. creates one or more patron SIF files for fixing data. use drop_crlf if necessary. Some details on the resources…

50. patrononeactive.pl queries Voyager, checking patrons’ active barcodes. if more than one is found, changes all but the most recent active barcodes to other . check the code carefully as it may need modification for your use. (incorporates patronsif.pl code) patrononeactive.ksh combines patrononeactive.pl and drop_crlf in a script suitable for cron use Some details on the resources…

51. Picture © 2008 by Roy Zimmer Thank you for listening. Roy Zimmer [email_address]