AAPB: National Federation of Community Broadcasters
•Télécharger en tant que PPTX, PDF•
1 j'aime•1,388 vues
This AAPB presentation was given at the 2016 National Federation of Community Broadcasters (NFCB) conference in Denver, Colorado. Panelists included Karen Cariani, Alan Gevinson, Rachel Curtis, and Casey Davis.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (16)
En vedette
Similaire à AAPB: National Federation of Community Broadcasters
Similaire à AAPB: National Federation of Community Broadcasters (20)
Plus de WGBH Media Library and Archives
Plus de WGBH Media Library and Archives (16)
Dernier
Dernier (20)
AAPB: National Federation of Community Broadcasters
- 1. a collaboration between WGBH and the Library of Congress Seeking to preserve and make accessible significant historical content created by public media, and to coordinate a national effort to save at-risk public media before its content is lost to posterity
- 3. KAREN CARIANI AAPB Project Director, WGBH Senior Director, WGBH Media Library and Archives Karen_Cariani@wgbh.org @kcariani
- 4. 40,000 hours of digital material initially from more than 100 stations – 5,000 hours from born digital files 2.5 million inventory records from 120 stations Identified more than 3 million items kept at stations, archives, producers, university collections across the country dating back to the 1950s Initial Collection
- 6. Launched New Projects • National Educational Television (NET) Collection Catalog Project – Build a national inventory of titles created for National Educational Television (pre-PBS) • AAPB National Digital Stewardship Residency – Expand the NDSR program to include geographically diverse residencies at organizations with public media collections • Improving Access to Time-Based Media through Crowdsourcing & Machine Learning
- 7. Public Broadcasting Metadata Dictionary (PBCore) – Continuing the development of PBCore as a metadata schema for media materials – Engaging the PBCore community for input in the development – Outreach to new adopters of PBCore – Collaborating with EBUCore to utilize their RDF ontology For more information: pbcore.org
- 8. Challenges of Access viewing/listening Rights issues – we don’t own rights to all the content, and potentially neither does the donor Risk assessment – How much can we rely on fair use and being an archive/library for educational and scholarly access? Large volume – Adds to difficulty of discovery, costs for access Not knowing what it all is – need for descriptive data for discovery – Not completely catalogued
- 12. Metadata
- 13. Public Broadcasting Metadata Dictionary – Descriptive – Administrative – Technical – Extensions pbcore.org
- 15. Archival Management System (AMS) github.com/avpreserve/ams • Developed by AVPreserve • LAMP stack application • Licensed under GPLv3 • Create, store and manage descriptive, technical and preservation metadata • Stations can log-in and access their metadata and stream content • Digitization vendor sends technical metadata to server using BagIt specification; preservation metadata sent to server through Google Spreadsheets API • Provides PBCore and CSV import and export • PBCore and PREMIS REST API
- 16. To fully catalog the AAPB collection, it would take one person 32 years working full time. So now we’re focusing on normalization & MVC. • AAPB Phases of Cataloging – Inventory (item-level) – Normalization: formatting dates, titles, splitting out types of titles; formatting existing data at a high level in CSV – Minimum Viable Cataloging (MVC): one-by-one cataloging, 15 minutes per item, spend most time reviewing opening and closing credits; adding dates, titles, creators, contributors, publishers, copyright information, topic, genre (format), and copyright information. This will take approx. 6 years. – Full Cataloging: I’ll come back as a volunteer and do this after I retire
- 17. Metadata Titles Contributing Organization Identifiers Description Date Asset Type Genres Creators Contributors Publishers Media Type Copyright Duration This is the metadata we expose to users on the public- facing website at americanarchive. org.
- 18. Our advanced search, following the Google model.
- 19. A way for users to explore the collection by geographic location.
- 20. A way for users who don’t know what they are looking for to dive in to the collection
- 22. There’s a lot of stuff on our site. Keeping the access facet always expanded w/ help text may help users avoid confusion.
- 23. SLIS Student Contributions • Working with interns to help catalog – They get experience, we get cataloging! • Working with students at Simmons – Evaluate website design and functionality
- 24. ALAN GEVINSON AAPB Project Director, Library of Congress Special Assistant to the Chief, National Audio-Visual Conservation Center, Library of Congress agev@loc.gov
- 25. Goal: A Centralized Web Portal for Discovery • All AAPB digitized content on specific topics discoverable through single searches • Direct links to public media on other sites • One-stop shopping for users • Helps solve the separate silos syndrome • DPLA as a model
- 26. State Percent Massachusetts 27% Maryland 11% Minnesota 8% New York 7% California 6% New Jersey 5% Montana 4% South Carolina 4% Illinois 4% Wisconsin 3% North Carolina 2% Connecticut 2% Overall Collection Stats: Place 38 states and 2 non-states (DC and Guam) submitted materials for digitization 12 states (on right) contributed above average amounts 12 states did not participate
- 27. Overall Collection Stats: Region Region Number of Assets % Northeast 19,202 31.1 Mid-Atlantic 14,974 24.2 Midwest 11,262 18.2 West 8,187 13.3 South 7,545 12.2 Pacific (AK, HI, GU) 905 1.5
- 28. Decade Number of Assets 1940 – 1949 3 1950 – 1959 1,057 1960 – 1969 3,687 1970 – 1979 2,950 1980 – 1989 6,587 1990 – 1999 7,999 21st Century 7,591 To the best of our knowledge, content in the collection ranges from 1947 – 2013, although only 44% of records are currently dated. Overall Collection Stats: Time
- 29. AAPB • national history • regional history • local history • news • public affairs • civic affairs • religion • education • environmental issues • music • art • literature • filmmaking • dance • poetry documents
- 30. AAPB can be of value for scholarship because of ... Geographical breadth • to uncover ways that national and global processes played out on the local scene Chronological reach • to document change (or stasis) over time
- 31. Scholars’ • “I have long been frustrated ... gaining access to the vast audiovisual record of my period” • “content [is] held in relative obscurity by the TV and radio networks and the public TV stations” • “programs remain almost impossible to locate and access ... locked in the collections of its many member stations” • “Working to document recent American history without access to the pictures has been a real challenge” • “key historical moments and events are lost to us forever” complaints
- 33. Steps to Determining ORR Access • Joints rights meeting (October 2014) • Reviewed existing contributor agreements • Quitclaims – 75% signed • Bucket analysis Decision: ORR or On Location
- 34. Genre Number of Assets Magazine 2,657 Talk Show 2,628 News Report 2,018 Event Coverage 414 Call-in 329 Documentary 286 Interview 112 Debate 98 Unedited Footage 70 More than 13,000 items (20% of the collection) are in the ORR Contributed by 65 participating organizations representing 36 states ORR content is 40% audio & 60% video Online Reading Room Stats
- 35. AAPB can be of value for scholarship because ... • scholarship pertaining to the period of 1973 onwards is “limited, fragmentary, and politically conflicted” • for the 1980s, “the archival and monographic work … has not yet been done” • accounts about the 1990s and later have “not really been history” Kim Phillips-Fein, “1973 to the Present,” in American History Now (2011)
- 36. The Importance of Local History ... • “emphasis on diversity” • “the history of the nation is many different stories, no one of which can be considered the ‘main’ story” • a “skepticism about finding common definitions of American nationalism or discovering common values” among many historians of the 1960s and 1970s History from the bottom up (quotes from Alan Brinkley)
- 37. The Importance of Local History for ... • relating “national experiences to larger processes and local resolutions.” Thomas Bender Rethinking American History in a Global Age (2002)
- 38. The Importance of Local Civil Rights History ... • “publication of local and state studies ... marked a major shift in the field” • “called into question many of the top-down generalizations” • “studying the importance of the movement’s local, indigenous base fundamentally alters our picture of the movement and its significance” • a “bottom-up perspective” can expose “students to a world beyond their immediate experience” Emilye Crosby, ed. Civil Rights History from the Ground Up (2011)
- 40. Civil Rights Movement • Montgomery, AL 1956 • Tallahassee, FL 1959-61 • Charleston, SC 1960 • McComb, MS 1961 • Baton Rogue, LS 1961 • Monroe, NC 1961 • Birmingham, AL 1963 • Savannah, GA 1963 • St. Augustine, FL 1964 • MS Freedom Summer 1964 • Natchez, MS 1965 radio stories in AAPB from ...
- 41. NAEB (National Association of Educational Broadcasters) Radio Network • “a first-class journalist coup [that] constituted a remarkable social document for the ear.” • “availability of extended time from assorted commercial pressures” • “depth of treatment and outspokenness seldom available elsewhere on the dial” -- Jack Gould (NYT) Birmingham: Testament of Nonviolence Broadcast June 10, 1963
- 42. RACHEL CURTIS AAPB Project Coordinator, Library of Congress Digital Conversion Specialist, Library of Congress rcur@loc.gov
- 43. Preservation at the Library • Our goal is to make sure these files are available for posterity • Files are converted into an “evergreen” format – Audio: BWF (broadcast wave format) RF64 – Video: 10 bit JPEG2000 reversible 5/3 in an .MXF OP1a wrapper • Checksums • Quality control
- 44. Checksums • A digit or hash representing the sum of the correct digits in a piece of stored or transmitted digital data, against which later comparisons can be made to detect errors in the data.
- 45. Quality Control • Detect possible issues that occurred during digitization • Cannot play through everything, but some things have to be looked at by human eyes • Some issues are related to the condition of the original tape
- 46. Metadata • Metadata is preserved along with the digitized files • We import the metadata given to WGBH for each collection • We add any additional preservation metadata
- 47. Ingestion process • Files arrive at the Library on LTO tape or hard drive with a master list of file names • Catalog records created • Files are offloaded • Checksums verified • QC performed • Files ingested into the digital archive and written to tape • Proxies created
- 48. Best Practices • Long term archival storage – Migrations every 3-4 years – Routinely validating checksums – Verifying archive contents • Updating workflows – Updating metadata as files are migrated – Keeping up with file and storage wrapper standards • Plan for the future of storage
- 49. Future of the Digital Archive • Learn from past collections • Prepare for born digital content • Prepare for complex media content
- 50. Ingestion Limitations • Technical limitations • Staff limitations • Working with differing specifications
- 51. Growing the Collection • Working with donors and vendors • Dealing with collections in different stages – Already digitized material – Born-digital material – Material that needs to be digitized
- 52. • PBS NewsHour • American Masters • NET programs • NHPR • Don Voegeli Collection • Southern California Public Radio • Ken Burns’ The Civil War interviews
- 53. CASEY E. DAVIS AAPB Project Manager, WGBH Casey_Davis@wgbh.org @caseyedavis1
- 54. Get your station on the map!
- 55. Why get involved in the AAPB? So that your station can begin to identify, manage and preserve your collection So that other producers can find and potentially license your content So that scholars can refer to your content in their research So that educators and students can access your content as primary source material So that lifelong learners can watch and listen to the programs they remember from the past Your station has created an archival record of your community and our shared cultural heritage. Making this historic content available fulfills public media’s core mission to educate, inspire, and enlighten.
- 56. Before you can contribute content to the AAPB, you first need to get a handle on what you have and where it is located.
- 57. Start with an item-level inventory Same row continued... Download a template: https://s3.amazonaws.com/americanarchive.org/resources/pbcore_excel_template.xls
- 58. Inventory recommendations Do not play back the tapes. Simply type into your spreadsheet the information you can glean from the tape label. Create unique identifiers for every asset in your collection. Label your tapes with the identifier and include the identifier as a column in your spreadsheet. Note where there are duplicates. For digital video/audio files, use the file name as the unique identifier. Include other fields such as Title, Title Type (Series, Program, Episode), Description, Date, Asset Type, Location (Room and shelf number), Format, Generation, Duration, & Physical Condition. Other descriptive info such as Topic & Genre are helpful for search and discovery. Use consistent vocabularies as much as possible.
- 59. Send us your spreadsheet inventory for ingestion into the AMS
- 60. We’ll make your records available on the AAPB public website.
- 62. Managing your digital files Many producers are creating “born digital” content. This material is even more vulnerable than your analog tapes. The cloud is not a long term archive! Use consistent and unique file naming conventions, and do not use special characters in file names. Store the highest quality files on hard drives or LTO tape, and organize the files in a way that would be understandable to someone else 10 years from now. Keep two versions of the files: a version that you store for long term, and a version that you access frequently. Exercise the hard drives regularly, and plan to migrate the files to new storage media every 3- 4 years.
- 63. Contributing digital files to AAPB AAPB can acquire a certain number of hours of digital content per year. For your collection to be considered, fill out our Collection Acquisitions Form: Download: https://s3.amazonaws.com/americanarchive.org/resources/AAPB_collection_acquis itions_form.pdf We also ask all donors to sign our AAPB Deed of Gift, an agreement between the collection donor and WGBH and the Library of Congress. Download: https://s3.amazonaws.com/americanarchive.org/resources/AAPB_model-deed-of- gift.docx
- 64. FAQ Q: Will my station be able to retain copyright ownership to our collection? A: Yes, though you are donating the digital copies to AAPB, you can retain copyright ownership. Q: Will we automatically make all of your content available online? A: No, we will follow our ORR policies to determine what we are comfortable making available online. We do prioritize acquisitions that are more likely to go in the Online Reading Room. Q: What happens if a researcher or producer wants to license my content? A: The researcher or producer is referred back to you.
- 65. AAPB Technical Specifications Once a collection has been approved for submission to AAPB, AAPB works closely with collection donors and their digitization vendors to clarify required and requested technical specifications. These include file formats, naming conventions, storage media for delivery of files, associated technical metadata, etc. We would be happy to work with you to develop your digitization vendor RFP. Current AAPB tech specs for donors are available for download here: https://s3.amazonaws.com/americanarchive.org/resources/AAPB_TechSpecs_1606 06.pdf
- 66. How can my station support this work? Hire an archivist. Advocate for staff resources to be allocated to archiving your content, and get them trained on best practices. Licensing and your own reuse of existing footage are potential ROIs. Hire Library Science graduate student interns. Check out ALA’s accredited library programs in your area: http://www.ala.org/accreditedprograms/directory Partner with a local university to preserve your analog collection. Seek funding from local foundations or donors interested in media and/or local history. Seek federal or other national grants targeted toward archiving.
- 67. Grant programs Institute of Museum and Library Services – imls.gov Council on Library and Information Resources – clir.org National Endowment for the Humanities – neh.gov National Endowment for the Arts – arts.gov Association for Recorded Sound Collections – arsc-audio.org NARA’s National Historical Publications & Records Commission – archives.gov/nhprc Grammy Foundation – grammy.org/grammy-foundation/grants
- 68. Get involved in the archiving community! Association of Moving Image Archivists 2016 Conference: Nov. 9-12, Pittsburgh Website: amianet.org Association for Recorded Sound Collections Website: arsc-audio.org Radio Preservation Task Force Website: radiopreservation.org
- 70. THANK YOU!
Notes de l'éditeur
- Our goal is to not only provide access and discoverability to the collection, but also long term preservation.
- The initial collection consists of 40,000 hours of digital media from over 100 stations across the country. 5,000 hours were ingested as already existing digital files the rest were created as a large analog to digital digitization project with a single vendor. Through an inventory project we collected 2.5 million records from 120 stations. These are not full catalog records, but inventories identifying that a physical object exists, and the format of that object (tape or file). In addition, we know there are over 3 million items sitting at station, archives, independent producers, universities, etc.
- The entire digitized collection is accessible on location at WGBH and the Library of Congress Wash DC reading room. About 13,500 items are available anywhere in the US via the website. The additional 2.5 million inventory records are also available on the website. 4 curated exhibits, and more to come, help highlight items in the collection that may be hard to find.
- Initial CPB funding has been extended for another year which will partially cover core staff. In addition, we’ve raised close to $2 million to launch 3 new projects which will help sustain our core staff for another 3 years and further our goals. The first is focused on the National Educational Television collection of pre PBS programing which contains some of the earliest public television programs covering historic events and issues of the 1950’s and 1960’s. The second is to place 7 Masters level graduates as paid fellows for 10 months to work on digital preservation projects at 7 stations or institutions holding public media content across the country. And third, working with the Pop-Up archive, we will create transcripts of all 40,000 hours using speech to text tools to improve searchability and create games for the public to correct and enhance the transcripts.
- As part of the American Archive we are also revitalizing PBCore –that’s a media data schema geared toward a/v collections. In particular we hope to map to a similar schema EUCore so PBcore users can take advantage of the rdf ontology EBUCore has developed.
- The challenges of access for large media collections are numerous. Certainly access to analog media is becoming more difficult as formats become obsolete and fragile. However, with digital files and the functionality of websites, technically access is much easier, and it usually boils down to legal and rights issues partly because the access becomes broader. What do we have the rights to make accessible, and of the analog material what is affordable to put into a digital form. Complicating the matter is that we don’t necessarily own the rights to al the content and neither does the donor, and often we don’t know who does. So it is often a risk assessment – how much can we rely on the law that grants certain rights to libraries and archives to allow access, and how much is risk – who will care enough to bring a legal case against us? And of course the larger the collection, means there is more data for the archivist to process and make decision and the user to scroll through. Fully cataloguing media means watching or listening to the entire item which is labor intensive.
- So we have launched our website and chose to to expose all the data we had regardless of how complete it was. The items with better data will show up in refined searches. The items with not so great data may seem like clutter to some, but to others may pique their interest. We’ve developed some screens to facet the searches so that there is perhaps less clutter and the user can choose how much they want. But the goal was to get the user to the content as quickly as possible.
- AMS – metadata creation and management, stations can log in and access their material, stores technical and preservation metadata, allows digitization vendor to import technical and preservation metadata, provides PBCore and PREMIS REST API
- AMS – metadata creation and management, stations can log in and access their material, stores technical and preservation metadata, allows digitization vendor to import technical and preservation metadata, provides PBCore and PREMIS REST API
- To fully catalog the whole collection we figured it would take 1 person 32 years and we can’t afford to hire an army of cataloguers, nor do we want to wait before making the material accessible. We decided to tackle the problem by cleaning up the data as quickly as possible and doing a minimal level of cataloguing – enough to allow better discoverability and some sense of what the content is. We are spending no more than 15 minutes per item and grabbing key data like verifying the title, the credits, noting if there is a date anywhere on the lead in countdown or on the copyright notice. Even this minimal data will take us about 6 years to complete the whole collection. We are all planning on volunteering after we retire.
- This list is what we expose to users on the website.
- We’ve designed the search to be as google like as possible
- We have a map visualization for users to explore by state.
- And we’ve thrown items into broad categories based purely on indexed terms that Solr pulls out – not based on any real cataloguing. (Until we have catalogued that item.) So you may find some clutter in these topics.
- The facets all users to narrow down their search results. I only want to see items that are digitized or streaming.
- AAPB hopes to provide a centralized web portal of discovery where researchers, educators, students – really anyone – can find relevant public broadcasting programs existing either on our own site or on sites belonging to other archives and stations. With approximately 1,250 public radio and television stations in existence, one access point will aid scholars interested in researching how national or even international topics have been covered in divergent localities over the past 60+ years. AAPB has made a start at becoming that portal. If stations and archives operating their own websites will send us metadata, we will provide direct links from AAPB to digitized files on the other sites. For a researcher, this would be one-stop shopping. This is how the Digital Public Library of America (DPLA) operates, and we plan soon to make AAPB files accessible through searches on the DPLA website as well as on our own. We want to help solve the separate silos syndrome.
- Our collection includes content from 38 states.
- The Northeast, Mid-Atlantic, and Midwest regions account for nearly three-quarters of the total number of files.
- We currently have metadata that includes dates for 44% of the total collection. Programs produced since 1990 account for more than half of the dated material, while the 1940s through the 1970s together account for only ¼ of the total.
- Places and years thus are skewed favoring some states, some regions, and some decades over others. Even with these imbalances, this remains an extraordinarily diverse collection of programming covering many localities across the U.S. and much that occurred throughout the nation during the past 60+ years. The materials offer unique television and radio programming that document the subjects you see here.
- Because of the geographic breadth of the material, researchers can use the collection to help uncover ways that national and even global processes played out on the local scene. The long chronological reach from the late 1940s to the present will supply researchers with previously inaccessible primary source material to document change over time.
- Scholars who have supported our work have repeatedly complained about the lack of access to audiovisual materials. A historian of the civil rights movement has written to us, “I have long been frustrated by difficulty [of] gaining access to the vast audiovisual record of my period.” A media historian similarly writes that public broadcasting programs, “remain locked in the collections of its many member stations . . . Bringing them out of obscurity . . . would be an immense boon to scholars not only of media history but of the era as well.”
- To help with these complaints, we launched our Online Reading Room in October 2015. We ask that users agree to our terms of use before accessing our digital files.
- We held a rights meeting in Oct 2014 with lawyers and advisors from the Library, WGBH and the Cybernet Unit of the Berkman Center for Internet & Society at Harvard. We reviewed our existing agreements with the donors and sent additional quit claims to them to sign. And we conducted an analysis of the types of programs that have been broadcast over public media to help us determine whether accessibility to programs should be confined to the Library and WGBH, or whether we were comfortable with making the programs available online to those who agreed to our terms of use.
- Over the past year, the Library and WGBH have worked diligently to clear more than 13,000 programs to become accessible in the Online Reading Room in order to provide streaming access to this material for research, educational, and informational purposes to users anywhere in the U.S., who agree to stipulated rules of use. Over the next year, we plan to clear many more additional programs.
- The material in the collection is especially important because of the era it reflects. There remains much basic excavation and interpretive work in recent American history for the present generation of scholars to accomplish. A recent essay noted that American history scholarship pertaining to the period of 1973 onwards is “limited, fragmentary, and politically conflicted.” Accounts about later periods, the author concluded, have “not really been history.”
- The American Archive collection contains a wealth of material produced locally for local audiences. These programs represent an untapped important resource. During the 1960s and 1970s, many historians began to focus on social history, history from the bottom up, instead of on national elites. This “emphasis on diversity,” Alan Brinkley has written, presumed “that the history of the nation is many different stories, no one of which can be considered the ‘main’ story.”
- More recently, some historians also have advocated for integrating the national story into wider contexts. The goal is to relate “national experiences to larger processes and also to local resolutions,” Thomas Bender has written.
- The history of the civil rights movement provides a good example of the importance of local studies. The publication in the mid-1990s of a number of key works on local civil rights history, “marked a major shift in the field,” Emilye Crosby has written. “Collectively,” she states, “these books highlighted how acknowledging and studying the importance of the movement’s local, indigenous base fundamentally alters our picture of the movement and its significance.” Crosby also argues that a focus on local studies can help people outside of academia, especially young people, to better appreciate the experiences of others, to give them, in her words, “the opportunity to understand more fully the power of perspective.” Local studies, she writes, can make civil rights history “relevant and accessible to many more people.”
- Our collection can help with this. Film, video, and audio often have the ability to convey experience more powerfully than the written word. It can turn students into vicarious witnesses of historical events. Audio at times can grab a student’s attention even more effectively than film or video due to its power to establish an experiential connection between listener and speaker.
- The materials in our collection are rich in primary source material documenting the southern civil rights movement in interviews, speeches, and lectures. We’ve developed a special curated exhibit on that topic called “Voices from the Southern Civil Rights Movement” to collect these programs in one spot.
- I want to end with a clip from that exhibit, from the six-part series “Birmingham: Testament of Nonviolence,” produced in May and June 1963 for the National Association of Educational Broadcasters. In 1951, the NAEB established a tape duplication exchange system at their headquarters in Urbana, Illinois, where programs from member stations were copied onto tape and sent to other stations for broadcast over the National Educational Radio network, a precursor of NPR. The University of Maryland, which holds the NAEB collection today, sent the American Archive nearly 5,500 recordings, including this series, which was produced by member station WRVR from the Riverside Church in New York, which sent reporters to cover the Birmingham movement. The series was praised by New York Times critic Jack Gould, who called it “a remarkable social document for the ear.” Noncommercial radio, Gould wrote, was able to provide "extended time free from assorted commercial pressures" that allowed for the broadcast of programs with "depth of treatment and outspokenness seldom available elsewhere on the dial." In this clip, one of the more than 3,000 students who participated in the Birmingham Children’s Crusade tells a WRVR reporter of her experiences in a Birmingham jail.
- Our goal in collaborating with WGBH is to not only provide access to these public media materials, but to ensure their preservation for posterity. While WGBH accepts the proxy files, which they make available through the AAPB website, the Library stores the preservation files at the Packard Campus. These files are in an “evergreen” format. This means JPG2000 for images and BWF (broadcast wave format) RF64 for audio files. These formats were chosen by the Library because they have the greatest potential for not becoming technically obsolete in the future (as most lossy compressed formats are expected to become) and because it has a clear licensing agreement that allows the Library to preserve J2k coded material in perpetuity (as opposed to proprietary formats such as MPEGs, DV, Apple, Windows Media). It also allows us to preserve the material with a common format so we don’t have to maintain different equipment and software types to reads all of the different formats. Also, if something were to every happen to the proxy/access file, the J2K file can be converted into whatever format is required. In addition to converting files to an evergreen format, part of the preservation process is to ensure that these files remain unchanged and viable for the future. This means verifying checksums and running files through a quality control system.
- A checksum is a digit or has representing the sum of the correct digits in a piece of stored or transmitted digital data, against which later comparisons can be made to detect errors in the data. Checksums are used to ensure the integrity of a file as it is moved between storage devices and to verify that it hasn’t been tampered with. Best practice is for a checksum to be generated as soon as a file is created and that data should travel with the file at each point of migration. Each time a checksum is re-calculated, the value should always be the same. A different value means something has gone wrong with the file and we’ll have to go to a back-up file. There are a few different types of checksums available (SHA-1, md5, SHA-2), but we’re only accepting md5 for incoming material. You can download a utility that will generate checksums for your files.
- Quality Control or QC is another important part of the preservation process. A file is no good to us if it is unviewable due to a problem during digitization. Additionally, issues that existed on the original analog material need to be documented, too, so they aren’t attributed to an error during digitization. These issues won’t be caught by a checksum verification. Because we are dealing with so many files for this project, looking and listening to each and every file we ingest isn’t a viable option. There is software available to automate this process, but manual checks will be required when a problem is discovered by the software. Ideally, QC will be performed right after digitization by the vendor and again when files are accepted at the Library. There are different software options available out there and vendors will have their own workflows for QC. Vendors should provide a QC report to you and us.
- Not only do we ingest the preservation file, but the metadata that is associated with that file. This includes descriptive, technical, and preservation metadata. Before the Library can ingest a preservation file, we have to create a record for that file in MAVIS, our collection management system. This record contains the descriptive metadata, technical metadata, and preservation metadata. It also includes the checksum value, GUID, QC report, and is linked to our ingestion platform, PCWA. A brief overview on the differences between these three types of metadata: Preservation metadata includes information related to digital preservation, technical metadata to the technical specifications of a digital object or analog tape, and descriptive metadata describes the content and what it is. Metadata is imported into the AMS and we at the Library harvest this metadata and map it to MAVIS, some amount of clean-up has to occur separate from cataloging.
- A brief and simplified overview of our process: We get the files on LTO Tape or hard drive. Catalog records are created, the files are offloaded, checksums verified and QC checks are performed. The files are then ingested into our digital archive and proxies are created. Even though proxy files are sent to WGBH, we prefer to create our own. There are two reasons for this, the first is to streamline the ingestion process. We’ve learned from the original 40,000 hours that dealing with preservation files and proxy files at the same time can create issues with our workflows. Second, the creation of proxies on our end can serve as a QC check of sorts – if a proxy cannot be created from a preservation file, it’s a good indicator that something is wrong with that preservation file.
- After the files are written to the archive, we’ll do migrations every 5-7 years onto new media. We store two copies of the preservation files at two geographically separate locations. We will also routinely validate checksums and verify the archive contents. The metadata will be updated as files are migrated. We also need to update our workflows and keep up with changes in best practices and file storage and file wrapper standards. This involves testing new products and methods before implementation. We plan to update our workflows as needed. Planning for future technological changes to software and hardware is a key part of digital preservation.
- During the original 40,000 hours, we learned a lot about what worked, what didn’t work, and what needs minor tweaking. Ingesting 40,000 hours from several different stations was a challenging task, but it did give us a good idea of what we need to do to prepare for future collections. We also need to be prepared for born-digital content. While we received some of this during the original 40,000 hours, we are still figuring out the best way to handle it. We also need to prepare for new file formats that we will encounter in the future.
- As much as we would love to take everything, we do have limitations to the amount of material we can accept per year. Our QC system can process only so many files per month. Our staff also has limitations and working through the negotiation process also takes time. Often, there are three to four institutions involved with each project. The Library, which takes the preservation files, WGBH, which takes the access copies, the donating individual/institution, and the vendor. Negotiating between these four entities is often time-consuming, but putting time into the front-end (specifying format requirements, delivery, contract deadlines, etc.) will definitely save everyone’s time later down the road.
- As we move beyond the original 40,000 hours, we are beginning to get a better handle on how much material we can accept per year. As we work with donors, we are dealing with collections that are in different stages. For example, some have already been digitized from original analog, while others have yet to be. We also have born-digital content and collections that might have material in more than one of these stages. Casey will talk about how to get your material into the AAPB in more depth, but we are happy to work with you to find the best solution. I’m going to talk a little bit about the material we’ve accepted since the original 40,000 were ingested.
- Here are some collections we have recently accepted or are in the process of acquiring. We are working with NewsHour Productions to digitize, preserve, and make publicly accessible on the AAPB website 32 years of NewsHour predecessor programs, from October 1975 to December 2007, that currently exist on obsolete analog formats. We have selected a digitization vendor, accessioned content at WETA, and instituted quality control procedures to ensure that all digitized files will be properly preserved for present and future generations. Thirteen Productions LLC has agreed to provide AAPB with metadata and nearly 2,500 files containing more than 1,800 hours of complete interviews conducted with key figures in American culture and the arts for American Masters between 1993 and 2015. They have also agreed to give us 67 programs distributed by National Educational Television, the precursor to PBS. We acquired from NHPR a collection of interviews and speeches by presidential candidates from 1996-2012. This equals about 100 hours of content and was added to the AAPB in January of this year. AAPB has made arrangements with Ken Burns to preserve and make accessible digital master copies of eight complete interviews with prominent historians and others conducted for The Civil War that have not previously been seen by the public. We will receive programs from Southern California Public Radio, including award-winning stories chronicling endangered species and environmental issues in California; recordings of music written for public radio by Donald Voegeli, composer of the All Things Considered theme; and award-winning public radio programs on conservation issues produced by James Voegeli. Each of these collections presented their own challenges. Some are very large, while others are very small. The larger collections require a lot of planning, especially when grants are involved. Civil War – we’re taking the original files, not preservation files. NHPR – this one was relatively easy, as it was a small collection of radio programs. Radio is always easier to process on our end than video. The file sizes are smaller and not as complex as video files, so they go through the ingestion process faster. American Masters and NewsHour – these two situations are more complex. Both involve analog tapes that need to be digitized. The NewsHour also involves multiple entities which hold the physical analog tapes that need to be digitized and it is a grant project through CLIR, so there are additional requirements that need to be met. A commonality among all these collections is the need for a Deed of Gift. These collections need to be able to be viewed on the AAPB website and it often takes a lot of time for this process to complete. With that, I’ll hand it over to Casey, who will go over how you can get your material into the AAPB.
- So we have launched our website and chose to to expose all the data we had regardless of how complete it was. The items with better data will show up in refined searches. The items with not so great data may seem like clutter to some, but to others may pique their interest. We’ve developed some screens to facet the searches so that there is perhaps less clutter and the user can choose how much they want. But the goal was to get the user to the content as quickly as possible.
- So we have launched our website and chose to to expose all the data we had regardless of how complete it was. The items with better data will show up in refined searches. The items with not so great data may seem like clutter to some, but to others may pique their interest. We’ve developed some screens to facet the searches so that there is perhaps less clutter and the user can choose how much they want. But the goal was to get the user to the content as quickly as possible.
- Our goal is to not only provide access and discoverability to the collection, but also long term preservation.