DPE Training materials

1. File formats and registries Manfred Thaller, University at Cologne October 2 nd , 2007

3. An image

4. An image 6 rows 5 columns

5. 5 rows 6 columns

6. An image 1 == yellow 0 == red 1 1 1 1 1 1 0 0 0 1 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1

7. An image 1 == violett 0 == green 1 1 1 1 1 1 0 0 0 1 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1

8. An image Store: 1,1,1,1,1,1,0,0,0,1,1,1,0,1,1,1,1,0,1,1,1,1,0,1,1,1,1,1,1,1 1 1 1 1 1 1 0 0 0 1 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1

9. An image Store: 6,1,3,0,3,11,0,4,1,1,0,4,1,1,0,7,1 1 1 1 1 1 1 0 0 0 1 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1

10. An image Store: 1,1,1,1,1,1,0,0,0,1,1,1,0,1,1,1,1,0,1,1,1,1,0,1,1,1,1,1,1,1 Uncompressed 1 1 1 1 1 1 0 0 0 1 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1

11. An image Store: 6,1,3,0,3,1,1,0,4,1,1,0,4,1,1,0,7,1 (Compressed)Run Length Encoded 1 1 1 1 1 1 0 0 0 1 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1

12. An image Store: SetSize: 5 by 6 SetBackgroundColor: Blue SetForegroundColor: Red SetLetterHeight: 4 MoveTo: 3,5 DrawLetter: T 1,1 2,1 3,1 4,1 5,1 1,2 2,2 3,2 4,2 5,2 1,3 2,3 3,3 4,3 5,3 1,4 2,4 3,4 4,4 5,4 1,5 2,5 3,5 4,5 5,5 1,6 2,6 3,6 4,6 5,6

13. An image 6 rows 5 columns 1 == yellow 0 == red Uncompressed

14. An image dimensions 1 == yellow 0 == red Uncompressed

15. An image dimensions photogrammetric interpretation Uncompressed

16. An image dimensions photogrammetric interpretation compression

17. An image <basic information> <rendering information> <storage information>

18. An image <basic information> (implicit / explicit) <rendering information> (implicit / explicit) <storage information> (implicit / explicit) … and the data?

19. An image <basic information> (implicit / explicit) <rendering information> (implicit / explicit) <storage information> (implicit / explicit) … and the data?

20. An image Data either as data stream 1,1,1,1,1,1, 0,0,0,1,1,1, 0,1,1,1,1,0, 1,1,1,1,0,1, 1,1,1,1,1,1 1 1 1 1 1 1 0 0 0 1 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1

21. An image Data either as data stream or as processing instructions SetSize: 5 by 6 SetBackgroundColor: Yellow SetForegroundColor: Red SetLetterHeight: 4 MoveTo: 3,5 DrawLetter: T 1 1 1 1 1 1 0 0 0 1 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1

22. File format <basic information> <rendering information> <storage information> <data>

23. File format <basic information> What to do? <rendering information> <storage information> <data>

24. File format <basic information> What to do? <rendering information> How to do it? <storage information> <data>

25. File format <basic information> What to do? <rendering information> How to do it? <storage information> How to move it from persistent to deployed form? <data>

26. File format <basic information> What to do? <rendering information> How to do it? <storage information> How to move it from persistent to deployed form? <data> What to deploy?

27. File format <basic information> What to do? <rendering information> How to do it? <storage information> How to move it from persistent to deployed form? <data> What to deploy?

28. File format <basic information> Mandatory <rendering information> Useful <storage information> Historical <data> Mandatory

29. File format A deterministic specification how the properties of a digital object can reversibly be converted into a linear bytestream (bitstream).

30. File format: TIFF

31. File format: PDF 1 0 obj << /Type /Page /Parent 281 0 R /Resources 2 0 R /Contents 3 0 R /StructParents 2 /MediaBox [ 0 0 612 792 ] /CropBox [ 0 0 612 792 ] /Rotate 0 >> endobj

32. File format: PDF 2 0 obj << /ProcSet [ /PDF /Text ] /Font << /TT2 292 0 R /TT4 288 0 R >> /ExtGState << /GS1 300 0 R >> /ColorSpace << /Cs6 289 0 R >> >> endobj

33. File format: PDF 3 0 obj << /Length 4605 /Filter /FlateDecode >> stream H‰„WÛŽÛÈ}×Wô#Œ4jR”¨`±Àø ™Í&quot; ¶(²5j›&quot;¹lräý‘|oêÖ-j —‹ udTÙÂ…fPnˆ¿ìþ>Ó›Ež²ÝÕ˽âä”uª2i*<<v ú[Óžk9Q‰¼‡x»XTP{ ‹ ±/[i²½Ö)}ÔÏö&ªÙH;<Cµ … and about 4000 bytes more ŠøL&quot;È÷ےƐ¬JYØÂm]j¥Ýqõ¥ÏººÕ™·²ôÒ·Ûº¤–÷.u-kP0 4“øTxM<é識9uôøˆòLi¦ØoTÖ m–;ǯ÷¤ÿlÕºvéU—Ë ±¤Lm°gŸˆu1Åëu5l3¯’¢O %òËTîü7?ìNdh endstream endobj

34. File format: XML (here: SVG) <?xml version=&quot;1.0&quot; encoding=&quot;UTF-16&quot;?> <svg:svg width=&quot;800&quot; height=&quot;1000&quot; xmlns:svg=&quot;http://www.w3.org ... <svg:rect x=&quot;0&quot; y=&quot;0&quot; width=&quot;800&quot; height=&quot;1000&quot; fill=&quot;white&quot; /> <svg:g transform=&quot;translate(-140,0)&quot;> <svg:line x1=&quot;600&quot; y1=&quot;20&quot; x2=&quot;500&quot; y2=&quot;20&quot; stroke=&quot;black&quot; … <svg:text x=&quot;600&quot; y=&quot;28.8&quot; font-size=&quot;6&quot; fill=&quot;black&quot; … </svg:g> <svg:g transform=&quot;translate(-140,0)&quot;> <svg:text x=&quot;500&quot; y=&quot;24.4&quot;> <svg:tspan font-size=&quot;4&quot; fill=&quot;black&quot;>Leiste</svg:tspan> </svg:text> </svg:g> <svg:defs> <svg:g id=&quot;halbeSaeuleLeiste0&quot;>

35. File format: XML (here SVG)

36. File format: XML (ETH: “column XML”) <?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?> <Autor name=&quot;Vitruv&quot;> <Ordnung name=&quot;Ionisch&quot; THz=&quot;&quot; THn=&quot;&quot; MH=&quot;&quot; TBz=&quot;&quot; TBn=&quot;&quot; … <Element name=&quot;Gebaelk&quot; original=&quot;&quot; THz=&quot;&quot; THn=&quot;&quot; MH=&quot;&quot; … <Element name=&quot;Gesims&quot; original=&quot;corona&quot; THz=&quot;&quot; THn=&quot;&quot; MH=&quot;&quot; … <Element name=&quot;Leiste&quot; original=&quot;&quot; THz=&quot;&quot; THn=&quot;&quot; MH=&quot;0.03&quot; … <Element name=&quot;Kyma&quot; original=&quot;sima&quot; THz=&quot;&quot; THn=&quot;&quot; … <Element name=&quot;Leiste&quot; original=&quot;&quot; THz=&quot;&quot; THn=&quot;&quot; MH=&quot;0.017&quot; … <Element name=&quot;Kyma_reversa&quot; original=&quot;cymatium&quot; THz=&quot;&quot; … <Element name=&quot;Platte&quot; original=&quot;corona&quot; THz=&quot;&quot; THn=&quot;&quot; … <Element name=&quot;Leiste&quot; original=&quot;&quot; THz=&quot;&quot; THn=&quot;&quot; MH=&quot;0.017&quot; … <Element name=&quot;Kyma_reversa&quot; original=&quot;cymatium&quot; THz=&quot;&quot; … <hElement name=&quot;Band&quot; typ=&quot;1&quot; dx=&quot;0.048&quot; r=&quot;0.019&quot;/> <hElement name=&quot;Band&quot; typ=&quot;1&quot; dx=&quot;0.048&quot; r=&quot;0.019&quot;/> </Element>

38. Bit rot An Image file before ….

39. Bit rot ... and after one byte is changed.

40. Bit rot ... and after one byte is changed. Undetectable by software.

41. Bit rot Processing dictionary Payload 002 004 234 123 234 156 127 178 221 221

42. Bit rot One byte is damaged, one byte cannot be displayed correctly. 002 004 234 123 234 156 127 xxx 221 221

43. Bit rot One byte is damaged, ten bytes cannot be displayed correctly. 002 xxx 234 123 234 156 127 178 221 221

44. Result: http://www.cflr.beniculturali.it/Progetti/Fixit.php www.cflr.beniculturali.it Franco Liberati [email_address] Università di Roma “La Sapienza” Dipartimento Informatica Centro Fotoriproduzione Legatoria e Restauro Paolo Buonora [email_address]

45. Paolo on JPEG JPEG2000 more robust against bit rot than TIFF.

46. Paolo on JPEG JPEG2000 more robust against bit rot than TIFF. So, to stinulate more empiricism …

57. What kind of file is this? Two ways to identify a file: (b) By internal characteristics („magic number“, „signature“). A TIFF file begins with … Bytes 0-1: The byte order used within the file. Legal values are: “ II” (4949.H) / “MM” (4D4D.H) Bytes 2-3 An arbitrary but carefully chosen number (42) that further identifies the file as a TIFF file.

59. What kind of file is this? The following 4 transparencies are a quotation from http://hul.harvard.edu/gdfr (see below).

62. Repository Format Dependencies Using the OAIS Reference Model Global Digital Format Registry DSpace User Group, March 2004

64. File format registries - URLs PRONOM : http://www.nationalarchives.gov.uk/pronom/ (does not only rely on extensions) Global Digital Format Registry : http://hul.harvard.edu/gdfr (predominantly project description) FileExt : http://filext.com (predominantly links to software)

65. Exercise I: A few experiments Group 1 Aist ė Abromaityt ė Tomasz Jablonski Aadi Kaljuvee Jurat ė Kuprien ė Violeta Meiliūnait ė

66. Exercise I: A few experiments Group 2 Libor Coufal Edvardas Germanas Hamid Rofoogaran Laima Šiudikiene Egl ė Žvinyt ė

67. Exercise I: A few experiments Group 3 Renata Balandien ė Thomas Guignard Edgars Jekabsons Elona Malaiškien ė Bjorn Ragnolf Ronning

68. Exercise I: A few experiments Group 4 Gražina Deveikyt ė Raimondas Malaiška Filip Kwiatek Marija Prokopčik Piret Randmae Jelena Saikovič

69. PART II – Formats in PLANETS: File characteristics

75. Confession

76. Confession Computer science does not really know what information is.

77. Computer science does not really know what information is. It is pretty good at representing and processing it, though.

78. Representations & migrations III == 3 == γ ‘ == ●●● Four representations of the idea / concept / model three

79. Representations & migrations I divided by III == 1 / 3 == 1.3333? I divided by III == 1 / 3 == 1.3 periodic Some ideas are handled more precisely by Some thinkers than others.

80. Representations & migrations 48 bit images on 24 and on 48 bit graphics cards. Some data is processed more adequately by some equipment than others

81. Representations & migrations A model for information before and after a migration must therefore potentially represent all information there, irrespective of the possibility to process it in a given environment.

82. XCEL / XCDL Languages are being processed … … development focus currently: dynamic handling of format specific algorithms.

83. XCEL / XCDL: image model (1) A pixel cube … Each pixel: MSB (channel 1), … LSB (channel 1), … MSB (channel n), … LSB (channel n), MSB (aux 1), … LSB (aux 1), … MSB (aux m), … LSB (aux m)

84. XCEL / XCDL: image model (2) A pixel cube … Accompanied by rendering info plus deployment info.

85. XCEL / XCDL: image model - example <property id=&quot;p4&quot; source=&quot;raw&quot; cat=&quot;descr&quot; > <name>imageType</name> <valueSet id=&quot;i_i1_s5&quot; > <rawValue>2</rawValue> <labValue> <val>truecolour</val> <type>fixedLabel</type> </labValue> <dataRef ind=&quot;normAll&quot; /> <propRel/> </valueSet> </property>

86. XCEL / XCDL: text model A text (= <object>) is composed of - data (<normData>) plus - interpretations of data according to the underlying format specification (=properties; <property>).

87. XCEL / XCDL: text model - example This is a text <refData id=&quot;1&quot;>54 68 69 73 20 69 73 20 61 20 74 65 78 74</refData> … <property> <name>fontsize</name> <rawVal> <val>00 18</val> <type>unsignedInt8</type> </rawVal> <dataRef> <!-- property refers to discrete part of reference data- -> <ref id=&quot;1&quot; start=&quot;0&quot; end=&quot;3&quot;/> <ref id=&quot;1&quot; start=“10&quot; end=&quot;12&quot;/> </dataRef> </property>

88. Exercise II: Abstract modelling Group 1: maps Group 2: music Group 3: excel sheets Group 4: „books“ … ever heard of FRBR?