Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Data Formats &
Protocols
Sven Van Caekenberghe
Pharo Object World
Pharo Object World
• Object Memory + Virtual Machine
• Graphics + Interaction, OS + Libraries
• Tools, IDE
Dream Environment
Pharo
• One simple language used everywhere
• True open source for all parts
• Understandable from high to low level
Island ?
Outside World
• Infinitely larger
• Our world is part of it
• We need to interact with it & affect it
Bridges
Types of Bridges
• Foreign Function Interface
• OS (Sub) Process
• Network Services
FFI
• Link to existing C libraries
• Huge availability, high quality
• Reuse, NIH
FFI
• Complex
• Opaque, not Object Oriented
• Not always that efficient, not always good fit
OS (Sub) Process
• Execute arbitrary programs as sub processes
• Communicate via standard input/output/error
• Huge availa...
OS (Sub) Process
• Large overhead
• Opaque, not Object Oriented
• Awkward interface, Not always good fit
Network Services
• Connect to some service
• Send requests and receive responses
• Local, LAN, WAN
Network Services
• Fully implemented in Pharo, Open standards
• Client/server, Distributed, IOT, DB
• Marshalling & commun...
Building Bridges
Building Bridges
• What ? - Data Formats
• How ? - Protocols
Data Format
• Given a stream
• Read/parse a stream into objects
• Write objects to a stream
Protocol
• Communication channel
• Network stream
• The conversation itself
Basics
Streams
Types of Streams
• Binary Streams
• Character Streams
Binary Streams
• File Stream
• Socket Stream
Character Streams
• One of the simplest data formats
• ASCII, Latin1, UTF-8, UTF-16, UTF-32
• Fully covered & implemented ...
Character Streams
• Orthogonal concern
• Compose, Wrap
• A solved issue
| array readStream |
array := ByteArray streamContents: [ :out |
(ZnCharacterWriteStream on: out encoding: #iso88591)
next...
Reference
http://files.pharo.org/books/enterprise-pharo/book/Zinc-
Encoding-Meta/Zinc-Encoding-Meta.html
Standard Protocols
• File Input/Output
• HTTP(S)
• Sub Process Standard Input/Output/Error
Data Formats
Internal Formats
• FUEL (binary)
• STON (textual)
• Very cool, very useful
• Limited to the Pharo World
Example
PVTRecord
• timestamp <DateAndTime>
• id <String>
• position <longitude@latitude> <Float@Float>
• speed <Integer>
Example
• as CSV
• as JSON
• as XML
CSV
CSV
• textual
• 1 record per line (any EOL)
• fields separated by delimiter
• fixed number of fields
CSV
• optionally quoted
• untyped strings
• conventional header
id,time,long,lat,speed
v1,2016-03-29T15:21:00+00:00,5.3403835,50.918625,37
v1,2016-03-29T15:22:00+00:00,5.3397698,50.91519...
Parsing CSV
(FileLocator desktop / 'pvtrecords.csv')
readStreamDo: [ :in |
(NeoCSVReader on: in) upToEnd ].
(FileLocator desktop / 'pvtrecords.csv')
readStreamDo: [ :in |
(NeoCSVReader on: in)
skipHeader;
addField;
addFieldConvert...
(FileLocator desktop / 'pvtrecords.csv')
readStreamDo: [ :in |
(NeoCSVReader on: in)
skipHeader;
recordClass: PVTRecord;
a...
Generating CSV
(FileLocator desktop / 'pvtrecords2.csv')
writeStreamDo: [ :out |
(NeoCSVWriter on: out)
writeHeader: #(id time long lat s...
"id","time","long","lat","speed"
"v1","2016-03-29T15:21:00+00:00","5.3403835","50.918625","37"
"v1","2016-03-29T15:22:00+0...
(FileLocator desktop / 'pvtrecords2.csv')
writeStreamDo: [ :out |
(NeoCSVWriter on: out)
writeHeader: #(id time long lat s...
Reference
http://files.pharo.org/books/enterprise-pharo/book/
NeoCSV/NeoCSV.html
JSON
JSON
• Textual
• Maps & Lists
• Numbers, Strings, Booleans, null
JSON
• Simple, universal, partially self describing
• Maps well to Pharo (Dictionary & Array)
• No class info, no complex ...
[
{
"id":"v1",
"time":"2016-03-29T15:21:00+00:00",
"long":5.3403835,
"lat":50.918625,
"speed":37
},
{
"id":"v1",
"time":"2...
Parsing JSON
(FileLocator desktop / 'pvtrecords.json')
readStreamDo: [ :in |
(NeoJSONReader on: in) next ].
(FileLocator desktop / 'pvtrecords.json')
readStreamDo: [ :in |
STONJSON fromStream: in ].
(FileLocator desktop / 'pvtrecords.json')
readStreamDo: [ :in |
(NeoJSONReader on: in)
for: DateAndTime customDo: [ :mappi...
Generating JSON
(FileLocator desktop / 'pvtrecords2.json')
writeStreamDo: [ :out |
(NeoJSONWriter on: out)
nextPut: {
{
#id->#v1.
#time->'...
[{"speed":27,"long":5.3403835,"lat":50.918625,
"time":"2016-03-29T15:21:00+00:00","id":"v1"}]
(FileLocator desktop / 'pvtrecords2.json')
writeStreamDo: [ :out |
(NeoJSONWriter on: out)
prettyPrint: true;
newLine: Str...
[
{
"speed" : 37,
"long" : 5.3403835,
"lat" : 50.918625,
"time" : "2016-03-29T15:21:00+00:00",
"id" : "v1"
},
{
"speed" : ...
Reference
http://files.pharo.org/books/enterprise-pharo/book/
NeoJSON/NeoJSON.html
XML
XML
• Textual
• Structured Markup (Tags & Attributes)
• Documents & Data
XML
• Large & Complex
• No inherent typing, Requires add ons
• Verbose
<records>
<pvt>
<id>v1</id>
<time>2016-03-29T15:21:00+00:00</time>
<long>5.3403835</long>
<lat>50.918625</lat>
<speed>37</...
Parsing XML
XMLDOMParser parseFileNamed:
(FileLocator desktop / 'pvtrecords.xml') fullName.
((XMLDOMParser parseFileNamed:
(FileLocator desktop / 'pvtrecords.xml') fullName)
allElementsNamed: 'pvt')
collect: [ :eac...
Generating XML
(FileLocator desktop / 'pvtrecords2.xml')
writeStreamDo: [ :out |
| writer |
(writer := XMLWriter on: out)
enablePrettyPri...
<?xml version="1.0"?>
<records>
<pvt>
<id>v1</id>
<time>2016-03-29T15:21:00+00:00</time>
<long>5.3403835</long>
<lat>50.91...
Protocols
Lingua Franca = HTTP
Web Services
HTTP
• Request / Response
• Transfer Any Entity Type
• URL / URI + Headers
Zinc HTTP
Components
Client & Server

part of standard image
Client Side
one tiny example
ZnClient new
url: 'http://easy.t3-platform.net/rest/geo-ip';
queryAt: 'address' put: '81.83.7.35';
get.
=> '{
"latitude" :...
ZnClient new
systemPolicy;
url: 'http://easy.t3-platform.net/rest/geo-ip';
queryAt: 'address' put: '81.83.7.35';
accept: Z...
Server Side
• Representational State Transfer (REST)
• Plain Zinc (BYO)
• Zinc-REST, Seaside-REST
• Teapot
2 arbitrary protocol examples
memcached
memcached
• memory caching server
• key/value - LRU
• standard architecture element
memcached
• lots of RAM (GB)
• distributed / shared
• similar to a database
memcached protocol
• simple, text / binary mix
• request / response
• small command set
| client |
client := MDBasicClient new.
[
client
at: 'foo-key'
ifAbsentPut: [ 'my-long-query-result' asByteArray ].
] ensu...
MOM
Message
Oriented
Middleware
MOM
• infrastructure supporting sending & receiving
messages between distributed systems
• heterogeneous & decoupled
• asy...
MOM
• exchanges / queues
• client / server
• producer / consumer
• routing / transformation
STOMP
Streaming Text Oriented
Messaging Protocol
STOMP
• simple & textual
• wire format / commands
• similar to HTTP, yet different
STAMP
• Implementation of STOMP
• Protocol spec 1.2
• Tested against RabbitMQ
| server |
server := self stampClient.
[
server open.
server subscribeTo: 'factorial'.
server runWith: [ :message | | numb...
| client request |
client := self stampClient.
client open.
request := client newSendFrameTo: 'factorial'.
request text: 4...
There are many other
formats & protocols
Finding Formats &
Protocols
Finding Formats & Protocols
• The Pharo Catalog
• Spotter
• http://catalog.pharo.org
• Ask on the mailing lists
NotFound ?
Consider doing your
own implementation !
The End
Q & A
#Pharo Days 2016 Data Formats and Protocols
#Pharo Days 2016 Data Formats and Protocols
#Pharo Days 2016 Data Formats and Protocols
#Pharo Days 2016 Data Formats and Protocols
#Pharo Days 2016 Data Formats and Protocols
#Pharo Days 2016 Data Formats and Protocols
#Pharo Days 2016 Data Formats and Protocols
Prochain SlideShare
Chargement dans…5
×

1

Partager

Télécharger pour lire hors ligne

#Pharo Days 2016 Data Formats and Protocols

Télécharger pour lire hors ligne

Interfacing Pharo with the World.
Sven Van Caekenberghe

Livres associés

Gratuit avec un essai de 30 jours de Scribd

Tout voir

#Pharo Days 2016 Data Formats and Protocols

  1. 1. Data Formats & Protocols Sven Van Caekenberghe
  2. 2. Pharo Object World
  3. 3. Pharo Object World • Object Memory + Virtual Machine • Graphics + Interaction, OS + Libraries • Tools, IDE
  4. 4. Dream Environment
  5. 5. Pharo • One simple language used everywhere • True open source for all parts • Understandable from high to low level
  6. 6. Island ?
  7. 7. Outside World • Infinitely larger • Our world is part of it • We need to interact with it & affect it
  8. 8. Bridges
  9. 9. Types of Bridges • Foreign Function Interface • OS (Sub) Process • Network Services
  10. 10. FFI • Link to existing C libraries • Huge availability, high quality • Reuse, NIH
  11. 11. FFI • Complex • Opaque, not Object Oriented • Not always that efficient, not always good fit
  12. 12. OS (Sub) Process • Execute arbitrary programs as sub processes • Communicate via standard input/output/error • Huge availability, high quality, reuse, NIH
  13. 13. OS (Sub) Process • Large overhead • Opaque, not Object Oriented • Awkward interface, Not always good fit
  14. 14. Network Services • Connect to some service • Send requests and receive responses • Local, LAN, WAN
  15. 15. Network Services • Fully implemented in Pharo, Open standards • Client/server, Distributed, IOT, DB • Marshalling & communication overhead
  16. 16. Building Bridges
  17. 17. Building Bridges • What ? - Data Formats • How ? - Protocols
  18. 18. Data Format • Given a stream • Read/parse a stream into objects • Write objects to a stream
  19. 19. Protocol • Communication channel • Network stream • The conversation itself
  20. 20. Basics
  21. 21. Streams
  22. 22. Types of Streams • Binary Streams • Character Streams
  23. 23. Binary Streams • File Stream • Socket Stream
  24. 24. Character Streams • One of the simplest data formats • ASCII, Latin1, UTF-8, UTF-16, UTF-32 • Fully covered & implemented in Pharo
  25. 25. Character Streams • Orthogonal concern • Compose, Wrap • A solved issue
  26. 26. | array readStream | array := ByteArray streamContents: [ :out | (ZnCharacterWriteStream on: out encoding: #iso88591) nextPutAll: 'Les élèves français' ]. readStream := array readStream. (ZnCharacterReadStream on: readStream encoding: #iso8859) upToEnd. -> 'Les élèves français'
  27. 27. Reference http://files.pharo.org/books/enterprise-pharo/book/Zinc- Encoding-Meta/Zinc-Encoding-Meta.html
  28. 28. Standard Protocols • File Input/Output • HTTP(S) • Sub Process Standard Input/Output/Error
  29. 29. Data Formats
  30. 30. Internal Formats • FUEL (binary) • STON (textual) • Very cool, very useful • Limited to the Pharo World
  31. 31. Example
  32. 32. PVTRecord • timestamp <DateAndTime> • id <String> • position <longitude@latitude> <Float@Float> • speed <Integer>
  33. 33. Example • as CSV • as JSON • as XML
  34. 34. CSV
  35. 35. CSV • textual • 1 record per line (any EOL) • fields separated by delimiter • fixed number of fields
  36. 36. CSV • optionally quoted • untyped strings • conventional header
  37. 37. id,time,long,lat,speed v1,2016-03-29T15:21:00+00:00,5.3403835,50.918625,37 v1,2016-03-29T15:22:00+00:00,5.3397698,50.915192,27 v1,2016-03-29T15:23:00+00:00,5.3415751,50.911633,47
  38. 38. Parsing CSV
  39. 39. (FileLocator desktop / 'pvtrecords.csv') readStreamDo: [ :in | (NeoCSVReader on: in) upToEnd ].
  40. 40. (FileLocator desktop / 'pvtrecords.csv') readStreamDo: [ :in | (NeoCSVReader on: in) skipHeader; addField; addFieldConverter: [ :string | DateAndTime fromString: string ]; addFloatField; addFloatField; addIntegerField; upToEnd ].
  41. 41. (FileLocator desktop / 'pvtrecords.csv') readStreamDo: [ :in | (NeoCSVReader on: in) skipHeader; recordClass: PVTRecord; addField: #identification:; addField: #timestamp: converter: [ :string | DateAndTime fromString: string ]; addFloatField: #longitude:; addFloatField: #latitude:; addIntegerField: #speed:; upToEnd ].
  42. 42. Generating CSV
  43. 43. (FileLocator desktop / 'pvtrecords2.csv') writeStreamDo: [ :out | (NeoCSVWriter on: out) writeHeader: #(id time long lat speed); nextPutAll: ( { PVTRecord one. PVTRecord two. PVTRecord three } collect: [ :each | { each identification. each timestamp. each longitude. each latitude. each speed } ] ) ].
  44. 44. "id","time","long","lat","speed" "v1","2016-03-29T15:21:00+00:00","5.3403835","50.918625","37" "v1","2016-03-29T15:22:00+00:00","5.3397698","50.915192","27" "v1","2016-03-29T15:23:00+00:00","5.3415751","50.911633","47"
  45. 45. (FileLocator desktop / 'pvtrecords2.csv') writeStreamDo: [ :out | (NeoCSVWriter on: out) writeHeader: #(id time long lat speed); addFields: #( identification timestamp longitude latitude speed); nextPutAll: { PVTRecord one. PVTRecord two. PVTRecord three } ].
  46. 46. Reference http://files.pharo.org/books/enterprise-pharo/book/ NeoCSV/NeoCSV.html
  47. 47. JSON
  48. 48. JSON • Textual • Maps & Lists • Numbers, Strings, Booleans, null
  49. 49. JSON • Simple, universal, partially self describing • Maps well to Pharo (Dictionary & Array) • No class info, no complex graphs
  50. 50. [ { "id":"v1", "time":"2016-03-29T15:21:00+00:00", "long":5.3403835, "lat":50.918625, "speed":37 }, { "id":"v1", "time":"2016-03-29T15:22:00+00:00", "long":5.3397698, "lat":50.915192, "speed":27 }, { "id":"v1", "time":"2016-03-29T15:23:00+00:00", "long":5.3415751, "lat":50.911633, "speed":47 } ]
  51. 51. Parsing JSON
  52. 52. (FileLocator desktop / 'pvtrecords.json') readStreamDo: [ :in | (NeoJSONReader on: in) next ].
  53. 53. (FileLocator desktop / 'pvtrecords.json') readStreamDo: [ :in | STONJSON fromStream: in ].
  54. 54. (FileLocator desktop / 'pvtrecords.json') readStreamDo: [ :in | (NeoJSONReader on: in) for: DateAndTime customDo: [ :mapping | mapping decoder: [ :string | DateAndTime fromString: string ] ]; for: PVTRecord do: [ :mapping | mapping mapAccessor: #speed; mapAccessor: #identification to: #id; mapAccessor: #longitude to: #long; mapAccessor: #latitude to: #lat. (mapping mapAccessor: #timestamp to: #time) valueSchema: DateAndTime ]; nextListAs: PVTRecord ].
  55. 55. Generating JSON
  56. 56. (FileLocator desktop / 'pvtrecords2.json') writeStreamDo: [ :out | (NeoJSONWriter on: out) nextPut: { { #id->#v1. #time->'2016-03-29T15:21:00+00:00'. #long->5.3403835. #lat->50.918625. #speed->27 } asDictionary } ].
  57. 57. [{"speed":27,"long":5.3403835,"lat":50.918625, "time":"2016-03-29T15:21:00+00:00","id":"v1"}]
  58. 58. (FileLocator desktop / 'pvtrecords2.json') writeStreamDo: [ :out | (NeoJSONWriter on: out) prettyPrint: true; newLine: String lf; for: DateAndTime customDo: [ :mapping | mapping encoder: [ :dateAndTime | dateAndTime asString ] ]; for: PVTRecord do: [ :mapping | mapping mapAccessor: #speed; mapAccessor: #identification to: #id; mapAccessor: #longitude to: #long; mapAccessor: #latitude to: #lat. (mapping mapAccessor: #timestamp to: #time) ]; nextPut: { PVTRecord one. PVTRecord two. PVTRecord three }; newline ].
  59. 59. [ { "speed" : 37, "long" : 5.3403835, "lat" : 50.918625, "time" : "2016-03-29T15:21:00+00:00", "id" : "v1" }, { "speed" : 27, "long" : 5.3397698, "lat" : 50.915192, "time" : "2016-03-29T15:22:00+00:00", "id" : "v1" }, { "speed" : 47, "long" : 5.3415751, "lat" : 50.911633, "time" : "2016-03-29T15:23:00+00:00", "id" : "v1" } ]
  60. 60. Reference http://files.pharo.org/books/enterprise-pharo/book/ NeoJSON/NeoJSON.html
  61. 61. XML
  62. 62. XML • Textual • Structured Markup (Tags & Attributes) • Documents & Data
  63. 63. XML • Large & Complex • No inherent typing, Requires add ons • Verbose
  64. 64. <records> <pvt> <id>v1</id> <time>2016-03-29T15:21:00+00:00</time> <long>5.3403835</long> <lat>50.918625</lat> <speed>37</speed> </pvt> <pvt> <id>v1</id> <time>2016-03-29T15:22:00+00:00</time> <long>5.3397698</long> <lat>50.915192</lat> <speed>27</speed> </pvt> <pvt> <id>v1</id> <time>2016-03-29T15:23:00+00:00</time> <long>5.3415751</long> <lat>50.911633</lat> <speed>47</speed> </pvt> </records>
  65. 65. Parsing XML
  66. 66. XMLDOMParser parseFileNamed: (FileLocator desktop / 'pvtrecords.xml') fullName.
  67. 67. ((XMLDOMParser parseFileNamed: (FileLocator desktop / 'pvtrecords.xml') fullName) allElementsNamed: 'pvt') collect: [ :each | PVTRecord new timestamp: (DateAndTime fromString: (each contentStringAt: 'time')); identification: (each contentStringAt: 'id'); longitude: (each contentStringAt: 'long') asNumber; latitude: (each contentStringAt: 'lat') asNumber; speed: (each contentStringAt: 'speed') asInteger; yourself ] as: Array.
  68. 68. Generating XML
  69. 69. (FileLocator desktop / 'pvtrecords2.xml') writeStreamDo: [ :out | | writer | (writer := XMLWriter on: out) enablePrettyPrinting; lineBreak: String lf; xml. writer tag: #records with: [ { PVTRecord one. PVTRecord two. PVTRecord three } do: [ :each | writer tag: #pvt with: [ writer tag: #id with: each identification; tag: #time with: each timestamp asString; tag: #long with: each longitude asString; tag: #lat with: each latitude asString; tag: #speed with: each speed asString ] ] ] ].
  70. 70. <?xml version="1.0"?> <records> <pvt> <id>v1</id> <time>2016-03-29T15:21:00+00:00</time> <long>5.3403835</long> <lat>50.918625</lat> <speed>37</speed> </pvt> <pvt> <id>v1</id> <time>2016-03-29T15:22:00+00:00</time> <long>5.3397698</long> <lat>50.915192</lat> <speed>27</speed> </pvt> <pvt> <id>v1</id> <time>2016-03-29T15:23:00+00:00</time> <long>5.3415751</long> <lat>50.911633</lat> <speed>47</speed> </pvt> </records>
  71. 71. Protocols
  72. 72. Lingua Franca = HTTP
  73. 73. Web Services
  74. 74. HTTP • Request / Response • Transfer Any Entity Type • URL / URI + Headers
  75. 75. Zinc HTTP Components Client & Server
 part of standard image
  76. 76. Client Side one tiny example
  77. 77. ZnClient new url: 'http://easy.t3-platform.net/rest/geo-ip'; queryAt: 'address' put: '81.83.7.35'; get. => '{ "latitude" : 50.8333, "address" : "81.83.7.35", "country" : "BE", "longitude" : 4.0 }'
  78. 78. ZnClient new systemPolicy; url: 'http://easy.t3-platform.net/rest/geo-ip'; queryAt: 'address' put: '81.83.7.35'; accept: ZnMimeType applicationJson; contentReader: [ :entity | STONJSON fromString: entity contents ]; get.
  79. 79. Server Side • Representational State Transfer (REST) • Plain Zinc (BYO) • Zinc-REST, Seaside-REST • Teapot
  80. 80. 2 arbitrary protocol examples
  81. 81. memcached
  82. 82. memcached • memory caching server • key/value - LRU • standard architecture element
  83. 83. memcached • lots of RAM (GB) • distributed / shared • similar to a database
  84. 84. memcached protocol • simple, text / binary mix • request / response • small command set
  85. 85. | client | client := MDBasicClient new. [ client at: 'foo-key' ifAbsentPut: [ 'my-long-query-result' asByteArray ]. ] ensure: [ client close ]
  86. 86. MOM Message Oriented Middleware
  87. 87. MOM • infrastructure supporting sending & receiving messages between distributed systems • heterogeneous & decoupled • asynchronous
  88. 88. MOM • exchanges / queues • client / server • producer / consumer • routing / transformation
  89. 89. STOMP Streaming Text Oriented Messaging Protocol
  90. 90. STOMP • simple & textual • wire format / commands • similar to HTTP, yet different
  91. 91. STAMP • Implementation of STOMP • Protocol spec 1.2 • Tested against RabbitMQ
  92. 92. | server | server := self stampClient. [ server open. server subscribeTo: 'factorial'. server runWith: [ :message | | number | message body = 'quit' ifTrue: [ ConnectionClosed signal ]. number := message body asInteger. server sendText: number factorial asString to: message replyTo ] ] fork.
  93. 93. | client request | client := self stampClient. client open. request := client newSendFrameTo: 'factorial'. request text: 42 asString. request replyTo: '/temp-queue/factorial'. client write: request. response := client readMessage. self assert: response body equals: 42 factorial asString. client sendText: 'quit' to: 'factorial'. client close.
  94. 94. There are many other formats & protocols
  95. 95. Finding Formats & Protocols
  96. 96. Finding Formats & Protocols • The Pharo Catalog • Spotter • http://catalog.pharo.org • Ask on the mailing lists
  97. 97. NotFound ?
  98. 98. Consider doing your own implementation !
  99. 99. The End Q & A
  • SaogatRab

    Nov. 2, 2016

Interfacing Pharo with the World. Sven Van Caekenberghe

Vues

Nombre de vues

635

Sur Slideshare

0

À partir des intégrations

0

Nombre d'intégrations

6

Actions

Téléchargements

3

Partages

0

Commentaires

0

Mentions J'aime

1

×