The document discusses using open source tools for document automation. It describes Neale Morison's background working for various tech companies including CSIRO. It then provides examples of using tools like JavaScript, XML, and HTML to dynamically generate documents from structured data for projects like the ASKAP radio telescope and Cisco chip documentation. Throughout, it emphasizes principles for open information like using open formats, minimizing repetition, and using freely available technology.
Unit-IV; Professional Sales Representative (PSR).pptx
Open Source Automated Documentation in a Development Environment
1. Using Open Source Tools
for Document Automation
Neale Morison
May 2012
2. Who Am I?
• Various jobs
• Technical writer
• Journalist
• Editor
• Web developer
• Software developer
• Electronic technician
• Various employers
• Startups – hardware and software
• IBM, Cisco, Microsoft, Fujitsu…
• Today, CSIRO:
Commonwealth Scientific and Industrial
Research Organisation
CSIRO. Using Open Source Tools for Document Automation 2
3. What is ASKAP?
• Australian SKA
Precursor
• CSIRO radio telescope
36 antennas in Western
Australia
• World’s fastest survey
radio telescope
at 0.7-1.8 GHz
• The system will process
2Tb/s from each antenna
• Processed data then • What will it do?
shipped via fibre to a Perth • Discover millions of new galaxies
supercomputer - exascale
• Investigate dark matter, dark energy
• Similar issues to LHC
(Large Hadron Collider)
CSIRO. Using Open Source Tools for Document Automation 3
5. Who Are You?
• Technical Writers
• Experience with:
• Web development
• Software development
• Hardware development
• System administration
• HTML
• XML
• JavaScript
• Python
CSIRO. Using Open Source Tools for Document Automation 5
6. What am I Talking About?
• Automation
Any kind of software or system that assists in
technical communication
• Open Source Tools
Broadly defined:
• Software: with an Open Source License
• Standards: non-proprietary or open
e.g. PDF, open since June 2008
• Languages: Python, JavaScript
(ECMAscript), HTML, XML,
• Apps: Apache HTTP server, MySQL
• Free Tools
Free (as in beer) apps – e.g. web browsers
CSIRO. Using Open Source Tools for Document Automation 6
7. Prologue - Fairlight 1985
• Fairlight CMI
(Computer Musical Instrument)
Series III
• Manual manual
literally cut and paste
• Index created automatically
• Simple algorithm
sample index.txt: sample output:
using code 32 a blog, writing 128
making an index 42 an index, making 42
writing a blog 128 blog, writing a 128
code, using 32
index, making an 42
making an index 42
using code 32
writing a blog 128
CSIRO. Using Open Source Tools for Document Automation 7
8. CSIRO ASKAP Antenna IDs – Problem
• Two different
identification systems
• Manufacturer: CETC54 ID
• ASKAP pad ID
• Tests refer to both
• Almost insoluble
• Which one to use?
• Why not both?
• Create a table showing the
correspondence
CSIRO. Using Open Source Tools for Document Automation 8
9. CSIRO ASKAP Antenna IDs
• Requirements
• Show ID correspondence
• Sort by either ID
• Link to test directories
• Show other useful information
• Portable – all contained in a
single small file
• Easy to update – antenna data
keeps changing
CSIRO. Using Open Source Tools for Document Automation 9
10. CSIRO ASKAP Antenna IDs - Solution
• Dynamic Document
• Fixed (static) HTML:
more maintenance
• Better to write JavaScript that writes the
HTML for you
• You have to maintain only the data:
tableRows = [
{'cetc54':1, 'askap':29, 'testDir':'ASKAP29_C01', 'FAT':'Y', 'SAT':'Y', 'MRO':'Y',},
{'cetc54':2, 'askap':3, 'testDir':'ASKAP03_C02', 'FAT':'Y', 'SAT':'Y', 'MRO':'Y',},
…
]
• Responds to user requirements
• Smaller than static HTML
• ASKAP Antenna Table
• Filtered ASKAP Antenna Table
CSIRO. Using Open Source Tools for Document Automation 10
11. CSIRO ASKAP Antenna IDs - Solution
• Dynamic Document
• Fixed (static) HTML:
more maintenance
• Better to write JavaScript that writes the
HTML for you
• You have to maintain only the data:
tableRows = [
{'cetc54':1, 'askap':29, 'testDir':'ASKAP29_C01', 'FAT':'Y', 'SAT':'Y', 'MRO':'Y',},
{'cetc54':2, 'askap':3, 'testDir':'ASKAP03_C02', 'FAT':'Y', 'SAT':'Y', 'MRO':'Y',},
…
]
• Responds to user requirements
• Smaller than static HTML
• ASKAP Antenna Table
• Filtered ASKAP Antenna Table
CSIRO. Using Open Source Tools for Document Automation 11
12. Principles of Open Information
• Open
Store information in an open format
• Terse
Provide only the necessary information
• DRY
Don’t Repeat Yourself - Single Source [3]
• Accessible
Easy to find, easy to use
• Free
As in speech, love, beer – technology, IP
• Re-usable
• Maintainable
• Simple
CSIRO. Using Open Source Tools for Document Automation 12
13. Cisco Radiata Group
• WiFi on a chip
• Hardware startup exploiting:
• wireless invention
by John O’Sullivan and CSIRO radiophysics team
• analog IC design innovations
• Sold to Cisco in 1999 for unheard of sum
(around $400 million)
CSIRO. Using Open Source Tools for Document Automation 13
14. Cisco Radiata Register Docs – Problem
• Not really a problem – an opportunity
• Developers used a single source register description
in a Python script:
• generate XML register description
• generate Verilog code from which IC is built
• Required – a document generated from XML
CSIRO. Using Open Source Tools for Document Automation 14
15. Cisco Radiata Register Docs - Solution
• XML can be reformatted using XSLT and XSL/FO
(Extensible Stylesheet Language / Formatting Objects)
• Processed with Java tools, e.g. Saxon, Xalan
• Various outputs:
• HTML
• Postscript -> PDF
• LaTeX
• FrameMaker - MIF
• Result: a 100 page register reference manual
in seconds
• Register document example
CSIRO. Using Open Source Tools for Document Automation 15
16. Cisco Radiata Register Docs - Principles
• Open XML
• Terse Describes only the registers
• DRY One XML file stores all information
• Accessible one pdf
• Free common, available technology:
• Adobe Reader, Web browsers
• XML / XSLT
• HTML
• Re-usable use for next chip
• Maintainable
change only if XML format changes
• Simple for the users
CSIRO. Using Open Source Tools for Document Automation 16
17. Cisco Radiata Test Data – Problem
• Tests run on chip simulation
• Result: large quantity of numerical data
• Difficult to determine how data fits
specification
• Provided visualisation facilities do not
meet requirement
CSIRO. Using Open Source Tools for Document Automation 17
18. Cisco Radiata Test Data - Solution
• Process the data to
produce a customised
visualisation
• Tools used:
• Perl
• Lisp-like script
• Gnuplot
• Many tables, plots,
in HTML document
• Test data
visualisation example
CSIRO. Using Open Source Tools for Document Automation 18
19. Cisco Radiata Test Data - Principles
• Open Proprietary, but documented
• Terse No repeated data
• DRY Everything generated
from single set of data
• Accessible
HTML presentation
• Free
common, available technology:
• Web browsers
• Perl, HTML
• GnuPlot
• Re-usable use for next tests
• Maintainable
• Simple for the users
CSIRO. Using Open Source Tools for Document Automation 19
20. VaST XML Reports
• Software models of embedded systems, system-on-chip
(SoC)
• Used by developers of cell phones, automotive systems,
handheld devices to model the chips they were designing
• Low-level, cycle-accurate model that ran at speeds
comparable to the chip, sometimes faster
• Huge savings in time and money – chip developers could
write the firmware and software for the chip before
fabrication
• Integrated Development Environment (IDE) ran on
Windows, now also Linux.
• Acquired by Synopsys in 2010
CSIRO. Using Open Source Tools for Document Automation 20
21. VaST XML Reports – Problem
• Models shown as a hierarchy
obscures connections
• Item details hidden
multiple clicks, right clicks required
• No way to print full details
or to view as a whole
CSIRO. Using Open Source Tools for Document Automation
22. VaST XML Reports - Solution
• Data stored in XML
• XML can be:
• transformed using XSLT
• output as HTML in a browser
• With a single button
click, a full model report
can be instantly
generated:
PMX report example
CSIRO. Using Open Source Tools for Document Automation
23. VaST XML Reports – Good News,, Bad News
• Tricky
to get it to work - XLST
• Co-operation required
from GUI developers
• You only have to do it once
• Works when you’re not around,
for the customer’s models
• Maintenance: infrequent
when model storage format changes
• Value permanently added
CSIRO. Using Open Source Tools for Document Automation 23
24. VaST XML Reports - Principles
• Open XML
• Terse Describes only the model
• DRY One XML file stores all information
• Accessible one click
• Free common, available technology:
• Web browsers
• XML / XSLT
• HTML
• Re-usable
• Maintainable
• Simple for the users;
for the developer, not so much
CSIRO. Using Open Source Tools for Document Automation
25. CSIRO ASKAP Repositories – Problem
• SharePoint
• Corporate choice
• Subversion
• Developer choice
• Other repositories in use
• Don’t want to discourage or confuse
• Don’t want to introduce yet another system
CSIRO. Using Open Source Tools for Document Automation 25
26. CSIRO ASKAP Repositories - Solution
• Document Database
• Python CGI SQL app
• Scans repositories
• Can enter manually
• Captures metadata
• Single central site
• Searchable
• Links to issue tracker
• ASKAP Document
Database
CSIRO. Using Open Source Tools for Document Automation 26
27. CSIRO ASKAP Repositories - Principles
• Open SQL
• Terse normal form db
• DRY Single SQL db
• Accessible standard
web presentation, form fill
• Free
• Web browsers
• Python
• SQL
• Re-usable – other databases use same code
• Maintainable – some issues - complex to scan repositories. Code is documented.
• Simple – for users; but can I pass this on to another maintainer?
CSIRO. Using Open Source Tools for Document Automation 27
28. CSIRO ASKAP Document Management – Problem
• Want to track:
?
• Review History Tracking
• Approval ? ?
• History
Document
Repositories
Database
• Tie in to repositories
• Tie in to document database ?
• Complex and costly to program into database -
violates simplicity rule
CSIRO. Using Open Source Tools for Document Automation 28
29. CSIRO ASKAP Document Management – Solution
• Use issue tracking system
Redmine, Bugzilla… Redmine Issue &
History Tracking
• Perform review and approval in
URL ID
issue tracker
• Capture history in issue Repositories
Document
Database
• Link to repositories ID
via document URL
• Link from document database via document ID
• Link back to document database via document ID
• Requires procedures, training – document in wiki
CSIRO. Using Open Source Tools for Document Automation 29
30. CSIRO ADE Diagrams – Problem
• How Engineers
think:
• Requirements
• Hierarchy of
clickable diagrams
• Links from diagrams
to sub-diagrams
• Document for every block in diagram
• Links from diagrams to documents
CSIRO. Using Open Source Tools for Document Automation 30
31. CSIRO ADE Diagrams – Problem
• SVG
No – IE, XP
• HTML5
No – IE, XP
• PDF
Maybe, but
awkward to add links
clunky browser display
• Graphviz
Good range of outputs
Gets busy quickly: A B C
Not the style wanted
Links only in SVG output
CSIRO. Using Open Source Tools for Document Automation 31
32. CSIRO ADE Diagrams – Solution A
• Use hierarchy, not diagrams
• Procedure:
• Go through diagrams and
manually transfer nodes, links
to custom data format
• Use Python to read data and output JSON
• Use JavaScript to generate a hierarchy map from JSON
• Add document names, responsibilities, metadata
• Generate documents using PowerShell, COM objects
• ADE Hierarchy
CSIRO. Using Open Source Tools for Document Automation 32
33. CSIRO ADE Diagrams – Solution B
• Do the diagrams
HTML Image Map, JavaScript
• Procedure:
• Create diagrams as simple graphics – PPT
• Save as individual images
• Create image maps – free program
• JavaScript code transforms simple image
map
• JavaScript links to the hierarchy page
• ADE System Block Diagrams
CSIRO. Using Open Source Tools for Document Automation 33
34. CSIRO ADE Diagrams - Principles
• Open Python, HTML, JavaScript
Doc creation not open, but free: PowerShell
• Terse Only data
• DRY One source
• Accessible standard web presentation
• Free
• Web browsers
• Python
• JavaScript
• PowerShell
• Re-usable
• Maintainable
• Simple
CSIRO. Using Open Source Tools for Document Automation 34
35. Conclusion
• Coding is there to be learned
• HTML, XML, JavaScript, C, Java, Perl, Python, SQL, LaTeX …
• Excellent online resources
• You have to learn something
– either a proprietary system, or open tools
• Advantages of open tools
• Skill Transfer
• Embody general principles
• Expert and helpful community
• You can solve otherwise intractable problems
• Free
The only cost is time
CSIRO. Using Open Source Tools for Document Automation 35
36. Glossary
• ASKAP Australian SKA Pathfinder
• CGI Common Gateway Interface
• DRY Don’t Repeat Yourself
• DRY Don’t Repeat Yourself
• FAT Factory Acceptance Test
• GNU GNU is Not Unix
• HTML HyperText Markup Language
• ID Identifier
• Recursion see Recursion
• SKA Square Kilometre Array
• SAT Site Acceptance Test
• SQL Structured Query Language
• SVG Scaleable Vector Graphics
• URL Uniform Resource Locator
• XML eXtensible Markup Language
• XSLT eXtensible Stylesheet Language Transformations
• XSL/FO eXtensible Stylesheet Language /Formatting Objects
CSIRO. Using Open Source Tools for Document Automation 36
37. References
• 1. Open Source Initiative
http://www.opensource.org
• 2. Fairlight Instruments Web site
fairlightinstruments.com.au
• 3. DRY – Don’t Repeat Yourself
The Pragmatic Programmer: From Journeyman to Master
by Andrew Hunt and David Thomas
http://pragprog.com/the-pragmatic-programmer
• 4. ASKAP
• http://www.atnf.csiro.au/projects/askap/
• 5. CSIRO
http://www.csiro.au/
• 6. CSIRO Wireless LAN patent
CSIRO. Using Open Source Tools for Document Automation
38. CSIRO Astronomy and Space Science
Neale Morison
ASKAP Documentation Manager
Phone: 02 9372 4726
Email: neale.morison@csiro.au
Web: www.csiro.au/org/cass
Thank you
Contact Us
Phone: 1300 363 400 or +61 3 9545 2176
Email: enquiries@csiro.au Web: www.csiro.au