Easy contributable internationalization process with Sphinx (PyCon APAC 2015 in Taiwan)

1. Takayuki Shimizukawa Sphinx co-maintainer Sphinx-users.jp 1

2. Who am I @shimizukawa 1. Sphinx co-maintainer 2. Sphinx-users.jp 3. PyCon JP committee  BePROUD co, ltd. 2

3. http://pycon.jp/ 3

4. Early Bird Registration: ２１０ tickets. Speaker Call for Proposals: until July 15th. 4 170 tickets gone

5. Agenda 1. Sphinx introduction & Setup for i18n 2. A easy way and Non easy ways to translate 3. Sphinx i18n feature 4. Automated translation process with several services 5. Tips, tricks, traps 5

7. What is Sphinx? Sphinx is a documentation generator Sphinx generates doc as several output formats from the reST text markup 7 1. Sphinx introduction & Setup for i18n Sphinx reSTreSTreStructuredText (reST) reST Parser HTML Builder ePub Builder LaTeX Builder texlive HTML theme Favorite Editor

8. Many docs are written by Sphinx For examples  Python libraries/tools: Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …  Non python libraries/tools: Chef, CakePHP(2.x), MathJax, Selenium, Varnish 8 1. Sphinx introduction & Setup for i18n

9. Relation between Sphinx and Docutils  Sphinx based upon Docutils library  Docutils/Sphinx supports reStructuredText it called reST, it's an extensible markup (PEP287 / accepted at 2002) 9 1. Sphinx introduction & Setup for i18n Sphinx reST Parser HTML Builder ePub Builder LaTeX Builder HTML theme docutils

10. Comparing Docutils and Sphinx Docutils Sphinx 1. Single page 2. Link to others with full name 3. Cross reference in a page 4. Writers: html, latex, man, xml, ... 5. Standard reST markup specs 1. Multiple Pages 2. toctree to connect all pages under single tree structure 3. Cross reference over each other pages 4. Additional writers: html(w/ themes), pdf(latex), texinfo, epub, … 5. Additional markup specs: autodoc directive and so on 10 1. Sphinx introduction & Setup for i18n

11. SPHINX docutils HTML Builder HTML theme (Jinja2) gettext Builder *.pot *.po I18N *.mo OmegaT Pootle Transifex Translation Tools, Services Favorite Editor Sphinx i18n feature (built-in) 11 1. Sphinx introduction & Setup for i18n reST Parser (directive / role) doctree (Intermediate) reSTreST reStructuredText (reST)

12. Sphinx internals and externals 12 1. Sphinx introduction & Setup for i18n SPHINX docutils Sphinx拡張 directive/role reSTreST reStructuredText (reST) Several formats HTML Builder EPUB Builder LaTeX Builder HTML theme (Jinja2) code highlighter (Pygments) man, texinfo, text, ... Builder gettext Builder XML, man, texinfo, text, winhelp, qthelp, ... TeXLive or else *.pot *.po I18N *.mo OmegaT Pootle Transifex Translation Tools, Services Favorite Editor Sphinx Extension Builder Custom Theme epub3, docx, dash, ... SphinxExtension directive/role reST Parser (directive / role) doctree (Intermediate)

13. $ pip install sphinx Support tools for translation How to install Sphinx with i18n tools To build a sphinx document $ pip install sphinx-intl $ pip install transifex-client=0.8 13 1. Sphinx introduction & Setup for i18n

14. $ git clone https://github.com/shimizukawa/deepthought.git $ cd deepthought/doc $ make html ... Build finished. The HTML pages are in _build/html. "make html" command generates html files into _build/html. make html once 14 1. Sphinx introduction & Setup for i18n

16. 16

17. What is internationalization? 17 2. Easy contributable internationalization process with Sphinx I18N

18. What is easy contributable? 18 2. Easy contributable internationalization process with Sphinx

19. Not a easy way (example 1/3) The manual are provided only in the HTML files Rewriting reST files 19 2. Easy contributable internationalization process with Sphinx

20. Not a easy way (example 2/3) The manual are generated from reST files and docstrings in the source files  Rewriting reST files  Rewriting docstrins in the source files. 20 2. Easy contributable internationalization process with Sphinx

21. Not a easy way (example 3/3) OK, we are engineers. We can use git!  Learn git  Learn GitHub  git clone to get the code  Translation (Yes! This is what I want to do!)  git commit  git pull  Sometimes resolve conflict  git push 21 2. Easy contributable internationalization process with Sphinx

22. Not easy vs easy Not a easy way Easy way 1. Learn git 2. Learn GitHub 3. git clone to get the code 4. Translation 5. git commit, pull, push 6. Resolve conflict 7. Build html your self 1. No git 2. No Github 3. No file 4. Translation 5. Update Automatically 6. No conflict 7. You can get a HTML output w/o hand-build. 22 2. Easy contributable internationalization process with Sphinx

23. 23

24. Two i18n features of Sphinx  Output pot files: from reST  Input po files: to generate translated HTML 24 3. Sphinx i18n feature reST pot reST html po

25. Translation flow  Generate pot  Translate pot into po  Generate Translated HTML 25 3. Sphinx i18n feature reST pot reST html po pot po Translator Please translate Translate! Thanks for your Contribution!

26. #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize ``obj`` to a JSON formatted :class:`str`." msgstr "" msgid "For example:" msgstr "" 26 Output pot files 3. Sphinx i18n feature $ make gettext ... Build finished. The message catalogs are in _build/gettext. $ ls _build/gettext api.pot examples.pot generated.pot index.pot generated.pot reST pot

28. 28 Preparing po files to translate $ sphinx-intl update -p _build/gettext -l zh_tw Create: locale/zh_tw/LC_MESSAGES/api.po Create: locale/zh_tw/LC_MESSAGES/examples.po Create: locale/zh_tw/LC_MESSAGES/generated.po Create: locale/zh_tw/LC_MESSAGES/index.po At first, sphinx-intl copy pot files and rename them pot po sphinx-intl $ make gettext $ sphinx-intl update -p _build/gettext -l zh_tw Not Changed: locale/zh_tw/LC_MESSAGES/api.po Updated: locale/zh_tw/LC_MESSAGES/examples.po +3, -1 After change the document, sphinx-intl update differences 3. Sphinx i18n feature

29. Translate po files #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize `òbj`` to a JSON formatted :class:`str`." msgstr "" generated.po pot po sphinx-intl Translator #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize `òbj`` to a JSON formatted :class:`str`." msgstr "msgstr "序列化 `òbj`` 要JSON格式 :class:`str`." generated.po Translate by using Vim, Emacs, OmegaT, ... 29 3. Sphinx i18n feature

30. Input po files 30 3. Sphinx i18n feature reST html po Translated $ make html ... Build finished. The HTML pages are in _build/html.

31. Big picture 31 3. Sphinx i18n feature reST pot html po make gettext sphinx-intl Translator make html Doc Author Translation Catalog Translated catalog

32. 32

33. Entire process to translate sphinx docs 33 4. Automated translation process with several services reST pot html po make gettext sphinx-intl make html Doc Author Translation Catalog Translated catalog Translator Translator Translator Autor / Translator upload Translator clone Translator

34. To Be 34 4. Automated translation process with several services reST pot html po make gettext sphinx-intl make html Doc Author Translation Catalog Translated catalog upload Translators To Be Automated To Be Parallel clone

35. Translation tool types  Vim / Emacs (Editors)  Edit local files  Translation support plugins are available.  OmegaT (Translation Tools)  Edit local files.  Translation support features.  Transifex (Services)  Edit online  Translation support features. 35 4. Automated translation process with several services po Translators To Be Parallel

36. Be Parallel by using Transifex Transifex provides... As API: Upload pot Download po As Web console: Glossary Translation memory Recommendation Auto-translation 36 4. Automated translation process with several services po Translators Parallel pot Upload pot Auto Update sphinx-intl transifex-client po transifex-client Download

37. Translation on Transifex web console 37 4. Automated translation process with several services Original Text Translated Text (you should keep reST syntax) Suggestions from Translation Memory (TM) Original (pot) Translation (po) Copy orig to translation Machine translation Save Review (if needed) Translators Parallel 1 2 4 3 5 6

38. To Be Automated 38 4. Automated translation process with several services po Translators Parallel pot Upload pot Auto Update sphinx-intl transifex-client po transifex-client Download reST html make gettext make html Doc Author upload To Be Automated clone

39. To Be Automated 39 4. Automated translation process with several services pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated

40. The procedure for automation 1. clone repository 2. make gettext 3. Upload pot 4. Download po 5. make html 6. Upload html 40 4. Automated translation process with several services pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated

41. 1. pip install sphinx sphinx-intl transifex-client==0.8 2. git clone https://github.com/shimizukawa/deepthought.git 3. cd deepthought/doc 4. sphinx-intl create-transifexrc # create ~/.transifexrc 5. sphinx-intl create-txconfig # create .tx/config 6. make gettext 7. sphinx-intl -p _build/gettext update-txconfig-resources # update .tx/config 8. tx push -s # push pot files to transifex 9. tx pull -l zh_tw # pull po files from transifex 10. make html SPHINXOPTS="-D language=zh_tw" Shell commands for automation run.sh $ export SPHINXINTL_TRANSIFEX_USERNAME=mice $ export SPHINXINTL_TRANSIFEX_PASSWORD=42 $ export SPHINXINTL_TRANSIFEX_PROJECT_NAME=deepthought-0_7 $ export SPHINXINTL_POT_DIR=_build/gettext $ run.sh 41 4. Automated translation process with several services

42. The drone.io 42 WebHook Deploy Clone repository Run shell script The drone.io is a continuous integration service. 4. Automated translation process with several services

43. GitHub + drone.io + S3 43 GitHub Amazon S3 Transifex 1. Clone repository 2. make gettext. 3. Upload pot 4. Download po 5. make html 6. Upload html 4. Automated translation process with several services 2 1 3

44. Be Automated 44 pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated Upload pot Download po upload WebHook clone 1 5 6 make html make gettext2 3 4 1 WebHook Automated 4. Automated translation process with several services

45. Automated by drone.io 45 Upload potDownload po upload WebHook clone 1 5 6 make html make gettext2 3 4 1 WebHook Automated 4. Automated translation process with several services

46. View from doc author  Doc Author doesn't require annoying procedure. Update Translation Source Doc Author Notify See Commit 46 4. Automated translation process with several services

47. View from doc translators  No git  No file  No conflict  Update Automatically  They can get a translated HTML output w/o hand-build. Translators Parallel See Translate Translated Pages 47 4. Automated translation process with several services

48. The entire automated process 48 Update Translation Source Doc Author Translators Parallel Notify See See Publish Translate Commit Download Translations Notify Automated 4. Automated translation process with several services

49. 49

50. TIP: Drone.io limits 15mins for a build  Drone.io limits 15mins for a build.  Install from wheel package may help you 50 5. Tips, tricks, traps 1. curl -L -s https://example.com/wheelhouse.tgz | tar vzxf - 2. export PIP_FIND_LINKS=./wheelhouse 3. pip install sphinx sphinx-intl transifex-client==0.8 run.sh ex. https://bitbucket.org/sphinxjp/docutils-translation/

51. TRAP: Version of transifex-client  transifex-client 0.11b3 is not stable (for me)  Especially for Windows users  If you have encountered trouble with using newer version, please try: 51 5. Tips, tricks, traps $ pip install "transifex-client=0.8"

52.  Drone.io only list-up repositories which you have admin permission.  Prepare a empty repository to create a drone.io project. 52 TRICK: Preparing Drone.io project 5. Tips, tricks, traps

53. Examples  Sphinx-1.3 doc for "ja" translation https://drone.io/bitbucket.org/shimizukawa/sphinx-doc13/admin  Sphinx-1.4 doc for "ja" translation https://drone.io/bitbucket.org/shimizukawa/sphinx-doc14/admin  Docutils doc for "ja" translation https://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin 53 5. Tips, tricks, traps

54. Questions? @shimizukawa Grab me at Night Market. 54

55. Thanks :) 55