Evaluation of nine major Machine Translation providers: performance, quality, language coverage, latency, API update frequency.

1. Machine Translation
Benchmark
by Intento
July 2017

2. • New MT providers: Global Tone Communications
(best for English-to-Chinese so far)
• Improved MT quality: Google, Microsoft, Baidu
• Language coverage: no changes
• Added latency analytics (slide 8)
• Added API changes analytics (slide 9)
July 2017© Intento, Inc.
Changes since May 2017

3. Overview
• General Purpose Machine Translation Services with Public APIs
• IBM Watson Language Translator
• SDL Cloud Machine Translation
• Google Cloud Translation API
• Yandex Translate API
• Baidu Translate API
• Microsoft Translator Text API
• Systran REST Translation API
• PROMT Cloud API
• GTCom YeeCloud MT
• Benchmark Dimensions
• Performance vs. Price
• Language Coverage
• API Latency
• API Change frequency
May 2017© Intento, Inc.

4. Performance vs. Price
Dataset WMT2016-17
Domain: News
Pairs: ru-en, en-ru, de-en, en-de, cs-
en, en-cs, tr-en, en-tr, fi-en, en-fi, ro-en,
en-ro, en-zh, zh-en
Performance
Normalised hLEPOR score
aggregated across all
language pairs in the dataset
Affordability = 1/price
Using public volume-based
pricing tiers
Legend
• performance range
• price range
• average position
© Intento, Inc. July 2017

5. Language coverage
Unique
language
pairs -
supported
exclusively by
one provider
1
100
10000
Google Yandex Microsoft Baidu Systran SDL MT PROMT IBM Watson GTCom
22
4
1
154
906
1 246
3 094
3
20
84
104109
756
3 306
8 556
10 506
Total
Unique
Supported language pairs
© Intento, Inc. July 2017

6. Language groups by
web popularity*:
P1 - ≥ 2.0% websites
P2 - 0.5%-2% websites
P3 - 0.1-0.3% websites
P4 - <0.1% websites
100% 94% 64%
88%
30%
P1 P2 P3 P4
P1
P2
P3
P4
59%
100%
100%
94%
64%
100% 94%
94%
63%
63% 59%
Supported language pairs by popularity
* https://w3techs.com/technologies/overview/content_language/all© Intento, Inc.
Language coverage
44%
of possible
language pairs
July 2017

7. by service provider
Google Cloud
Translation API
Microsoft
Translator Text API
Yandex
Translate API
Systran
REST Translation API
SDL Cloud
Machine Translation
PROMT
Cloud API
IBM Watson
Language Translation
July 2017© Intento, Inc.
Language coverage
Baidu
Translate API

8. Based on 100
1-segment
requests for
en-de pair
Service latency
Latency,seconds
0,1
1
10
100
Google Microsoft PROMT Yandex SDL MT GTCom IBM Baidu Systran
75% 95% 99.9%
July 2017© Intento, Inc.

9. API & Documentation changes in July 2017:
Change frequency
SDL MT 27
IBM 14
Google 13
Baidu 3
July 2017© Intento, Inc.

10. Discover the best service providers for
your AI task
Evaluate performance on your own data
at a fraction of the potential cost
saving
Access any provider with no effort
using to our Single API
Intento Service Platform
July 2017© Intento, Inc.

11. Intento
https://inten.to
Konstantin Savenkov
CEO Intento, Inc.
<ks@inten.to>