"Nature Language Processing for proteins" by Amélie Héliou, Software Engineer @ Google Research Abstract: Over the past few months, Large Language Models have become very popular. We'll see how a simple LLM works, from input sentence to prediction. I'll then present an application of LLM to protein name prediction. Twitter: @Amelie_hel

1. Amélie Héliou
NLP for Proteins

2. NLP for
Proteins
Encoder-Decoders
Transformers
Application to Proteins
Language representation

3. Application to
proteins
Predict proteins name
MRYRLAWLLHPALPSTFRSVLGARLPPPERLCGFQKKTYSKMNNP
AIKRIGNHITKSPEDKREYRGLELANGIKVLLISD …
P14735 Insulin-degrading enzyme
https://www.uniprot.org/uniprotkb/P14735/entry

4. Application to
proteins
Predict proteins name
● Names give an idea about the function of
proteins.
● Names are given by experts or rule based
annotations.
● ~ 50 millions of proteins without a name.
(20%).

5. Language
representation
Tokenization
Vocabulary
Embeddings
Input sentence:
Women are powerful.
Tokenization:
Women are powerful.
Vocabulary:
10 123 25
Embeddings:

6. Encoder
Example: BERT
Learnt token embeddings.
Can be trained on masked
tasks.
Input sentence:
Women are powerful.
Encoder

7. Decoder
Example: PaLM
Next token prediction
tasks.
Input sentence:
Women are powerful.
Decoder
Next token probabilities.

8. Encoder -
Decoder
Input sentence:
Women are powerful.
Encoder Decoder
Next token probabilities.

9. Autoregressive
tasks
Predict tokens one by
one.
Input sentence:
Women are powerful.
Encoder Decoder
Les
<s>

10. Autoregressive
tasks
Predict tokens one by
one.
Input sentence:
Women are powerful.
Encoder Decoder
femmes
<s> Les

11. Autoregressive
tasks
Predict tokens one by
one.
Input sentence:
Women are powerful.
Encoder Decoder
sont
<s> Les femmes

12. Autoregressive
tasks
Predict tokens one by
one.
Input sentence:
Women are powerful.
Encoder Decoder
puissantes
<s> Les femmes sont

13. Autoregressive
tasks
Predict tokens one by
one.
Input sentence:
Women are powerful.
Encoder Decoder
<eos>
<s> Les femmes sont puissantes

14. Transformers
Self-Attention
Nice explanation:
https://jalammar.github.io/illust
rated-transformer/

15. Transformers
Self-Attention
Nice explanation:
https://jalammar.github.io/illust
rated-transformer/
Women
are
powerful
Women are powerful
keys = XWk
queries
=
XWq

16. Transformers
Self-Attention
Nice explanation:
https://jalammar.github.io/illust
rated-transformer/
Women
are
powerful
Women are powerful
Women = Women x + are x
are = Women x + are x
powerful = Women x + powerful x
queries
=
XWq
keys = XWk
values = XWv

17. Transformers
Position
feed-forward
● Position is added to the initial
embedding.
● After attention:
○ Residual connection
(input+output)
○ Feed-forward

18. Transformers
Overall architecture.
Attention is all you
need, Vaswani et al.
2017

19. Application to
proteins
Predict proteins name
Input sentence:
MRYRLAWLLHPALPSTFRSVLGARLPPPERL
Encoder Decoder
Insulin
<s>

20. Application to
proteins
Predict proteins name
Input sentence:
MRYRLAWLLHPALPSTFRSVLGARLPPPERL
Encoder Decoder
degrading
<s> Insulin

21. Application to
proteins
Predict proteins name
Input sentence:
MRYRLAWLLHPALPSTFRSVLGARLPPPERL
Encoder Decoder
enzyme
<s> Insulin degrading

22. Application to
proteins
Predict proteins name
Input sentence:
MRYRLAWLLHPALPSTFRSVLGARLPPPERL
Encoder Decoder
<eos>
<s> Insulin degrading enzyme

23. Application to proteins

24. Thank you