Kyiv Elixir Meetup 3.1 (3.11.2016) We will learn how the Erlang virtual machine handles code loading, how OTP releases are generated and how to prepare a release in Elixir so that it can be upgraded on the fly, without stopping the running application. We will also learn the intricate details of what goes on under the hood of the release upgrade process and we'll look at the pros and cons of performing hot code replacement in a production environment. Video here: https://youtu.be/uQE5dAWPAPc #kyivelixirmeetup #elixirmeetup

1. Releases and Hot Code
Replacement in Elixir
Alexei Sholik
Kyiv Elixir Meetup, 3 Nov 2016

2. What we'll learn
• The importance of separating code and data
• Code loading in BEAM
• Fundamentals of release upgrades in OTP
• Building an upgrade using Distillery

3. Separation of code
and data

4. OOP

5. OOP
A program is an object graph

6. OOP
def some_method(this, x):
y = this.method_foo(x)
# ...
this.method_bar(y)
What if we change
the code at this
point?

7. Code reloading in OOP
DOES NOT WORK

8. Functional
programming

9. Functional programming
Code and data are separate
data
data
code

10. Code reloading in
functional programming
TRIVIAL

11. Code reloading in
functional programming
OR IS IT?

12. Code reloading in
functional programming
def some_function(x) do
y = function_foo(x)
# ...
function_bar(y)
end
What if we change
the code at this
point?

13. Code reloading in
functional programming
def some_function(x) do
y = function_foo(x)
# ...
M.function_bar(y)
end
What if we change
the code at this
point?

14. Code loading in
BEAM

15. Code loading in BEAM
A process is a unit of concurrency
Inside a single process all code is executed
sequentially
Concurrency is achieved by interleaving the
execution of many processes

16. Code loading in BEAM
A module is a unit of code
A module is loaded or reloaded as a whole
The VM keeps at most two versions of the same
module
Reloading a module creates a new version of it in
memory

17. Code loading in BEAM
The most recently loaded version is called current
The previous version is called old

18. Code loading in BEAM
When a process is executing the code from a module
(basically, when it is inside a function call),
it is said to be running in that version of the module.
If the module is purged (by the VM or manually), the
process is killed immediately.

19. Local vs fully qualified
function calls
function_foo()
ModuleBar.function_baz()
local call
fully qualified
call

20. Local vs fully qualified
function calls
Local function call always invokes code from
the same module version that is being executed

21. Local vs fully qualified
function calls
Fully qualified function call always invokes code from
the latest module version

22. Code reloading in BEAM
def some_function(x) do
y = function_foo(x)
# ...
function_bar(y)
end
Changing the code
at this point won't
affect the function

23. Code reloading in BEAM
def some_function(x) do
y = function_foo(x)
# ...
M.function_bar(y)
end
Changing the code
for M at this point
will result in running
the new code for
function_bar()

24. Code reloading in BEAM
Doing a fully qualified function call allows a process
to switch to the new code.
But there are caveats we have to take into account...

25. Deterministic code
reloading in the face of
massive concurrency
In other words, "Would would OTP do?"

26. Code reloading with OTP
We have two problems to solve
1. Switching a process to the newly loaded module
version
2. Migrating the process state

27. Code reloading with OTP
How do we ensure our gen servers keep running
in those unforgiving circumstances?
By building a release, of course!

28. Release upgrades in
OTP

29. Quick guide to release
upgrades

30. Quick guide to release
upgrades
• Change the code. Make sure to handle all
necessary data migrations
• Update the app version
• Write an <app name>.appup file with instructions
on how each changed or new module should
be loaded

31. Quick guide to release
upgrades (continued)
• Generate a relup file with low-level instructions
for the release upgrade process
• Build a release, package it up into a .tar file
and place it in the releases/ directory on the
target machine
• Execute a series of calls to :release_handler in
order to install the new release in the running
system

32. .appup instructions
{update, Mod}
{update, Mod, supervisor}
{update, Mod, Change}
{update, Mod, DepMods}
{update, Mod, Change, DepMods}
{update, Mod, Change, PrePurge, PostPurge, DepMods}
{update, Mod, Timeout, Change, PrePurge, PostPurge, DepMods}
{update, Mod, ModType, Timeout, Change, PrePurge, PostPurge, DepMods}
{load_module, Mod}
{load_module, Mod, DepMods}
{load_module, Mod, PrePurge, PostPurge, DepMods}
{add_module, Mod}
{add_module, Mod, DepMods}
{delete_module, Mod}
{delete_module, Mod, DepMods}
{add_application, Application}
{add_application, Application, Type}
{remove_application, Application}
{restart_application, Application}
...

33. Thankfully, the actual
upgrade process is
handled by OTP

34. Release upgrade process
• Find all processes running the changed
modules and suspend them
• Load the new code for those modules
• Invoked the code_change() callback in all
relevant processes
• Resume suspended processes

35. Release upgrade process
(optional extra steps)
• Update the init() callback of a supervisor
process
• Add or remove an OTP application
• Restart a running OTP application
• Restart the whole node
• ... (there are more possibilities)

36. Upgrading a release
with Distillery

37. Upgrading a release with
Distillery
• Change the code. Make sure to handle all
necessary data migrations
• Update the app version
• Build a release using mix release --upgrade
• Activate and install the new release in the shell

38. Update the app version

39. Build a release
$ mix release --upgrade
==> Assembling release..
==> Building release chat:0.2.0 using environment prod
==> Including ERTS 8.1 from /...
==> Generated .appup for chat 0.1.0 -> 0.2.0
==> Relup successfully created
==> Packaging release..
==> Release successfully built!

40. Install the new release
$ rel/chat/bin/chat console
iex(chat@127.0.0.1)1>
:release_handler.set_unpacked 'releases/0.2.0/chat.rel', []
{:ok, '0.2.0'}
iex(chat@127.0.0.1)2> :release_handler.install_release '0.2.0'
{:ok, '0.1.0', []}
iex(chat@127.0.0.1)3> :release_handler.which_releases
[{'chat', '0.2.0', [...], :current},
{'chat', '0.1.0', [...], :permanent}]

41. Install the new release
(continued)
iex(chat@127.0.0.1)4> :release_handler.make_permanent '0.2.0'
:ok
iex(chat@127.0.0.1)5> :release_handler.which_releases
[{'chat', '0.2.0', [...], :permanent},
{'chat', '0.1.0', [...], :old}]

42. Distillery helps a lot

43. But you still have to
know the underlying
details

44. Conclusions
• The basic code reloading is trivial to do in
BEAM
• In a real application, though, there are a lot
more things to consider
• It's easy to make a mistake and cause the
application to misbehave
• Rule of thumb: don't use hot code replacement
unless absolutely necessary and worth the cost

45. Questions?
Alexei Sholik
github.com/alco
twitter.com/true_droid