ManageIQ currently runs on Ruby 1.9.3. This presentation is about the effort to move ManageIQ to Ruby 2.x to take advantage of new features and performance in the language and runtime engine.
For more on ManageIQ, see http://manageiq.org/
4. Agenda
1. History
2. Why upgrade?
3. Ruby 2.1
4. Ruby 2.0
5. "Fall cleanup" of old code
6. Slow tests
7. Building 2.0 appliances
8. Developer setup
9. Links
10. Questions?
5. History
Tue Nov 1 2011:
First ruby 1.8.7 -> 1.9.3 related commit on ManageIQ
6. History
Tue Nov 1 2011:
First ruby 1.8.7 -> 1.9.3 related commit on ManageIQ
Tue Apr 23 2013:
Ruby 1.9.3 finally...
7. History
Tue Nov 1 2011:
First ruby 1.8.7 -> 1.9.3 related commit on ManageIQ
Tue Apr 23 2013:
Ruby 1.9.3 finally...
540 days???
8. History
Tue Nov 1 2011:
First ruby 1.8.7 -> 1.9.3 related commit on ManageIQ
Tue Apr 23 2013:
Ruby 1.9.3 finally...
540 days???
Lesson learned: Don't wait to upgrade!
10. Why upgrade?
We're behind!!!
Ruby 1.9.3 is ending
In maintenance until February 23, 2014
Security only mode until February 23, 2015
Ruby 2.0.0 is nearly 20 months old
Ruby 2.1.0 is nearly 10 months old
Ruby 2.2.0 is scheduled for a Christmas release
11. Why upgrade? ... Because ruby 2.1!
Generational mark and sweep garbage collector
http://tmm1.net/ruby21-rgengc/
String#freeze - reuse String objects
Less objects == less memory == less GC time
Object allocation tracing
http://tmm1.net/ruby21-objspace/
https://github.com/srawlins/allocation_stats
Required keyword arguments
def returns method name
Exception#cause - ActiveRecord::StatementInvalid#cause -> Real error
More...
12. Why upgrade? ... Because ruby 2.1!
Useless benchmark
bundle exec rspec spec/models/ems_refresh/refreshers
1.9.3-p545
70.85s user 2.23s system 96% cpu 1:15.98 total
71.21s user 2.22s system 96% cpu 1:16.27 total
13. Why upgrade? ... Because ruby 2.1!
Useless benchmark
bundle exec rspec spec/models/ems_refresh/refreshers
1.9.3-p545
70.85s user 2.23s system 96% cpu 1:15.98 total
71.21s user 2.22s system 96% cpu 1:16.27 total
2.0.0-p576
54.02s user 2.03s system 95% cpu 58.980 total
49.96s user 2.14s system 94% cpu 54.923 total
14. Why upgrade? ... Because ruby 2.1!
Useless benchmark
bundle exec rspec spec/models/ems_refresh/refreshers
1.9.3-p545
70.85s user 2.23s system 96% cpu 1:15.98 total
71.21s user 2.22s system 96% cpu 1:16.27 total
2.0.0-p576
54.02s user 2.03s system 95% cpu 58.980 total
49.96s user 2.14s system 94% cpu 54.923 total
2.1.3
36.52s user 2.79s system 91% cpu 42.930 total
35.68s user 2.22s system 92% cpu 40.768 total
15. Why upgrade? ... Because ruby 2.1!
Example allocation information:
Line number
Number of allocations by object type
Such as:
Running: ./spec/controllers/application_controller/buttons_spec.rb:91
223730 Arrays @ .../activerecord/lib/active_record/result.rb:35
203280 Strings @ .../activerecord/lib/active_record/relation.rb:27
16. Why upgrade? ... Because ruby 2.1!
Example allocation information:
Line number
Number of allocations by object type
Such as:
Running: ./spec/controllers/application_controller/buttons_spec.rb:91
223730 Arrays @ .../activerecord/lib/active_record/result.rb:35
203280 Strings @ .../activerecord/lib/active_record/relation.rb:27
See Issue 241 and 762
20. Faster Rails startup
Optimizations were made to speed up 'require'
(master) (1.9.3-p545) + time bundle exec rake environment
bundle exec rake environment 3.51s user 0.63s system 99% cpu 4.148 total
(master) (2.0.0-p576) + time bundle exec rake environment
bundle exec rake environment 2.60s user 0.53s system 99% cpu 3.132 total
21. Faster Rails startup
Optimizations were made to speed up 'require'
(master) (1.9.3-p545) + time bundle exec rake environment
bundle exec rake environment 3.51s user 0.63s system 99% cpu 4.148 total
(master) (2.0.0-p576) + time bundle exec rake environment
bundle exec rake environment 2.60s user 0.53s system 99% cpu 3.132 total
25% faster loading of rails environment!
22. Faster Rails startup
Optimizations were made to speed up 'require'
(master) (1.9.3-p545) + time bundle exec rake environment
bundle exec rake environment 3.51s user 0.63s system 99% cpu 4.148 total
(master) (2.0.0-p576) + time bundle exec rake environment
bundle exec rake environment 2.60s user 0.53s system 99% cpu 3.132 total
25% faster loading of rails environment!
Most obvious when:
Running tests
Loading Rails console
30. Module#prepend
Using alias_method:
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
def run
puts "Sub"
super
end
def run_with
puts "DebugIt"
run_without
end
alias_method :run_without, :run
alias_method :run, :run_with
end
31. Module#prepend
Using alias_method:
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
def run
puts "Sub"
super
end
def run_with
puts "DebugIt"
run_without
end
Sub is a class with the slow run method...
alias_method :run_without, :run
alias_method :run, :run_with
end
32. class Parent
def run
puts "Parent"
end
end
class Sub < Parent
def run
puts "Sub"
super
end
def run_with
puts "DebugIt"
run_without
end
Sub is a class with the slow run method...
alias_method :run_without, :run
alias_method :run, :run_with
end
irb(main):01:0> Sub.new.run
DebugIt
Sub
Parent
Module#prepend
Using alias_method:
33. class Parent
def run
puts "Parent"
end
end
class Sub < Parent
def run
puts "Sub"
super
end
def run_with
puts "DebugIt"
run_without
end
Sub is a class with the slow run method...
alias_method :run_without, :run
alias_method :run, :run_with
end
irb(main):01:0> Sub.new.run
DebugIt
Sub
Parent
YAY! But that's really dirty!
(alias_method_chain)
Module#prepend
Using alias_method:
34. Module#prepend
Using include:
module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
include DebugIt
def run
puts "Sub"
super
end
end
35. module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
include DebugIt
def run
puts "Sub"
super
end
end
Sub is a class with the slow run method...
Module#prepend
Using include:
36. module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
include DebugIt
def run
puts "Sub"
super
end
end
Sub is a class with the slow run method...
irb(main):002:0> Sub.ancestors
=> [Sub, DebugIt, Parent, Object, Kernel, BasicObject]
Module#prepend
Using include:
37. module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
include DebugIt
def run
puts "Sub"
super
end
end
Sub is a class with the slow run method...
irb(main):002:0> Sub.ancestors
=> [Sub, DebugIt, Parent, Object, Kernel, BasicObject]
irb(main):01:0> Sub.new.run
Sub
DebugIt
Parent
Module#prepend
Using include:
38. module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
include DebugIt
def run
puts "Sub"
super
end
end
Sub is a class with the slow run method...
irb(main):002:0> Sub.ancestors
=> [Sub, DebugIt, Parent, Object, Kernel, BasicObject]
irb(main):01:0> Sub.new.run
Sub
DebugIt
Parent
UGH, Sub's method comes first!
Module#prepend
Using include:
39. Module#prepend
Using prepend:
module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
prepend DebugIt
def run
puts "Sub"
super
end
end
40. module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
prepend DebugIt
def run
puts "Sub"
super
end
end
irb(main):002:0> Sub.ancestors
=> [DebugIt, Sub, Parent, Object, Kernel, BasicObject]
Module#prepend
Using prepend:
41. module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
prepend DebugIt
def run
puts "Sub"
super
end
end
irb(main):002:0> Sub.ancestors
=> [DebugIt, Sub, Parent, Object, Kernel, BasicObject]
irb(main):01:0> Sub.new.run
DebugIt
Sub
Parent
Module#prepend
Using prepend:
42. module DebugIt
def run
puts "DebugIt"
super
end
end
class Parent
def run
puts "Parent"
end
end
class Sub < Parent
prepend DebugIt
def run
puts "Sub"
super
end
end
irb(main):002:0> Sub.ancestors
=> [DebugIt, Sub, Parent, Object, Kernel, BasicObject]
irb(main):01:0> Sub.new.run
DebugIt
Sub
Parent
Ship it!
... but don't forget to call super!
Module#prepend
Using prepend:
47. Refinements
Goal: localize monkey patches
I'm not going to explain them because:
Ruby's open classes
Many gotchas...
See Charles Nutter (@headius/jruby guy) explanation:
http://blog.headius.com/2012/11/refining-ruby.html
49. Enumerable#lazy
Enumerable methods evaluate left to right
With lazy, chains of enumerations are evaluated right to left
Ruby may "cheat":
May skip creating intermediate objects
Large collection operations may be optimized
50. Enumerable#lazy
Example benchmark
(0...1000).select(&:odd?).take(5).to_a
(0...1000).lazy.select(&:odd?).take(5).to_a
What is this doing?
51. Enumerable#lazy
Example benchmark
(0...1000).select(&:odd?).take(5).to_a
(0...1000).lazy.select(&:odd?).take(5).to_a
What is this doing?
First 5 odd numbers
=> [1, 3, 5, 7, 9]
52. Enumerable#lazy
require 'benchmark/ips'
Benchmark.ips do |x|
x.report("normal") { (0...1000).select(&:odd?).take(5).to_a }
x.report("lazy") { (0...1000).lazy.select(&:odd?).take(5).to_a }
end
53. Enumerable#lazy
require 'benchmark/ips'
Benchmark.ips do |x|
x.report("normal") { (0...1000).select(&:odd?).take(5).to_a }
x.report("lazy") { (0...1000).lazy.select(&:odd?).take(5).to_a }
end
Calculating -------------------------------------
normal 1539 i/100ms
lazy 5778 i/100ms
-------------------------------------------------
normal 15123.2 (±2.5%) i/s - 76950 in 5.091373s
lazy 59797.4 (±4.0%) i/s - 300456 in 5.033123s
See http://patshaughnessy.net/2013/4/3/ruby-2-0-works-hard-so-you-can-be-
lazy
54. __dir__
__dir__ path of the script without the filename
# cat test.rb
puts __dir__
# ruby test.rb
/Users/joerafaniello/Code/test
55. __dir__
__dir__ path of the script without the filename
# cat test.rb
puts __dir__
# ruby test.rb
/Users/joerafaniello/Code/test
We have 984 instances of File.dirname(__FILE__)!
WAT...Why?
56. Ruby 2.0 breaking changes and deprecations
Note: We're green on travis, so we're getting close...
57. Objects don't respond_to? to protected methods
Ruby 1.9.3:
respond_to?(symbol) => public and protected methods
respond_to?(symbol, true) => all methods
58. Objects don't respond_to? to protected methods
Ruby 1.9.3:
respond_to?(symbol) => public and protected methods
respond_to?(symbol, true) => all methods
Ruby 2.0.0
respond_to?(symbol) => public methods only
respond_to?(symbol, true) => all methods
60. Objects don't respond_to? to protected methods
class Worker
protected
def run
end
end
Worker.new.respond_to?(:run)
1.9.3 => true
2.0.0 => false
61. Objects don't respond_to? to protected methods
class Worker
protected
def run
end
end
Worker.new.respond_to?(:run)
1.9.3 => true
2.0.0 => false
Pass true as second argument...
Worker.new.respond_to?(:run, true)
1.9.3 => true
2.0.0 => true
62. Objects don't respond_to? to protected methods
class Worker
protected
def run
end
end
Worker.new.respond_to?(:run)
1.9.3 => true
2.0.0 => false
Pass true as second argument...
Worker.new.respond_to?(:run, true)
1.9.3 => true
2.0.0 => true
See Pull #685 - default_value_for gem
63. UTF-8 is the default character encoding of ruby
scripts
# cat test.rb
FOO = "222dL256"
64. UTF-8 is the default character encoding of ruby
scripts
# cat test.rb
FOO = "222dL256"
Ruby 1.9.3
US-ASCII is the default encoding of ruby scripts
65. UTF-8 is the default character encoding of ruby
scripts
# cat test.rb
FOO = "222dL256"
Ruby 1.9.3
US-ASCII is the default encoding of ruby scripts
Binary string literals become ASCII-8BIT:
66. UTF-8 is the default character encoding of ruby
scripts
# cat test.rb
FOO = "222dL256"
Ruby 1.9.3
US-ASCII is the default encoding of ruby scripts
Binary string literals become ASCII-8BIT:
irb(main):001:0> require './test'
=> true
irb(main):002:0> FOO.encoding
=> #<Encoding:ASCII-8BIT>
67. UTF-8 is the default character encoding of ruby
scripts
# cat test.rb
FOO = "222dL256"
68. UTF-8 is the default character encoding of ruby
scripts
# cat test.rb
FOO = "222dL256"
Ruby 2.0.0
UTF-8 is the default encoding
69. UTF-8 is the default character encoding of ruby
scripts
# cat test.rb
FOO = "222dL256"
Ruby 2.0.0
UTF-8 is the default encoding
Binary string literals become UTF-8 even if invalid:
irb(main):001:0> require './test'
=> true
irb(main):002:0> FOO.encoding
=> #<Encoding:UTF-8>
irb(main):003:0> FOO.valid_encoding?
=> false
70. UTF-8 is the default character encoding of ruby
scripts
So, what's the problem?
Binary strings are used in many places for vm "fleecing"
Invalid UTF-8 encoded strings != raw binary:
71. UTF-8 is the default character encoding of ruby
scripts
So, what's the problem?
Binary strings are used in many places for vm "fleecing"
Invalid UTF-8 encoded strings != raw binary:
irb(main):003:0> FOO.valid_encoding?
=> false
irb(main):004:0> FOO == "222dL256".force_encoding("ASCII-8BIT")
=> false
72. UTF-8 is the default character encoding of ruby
scripts
Solutions:
Force binary on individual Strings:
1.9.3/2.0.0 compatible
Painful on files with many binary string literals
73. UTF-8 is the default character encoding of ruby
scripts
Solutions:
Force binary on individual Strings:
1.9.3/2.0.0 compatible
Painful on files with many binary string literals
# cat test.rb
FOO = "222dL256".force_encoding("ASCII-8BIT")
74. UTF-8 is the default character encoding of ruby
scripts
Solutions:
Force binary on individual Strings:
1.9.3/2.0.0 compatible
Painful on files with many binary string literals
# cat test.rb
FOO = "222dL256".force_encoding("ASCII-8BIT")
irb(main):001:0> require './test'
=> true
irb(main):002:0> FOO.encoding
=> #<Encoding:ASCII-8BIT>
irb(main):003:0> FOO.valid_encoding?
=> true
irb(main):004:0> FOO == "222dL256".force_encoding("ASCII-8BIT")
=> true
75. UTF-8 is the default character encoding of ruby
scripts
Solutions:
Use String#b on individual strings
Not compatible with ruby 1.9.3
Copies the String in ASCII-8BIT encoding
Note: String#force_encoding("ASCII-8BIT") modifies the receiver
76. UTF-8 is the default character encoding of ruby
scripts
Solutions:
Add #encoding magic comment at top
1.9.3/2.0.0 compatible
Good option when binary strings are expected
77. UTF-8 is the default character encoding of ruby
scripts
Solutions:
Add #encoding magic comment at top
1.9.3/2.0.0 compatible
Good option when binary strings are expected
# cat test.rb
# encoding: US-ASCII
FOO = "222dL256"
78. UTF-8 is the default character encoding of ruby
scripts
Solutions:
Add #encoding magic comment at top
1.9.3/2.0.0 compatible
Good option when binary strings are expected
# cat test.rb
# encoding: US-ASCII
FOO = "222dL256"
irb(main):001:0> require './test'
=> true
irb(main):002:0> FOO.encoding
=> #<Encoding:ASCII-8BIT>
79. Descriptors except 0, 1, 2 are closed in child
processes
Prevents file descriptor leakage
See Pull #682, Issue #459, and https://bugs.ruby-lang.org/issues/5041
80. Descriptors except 0, 1, 2 are closed in child
processes
Prevents file descriptor leakage
See Pull #682, Issue #459, and https://bugs.ruby-lang.org/issues/5041
Example: shared pipe to communicate data between two processes
81. Descriptors except 0, 1, 2 are closed in child
processes
Prevents file descriptor leakage
See Pull #682, Issue #459, and https://bugs.ruby-lang.org/issues/5041
Example: shared pipe to communicate data between two processes
Use IO#close_on_exec = false
reader, writer = IO.pipe
writerfd = writer.fileno
my_env["WRITER_FD"] = writerfd.to_s
+
+ writer.close_on_exec = false
+
pid = Kernel.spawn(my_env, "ruby #{SERVER_PATH}VixDiskLibServer.rb",
[:out, :err] => [LOG_FILE, "a"],
:unsetenv_others => true,
82. String#lines returns an array
Also String#chars, #bytes and #codepoints
Previously, returned enumerators
Above are deprecated for StringIO, IO and friends
83. String#lines returns an array
Also String#chars, #bytes and #codepoints
Previously, returned enumerators
Above are deprecated for StringIO, IO and friends
StringIO#lines deprecated, so use #each_line instead:
@log_stream.rewind
- lines = @log_stream.lines.to_a
+ lines = @log_stream.each_line.to_a
lines.length.should == 1
line = lines.first.chomp
84. String#lines returns an array
Also String#chars, #bytes and #codepoints
Previously, returned enumerators
Above are deprecated for StringIO, IO and friends
StringIO#lines deprecated, so use #each_line instead:
@log_stream.rewind
- lines = @log_stream.lines.to_a
+ lines = @log_stream.each_line.to_a
lines.length.should == 1
line = lines.first.chomp
See Pull #714
86. "Fall cleanup" of old code
"...Now I've only been an OpenBSD developer for 11 years, one year less than
this header has existed, but in that brief time, I've learned a thing or two
about deleting obsolete code. It doesn't delete itself. And worse, people
will continue using it until you force them onto a better path."
http://freshbsd.org/commit/openbsd/68dc781944a2c5b90f8b6e1069a4201750c67f94
87. "Fall cleanup" of old code
"...Now I've only been an OpenBSD developer for 11 years, one year less than
this header has existed, but in that brief time, I've learned a thing or two
about deleting obsolete code. It doesn't delete itself. And worse, people
will continue using it until you force them onto a better path."
http://freshbsd.org/commit/openbsd/68dc781944a2c5b90f8b6e1069a4201750c67f94
Git is our friend if we really want it back
89. "Fall cleanup" of old code
Path to ruby 2.0...
83 commits
565 lines added
5,553 lines deleted
90. "Fall cleanup" of old code
Path to ruby 2.0...
83 commits
565 lines added
5,553 lines deleted
A good start?
91. "Fall cleanup" of old code
More "opportunities"
host directory
soap4r/actionwebservice (fork)
handsoap (fork) - note, useful but still forked :-(
ruport (fork)
ziya (fork) - patches rails!
prototype
old rails plugins
old gems
old monkey patches
92. Slow tests
Tests take 30+ minutes on CI servers
Separate tests
Minimizing setup (database inserts)
Remove invalid/not useful/duplicate tests
Allocation tracing with ruby 2.1
Cut support for 1.9.3 when 2.0 is stable
93. Building 2.0 appliances
Verified Ruby 2.0 on CentOS appliance:
Appliance startup
SmartState Analysis "fleecing" using vddk
vCenter inventory
Basic reporting
94. Building 2.0 appliances
Goal: automate building ruby 2.0 appliances
We currently use ruby 1.9.3 through SCL rpms
Not multi-platform
Restricts updating of some gems
Ruby 2.1 is not yet packaged as SCL rpms
95. Building 2.0 appliances
Solution:
rpms for base CentOS OS
rpms needed to build ruby and compiled gems
libxml2-devel, libxslt-devel, etc.
96. Building 2.0 appliances
Solution:
rpms for base CentOS OS
rpms needed to build ruby and compiled gems
libxml2-devel, libxslt-devel, etc.
Use ruby-install or ruby-build for building ruby
https://github.com/postmodern/ruby-install
https://github.com/sstephenson/ruby-build
97. Building 2.0 appliances
Solution:
rpms for base CentOS OS
rpms needed to build ruby and compiled gems
libxml2-devel, libxslt-devel, etc.
Use ruby-install or ruby-build for building ruby
https://github.com/postmodern/ruby-install
https://github.com/sstephenson/ruby-build
Let bundler handle what it does well...
98. Developer setup
2.0.0 is not much different from 1.9.3:
Install using rvm, ruby-install, or ruby-build
Manage with rvm, rbenv, or chruby
Need guinea pigs to try it and document any issues
Others tools, such as rubymine, may require some configuration
99. Links
2.0 open issues: https://github.com/ManageIQ/manageiq/labels/ruby%202
2.0 closed issues: https://github.com/ManageIQ/manageiq/issues?
q=label%3A%22ruby+2%22+is%3Aclosed
2.0 in depth: http://globaldev.co.uk/2013/03/ruby-2-0-0-in-detail/
2.1 in depth: http://globaldev.co.uk/2014/05/ruby-2-1-in-detail/
Slides available here: https://github.com/jrafanie/manageiq_summit_ruby20
Slides written in markdown using remarkjs: http://remarkjs.com/#1