This document discusses the thread and process safety of the Ruby Logger. It was found that while Logger is thread-safe due to use of a mutex, it is not process-safe as the mutex does not work across processes. The author contributed a fix to Ruby core that uses flock to lock access to logs across processes, ensuring process-safe logging. The fix will be included in Ruby 2.1.0, making the standard Ruby Logger thread and process-safe.

1. Is Ruby Logger
Thread(Process)safe?
RubyConf 2013 @Miami
SEO Naotoshi (@sonots)
2013/11/09

2. Naotoshi SEO
like Now-toshi Say-oh
@sonots (twitter, github)
Dev Engineer for Operations
Became as a commiter of Fluentd

3. •realtime pipelining of log streams
•plugins written in ruby

4. Log Management Tool
•syslog-ng / rsyslog Only collection
Not Free
•splunk
No Plugin
•scribed
Java Plugin
•flumeNG
Ruby Plugin (Not Gem)
•logstash
•fluentd Ruby Plugin (Gem Support)

5. Try the
fluentd!
It s very nice tool, check it out!
http://fluentd.org/

6. Today s
Talk

7. Logger

8. We needed a
Logger which
works safely
in multiprocess
for Fluentd

9. Examined Ruby Logger
• 1) Will logs not be mixtured in multi-threads?
• 2) Will logs not be mixtured in multi-processes?
• 3) Does log rotation work safely in multi-threads?
• 4) Does log rotation work safely in multi-processes?

10. 1) Will logs not be mixtured in multi-threads?
Code
require 'logger'
require 'parallel'
logger = Logger.new("/tmp/test.log")
Parallel.map(['a', 'b'], :in_threads => 2) do |letter|
10000.times do
logger.info letter * 5000
end
end
Result
$ egrep -e 'ab' -e 'ba' /tmp/test.log
[empty]
OK

11. WHY?
logger.rb#L559
def write(message)
begin
@mutex.synchronize do
...
@dev.write(message)
...
end
Ruby’s Logger is taking mutex to be thread-safe.

12. 2) Will logs not be mixtured in multi-processes?
Code
require 'logger'
require 'parallel'
logger = Logger.new("/tmp/test.log")
Parallel.map(['a', 'b'], :in_processes => 2) do |letter|
10000.times do
logger.info letter * 5000
end
end
Result
$ egrep -e 'ab' -e 'ba' /tmp/test.log
[empty]
OK

13. Really? Why?
Mutex does not work to exclusively lock in multiprocesses environment.
Atomic/non-atomic: A write is atomic if the whole amount written in one
operation is not interleaved with data from any other process. This is
useful when there are multiple writers sending data to a single reader.
Applications need to know how large a write request can be expected to
be performed atomically. This maximum is called {PIPE_BUF}. This volume
of IEEE Std 1003.1-2001 does not say whether write requests for more
than {PIPE_BUF} bytes are atomic, but requires that writes of {PIPE_BUF} or
fewer bytes shall be atomic.
But, Linux system call write(2) itself is atomic as long
as writing to a local file.
OK

14. 3) Does log rotation work safely multi-threads?
Code
require 'logger'
require 'parallel'
logger = Logger.new("/tmp/test.log", 3, 1024 * 10)
Parallel.map(['a', 'b'], :in_threads => 2) do |letter|
10000.times do
logger.info letter * 5000
end
end
Result
$ ls -l /tmp/test*
-rw-r--r-- 1 sonots sonots 10806
9月 29 23:03 /tmp/test.log
-rw-r--r-- 1 sonots sonots 10806
9月 29 23:03 /tmp/test.log.0
-rw-r--r-- 1 sonots sonots 10806
9月 29 23:03 /tmp/test.log.1
No Error, OK

15. 4) Does log rotation work safely in multi-processes?
Code
require 'logger'
require 'parallel'
logger = Logger.new("/tmp/test.log", 3, 1024 * 10)
Parallel.map(['a', 'b'], :in_processes => 2) do |letter|
10000.times do
logger.info letter * 5000
end
end
Result
log
log
log
log
...
writing failed. closed stream
shifting failed. closed stream
writing failed. closed stream
shifting failed. closed stream
Oops, Bad!

16. WHY?
logger.rb#L559
def write(message)
begin
@mutex.synchronize do
...
check_shift_log
...
end
Ruby’s Logger is taking mutex to be thread-safe,
but was not treating anything for multi-processes.

17. We(@frsyuki and me) Fixed
• The approach is to use flock(2) without creating
another *.lock file

18. Pull requested to ruby
My first contribution to ruby

19. Merged!!
Thanks to @naruse for his review and advice!!

20. Will be released
with 2.1.0!!
on 12/25

21. CONCLUSION
• Ruby Logger was not safe in multiprocesses, but it is safe now!!!
• Try ruby-trunk or
https://github.com/sonots/
process_safe_logger

22. Try the
fluentd!
It s very nice tool, check it out!
http://fluentd.org/