Some ideas about debugging and monitoring live applications: logging, remote-shells using Twisted (even in non-twisted apps), python debuggers, and creating IM bots so your apps can talk to you. Presented at Kiwi Pycon 2009

1. Monitoring & Debugging
Live Applications
Rob Coup
robert@coup.net.nz

2. Who am I?
• Koordinates
• Open data
open.org.nz
• Geek
• Pythonista

3. Crash!
• trace ➭ reproduce ➭ simplify ➭ solve
• post-mortems suck

4. log everything

5. print "name=", name

6. print >>sys.stderr, "name=", name

7. # http://www.flickr.com/photos/larimdame/2575986601

8. log.debug("name=%s", name)

9. Separation
• where the log messages are generated
• where the log output goes to
• runtime configuration

10. http://www.flickr.com/photos/theeerin/1108095143

11. “too Java”
http://www.flickr.com/photos/lumaxart/2136948489

12. Too hard?
import logging
logging.basicConfig(level=logging.DEBUG,
file="/tmp/my.log")
log = logging.getLogger("somewhere")

13. Logger
• named (foo.bar.baz)
• debug() info() warn() error()
critical() exception()
• % formatting

14. Handler
• does something with log messages
• stderr, files, syslog, HTTP, email, …
• filtering on logger hierarchy & level

15. Formatter
• formats the log output for a Handler
• time, file/line/function, thread, process, …

16. Logging Example
• log everything to /tmp/machine.log
• log startup and shutdown events to syslog
• quieten some noisy libraries (log warnings
& errors only)
• email errors in machine.core to
admin@example.com

17. In the code
logging.config.fileConfig("machine-logging.ini")
L = logging.getLogger("machine.startstop.start")
L.info("Starting the machine...")
L = logging.getLogger("machine.core.reactor")
L.error("Meltdown in Sector 7-G!")

18. Logging Example
# our normal formatter
[formatter_normal]
format=%(asctime)s machine[%(process)d]: %(name)s %(module)s
(%(lineno)d): %(message)s
# root logger, send everything to the file log
[logger_root]
level=NOTSET
handlers=file
# log to a file
[handler_file]
class=FileHandler
formatter=normal
args=('/tmp/machine.log','a')

19. Logging Example
# send startup/shutdown notices to syslog
[logger_startstop]
qualname=machine.startstop
handlers=syslog
# log to syslog
[handler_syslog]
class=handlers.SysLogHandler
formatter=normal
args=('/dev/log',)

20. Logging Example
# ignore messages below WARN for noisy libraries
[logger_noisylibs]
level=WARN
qualname=noisy,chatty,loud
propagate=0
handlers=

21. Logging Example
# send machine.core errors to the email handler
[logger_core]
qualname=machine.core
level=ERROR
handlers=email
# log to email
[handler_email]
class=handlers.SMTPHandler
formatter=email
args=('localhost', 'logs@a.com', ['admin@a.com'], 'LOG NOTICE')

22. Other Log Tools
• syslog-ng - centralise
• ack - grep on steroids
• less - a real pager
• Splunk - match & search & filter in bulk

23. Debuggers
• pdb works quite well
• even nicer with IPython
• pydbgr (Pydb) & WinPDB

24. Attach to running
process
• I want “pdb -p 12345” (like gdb)
• Alternative:
import pdb, signal
signal.signal(signal.SIGUSR1, lambda x,y: pdb.set_trace())
• Then:
kill -s SIGUSR1 <pid>

25. Remote Consoles
http://www.flickr.com/photos/pasukaru76/3959355664/

26. Twisted Manhole
• Telnet/SSH server
• Access to interpreter namespace
• Interact with variables, call methods, …

27. Pretty easy
from twisted.conch import manhole_tap
manhole = manhole_tap.makeService({
'namespace': namespace,
# listen for Telnet connections on localhost:4777
'telnetPort' : 'tcp:4777:interface=127.0.0.1',
'sshPort' : None,
# path to passwd-style file (username:password, plaintext)
'passwd' : 'passwd',
})
manhole.setServiceParent(application)

28. But, uh, Twisted?!
• run the Twisted manhole in a thread
• leave normal Python code alone
• still get remote console
• need to handle concurrency!

29. Bots
http://saiogaman.deviantart.com/art/Danbo-Wallpaper-107237965

30. Why?
• interact with our apps from our desktop
• have our apps interact with us
• great for non-technical users

31. Twisted Words
• for accessing IM services - XMPP, IRC, …
• Wokkel makes XMPP even easier

32. Abuse Bot
class TalkProtocol(xmppim.MessageProtocol):
def onMessage(self, message):
# we've received a message!
reply = "nah, you're a " + str(message.body)
response = domish.Element((None, "message"))
response["to"] = message['from']
response.addElement((None, 'body'), content=reply)
self.send(response)

33. Outcomes
• Use logging
• Talk to your apps (console, bots)
• Have your apps talk to you (bots)
• You don’t need to have a Twisted app

34. Monitoring & Debugging
Live Applications
Rob Coup
robert@coup.net.nz