Experiences from the trenches: Developing a commercial open source project in Python.

1. Developing a Commercial Open Source Project
with Python
OSDC 2007, Brisbane
Juergen Brendel
Principal Software Engineer

2. Agenda
• The SnapLogic project
• Open source opportunities / challenges
• Python opportunities / challenges
Developing a Commercial Open Source Project with Python Slide 2
OSDC 2007, Brisbane

3. The SnapLogic project
Started in 2005
•
• Open source data integration framework
• REST
• Targeted at developers
Developing a Commercial Open Source Project with Python Slide 3
OSDC 2007, Brisbane

4. Example: Browser based GUI
Developing a Commercial Open Source Project with Python Slide 4
OSDC 2007, Brisbane

5. Example: Resources
HTTP://server1.example.com/customer_list
Databases
SnapLogic Server
Client
Files HTTP
HTTP
Component
Request
Applications
and
Atom / RSS
Response
Resource
Definition
• Resource Name
• HTTP://server1.example.com/customer_list
• Component and Behavior
• SQL Query or filename
• credentials
• parameters
Developing a Commercial Open Source Project with Python Slide 5
OSDC 2007, Brisbane

6. Example: Resources
HTTP://server1.example.com/processed_customer_list
Databases
SnapLogic Server
Client
Files
HTTP
HTTP
Component Component Component
Request
Applications
and
Atom / RSS
Response
Resource Resource Resource
Definition Definition Definition
Read Geocode Sort
Developing a Commercial Open Source Project with Python Slide 6
OSDC 2007, Brisbane

7. Why open source?
• Practical reasons
• Philosophical reasons
• Business reasons
Developing a Commercial Open Source Project with Python Slide 7
OSDC 2007, Brisbane

8. Business reason: Lower cost
Traditional sales funnel
Potential customers
Interested
Qualified
Opportunity
Sale
Developing a Commercial Open Source Project with Python Slide 8
OSDC 2007, Brisbane

9. Business reason: Lower cost
Open source sales funnel
Potential customers
Interested
Qualified
Opportunity
Sale
Developing a Commercial Open Source Project with Python Slide 9
OSDC 2007, Brisbane

10. Business reason: Lower cost
Traditional FOSS
Sales cost
Development cost
Income
Developing a Commercial Open Source Project with Python Slide 10
OSDC 2007, Brisbane

11. Business reason: Broad adoption
• Broad adoption
– More 'sales'
– More feedback
– Contributions
– Lower development cost
• OSS gets in through the side door
Developing a Commercial Open Source Project with Python Slide 11
OSDC 2007, Brisbane

12. Challenge: Little OSS experience
• The open source community
• What's public, what's private?
• You are being watched!
• Solution:
– Clarify policies
– Embrace openness
Developing a Commercial Open Source Project with Python Slide 12
OSDC 2007, Brisbane

13. Challenge: Licenses
• Which license to choose?
• Modified standard license?
• Solution:
– Get professional advice
– Keep it simple
– Use GPL
Developing a Commercial Open Source Project with Python Slide 13
OSDC 2007, Brisbane

14. Challenge: Source as a feature
• People choose OSS because of source
• Source becomes a feature (!) of the product
• Solution: Make source beautiful
– Documentation
– Consistency
– Style and documentation guide
Developing a Commercial Open Source Project with Python Slide 14
OSDC 2007, Brisbane

15. Challenge: Enabling contributions
• Open APIs more important than open
source!
• Enable 'deeper' contributions
• Solution:
– Don't rest on your laurels
– Use your unique advantage: Deep contributions
– Repeat after me: “APIs, modules, APIs, modules,
APIs, modules, APIs, modules...”
Developing a Commercial Open Source Project with Python Slide 15
OSDC 2007, Brisbane

16. Why Python?
One of the founders likes it
•
• Rapid prototyping and development
• Reasonably fast
• Readable code
Suitable even for large projects
•
Developing a Commercial Open Source Project with Python Slide 16
OSDC 2007, Brisbane

17. Challenge: Little experience
• Can't hire Python developers
• Even in the Silicon Valley
• Solution:
– Just hire good (generic) developers
– They can learn
Developing a Commercial Open Source Project with Python Slide 17
OSDC 2007, Brisbane

18. Challenge: Tainted by past life
• Team experience in other languages
• Not using Python to its full potential
• Solution:
– Understand pythonic
– Read: Python is not Java
– Share knowledge: Code reviews
Developing a Commercial Open Source Project with Python Slide 18
OSDC 2007, Brisbane

19. Challenge: Getting carried away
• Powerful and very (!) dynamic:
>>> class Foo(object):
>>> def hello(self):
>>> print “Hello World!”
>>> x = Foo()
>>> dir(x)
[ .......... , 'hello']
>>> x.bar = 123
>>> dir(x)
[ .......... , 'bar', 'hello']
• Solution:
– Create conventional looking APIs
– Use those APIs as documentation
Developing a Commercial Open Source Project with Python Slide 19
OSDC 2007, Brisbane

20. Challenge: Why didn't you just use...?
• Python comes with Batteries included
• Don't know all the libraries have to offer
• Write yourself? Oh, the temptation!
• Solution:
– Learn to use Google
– Read Python Cookbook and others
– Share knowledge: Code reviews
Developing a Commercial Open Source Project with Python Slide 20
OSDC 2007, Brisbane

21. Challenge: Third-party package risks 1
• Import chain madness
• Some packages rely on C extensions
• Normally: Don't reinvent the wheel
• Solution:
– Watch what you import
– Inventing a little wheel can be helpful
Developing a Commercial Open Source Project with Python Slide 21
OSDC 2007, Brisbane

22. Challenge: Third-party package risks 2
• So far, no bugs in core libraries
• Bugs in third-party packages
• Included patched packages in the installer
• Solution:
– Speed up acceptance time for submitted patches
– Provide patch, test cases, detailed docs
Developing a Commercial Open Source Project with Python Slide 22
OSDC 2007, Brisbane

23. Challenge: The GIL
• Can't use multiple cores with threads
• Solution 1:
– Use multi-processing
• Solution 2:
– mod_wsgi
Developing a Commercial Open Source Project with Python Slide 23
OSDC 2007, Brisbane

24. Challenge: Hidden bugs
• No compiler to catch typos
• Duck typing
• Solution:
– Unit testing: pyunit, buildbot
– Coverage testing: figleaf, buildbot
Developing a Commercial Open Source Project with Python Slide 24
OSDC 2007, Brisbane

25. The end
Any questions?
jbrendel@snaplogic.org
Developing a Commercial Open Source Project with Python Slide 25
OSDC 2007, Brisbane