This document discusses configuring HTCondor to flock jobs between two HTCondor pools running on different cloud platforms. It describes installing HTCondor on a Windows Server 2008 machine on Rackspace as pool 1 and another on Amazon EC2 as pool 2. It then explains modifying the condor_config file on both pools to specify the other pool's Central Manager IP address in the FLOCK_FROM and FLOCK_TO statements, allowing jobs to flock between the two pools. Testing shows the flocking works as each pool can see and submit jobs to the other.

1. Flocking HTCondor
Between
Two Clouds
Guy Tel-Zur
tel-zur@computer.org
March 2013

2. Goal
• Submit HTCondor jobs from Amazon EC2
HTCondor pool and flock to another HTCondor
pool on Rackspace.
• Both pools use Windows Server 2008

4. Outline
• Install Windows Server 2008 with HTCondor
on Rackspace (Pool 1)
• Install Windows Serve 2008 with HTCondor on
EC2 (Pool 2)
• Modify the condor_config file
FLOCK_FROM and FLOCK_TO specifying the IP
addresses of the Central Managers.
• Verify the Flocking works

6. Open terminal
from here

7. The server is up

8. In the next slides we show HTCondor
Installation steps on the first node

17. Verify that condor service is running

18. condor_status works

19. Before first submission
store your credentials

20. A simple test
program

21. Now we repeat the above steps and
install a HTCondor node on AWS EC2

26. The new machine is ready to be
launched

27. EC2 dashboard

28. Connecting with remote
desktop

30. The test program works also on EC2

31. Add Flocking
• Edit condor_config
• At both ends:
– Set the IP addresses of the condor master nodes
in the flock_from and flock_to statements
– Save and restart condor
• That’s it !!!

32. Check Flocking at
both ends
Each pool “sees”
the other pool
NOTE! In this demo for simplicity firewalls were disabled. This is not recommended in
real production runs!

33. Each pool can see the other pool

35. Reference
• HTCondor:
http://research.cs.wisc.edu/htcondor/
• Rackspace: http://www.rackspace.com/
• AWS: http://aws.amazon.com/