1.
MERGE 2013 THE PERFORCE CONFERENCE SAN FRANCISCO • APRIL 24−26
Perforce Administrative
Self- Services at SAP
Wolfram Kramer, SAP AG
Perforce White Paper
SAP changed the processes for user and group
administration and for project management step by
step from a purely centralistic approach to a much
leaner one.

2. 2 Perforce Administrative Self-Services at SAP
Background
SAP AG has used Perforce as its version control system for almost 15 years. The use of
Perforce has grown over time; currently, SAP hosts one of the largest Perforce server
installations in the world in terms of the number of licensed users: 10,000. In terms of the
number of productively used Perforce server instances, it is the largest one, with approximately
100. Many of the developers accessing Perforce use it to work with more than one instance.
Perforce implements a per-server user and group management for which instance-specific
protection tables are defined. It is evident that on the one hand, a synchronization of users,
groups, and also protection tables is necessary to ensure that any user gets access to all the
servers where he or she expects it. On the other hand, it is also clear that such a
synchronization of user and group data cannot be done manually on all the servers.
For this reason, early in the 21st century SAP implemented a central user and group storage
based on a relational database. The user and group data for each Perforce server instance are
synchronized from there. Manual user and group administration directly on a Perforce instance
(via p4 user or p4 group) was not done any more. The leading paradigm for the
implementation of the central user and group storage was a centralistic approach of user
management and any user and group management activities were done by a central team.
Tightly coupled with the scheme of permission groups and protections is the centralized
governed depot structure on any Perforce server at SAP. Versioned files are organized in so-
called projects as folders on the second level on the Perforce server directly below the depots.
The creation of projects and their code lines (organized in the third level below) is strictly
standardized and can only be done through a central infrastructure. Perforce triggers prohibit
adding new files outside an already existing project and/or code line. The design of the project
and code line creation process followed the paradigm of central administration by a central
team.
Although central teams are good at providing services in a workflow-like manner for
implementing and empowerment of governance, they are a hindrance in the context of lean
and agile principles because they are bottlenecks for the flow. They produce waiting time for
the requestors (project managers or developers) because the workflow still needs human
interaction by the central team. This white paper describes how SAP changed the processes
for user and group administration and for project management step by step from a purely
centralistic approach to a much leaner one with—in the ideal case—no more waiting time for
consumers.
Perforce Server Landscape and Centralized User Storage
The Perforce server landscape at SAP consists of:
• Approximately 100 Perforce server instances
• A relational database named “P4SAP” containing central metadata such as user, group,
and project data
• The P4 Management System P4MS as a framework of tools and services for data
manipulation on the P4SAP database

3. 3 Perforce
Administrative
Self-­‐Services
at
SAP
• Tools and services for data synchronization from the P4SAP database to Perforce
servers, also integrated into P4MS
• Self-services and web services around P4MS to enable user interaction
Figure 1 illustrates the architecture:
Figure 1: Perforce server and user storage at SAP
Users and groups as parts of the metadata are stored and maintained on the P4SAP database
via the P4 Management System (P4MS). Each group that is stored in the database has a
specific validity scope of Perforce servers to which it is synchronized. This is because the
sources of certain larger products are distributed over more than one Perforce server. The
definitions of groups affecting accesses to the code base therefore have to be the same on
each Perforce server. This is ensured by the synchronization of the group from the P4SAP
database to all Perforce servers within the validity scope of the group.
Synchronization of user data to Perforce servers happens together with group synchronization.
The protection scheme at SAP assumes that any user gets permission through a group in
which the user is a member. As soon as the first group that contains the user as a member is
synchronized to a dedicated server, the user data is synchronized as well. This
synchronization ensures that any member of a group that exists on one Perforce server is also
a valid user defined on the Perforce server. From a technical point of view, P4MS uses p4 user
–i to create a user on the relevant Perforce servers.
Permission groups control the access to files within Perforce depots. At SAP, a standard
Perforce depot is structured along projects. A project is a location in the Perforce depot tree
that hosts the code for a subsystem of an SAP application. Because subsystems can have
their own release cycles, the code line level is below the project level, which ends up in a
Perforce depot structure (see Error! Reference source not found.).
Ruby
Dele

4. 4 Perforce Administrative Self-Services at SAP
Perforce creates folders as pure representations of the full path of versioned files, appearing
whenever a file having a new path is submitted. To prohibit wild creation of folder structures at
SAP’s Perforce servers, pre-submit triggers prevent the creation of new project and code line
folders. Creation of new projects is therefore also a task for the central Perforce team that
operates on request by the development teams. It has permission to circumvent the pre-submit
trigger for the creation of a first, empty file in the path of the new project. The purpose of this
first “createdir.txt” file is to make the path visible in the Perforce navigation tree. By this
operation, the pre-submit trigger is deactivated for future submits on the project path.
Currently, there are more than 9,000 projects; each has a lot of code lines. New code lines of a
project are usually needed for release branching. A service offered in the P4MS framework
automatically creates the needed code line as a folder in Perforce. Nevertheless, the creation
of new projects is still a manual, central activity.
Figure 2: At SAP, a standard Perforce depot is structured along projects
Design of Perforce Access Groups and Permissions
In the past, development teams followed a lot of different processes that slightly differed in
requirements concerning code line patterns, their naming, and their permission groups. But
one pattern applied to almost all variants contained two basic groups:
• A dev group that has permissions for writing to the dev code line where all new changes
are created; developers are typically members of this group.
• A patcher group that has permissions for writing on the qual code line that contains the
qualified, stable code; permissions to this code line usually have build operators and
quality managers.

5. 5 Perforce
Administrative
Self-­‐Services
at
SAP
//product_X
project_A
dev
qual
dev
group
project_A-­‐dev
patcher
group
project_A-­‐patcher
write
read
write
write
Figure 3: Permissions are simplified for project admins at SAP
Although it is possible to apply this pattern on a per-project level, in most cases, this approach
is too finely granular. In general, a product consists of several projects that are contained in
one depot. The same people are working on any of the projects within the depot. Therefore,
the group pattern is often applied on a depot level: instead of groups like project_A-dev and
project B-dev, product_X-dev is used. This simplification means the project admin has to
maintain dev and patcher groups only once and not for each project separately (see Figure 3.)
Also the central Perforce team that creates new projects does not have to create new groups
or make new entries into the protection table. The latter are covered by entries using wildcards
such as:
write group product_X-dev * //product_X/*/dev/...
read group product_X-dev * //product_X/*/qual/...
write group product_X-patcher * //product_X/*/dev/...
write group product_X-patcher * //product_X/*/qual/...
Centralized User Management
Shortly after the introduction of Perforce in 2001, SAP began to distribute projects to more than
one Perforce server instance. After first developing intermediate solutions using script-based
copy of users from one server to another one, the central user and group storage described
above was introduced (see Figure 4). After that, it was possible for the central Perforce
operating team to use a web front end to maintain users and groups in the central database.
Groups were assigned to one or more Perforce instances. Publishing users and groups to the
Perforce server was done instantaneously by pressing the save button.
Ruby
Dele

6. 6 Perforce Administrative Self-Services at SAP
Figure 4: A web front end is used to maintain users and groups in the central database
It was a great step forward because per-instance maintenance of groups was no longer
needed. But still all requirements from developer teams were handled by processes based on
workflows. Typical scenarios handled by the central operating team were:
• Adding a user to a permission group to grant read or write access to a project; keep in
mind that the admin groups already mentioned did not yet exist.
• Creation of a new project including its permission groups.
There were of course tools that made the creation of a project as comfortable as possible for
the operation team; creation of the project and the groups was possible in one step. But the
strict centralization of any activity concerning user and project management turned out to be a
bottleneck: it was not scalable with the increasing number of projects. Instead, it created
waiting time for the developer teams and it ate up the time of the central Perforce operating
team, preventing it from investing in improvements to the infrastructure.
Group Member Self-Administration
The consequence was the insight that user management had to be decentralized. In fact, the
decision whether permissions on a project shall be granted to a specific user was not done by
the central team but by the responsible development manager of the project. This was when
the admin group for a project was introduced (see Figure 5).
The admin group has the purpose to modify the dev group and patcher group for a project and
for itself. It exists only inside P4MS and is in general not synchronized to Perforce servers at
all. Project managers are usually members of this group since they decide who should join as
developer or build operator of the project.

7. 7 Perforce
Administrative
Self-­‐Services
at
SAP
//product_X
project_A
dev
qual
dev
group
project_A-­‐dev
patcher
group
project_A-­‐patcher
admin
group
project_a-­‐admin
change
change
changewrite
read
write
write
Figure 5 An admin group was established to decentralize user management
P4MS has an own authorization concept that is able to assign P4MS-application specific
permissions to groups. Members of the admin groups of a project can modify the dev and
patcher groups in P4MS and thus in the P4SAP database. The synchronization mechanism
described above does the job to make these groups productive on the dedicated Perforce
servers.
By what has been described, the use case “I as a project manager would like to define the
group members who have access to my project” is covered. For the use case “I as a developer
would like to get access for a dedicated project,” the mechanism is not sufficient. The
assumption is unfortunately wrong that the developer could just write an email to his or her
project manager to add him or her into the project dev group. Often the developer does not
know who the project manager is.
Figure 6: Any employee is able to request access to a project in Perforce at SAP
To cover the second use case, P4MS offers a service by which any employee is able to
request access to a project in Perforce. This can be done in two ways: either the user knows
the name of the access group of the project he/she wants to be join or—and this happens in

8. 8 Perforce Administrative Self-Services at SAP
most cases because most people are not familiar with the group names—the user provides the
Perforce path of the project he or she want to get access to. In the latter case, the system finds
out by itself which project the path points to and what the name of the dev group is (see Figure
6).
Then the system determines the admin group that is able to maintain the dev group, and an
email is sent to members of the admin group. Via the link that is included in this email, any of
the addressed administrators is able to open the maintenance dialog for the dev group the
requesting user wants to join. After having pressed the button for approval, the user is added
to the group within the P4SAP database. The synchronization of the group to the affected
Perforce servers is scheduled and takes place within a few minutes asynchronously.
What makes this dialog so powerful is that a new Perforce user is created automatically on any
affected Perforce server where the user did not exist before. In this case, a strong random
password is created, and the user receives a generated email to set a password of his or her
choice. Hence the outlined workflow applies to already existing Perforce users who get access
to an additional project as well as for people who did not have a Perforce user before.
At this point, it is worthwhile to outline the procedure of setting passwords on SAP’s Perforce
servers. As already mentioned, many of the users have to connect to more than one Perforce
server. Letting them maintain and synchronize their passwords by themselves by applying the
p4 passwd command on each server would create much frustration. Therefore, the password
setting and distribution occur via a central service in P4MS that sets the password
automatically on each Perforce server on which the user is synchronized. For security reasons,
the passwords are not stored (not even encrypted) in the P4SAP database. Hence whenever
users get access to an additional server, they have to go through this service once again (but
with the advantage that, afterward, the password is the same on all Perforce servers).
Limited Local Perforce Project Control
Group administration has been successfully decentralized, but the creation of new projects and
code lines until recently was completely a central responsibility. But agility for the development
teams also means being able to create very quickly the prerequisites to start a new prototype
without any unnecessary loss of time. A project on a Perforce server belongs to the most basic
infrastructural pieces that enable developers to do their work. A central workflow to create a
project with all its waiting time and involvement of central people that does not directly add
value to the product does not fit any more. Therefore, SAP introduced a self-service for project
creation in a dedicated area where speed is crucial: projects hosting code for mobile
applications.
Ruby
Dele

9. 9 Perforce
Administrative
Self-­‐Services
at
SAP
Figure 7: Self-service project creation at SAP
The solution enables project managers to create a project by themselves without having to
request project creation from a central team. The self-service asks the project manager for a
project name; provided that the name is not yet in use, the project is created (i.e., the
corresponding path on the Perforce server is created) (see Figure 7).
So far, this self-service is only able to create the project in one single depot on one single
server hosting all SAP mobile applications. Also the permission concept is so far rather
pragmatically solved: for all the projects within this depot, there is only one dev group, one
patcher group, and one admin group. As long as no special, more granular permissions on a
per-project level are needed, this is sufficient. Otherwise, in rare cases, interaction by the
central operating team is still required.
The Future: Full Local Perforce Project Control
For the future, SAP plans to extend this offering to any kind of projects in any depot and on
any Perforce server such that any project manager at SAP could get rid of lengthy workflows
for project creation.
But this will not be possible without solving a structural problem: the right choice of the hosting
depot and Perforce instance for a completely new project still needs some central governance.
Often the creation of a completely new depot is required; in the future, this will be a task of
central administration. But luckily, that only happens when completely new products are
invented. The much more frequent use case for a project manager is: “I need a new project in
the same depot where all my other projects are already located.” Specifying an already
existing project in a dialog or choosing it from a drop-down box is then an easy task for the
project manager and a feasible approach of a generic self-service for project creation.

10. 1
0
Perforce Administrative Self-Services at SAP
Another challenge comes with the creation of protections on the Perforce server.
Unfortunately, there is no common pattern for code lines and groups that have read or write
access to them that fits for all development teams. Much worse, development teams need the
ability to create new code lines (e.g., for prototyping that does not follow any name pattern). To
avoid involving central manual interaction again, the protections on the Perforce servers have
to be adapted as well: the self-service must be able to add lines into the protection table to
grant permissions on a code line defined by the project administrator.