How To Make Git Scale Effectively 2. 2© Perforce Software Inc. All Rights Reserved.
What We’re Talking About
1
Git Overview
2
Perforce Technology
3
Solution Mapping
3. 3© Perforce Software Inc. All Rights Reserved.
• Popular
• Local operations can be fast
• Lots of publicly clone-able code
• Distributed
• Great for small projects that
are mostly code files
What’s So Great About Git?
4. 4© Perforce Software Inc. All Rights Reserved.
• Large binaries are problematic
• Repo sprawl
• Mixed-team composition
• Build + CI
• LFS
• Dependencies
• Distance
Git at Large Scale
6. 6© Perforce Software Inc. All Rights Reserved.
• New depot type (graph) for storing
tree-based git commits, objects, etc.
• Simple grant-based permission model for
single repos and collections
• Supports many “classic” style p4
commands: sync, edit, submit, merge, etc.
• Includes git-lfs support
• Git users connect to Helix Core Server
(P4D) via Git Connector
• Depot/repo/project management and code
review through Helix TeamHub
What Is Helix4Git?
7. 7© Perforce Software Inc. All Rights Reserved.
• Supports SSH/HTTP(S) authentication
• Authorizations are all handled by central
Helix Core Server (P4D)
• Translates between user’s git pull/push
to depot
• Keeps local cache of native git repo
• Git developers don’t even know it’s there
What Is Git Connector?
8. 8© Perforce Software Inc. All Rights Reserved.
• Git Connector and Helix4Git
have built-in mirroring
• Mirrored Helix4Git repos are
protected from normal commits
• Easily integrates with
Gerrit workflow
• Includes GitHub/GitLab
mirroring capabilities
Mirroring Repos and CI
9. 9© Perforce Software Inc. All Rights Reserved.
• Directly supported in the
Helix4Git repo
• Same local git-lfs configuration
• Stored in archive files on the
Helix Core Server (P4D)
• Fully accessible by p4 clients
(sync and edit)
Git LFS
10. 10© Perforce Software Inc. All Rights Reserved.
• Clients – a view into your
repo data
• Helix Core Server’s clients
are a lot like git shallow
clones
• Client views can span:
• Depots
• Repos
• Depot types (classic,
streams, and graph)
Graph Clients
13. 13© Perforce Software Inc. All Rights Reserved.
CI: 40-80% Faster Builds; 18% Less Storage
All tests performed with shallow clone of Linux kernel on a 1 Gbps link, and four
parallel threads of p4 sync. For WAN test, a 200 ms round-trip latency was added
between client and server.
Sync Linux kernel head
revision over WAN
Sync Linux kernel head
revision over LAN
Resulting sync sizes
0
10
20
30
40
50
Git Clone P4 Sync
796
MB
650
MB
43 (s)
13 (s)
47 (s)
25 (s)
14. 14© Perforce Software Inc. All Rights Reserved.
• Faster build sync times
with parallel sync enabled
• Dedicated build replicas to
reduce master overhead
• Can pull all project
components in single client
• Integration with GitLab,
GitHub, and Gerrit
Build + CI
15. 15© Perforce Software Inc. All Rights Reserved.
• A single Helix4Git depot can
contain all repos for a single
project
• Centralized management of
repos and authorization
• Centralized management of
externally sourced project
components
• Helix TeamHub for
management of projects
Repo Sprawl
16. 16© Perforce Software Inc. All Rights Reserved.
• Keep all assets for your mobile/web
app in a single project in TeamHub
Multi-Repo Example — Helix TeamHub
Open source app
framework code
My mobile app
source code
Graphics for my UI
Dependencies for
Android platform
Project docs
17. 17© Perforce Software Inc. All Rights Reserved.
• Code developers just want to pull
and develop
• Use git local and pull/push to
Helix4Git repo
• Digital asset editors don’t need
to build
• They need to find assets and edit them,
but have it pull-able through LFS
• Build team pulls it all together
• Multiple repos, some external,
controllable inputs to the build system
Mixed-Team Composition
18. 18© Perforce Software Inc. All Rights Reserved.
• Any significant project using git
for VCS is using multiple repos
• Client/server
• Frontend/backend
• Microservices
• Libraries and versions
• Philosophy of component
dev = be backwards/forwards
compatible
Project Dependencies
21. 21© Perforce Software Inc. All Rights Reserved.
• Helix TeamHub has the only
multi-repo code review in market
• Synchronize atomic commits
across multiple repos
• Keep frontend and backend
changes aligned
• Stop intermittent build failures
• Pre-flight changes to production
Multi-Repo Merge
22. 22© Perforce Software Inc. All Rights Reserved.
• p4 sync is your
best friend
• Custom clients can map
and re-map how you want
• Allows commits that are
fully git-compatible
• Git developers keep using
the tool that makes them
efficient
Large Projects
23. 23© Perforce Software Inc. All Rights Reserved.
• Co-locate Git Connectors
with your teams
• Set up pull-polling to reduce
developer hang time
• Multiple Git Connectors to
spread the load
• Build replica servers to
decrease master load
Geography
24. 24© Perforce Software Inc. All Rights Reserved.
Recap
Problem Solution
Large binaries are problematic LFS/p4 sync
Repo sprawl Centralized commit server + mirroring
Mixed-team composition Git+ LFS + p4 sync
Build Parallel sync, replicas
CI Mirroring
LFS Out-of-the-box support + p4 sync
Dependencies Multi-repo code review + merge
Distance
Inherent Git Connector replication,
federated servers