better living through new releases - perforce · •1 better living through new releases by richard...
TRANSCRIPT
• 1
Better Living Through New Releases
by Richard E. Baum
Overview
• Server performance-related functionality • Upgrades contain much more than bug fixes • This is a high level view • Close to chronological order
• By topic where appropriate • Not an exhaustive list • Release notes are your friend
• 2
Automatic labels (2006.2)
• Create labels based on changelist and path
• No ‘p4 labelsync’ needed • Label ‘Revision:’ field applied to view lines
Automatic labels (2006.2)
• Changelist alias:
• Files in a single changelist:
Label: nightly_build_01-Apr-2009 Owner: buildmeister Descr: Nightly build process View: //depot/... Revision: @4567
Label: patch-78901 Owner: mr-patchit Descr: Patch for bug 8471 View: //depot/... Revision: @7890,7890
• 3
p4 license (2006.2)
• Add licensed users without server restart • From command line • From Perforce Administration GUI
• IP change requires a restart
+Sn filetypes (2007.2)
• Only store most recent n revisions • Older revisions purged at submit • Older metadata remains intact • Values for n:
• Improves handling of large binary objects
1-10, 16, 32, 64, 128, 256, 512
• 4
Journal file buffering (2006.2)
• Journal buffer flushed periodically • No longer after every transaction
• Reduces writes • Reduces locks taken • Improves overall performance
Faster journal / checkpoint rotation (2007.3)
• Rotation can now be a rename • UNIX / Linux
• Rotation can be instant on UNIX / Linux
• Server copies and truncates if • -z flag used for compression • Rename fails • Non UNIX / Linux server used
p4d -jc or p4d -jj
• 5
Checkpoint / Journal locking (2007.3)
• Only read locks taken during:
• Most read operations can continue p4d –jc p4d –jj
TCP/IP buffering (2008.1)
• Auto-discover / adjust buffering levels • Make better use of buffering space • Improved network performance
• Particularly over high latency connections
• 6
Proxy caches on submit (2007.3)
• Files immediately ready for sync via proxy • No additional sync required
• MD5 hash values used for validation • Timestamps used only as a fallback
Improved Proxy cache load (2007.3)
• Update cache without sending files to client:
• Intended to be used with dedicated client • Have list assists with file selection • Client workspace remains empty
p4 –Zproxyload sync
• 7
Sensible proxy compression (2008.1)
• When syncing through Perforce Proxy • Eliminates proxy decompress/compress • Less overall work
• Saves CPU on proxy • Speeds up overall processing
• Start server/proxy compression earlier
Obliterate improvements (2008.1)
• We now compare: • Number of lazy copies • Number of revisions purged
• If equal, purge the archive file! • Fewer scans of database • Mapping optimizations
• 8
Locking – (various releases)
• Overall huge improvements • Incremental changes every release • Targeted rewrites for better concurrency • New locking algorithm (2008.1)
MaxLockTime (2006.2)
• p4 group field • Max time an operation can lock a db file • Server request fails when limit is reached • Group spec:
• User output:
• Log output:
Group: all_users MaxLockTime: 10000 Users: jerry
Operation took too long (over 10.00 seconds); see 'p4 help maxlocktime'.
--- killed by MaxLockTime
• 9
Group ‘unlimited’ values (2008.1)
• p4 group limit maximums • MaxScanRows, MaxResults, MaxLockTime, Timeout
• Allows use of finer granularity • These both set no limit:
• ‘unlimited’– other group settings don’t apply • ‘unset’ – other group settings apply
• Timeout of ‘0’ no longer allowed (use ‘unset’)
• Old behavior maintained on upgrade • ‘unlimited’ becomes ‘unset’
New locking methodology (2008.1)
• Old, blocking methodology: • Take locks as needed • If needed lock is unavailable, wait until it is • Perform operation • Release locks
• Locks held during wait • Blocks other processes
• 10
Old methodology locking log fragment: 'user-client -d build_client' -- db.domain -- locks wait+held read/write 0ms+0ms/0ms+252838ms -- db.view -- locks wait+held read/write 0ms+1ms/0ms+252838ms -- db.have -- locks wait+held read/write 115ms+10ms/3942ms+252837ms -- db.label -- locks wait+held read/write 0ms+0ms/0ms+252836ms -- db.resolve -- locks wait+held read/write 0ms+0ms/0ms+252836ms -- db.locks -- locks wait+held read/write 0ms+0ms/0ms+252835ms -- db.working -- locks wait+held read/write 0ms+0ms/0ms+252835ms -- db.change -- locks wait+held read/write 0ms+0ms/252831ms+4ms
• 7 tables blocked 4+ minutes… • …for 4ms of access to blocking table
New locking methodology (2008.1)
• New methodology: • Locks requested until a request blocks • Release acquired locks • Attempt to acquire blocking lock • When granted, drop and start again • After third try, use old behavior
• Only one lock held while waiting • Does not block additional processes
• 11
Batched updates (various)
• Many database updates now batched • Rather than updating revisions as changed • Changes buffered to reduce I/O, locking • Table update only every 100 records
• More commands added each release (sync, resolve, edit, revert, integ, submit, etc.)
Server tunables (2008.2)
• Don’t try this at home! • Configure server limits/values
• Previously hard-coded values
• See documentation for more details: • p4 help tunables • p4 admin tunables
db.isalive 10K Rows scanned before maxLockTime check db.trylock 3 Attempts to avoid locks that block dm.flushtry 100 update buffer for sync, resolve, edit
• 12
Enhanced ‘p4d -xv’ (2008.2)
• Newly built db files include data checksums • For db files built with 2008.2 and later server • Reported by ‘p4 admin dbstat’
• Now, ‘p4d -xv’ checks the checksum • Previously only table structure was checked • ‘p4d -xr’ sets new checksums during repair
• Only repair with assistance from support!
Enhanced ‘p4d -xv’ (2008.2)
• Example: %p4 admin dbstat db.rev db.rev internal+leaf 0+1 page size 8k end page 1 generation 5 levels 1 fanout 0 Checksum 325666531
%p4d -xv Validating db.counters . . .
Validating db.rev **** Checksum mismatch **** possible data corruption
• 13
Summary
• Performance-related changes every release
• Release notes detail many modern marvels!
• Modernize your depot – upgrade! • Need help? Call!
• Perforce Technical Support • Perforce Consulting Services
Q & A
• 14
The End