git lfs at autodesk - meetupfiles.meetup.com/20767392/larsxschneider-gitlfs-at... · git lfs tips...

Git LFS at Autodesk

LARS SCHNEIDER

@kit3bus

@kit3bus

Lars Schneider Autodesk Inc.

Git and Git LFS contributor

Technical Lead forGit at Autodesk

@kit3bus

• Best known for AutoCAD2D and 3D computer-aided design

• 33 years in business

• 4000 engineers, hundreds of products, terabytes of code and asset data

Lars Schneider Autodesk Inc.

@kit3bus

Architecture, Engineering & Construction

Image by Dave Tyner, Autodesk Plant 3D - P&ID

@kit3bus

Manufacturing

@kit3bus

Media & Entertainment

@kit3bus

3D Printing

"Future of Making Things"

Image courtesy of Local Motors Inc.

@kit3bus

Git Git LFS

+

@kit3bus

Why are large files a problem?

All history is local. Good for source files.

Problem for large files.

@kit3bus

Elephant-Video.mp4

Why are large files a problem?

@kit3bus

What files are not suited for Git?

Files that do not compress well and

change frequently....

@kit3bus

Mon Tue Wed

What files are not suited for Git?

Files that do not compress well and

change frequently.

@kit3bus

Example: Git repowith large files

codecode

Git Repo

mastercodecode

301MB

codecode

@kit3bus

Git LFS to the rescue!

git lfs track "*.mp4"

@kit3bus 1 MB

Example: Git LFS repo with large files

master

LFS Server

LFS Ptr

code

Git LFS Repo

LFS Ptr

codecode

codecode LFS Ptr

code

@kit3bus 1 MB

Example: Git LFS repo with large files

master

LFS Server

LFS Ptr

code

LFS Ptr

codecode

codecode LFS Ptr

code

Git LFS Repo

Clo

ne

@kit3bus 1 MB

Example: Git LFS repo with large files

master

LFS Server

LFS Ptr

code

LFS Ptr

codecode

codecode LFS Ptr

code

Git LFS Repo

Clo

ne +

Che

ckou

t

@kit3bus

Technical Details

https://autode.sk/git-lfs-talk

@kit3bus

What do we use Git LFS for?

Integration Test Data(3D Models, ...)

Auxiliary Data(Documentation, Images, Videos, ...)

Build Artifacts(not recommended)

@kit3bus

D E V E L O P E R

M I G R ATO R

A D M I N I S T R ATO R

- Git LFS - What have we learned?

@kit3bus

Migrator

@kit3bus

Migration Process

1 Identify an engineer with deep code knowledge

Create a "demo" migration on Git migration server

Iterate on "demo" migration until repo and CI are OK

Ask broader team to "play" with the "demo" migration

Perform migration on Git production server

2

3

4

5

@kit3bus

How to migrate?

+

git-svn / git-p4 / git-tfs ...

git filter branch / git-lfs-migrate

@kit3bus

How to migrate?

+

git-p4

git-svn / git-p4 / git-tfs ...

git filter branch / git-lfs-migrate

@kit3bus

Git LFSmigration gotchas

Discard large file history

1998 2007 2016

code code code +++

@kit3bus

Git LFSmigration gotchas

master LFS Ptr

code

LFS ServerGit LFS Repo

Avoid "orphaned" LFS files

@kit3bus

Git LFSmigration gotchas

rebasedmaster code

LFS ServerGit LFS Repo

?

Avoid "orphaned" LFS files

@kit3bus

( INCLUDES DESIGNER, TESTER, . . . )

Developer

@kit3bus

Mon Tue Wed

Teach why "Large" files are a problem!

Files that do not compress well and

change frequently.

@kit3bus

Files smaller than 500kb are OK.

Rule of Thumb

Teach why "Large" files are a problem!

@kit3bus

git lfs track "*.png"How to track Git LFS files?

@kit3bus

git lfs track "*.lfs.*"

e.g. /images/elephant.lfs.png

How to track Git LFS files?

@kit3bus

git lfs track "/big/*"

e.g. /big/elephant.png

How to track Git LFS files?

@kit3bus

git lfs track "/xxl.png"How to track Git LFS files?

@kit3bus

Less than 1000 files in LFS are OK.

Rule of Thumb

How to track Git LFS files?

@kit3bus

Less than 1000 files in LFS are OK.

Rule of Thumb

How to track Git LFS files?

https://autode.sk/git-filter

"Git checkout for a test repo with 12k LFS files on Windows takes 57 seconds instead of 55 minutes!"

Git 2.11 + GitLFS 1.5

@kit3bus

git lfs track "*.png"

git lfs track "*.[pP][nN][gG]"

Case sensitive:

Case in-sensitive:

Git LFS Tips & Tricks

@kit3bus

No line ending conversions on

LFS files!Git LFS Tips & Tricks

FirstLineSecondLine

FirstLine▯SecondLineWindowsmacOS

@kit3bus

git lfs clone <URL>Git LFS Tips & Tricks

@kit3bus

Setup your Git credential helper

(or use SSH)!

Git LFS Tips & Tricks

@kit3bus

Use the latestGit / Git LFS

version!

Git LFS Tips & Tricks

@kit3bus

Administrator

@kit3bus

How to make sure Git LFS is used properly?

Setup Git LFS on all dev machines!

Enterprise Config for Githttps://autode.sk/git

LFS Ptr

@kit3bus

How to make sure Git LFS is used properly?

"What happens in Git, stays in Git."

@kit3bus

How to make sure Git LFS is used properly?

Rewriting history can cause a lot of

trouble!

@kit3bus

How to make sure Git LFS is used properly?

Configure file size limit on Git server!

@kit3bus

How to make sure Git LFS is used properly?

Configure file size limit with localGit pre-commit

hooks!

@kit3bus

How to make sure Git LFS is used properly?

Initially, perform Git LFS usage reviews.

@kit3bus

Takeaways

• Git LFS works at scale

• Use the latest Git/Git LFS versions

• Automate Git LFS setuphttps://autode.sk/git

• Use fast clone commandgit lfs clone <URL>

• Use smart tracking patternsgit lfs track "*.lfs.*"

• Reject large files in Git