smart repository management (openblend 2011, ljubljana, slovenia)

Transforming Software Development

Smart Repository ManagementWith Nexus

MMXI. Ljubljana, Slovenia

Thursday, September 22, 2011


Smart Repository Management

The Inception2005




The Inception

Bandwidth waste and congestion

Depending on network and external repository up-times

Unsolved “hosting” and collaboration

Exposed to mismanaged repositories




The Inception

Laborious discovery of artifacts, plugins

Infrastructure maintenance hell (deploying, site publishing)

Non-trivial entry for new developers

Key-based security – who entered the office had full access




Proxy Repositories

Pluggable “remote” transport, caching locally

Isolation from outer world

Crossing two worlds: mirrors (content) and proxies (protocol)

Get what you need, when you need it




Hosted Repositories

Pluggable “local” storage

Simplify the needed infrastructure

Get all content treated same way

Keep your artifacts properly separated




Grouping

“Aggregating” multiple repositories into one

Hiding the actual source and URL(s) of Repositories

Order of processing in a group is important

Became de facto “standard” setup




Typical Usage

Nexus

Apache

Codehaus

JBoss

Terracotta

Dev

Test

QAProduction

Open Source Ghetto

Precious

DevelopersYour Ghetto




Grouping Optimizations

Is corollary of Grouping

Based on repository policy

Based on repository routes

Parallel processing

Or ultimate: stop using grouping! (today not quite possible)




Bridging and Mediation

Do it transparently

First application was to support Maven1 repositories

Layout changes (Maven1 – Maven2 bidi)

Content changes (GEM, OSGi recipes)




Indexing

Pluggable Index Creators (POM, Classes, OSGi, ...)

Incremental updating and publishing

De facto standard, embraced by all IDEs, MRMs...

Source donated to Apache Maven Project as Maven Indexer




Security

Integrated Apache Shiro

Supporting XML, LDAP, Atlassian Crowd, URL delegation...

Content secured using Repository Targets

Pluggable Realms




Staging and Promotion

Dogma of Maven is immutable releases, no recall once deployed

To not release incremental versions, or use temporary locations

Single deployment URL, just set it in organization POM

Notification and Automation




Artifact Procurement

Before was laborious, error prone and difficult

Firewall with user extensible rules

Strategies: “Ask First”, “Check Later”

Easy integration




ConclusionsWhat We Learned




Maven is usually wrongly recognized as a “CLI build tool”, but is much more than that, it’s

infrastructure.

Maven Fact




Maven Artifact Repository

Static Sites Repository

Eclipse P2 Bundle Repository

Apache Felix OSGi Bundle Repository

Ruby GEM Repository

OSGi Recipes

Anything we missed?

Refreshing Diversity – Unified!




Why do you need it?Hosts what your build outputs

Proxies what your build consumes

Bridges and mediates between different kind of repositories

Has motto “no content duplication” encoded in it’s DNA

Eases “every day routine”

Gives tools to implement complex workflows

Easy integration using REST API




What to proxy (outbound)?

Do not proxy “aggregated” repositories (information loss)

Do not proxy “staging” repositories (releases disappears)

Do not proxy snapshot repositories, unless a must




How to consume (inbound) – today

Groups in Nexus are cheap

Groups of groups

Per project groups

Groups per developers, CI and public




How to consume (inbound) – futureMirror of all – The typical setup since Proximity, has it’s known problems, information loss

Mirror per Repository – The most precise control, but not maintainable

Dynamic mirror discovery – Intelligent mirror selection and proper fallback, magic on both ends, no information loss

HTTP Proxy – Simple maintenance, still needs user side configuration, but magic happens on Nexus side, no information loss




Repositories in POM, yes or no?

No doubt:

yes




In-house Snapshots

SCM CI Maven NxMirko

Radirko

LocalRepo

Maven




Third Party Snapshots

“Freeze” them at least,

but better is to build from source




Central RepositoryFacts and Stats




Tamas CservenakSonatype Nexus Lead Developer

http://twitter.com/cstamas

Thanks,~t~

Questions?


smart repository management (openblend 2011, ljubljana, slovenia)

Technology