efs overview

EFS Overview

EFS in a nutshellEFS stands for “Enterprise File System”

EFS is: A special-purpose Unix global file system

A collaborative mechanism for developers to quickly and easily share and deploy software

A system for tracking releases of software (whether internally-produced, open-source, or third-party vendor)

Time Resources Budget

Goal: Overall Reduction in…

EFS BenefitsLess time spent deploying software to internal clients

If the software is in EFS, any client attached to EFS will have access to all core applications available

Less time administering individual boxes

Development/Deployment Teams have full responsibility for deployment; no coordination between teams which can lead to roadblocks, misunderstandings and additional time to address these issues

Recovery is quick and relatively painless:

Roll-back of software is done in near-seconds instead of hours or days

Because of the centralized repository of EFS, no re-installing software and libraries on different servers to run software

How can I Use EFS?

It could mean several things:• Developers and users making use of the

software already deployed in EFS to help them do their jobs efficiently, without the need for software-request tickets

• Using EFS to make releases of software developed internally available to regions worldwide

• Using EFS to manage the release management and deployment of software run within the firm’s sites worldwide

Access Development Deployment

What does it mean to “use” EFS?

We refer to these three uses, respectively, as:

An EFS Architecture Overview

Cell

A NAS filer containing the EFS data is called an EFS “Cell”A cell consists of every part of EFS’s infrastructure in a single location:

Storage EFS management servers Caches

CellSet

The collection of cells within a region is referred to as a “Cell Set” or “Regional Cell”

When any UNIX server is connected to EFS, they access the “EFS namespace.”EFS

namespace

DEV The DEV environment supports /efs/dev (read-write)

and /efs/dist (read-only, simulating PROD, used for testing, debugging, QA and UAT)

PROD (read-only) The PROD environment contains a “locally frozen” form of /efs/dist only

The /efs/dist data in both DEV and PROD is replicated worldwide to all Cell SetsThis replication is on-demand; commands are executed to cause freezing and replication

There are two environments in EFS:

EFS Internal structure

• Namespace begins with /efs• DEV environments hosts 2 areas:

– /efs/dev• Read/write space to input binaries and test configuration/installation. This is the

developer’s “sandbox” environment– /efs/dist

• Testing area, from initial testing to final testing before moving to production• Read only, no writes are permitted or possible on this mount• “Simulates” PROD environment• Think of this as a QA space to prepare for PROD

• PROD environment hosts 1 area– /efs/dist

• This area is always a subset of the information available in /efs/dist in the DEV cells– Releases in DEV are being tested and may never be disted to PROD– DEV has development tools deemed inappropriate for distribution into the PROD

environment– PROD environment is on an entirely different filer, even though mounts are

named the same on both DEV and PROD cells

EFS commands keep the environment very structured

MPRs

Form the top-level content-oriented hierarchy in EFS

A development group’s own area and must be defined by the EFS group

Permissions, user lists, and access rights are specified at this level

A specific software product consisting of files that are built together

An application, library or development project

Has a “home cell” which is the geographic location where most writes originate from

A specific version of a project

The location where the actual files reside

Releases are versioned and ordinarily immutable (“frozen”) once replicated (“disted”)

Under both /efs/dist and /efs/dev, the next three levels of the file system hierarchy is called the (MPR) Metaproject Project Release

Metaproject: Release:Project:

Indirection for stability and ease of use

• Developers compile for specific chip architectures

– For languages like C and C++, binaries must be compiled for the specific chip set (i.e., Linux x86_64, Solaris SPARC)

– Teams working from that OS compile the code and place binaries in specific EFS location

– For languages like Java which do not require compilation per chip set, binaries are placed in a single location

– Same process is done for bitness (e.g., 32 bit, 64 bit)

• To run binaries users go to the same location on any platform

– EFS indirection determines what chip set and bitness and accesses the appropriate binaries from paths resolved at mount-time.

• Use “ReleaseAiliases” to access binaries lets dev team control access

• ReleaseAliases are just symbolic links to specific releases

– /efs/dist/foo/bar/1.2.3 may point to /efs/dist/foo/bar/1.2.3-build001,/efs/dist/foo/bar/1.2.3-build097, etc.

– By changing the releasealias, your users get the release you want them to see

– Patches to releases can be done by creating a new release (e.g.1.2.3-build002, 1.2.3-build003, etc.) and point the releaselink (/efs/dist/foo/bar/1.2.3) to the specific release

– Users access new release transparently next time they start application

– No user reconfiguration or application “re-tooling” is required

Addressing multiple chipsets Using Release Aliases

EFS Use Case

• If global, you may have regional abnormalities; Standards for deployment in one region may be totally different in another

• Are all patches in place? For each operating system?

• Are they the same ones you tested with?

• Do you need to do a different build for every Region? Every Platform? Every machine?

• Libraries may or may not be available. If there is a question, do you include every single library in every installation?Isn’t that really inefficient?

Challenges: Distributing software without EFS

#1 Challenge: Many moving parts, little control

Document installing custom software, include any custom settings, based on your DEV environment or

Create tarball or other compressed package

Find QA and UAT machines (hardware)

Set up the machine (all software, all libraries, all language-specific items)

Check to be sure all the libraries are installed in the correct location with the correct parameters

If not, get these re-installed; change the documentation to reflect the change

Have SAs install your software, based on your instructions

Is this the right version? Are there undocumented changes?

When completed, do initial testing to be sure everything is in place. If not, contact SAs to schedule time for a reinstall

Document the changes, re-create tarball or other compressed package

When errors occur: Is it your application or the environment? Go back and test the environment to be sure you are not missing anything.

Repeat for next machine

EFS Use CaseChallenges: Distributing software without EFS (continued)

Getting Started Checking and Re-checking

Getting Started Easy deployment to PROD

Dist the software to the /efs/dist areain the DEV environment

It is now available for QA worldwide

11010101011010101000101000011101010

When errors occur: Fewer places to debug, since the DEV and PROD environments mirror each other

Dist to the PROD environment

It is now available to whatever worldwide cells you need it.

Change the stage of the release

It is now ready to be deployed, when you pull the switch

• Since EFS takes care of the distribution and all the clients are connected to EFS, running one command distributes your binaries throughout the enterprise

• EFS DEV environment has a stage that mirrors PROD (/efs/dist), so testing is done in an environment that simulates your target environment

• All libraries are in exactly the same place in DEV and PROD, so missing libraries is one less thing to check

HOW?

EFS Use CaseChallenges: Distributing software with EFS

Changes in process

Redesign application to have read/write space outside application namespace

Application must run from its home directory down

Do not mix software with configuration

Be aware of EFS: If EFS client is not installed, your application will not work

Need to install EFS client on servers

Developers

Application Support Teams

System Administrators

Further contact

http://openefs.org Edward Inderrieden

Manager, EFS [email protected]