OpenEmbedded and Yocto

introduction

Nicolas Dechesne <nicolas.dechesne@linaro.org>

Riku Voipio <riku.voipio@linaro.org>

Trevor Woerner <trevor.woerner@linaro.org>

LCE13, Dublin

2

Agenda

● Intro to OpenEmbedded / Yocto

● Overview of Yocto Project, organization and releases

● Yocto Project vs. DIY

● Introduction to OpenEmbedded concepts (recipes,

packages, images, layers, configuration, SDK, …)

● OpenEmbedded/Yocto and Linaro

● 'OpenEmbedded/Yocto Lab' (this afternoon, 3PM)

● Beginners practical introduction

● Guide you through a 'qemuarm' console image build

● Q&A

3

What is the Yocto Project?

It provides all the basic bits that every Linux product needs, pre-integrated and

tested to let you spend more time on differentiators that matter to you!

4

The Yocto Project ● Open source project, hosted by the Linux Foundation.

● Not the “new kid on the block”, OpenEmbedded has been around since

2003!

● A collaboration project, an “umbrella” project. A collection of projects that

make up the “Yocto Project”

● Bitbake

● OpenEmbedded-Core

● Poky

● Application Development Toolkit

● And even more...

● It helps you build a 'tailored' Linux distribution/product for your exact needs.

● Support for ARM, PPC, MIPS, x86

5

Yocto Project organization

● Hierarchical technical leadership:

● Richard Purdie as the project architect

● Sub system maintainers

● Somehow similar to Linux kernel technical leadership

● Yocto Project Advisory Board for administrative leadership

● community, finance, infrastructure, advocacy

● From linaro: TI, Enea, Huawei

● Open to all contributors

● Yocto Compliance program

● Yocto project participants

● Yocto project compatible

6

Yocto/OE releases ● 1 new release ~6 months

● Yocto Project releases in sync with OE, bitbake projects

releases

● Yocto Project release is the aggregation of oe-core, bitbake and

Poky layers/tools/documentation.

● Stable releases

● v1.4 (aka dylan), v1.3 (aka danny)

● No new feature, only bug fixes

● Development / master branch

● New features (systemd, wayland, Qt5, build engine improvements, ...)

● Components version updates

● Does not 'work' every single day

7

Yocto/OpenEmbedded vs DIY

● Spend less time on things that bring no value to your business:

● OE provides and maintains thousands of up-to-date recipes

● Build an entire Linux system from sources in less than 1 hour, using a validated set of

packages (toolchain, busybox, libc, init system...)

● Built in support for package management

● Predictable and reproducible builds

● Provides set of standard tools and build guidelines.

● What you build for a product can be reused for other products.

● autobuilder 'bot' for continuous integration

● Minimal dependencies on host, build as standard user (no root)

● Still, you have all the flexibility you need to

● patch as needed any OE component for your needs/product

● Add any unsupported component, even proprietary bits

● Fully configure the system, and make it exactly the way you want.

8

OpenEmbedded metadata

● OpenEmbedded does not contain components source code, only their metadata

● Bitbake is the build engine, a metadata parser, written in Python

● Recipes

● Component (.bb, .bbappend)

● Image (.bb)

● Class (.bbclass)

● Configuration files: behavior largely controlled by variables

● bitbake.conf

● <machine>.conf

● <distro>.conf

● local.conf, site.conf

● bblayers.conf

● Patches

9

OpenEmbedded overview

10

OpenEmbedded packages ● A recipe generates 1 or more binary packages

● Each recipe build is split into 'tasks'

● Tasks are scheduled by bitbake based on parallelism and dependencies

● fetch, unpack, patch, configure, build, install, package

● All packages are built from scratch with same compiler configuration (CPU,

optimizations, tuning, …) according to the underlying target CPU.

● Standard packages are created managed automatically (most of the time)

● Binaries are stripped automatically, and debug binaries are placed in -dbg

packages

● Development packages with .h , .so, .a, .pc, …

● ...

● Packages dependencies

● build depends (DEPENDS) are mandatory and explicit in recipes

● Run time depends (RDEPENDS) are semi-automatic (shlibdeps, pcdeps)

11

OpenEmbedded images ● Images are standard recipe (e.g. .bb file), with a different 'task set'

● Images are expected to be 'deployed' on target

● Most standard image format are supported (ext2/3, tarball, jffs2, cramfs, ramdisk,

…)

● Support for image generation 'post scripting'

● Packages are assembled into the final root file system to make up images

● Image recipe lists all components/applications required in the image

● RDEPENDS used to pull all the needed libraries, ...

● Multiple images can be built from the same 'build', by assembling different set of

packages

● OpenEmbedded has (optional) support for package management in the target

image: .rpm, .ipk, .deb.

● Can be used to implement a binary package based distro (with package feed)

● However OE always uses packages 'internally' to build images.

12

OpenEmbedded layers

● Couple of years ago, OpenEmbedded used to be a monolithic repo with thousands of

recipes, and became impractical to maintain and use

● Layer concept introduced

● Layers contain additional metadata (recipes, machine, distro, …)

● Oe-core is the base layer for the most fundamental components

● layers provide additional features: meta-systemd, meta-gnome, …

● Layers can be inter-dependent

● Layers can be hosted in different 'trees' (git)

● All layers must work with oe-core

● Layers can add, modify content/beahvior, and/or alter, extend metadata from other

layers.

● Generally, each BSP or distro has its own layer(s)

● meta-linaro, https://git.linaro.org/gitweb?p=openembedded/meta-linaro.git

● OpenEmbedded metadata on line index: http://layers.openembedded.org/

13

Poky

● Poky is a reference distro implementation.

● Minimal console only image

● Full Linux system with GUI using “sato”

● Generally used as a starting point (or an example) for building one's own

(commercial) distro: Angstrom, ELDK, Mentor Linux, Wind River Linux, ENEA

Linux, ...

● “it is often just a starting point, not an end goal”

● Built with bitbake and these layers: oe-core, meta-yocto, meta-yocto-bsp

(and yes, the name is confusing!)

● Used to test/validate OpenEmbedded by Yocto Project developers

● “Poky-tiny” is a variant that provides size optimized image.

● “Poky-bleeding” is a variant that uses bleeding edge software version

14

SDK ● Application Development Toolkit (ADT) provides custom built,

cross development SDK tailored for the image being built

● Cross toolchain

● Matching sysroot

● Eclipse IDE Yocto plugin

● Support SDK for x86 and/or x86-64 hosts

● ADT can be installed on 'OEM', customer hosts without the

need to rebuild the entire image packages

● “bitbake meta-toolchain” : includes only the minimal cross toolchain and

libs

● “bitbake -c populate_sdk <image>” : includes all the development

headers, libs, … for every package included in the image. e.g. build SDK

that matches the target image

15

Anatomy of a build folder ● Lots of space … ~20 Gb

● Entirely contained in '<build>' folder

● Sources used are in '<build>/downloads'

● Builds are in '<build>/tmp/work/<arch>'

● 'deploy' has

● Ready to use images

● Packages

● license information

● Buildhistory records information about content of

images, and package build after build (with diffs)

● 'flat' rootfs also available in image <work> folder

● INHERIT += “rm_work” cleans up each <work> folder

upon successful build

OpenEmbedded and Linaro

17

Background

● Originally we started using OE to bootstrap

Aarch64 support

● Also used by Toolchain WG to validate compiler

● Extended interest by networking WG and others

● Integrated to our CI loop

18

Quickstart

● Git clone git://git.linaro.org/openembedded/jenkins-setup.git

● $ sudo bash pre-build-root-install-dependencies.sh

● $ init-and-build.sh linaro-image-minimal

● Have a nice cup

● of coffee!

19

Show me the code

● git.l.org/openembedded/jenkins-setup.git

● Setup and build script as used by integration

● git.l.org/openembedded/manifest.git

● Repo manifest, describes used git repositores

● git.l.org/openembedded/meta-linaro.git

● Contains Linaro, toolchain and aarch64 layers

● Branches for head and stable (dylan)

● Poky support being worked on

20

Continuous integration

● Jenkins: builds – LAVA runtime testing

● https://ci.linaro.org/jenkins/job/openembedded-armv8-rootfs/

● Building the minimal, lamp and Java Images

● https://git.linaro.org/gitweb?p=qa/test-

definitions.git;a=tree;f=openembedded

● Test definitions for LAVA

● http://validation.linaro.org/lava-server/dashboard/image-

reports/minimal-armv8

● Test results in LAVA

● Similar pages for LAMP and JAVA images

21

Future plans

● Switching to Poky/Yocto or stable OE

● Changes in OE head disruptive

● Repo manifest tagging for releases

● For reproducibility

● CI integration of ARMv7 and big endian

● Bitbake world -k for toolchain validation

● Try to keep things simple instead of supporting every possible

usecase.

22

Resources

● https://www.yoctoproject.org/

● http://openembedded.org/wiki/Main_Page

● http://www.slideshare.net/rossburton/the-yocto-project

● https://www.yoctoproject.org/documentation/current

● http://layers.openembedded.org/layerindex/

● http://www.linuxfoundation.org/news-media/blogs/browse/2013/05/new-paper-how-engineering-

leaders-can-use-yocto-project-solve

● http://www.linuxfoundation.org/news-media/announcements/2010/10/linux-foundation-and-

consumer-electronics-linux-forum-merge

● http://www.linuxfoundation.org/news-media/blogs/browse/2010/10/while-apple-debates-open-vs-

integrated-we-want-best-both-worlds

23

Thank you!

24

OpenEmbedded/Yocto 'lab'

25

Demystify OE/Yocto

build through a practical exercise

● Build a minimal console image for qemuarm

● Run the image in QEMU

● Explore and understand the OE build directory

structure

● Look at some basic OE developers tasks

● Build/run an application with SDK

26

Practical information

● Source and build cache

● “repo” tool

● Sample of recipe

● Lab instructions are here:

● Part1: https://collaborate.linaro.org/x/NYBm

● Part2: https://collaborate.linaro.org/x/q4Bm

● Part3: https://collaborate.linaro.org/x/zIBm

● Ask questions, we are here to answer them!

27

Interesting things to do...

● Explore configuration files

● Checkout some recipes

● Explore 'WORKDIR' and 'deploy' folders (image and package

feed)

● Customize your image

● Add a new recipe, create your own layer

● Exercise with .bbappend

● Create a patch, and apply in recipe

● Use/understand bitbake 'tasks' and experiment typical

developer workflow (build, debug, edit, build, debug, ...)

28

Sources mirror, build cache

● OpenEmbedded 'trees' contain no source code

● By design, everything is built from scratch.

● Source code for all recipes is fetched during the build

● 'fetchall' target let you download all source first

● PREMIRROR let you configure OE to use an alternative mirror before going to the Internet

(can be local/intranet)

● OE implements “shared state code” (aka sstate) to support incremental builds

● OE uses hash/signature on each 'task' input variables to decide if task needs to be rebuilt.

● Prebuilt components are stored in “sstate” cache, which can be local or http, and can be

shared across multiple users.

● Even after deleting the entire working folder, a rebuild is 'instantaneous'

● We prepared 'tarballs' for 'download' and 'sstate' to make this workshop more

efficient, hopefully you've downloaded everything already

● http://people.linaro.org/~trevor.woerner/LCE2013/

● Otherwise we have a couple of HDDs

29

“repo” tool

● Implemented initially by the Android team to manage Android source repositories

● Open source tool: https://code.google.com/p/git-repo/

● Wrapper on top of git

● Manage 'forest' of git trees

● Support multiple 'product' branches and easy switch from one checkout to another

● Trees metadata stored in 'XML manifest' file

https://gerrit.googlesource.com/git-repo/+/master/docs/manifest-format.txt

● Manifest can checkout:

● Latest commit on branch (development)

● Fix commit/tag (release)

● All Linaro OE projects are moving to “repo”

● Repo can (optionally) be used with Gerrit code review

30

Manifest sample

<?xml version="1.0" encoding="UTF-8"?> <manifest> <remote name="linaro" fetch="git://git.linaro.org" /> <remote name="oe" fetch="git://git.openembedded.org" /> <remote name="beagleboard" fetch="git://github.com/beagleboard" />

<default remote="oe" revision="master" sync-j="2" />

<project name="openembedded-core" /> <project name="meta-openembedded" /> <project path="openembedded-core/bitbake" name="bitbake" /> <project path="meta-linaro" name="openembedded/meta-linaro" remote="linaro" /> <project name="meta-beagleboard" remote="beagleboard"/>

</manifest>

31

How to use “repo”

# install repo wrapper (once on each machine) $ curl https://dl-ssl.google.com/dl/googlesource/git-repo/repo > /tmp/repo $ chmod a+x /tmp/repo $ sudo mv /tmp/repo /usr/local/bin/ # initialize the repo, and download the manifest $ mkdir <mydir> && cd <mydir> $ repo init -u <manifest.git> -b <branch> # download and checkout all projects $ repo sync # build, edit, commit, debug, … # update my working copy $ repo sync # switch to another branch/product $ repo init -b <my other branch> $ repo sync

commands: list status info diff grep

32

Hello World! DESCRIPTION = "Simple helloworld application" SECTION = "examples" LICENSE = "MIT" LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" PR = "r0" SRC_URI = "file://helloworld.c" S = "${WORKDIR}" do_compile() { ${CC} helloworld.c -o helloworld } do_install() { install -d ${D}${bindir} install -m 0755 helloworld ${D}${bindir} }

Packages:

helloworld

helloworld-dev

helloworld-dbg

33

smartmontools

# meta-oe/recipes-extended/smartmontools/smartmontools_6.1.bb SECTION = "console/utils" DESCRIPTION = "Control and monitor storage systems using S.M.A.R.T." HOMEPAGE = "http://smartmontools.sourceforge.net/"

LICENSE = "GPLv2" LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"

SRC_URI = "${SOURCEFORGE_MIRROR}/smartmontools/smartmontools-${PV}.tar.gz" SRC_URI[md5sum] = "83a3a681f8183ed858392d550ae1cca6" SRC_URI[sha256sum] = "a9003b8bccc82682f658ce76d70edb1842411e51dc56d4cd6b56618da1d9ce07"

inherit autotools

http://cgit.openembedded.org/meta-openembedded/commit/?id=8fec906a4584f9cbc467fb0e406bb6a705848b64

34

Using layer to alter recipes

$ more meta-linaro/recipes-core/busybox/busybox_1.20.2.bbappend SRC_URI += "file://linaro.cfg"

$ more meta-aarch64/recipes-sato/webkit/webkit-gtk_1.8.3.bbappend SRC_URI_append = "file://aarch64.patch”

● “<packagename>_<version>.bbappend” files can be used in layers

● Any variable from the original recipe can be modified

● Very neat for patching! Avoid duplication, fragmentation.

● “bitbake-layers show-appends”

35

Backup slides

36

Tuning compiler ● CPU compiler options, FPU and ABI can be fully tuned from the machine

configuration file

● Each 'architecture' gets a set of pre-defined 'tunes'

● For ARM : VFP vs neon, Thumb vs ARM instructions, Little vs big endian, Calling

conventions (hardfp vs softfp, …)

TUNEVALID[neon] = "Enable Neon SIMD accelerator unit." TUNE_CCARGS .= "${@bb.utils.contains("TUNE_FEATURES", "neon", " -mfpu=neon", "" ,d)}" ARMPKGSFX_FPU .= "${@bb.utils.contains("TUNE_FEATURES", "neon", "-neon", "" ,d)}"

TUNEVALID[vfp] = "Enable Vector Floating Point (vfp) unit." ARMPKGSFX_FPU .= "${@bb.utils.contains("TUNE_FEATURES", "vfp", "-vfp", "" ,d)}" TUNEVALID[callconvention-hard] = "Enable EABI hard float call convention, requires VFP." TUNE_CCARGS .= "${@bb.utils.contains("TUNE_FEATURES", "vfp", bb.utils.contains("TUNE_FEATURES", "callconvention-hard", " -mfloat-abi=hard", " -mfloat-abi=softfp", d), "" ,d)}" ARMPKGSFX_EABI .= "${@bb.utils.contains("TUNE_FEATURES", [ "callconvention-hard", "vfp" ], "hf", "", d)}"

37

Layer configuration file

# LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf # changes incompatibly LCONF_VERSION = "5" BBPATH = "${TOPDIR}" BBFILES ?= "" BBLAYERS ?= " \ ${TOPDIR}/openembedded-core/meta \ ${TOPDIR}/meta-openembedded/toolchain-layer \ ${TOPDIR}/meta-openembedded/meta-multimedia \ ${TOPDIR}/meta-openembedded/meta-oe \ ${TOPDIR}/meta-linaro/meta-linaro \ ${TOPDIR}/meta-linaro/meta-linaro-toolchain \ ${TOPDIR}/meta-qt5 \ " BBLAYERS_NON_REMOVABLE ?= " \ ${TOPDIR}/openembedded-core/meta \ "

38

Machine configuration file #@TYPE: Machine #@NAME: common_pc #@DESCRIPTION: Machine configuration for running a common x86

PREFERRED_PROVIDER_virtual/xserver ?= "xserver-xorg" PREFERRED_PROVIDER_virtual/libgl ?= "mesa" PREFERRED_PROVIDER_virtual/libgles1 ?= "mesa" PREFERRED_PROVIDER_virtual/libgles2 ?= "mesa"

require conf/machine/include/qemu.inc require conf/machine/include/tune-i586.inc

KERNEL_IMAGETYPE = "bzImage"

SERIAL_CONSOLE = "115200 ttyS0"

XSERVER = "xserver-xorg \ mesa-driver-swrast \ xf86-input-vmmouse \ xf86-input-keyboard \ xf86-input-evdev \ xf86-video-vmware"

MACHINE_FEATURES += "x86"

MACHINE_ESSENTIAL_EXTRA_RDEPENDS += "v86d"

qemux86.conf

39

User local configuration file

# Parallelism Options #BB_NUMBER_THREADS = "4" #PARALLEL_MAKE = "-j 4" MACHINE = "qemuarm" DISTRO = “poky” PACKAGE_CLASSES ?= "package_ipk" # "dbg-pkgs" - add -dbg packages for all installed packages # (adds symbol information for debugging/profiling) # "tools-testapps" - add useful testing tools (ts_print, aplay, arecord etc.) # "debug-tweaks" - make an image suitable for development # e.g. ssh root access has a blank password EXTRA_IMAGE_FEATURES = "debug-tweaks" # - 'buildstats' collect build statistics # - 'image-prelink' in order to prelink the filesystem image USER_CLASSES ?= "buildstats image-prelink" #SSTATE_MIRRORS ?= "\ #file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \ #file://.* file:///some/local/dir/sstate/PATH"

40

Yocto Project Branding/Compliance

● Compliance controls the usage of the Yocto project name, logo

and brand.

● Yocto Project Participants

● Open source project, non profit, small business

● Committed to promoting and contributing to Yocto Projects (oe-core,

bitbake, yocto layers)

● Yocto Project Compatible

● Products, BSP, layers maintained by an open source, non profit, or Yocto

Project member organisation

● Compatible with a specific release of Yocto

● Contribute all Yocto Projects patches upstream

● All Yocto recommendations/processes implemented