slides

1Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http//freeelectrons.com

Free Electrons

Linuxkernel and driver

developmenttraining

Thomas PetazzoniMichael Opdenacker

Custom Development

System integrationEmbedded Linux demos and prototypesSystem optimizationApplication and interface development

Free ElectronsOur services

Embedded Linux Training

All materials released with a free license!

Unix and GNU/Linux basicsLinux kernel and drivers developmentRealtime Linux, uClinuxDevelopment and profiling toolsLightweight tools for embedded systemsRoot filesystem creationAudio and multimediaSystem optimization

Consulting and technical support

Help in decision makingSystem architectureSystem design and performance reviewDevelopment tool and application supportInvestigating issues and fixing tool bugs

Linux kernel

Linux device driversBoard support codeMainstreaming kernel codeKernel debugging


Rights to copy

Attribution – ShareAlike 3.0You are free

to copy, distribute, display, and perform the workto make derivative worksto make commercial use of the work

Under the following conditionsAttribution. You must give the original author credit.Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one.

For any reuse or distribution, you must make clear to others the license terms of this work.Any of these conditions can be waived if you get permission from the copyright holder.

Your fair use and other rights are in no way affected by the above.License text: http://creativecommons.org/licenses/bysa/3.0/legalcode

© Copyright 20042010, Free [email protected]

Electronic version of this document available onhttp://freeelectrons.com/doc/training/linuxkernel

Document updates will be available onhttp://freeelectrons.com/doc/training/linuxkernel

Corrections, suggestions,contributions and translations are welcome!

Latest update: Mar 5, 2010

http://creativecommons.org/licenses/by-sa/3.0/legalcode

http://free-electrons.com/doc/training/linux-kernel/

http://free-electrons.com/doc/training/linux-kernel


Hardware used in this training session

AT91SAM9263 ARM CPU

64 MB RAM256 MB flash

2 USB 2.0 host1 USB device

100 Mbit Ethernet port

Powered by USB!Serial and JTAG throughthis USB port.

Multiple extension boards.

162 EUR

Calao Systems USBA9263

Supported in mainstream Linux since version 2.6.27!


Hyperlinks in this document

Links to external sitesExample: http://kernel.org/

Kernel source filesOur links let you view them in your browser.Example: kernel/sched.c

Kernel source code:Identifiers: functions, macros, type definitions...You get access to their definition, implementation and where they are used. This invites you to explore the source by yourself!wait_queue_head_t queue;init_waitqueue_head(&queue);

Table of contentsDirectly jump to the corresponding sections.Example: Kernel configuration

click

Usable in the PDF and ODP formatsTry them on this page!

http://kernel.org/

http://lxr.free-electrons.com/source/kernel/sched.c

http://lxr.free-electrons.com/ident?i=wait_queue_head_t

http://lxr.free-electrons.com/ident?i=init_waitqueue_head


Cooperate!

As in the Free Software and Open Source community,cooperation during practical labs is valuable in this training session:

Don't hesitate to share your questions with other participants.They can make some points easier to understand, by using their own words.

Explaining what you understood to other participantsalso helps to consolidate your knowledge.

If you complete your labs before other people, don't hesitate to help other people and investigate the issues they face. The faster we progress as a group, the more time we have to explore extra topics.

Don't hesitate to report potential bugs to your instructor.

Don't hesitate to look for solutions on the Internet as well.


Free Electrons

Linux kernelintroduction

Michael OpdenackerThomas Petazzoni

Free Electrons

© Copyright 20042009, Free Electrons.Creative Commons BYSA 3.0 licenseLatest update: Mar 5, 2010, Document sources, updates and translations:http://freeelectrons.com/docs/kernelintroCorrections, suggestions, contributions and translations are welcome!

http://free-electrons.com/docs/kernel-intro


Embedded Linux driver development

Kernel overviewLinux features


Userspace

Linux kernel in the system

C library

Linux Kernel

Hardware

Library A User app A

User app B

Event notification, information exposition

Call to services

Event notificationManage hardware


History

The Linux kernel is one component of a system, which also requires libraries and applications to provide features to end users.

The Linux kernel was created as a hobby in 1991by a Finnish student, Linus Torvalds.

Linux quickly started to be used as the kernel for free software operating systems

Linus Torvalds has been able to create a large and dynamic developer and user community around Linux.

Nowadays, hundreds of people contribute to each kernel release, individuals or companies big and small.


Linux license

The whole Linux sources are Free Software releasedunder the GNU General Public License version 2 (GPL v2).

For the Linux kernel, this basically implies that:

When you receive or buy a device with Linux on it,you should receive the Linux sources, with the right to study, modify and redistribute them.

When you produce Linux based devices, you must release the sources to the recipient, with the same rights, with no restriction.

See our http://freeelectrons.com/articles/freesw/ training for exact details about Free Software and its licenses.

http://free-electrons.com/articles/freesw/


Linux kernel key features

Portability and hardware supportRuns on most architectures.

ScalabilityCan run on super computers as well as on tiny devices(4 MB of RAM is enough).

Compliance to standards and interoperability.

Exhaustive networking support.

SecurityIt can't hide its flaws. Its code is reviewed by many experts.

Stability and reliability.

ModularityCan include only what a system needs even at run time.

Easy to programYou can learn from existing code. Many useful resources on the net.


Supported hardware architectures

2.6.31 status

See the arch/ directory in the kernel sources

Minimum: 32 bit processors, with or without MMU, and gcc support

32 bit architectures (arch/ subdirectories)arm, avr32, blackfin, cris, frv, h8300, m32r, m68k, m68knommu, microblaze, mips, mn10300, parisc, s390, sparc, um, xtensa

64 bit architectures:alpha, ia64, sparc64

32/64 bit architecturespowerpc, x86, sh

Find details in kernel sources: arch/<arch>/Kconfig, arch/<arch>/README, or Documentation/<arch>/

http://lxr.free-electrons.com/source/arch/


http://lxr.free-electrons.com/source/arch/arm/

http://lxr.free-electrons.com/source/arch/avr32/

http://lxr.free-electrons.com/source/arch/blackfin/

http://lxr.free-electrons.com/source/arch/cris/

http://lxr.free-electrons.com/source/arch/frv/

http://lxr.free-electrons.com/source/arch/h8300/

http://lxr.free-electrons.com/source/arch/m32r/

http://lxr.free-electrons.com/source/arch/m68k/

http://lxr.free-electrons.com/source/arch/m68knommu/

http://lxr.free-electrons.com/source/arch/microblaze/

http://lxr.free-electrons.com/source/arch/mips/

http://lxr.free-electrons.com/source/arch/mn10300/

http://lxr.free-electrons.com/source/arch/parisc/

http://lxr.free-electrons.com/source/arch/s390/

http://lxr.free-electrons.com/source/arch/sparc/

http://lxr.free-electrons.com/source/arch/um/

http://lxr.free-electrons.com/source/arch/xtensa/

http://lxr.free-electrons.com/source/arch/alpha/

http://lxr.free-electrons.com/source/arch/ia64/

http://lxr.free-electrons.com/source/arch/sparc64/

http://lxr.free-electrons.com/source/arch/powerpc/

http://lxr.free-electrons.com/source/arch/x86/

http://lxr.free-electrons.com/source/arch/s390/


System calls

The main interface between the kernel and userspace is the set of system calls

About ~300 system calls that provides the main kernel services

File and device operations, networking operations, interprocess communication, process management, memory mapping, timers, threads, synchronization primitives, etc.

This interface is stable over time: only new system calls can be added by the kernel developers

This system call interface is wrapped by the C library, and userspace applications usually never make a system call directly but rather use the corresponding C library function


Virtual filesystems

Linux makes system and kernel information available in userspace through virtual filesystems (virtual files not existing on any real storage). No need to know kernel programming to access such information!

Mounting /proc:sudo mount t proc none /proc

Mounting /sys:sudo mount t sysfs none /sys

Filesystem type Raw deviceor filesystem imageIn the case of virtual

filesystems, any string is fine

Mount point


/proc details

A few examples:

/proc/cpuinfo: processor information

/proc/meminfo: memory status

/proc/version: kernel version and build information

/proc/cmdline: kernel command line

/proc/<pid>/environ: calling environment

/proc/<pid>/cmdline: process command line

... and many more! See by yourself!

Lots of details about the /proc interface are available in Documentation/filesystems/proc.txt(almost 2000 lines) in the kernel sources.

http://free-electrons.com/kerneldoc/latest/filesystems/proc.txt


Embedded Linux usage

Kernel overviewLinux versioning scheme and development process


Until 2.6 (1)

One stable major branch every 2 or 3 years

Identified by an even middle number

Examples: 1.0, 2.0, 2.2, 2.4

One development branch to integrate new functionalities and major changes

Identified by an odd middle number

Examples: 2.1, 2.3, 2.5

After some time, a development version becomes the new base version for the stable branch

Minor releases once in while: 2.2.23, 2.5.12, etc.


Until 2.6 (2)

2.4.0 2.4.1 2.4.2 2.4.3 2.4.4 2.4.5 2.4.6 2.4.7 2.4.8

2.5.0 2.5.1 2.5.2 2.5.3 2.5.4 2.6.0 2.6.1

Stable version

Development Stable

Note: in reality, many more minor versions exist inside the stable and development branches


Changes since Linux 2.6 (1)

Since 2.6.0, kernel developers have been able to introduce lots of new features one by one on a steady pace, without having to make major changes in existing subsystems.

Opening a new Linux 2.7 (or 2.9) development branch will be required only when Linux 2.6 is no longer able to accommodate key features without undergoing traumatic changes.

Thanks to this, more features are released to users at a faster pace.


Changes since Linux 2.6 (2)

Since 2.6.14, the kernel developers agreedon the following development model:

After the release of a 2.6.x version, a twoweeks merge window opens, during which major additions are merged.

The merge window is closedby the release of test version 2.6.(x+1)rc1

The bug fixing period opens, for 6 to 10 weeks.

At regular intervals during the bug fixing period,2.6.(x+1)rcY test versions are released.

When considered sufficiently stable,kernel 2.6.(x+1) is released, and the process starts again.


Merge and bug fixing windows

Merge window Bug fixing period

2.6.21 2.6.22rc1

2.6.22rc2

2.6.22rc3 2.6.22rc5

2.6.22rc4 2.6.22

2 weeks 6 to 10 weeks

2.6.21.1 2.6.21.22.6.21.2 2.6.21.3 2.6.21.4 2.6.21.5

Bug fix updates

2.6.22.1


More stability for the 2.6 kernel tree

Issue: bug and security fixes only released for last (or last two) stable kernel versions (like 2.6.16 and 2.6.17), and of course by distributions for the exact version that you're using.

Some people need to have a recent kernel, but with long term support for security updates.

You could get long term support from a commercial embedded Linux provider.

You could reuse sources for the kernel used in Ubuntu Long Term Support releases (5 years of free security updates).

You could choose Linux 2.6.27 for your project, which will be maintained by kernel.org for a long time, unlike other versions.


What's new in each Linux release?

The official list of changes for each Linux release is just a huge list of individual patches!

Very difficult to find out the key changes and to get the global picture out of individual changes.

Fortunately, a summary of key changeswith enough details is available on http://wiki.kernelnewbies.org/LinuxChanges

commit 3c92c2ba33cd7d666c5f83cc32aa590e794e91b0Author: Andi Kleen <[email protected]>Date: Tue Oct 11 01:28:33 2005 +0200

[PATCH] i386: Don't discard upper 32bits of HWCR on K8 Need to use long long, not long when RMWing a MSR. I think it's harmless right now, but still should be better fixed if AMD adds any bits in the upper 32bit of HWCR. Bug was introduced with the TLB flush filter fix for i386 Signedoffby: Andi Kleen <[email protected]> Signedoffby: Linus Torvalds <[email protected]>...

??!

http://wiki.kernelnewbies.org/LinuxChanges



Kernelsource code


Free Electrons

© Copyright 20042010, Free Electrons.Creative Commons BYSA 3.0 licenseLatest update: Mar 5, 2010, Document sources, updates and translations:http://freeelectrons.com/docs/kernelsources/Corrections, suggestions, contributions and translations are welcome!

http://free-electrons.com/docs/kernel-sources/



Linux code and device drivers


Supported kernel version

The APIs covered in these training slidesshould be compliant with Linux 2.6.31.

We may also mention features in more recent kernels.


Programming language

Implemented in C like all Unix systems.(C was created to implement the first Unix systems)

A little Assembly is used too:

CPU and machine initialization, exceptions

Critical library routines.

No C++ used, see http://www.tux.org/lkml/#s153

All the code compiled with gcc, the GNU C Compiler

Many gccspecific extensions used in the kernel code, any ANSI C compiler will not compile the kernel

A few alternate compilers are supported (Intel and Marvell)

See http://gcc.gnu.org/onlinedocs/gcc4.3.3/gcc/CExtensions.html

http://www.tux.org/lkml/#s15-3

http://gcc.gnu.org/onlinedocs/gcc-4.3.3/gcc/C-Extensions.html


No C library

The kernel has to be standalone and can't use userspace code.Userspace is implemented on top of kernel services, not the opposite.Kernel code has to supply its own library implementations(string utilities, cryptography, uncompression ...)

So, you can't use standard C library functions in kernel code.(printf(), memset(), malloc()...).You can also use kernel C headers.

Fortunately, the kernel provides similar C functions for your convenience, like printk(), memset(), kmalloc() ...

http://lxr.free-electrons.com/ident?i=printk

http://lxr.free-electrons.com/ident?i=memset

http://lxr.free-electrons.com/ident?i=kmalloc


Portability

The Linux kernel code is designed to be portable

All code outside arch/ should be portable

To this aim, the kernel provides macros and functions to abstract the architecture specific details

Endianness(cpu_to_be32, cpu_to_le32, be32_to_cpu, le32_to_cpu)

I/O memory access

Memory barriers to provide ordering guarantees if needed

DMA API to flush and invalidate caches if needed


http://lxr.free-electrons.com/ident?i=cpu_to_be32

http://lxr.free-electrons.com/ident?i=cpu_to_le32

http://lxr.free-electrons.com/ident?i=be32_to_cpu

http://lxr.free-electrons.com/ident?i=le32_to_cpu


No floating point computation

Never use floating point numbers in kernel code. Your code may be run on a processor without a floating point unit (like on arm).

Don't be confused with floating point related configuration options

They are related to the emulation of floating point operation performed by the user space applications, triggering an exception into the kernel

Using softfloat, i.e. emulation in userspace, is however recommended for performance reasons


No stable Linux internal API (1)

The internal kernel API to implement kernel code can undergo changes between two 2.6.x releases. A standalone driver compiled for a given version may no longer compile or work on a more recent one.See Documentation/stable_api_nonsense.txtin kernel sources for reasons why.

Of course, the external API must not change (system calls, /proc, /sys), as it could break existing programs. New features can be added, but kernel developers try to keep backward compatibility with earlier versions, at least for 1 or several years.

Whenever a developer changes an internal API, (s)he also has to update all kernel code which uses it. Nothing broken!

Works great for code in the mainline kernel tree.Difficult to keep in line for out of tree or closedsource drivers!

Kernel API changes reference: http://lwn.net/Articles/2.6kernelapi/

Feature removal schedule: Documentation/featureremovalschedule.txt

http://lxr.free-electrons.com/source/Documentation/stable_api_nonsense.txt

http://lwn.net/Articles/2.6-kernel-api/

http://free-electrons.com/kerneldoc/latest/feature-removal-schedule.txt


No stable Linux internal API (2)

USB example

Linux has updated its USB internal API at least 3 times (fixes, security issues, support for highspeed devices) and has now the fastest USB bus speeds (compared to other systems)

Windows XP also had to rewrite its USB stack 3 times. But, because of closedsource, binary drivers that can't be updated, they had to keep backward compatibility with all earlier implementation. This is very costly (development, security, stability, performance).

See “Myths, Lies, and Truths about the Linux Kernel”, by Greg K.H., for details about the kernel development process: http://kroah.com/log/linux/ols_2006_keynote.html

http://kroah.com/log/linux/ols_2006_keynote.html


Kernel memory constraints

Who can look after the kernel?

No memory protectionAccessing illegal memory locations result in (often fatal) kernel oopses.

Fixed size stack (8 or 4 KB)Unlike in userspace,no way to make it grow.

Kernel memory can't be swapped out (for the same reasons).

Userprocess

KernelIllegal

memorylocation

Attemptto access

Exception(MMU)

SIGSEGV, kill

Userspace memory managementUsed to implement: memory protection stack growth memory swapping to disk demand paging


Linux kernel licensing constraints

The Linux kernel is licensed under the GNU General Public License version 2

This license gives you the right to use, study, modify and share the software freely

However, when the software is redistributed, either modified or unmodified, the GPL requires that you redistribute the software under the same licence, with the source code

If modifications are made to the Linux kernel (for example to adapt it to your hardware), it is a derivative work of the kernel, and therefore must be released under GPLv2

The validity of the GPL on this point has already been verified in courts

However, you're only required to do so

At the time the device starts to be distributed

To your customers, not to the entire world


Proprietary code and the kernel

It is illegal to distribute a binary kernel that includes statically compiled proprietary drivers

The kernel modules are a grey area : are they derived works of the kernel or not ?

The general opinion of the kernel community is that proprietary drivers are badhttp://www.linuxfoundation.org/en/Kernel_Driver_Statement

From a legal point of view, each driver is probably a different case

Is it really useful to keep your drivers secret ?

There are some examples of proprietary drivers, like the Nvidia graphics drivers

They use a wrapper between the driver and the kernel

Unclear whether it makes it legal or not

http://www.linuxfoundation.org/en/Kernel_Driver_Statement


Advantages of GPL drivers

You don't have to write your driver from scratch. You can reuse code from similar free software drivers.

You get free community contributions, support, code review and testing. Proprietary drivers (even with sources) don't get any.

Your drivers can be freely shipped by others (mainly by distributions).

Closed source drivers often support a given kernel version. A system with closed source drivers from 2 different sources is unmanageable.

From the driver developer / decision maker point of view

Users and the community get a positive image of your company. Makes it easier to hire talented developers.

You don't have to supply binary driver releases for each kernel version and patch version (closed source drivers).

Drivers have all privileges. You need the sources to make sure that a driver is not a security risk.

Your drivers can be statically compiled into the kernel.


Advantages of intree kernel drivers

Advantages of having your drivers in the mainline kernel sources

Once your sources are accepted in the mainline tree, they are maintained by people making changes.

Costfree maintenance, security fixes and improvements.

Easy access to your sources by users.

Many more people reviewing your code.


Userspace device drivers (1)

Possible to implement device drivers in userspace!

Such drivers just need access to the devices through minimum, generic kernel drivers.

Examples

Printer and scanner drivers(on top of generic parallel port / USB drivers)

X drivers: low level kernel drivers + user space X drivers.

Userspace drivers based on UIO,framework introduced in 2.6.23


Userspace device drivers (2)

AdvantagesNo need for kernel coding skills. Easier to reuse code between devices.Drivers can be written in any language, even Perl!Drivers can be kept proprietary.Driver code can be killed and debugged. Cannot crash the kernel.Can be swapped out (kernel code cannot be).Can use floatingpoint computation.Less inkernel complexity.

DrawbacksLess straightforward to handle interrupts.Increased latency vs. kernel code.

See UIOHowto in the kernel documentation for detailed informations and the «Using UIO on an Embedded platform» talk at ELC 2008, http://www.celinux.org/elc08_presentations/uio080417celfelc08.pdf.

http://www.celinux.org/elc08_presentations/uio080417celfelc08.pdf



Linux sources


Location of kernel sources

The official version of the Linux kernel, as released by Linus Torvalds is available at http://www.kernel.org

This version follows the welldefined development model of the kernel

However, it may not contain the latest development from a specific area, due to the organization of the development model and because features in development might not be ready for mainline inclusion

Many kernel subcommunities maintain their own kernel, with usually newer but less stable features

Architecture communities (ARM, MIPS, PowerPC, etc.), device drivers communities (I2C, SPI, USB, PCI, network, etc.), other communities (realtime, etc.)

They generally don't release official versions, only development trees are available

http://www.kernel.org/


Linux sources structure (1)

arch/<arch> Architecture specific codearch/<arch>/<mach> Machine / board specific codeblock/ Block layer coreCOPYING Linux copying conditions (GNU GPL)CREDITS Linux main contributorscrypto/ Cryptographic librariesDocumentation/ Kernel documentation. Don't miss it!drivers/ All device drivers except sound ones (usb, pci...)fs/ Filesystems (fs/ext3/, etc.)include/ Kernel headersinclude/asm<arch> Architecture and machine dependent headersinclude/linux Linux kernel core headersinit/ Linux initialization (including main.c)



http://lxr.free-electrons.com/source/block/

http://lxr.free-electrons.com/source/COPYING

http://lxr.free-electrons.com/source/CREDITS

http://lxr.free-electrons.com/source/crypto/

http://lxr.free-electrons.com/source/Documentation/

http://lxr.free-electrons.com/source/drivers/

http://lxr.free-electrons.com/source/fs/

http://lxr.free-electrons.com/source/fs/ext3/

http://lxr.free-electrons.com/source/include/



http://lxr.free-electrons.com/source/init/

http://lxr.free-electrons.com/source/init/main.c


Linux sources structure (2)

ipc/ Code used for process communicationKbuild Part of the kernel build systemkernel/ Linux kernel core (very small!)lib/ Misc library routines (zlib, crc32...)MAINTAINERS Maintainers of each kernel part. Very useful!Makefile Top Linux makefile (sets arch and version)mm/ Memory management code (small too!)net/ Network support code (not drivers)README Overview and building instructionsREPORTINGBUGS Bug report instructionssamples/ Sample code (markers, kprobes, kobjects)scripts/ Scripts for internal or external usesecurity/ Security model implementations (SELinux...)sound/ Sound support code and driversusr/ Code to generate an initramfs cpio archive.

http://lxr.free-electrons.com/source/ipc/

http://lxr.free-electrons.com/source/Kbuild

http://lxr.free-electrons.com/source/kernel/

http://lxr.free-electrons.com/source/lib/

http://lxr.free-electrons.com/source/MAINTAINERS

http://lxr.free-electrons.com/source/Makefile

http://lxr.free-electrons.com/source/mm/

http://lxr.free-electrons.com/source/net/

http://lxr.free-electrons.com/source/README

http://lxr.free-electrons.com/source/REPORTING-BUGS

http://lxr.free-electrons.com/source/samples/

http://lxr.free-electrons.com/source/scripts/

http://lxr.free-electrons.com/source/security/

http://lxr.free-electrons.com/source/sound/

http://lxr.free-electrons.com/source/usr/


Accessing development sources (1)

Useful if you are involved in kernel developmentor if you found a bug in the source code.

Kernel development sources are now managed with git:http://kernel.org/pub/software/scm/git/

You can browse Linus' git tree (if you just need to check a few files):http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux2.6.git;a=tree

You can also directly use git on your workstation

Debian / Ubuntu: install the gitcore package

If you are behind a proxy, set Unix environment variables defining proxy settings. Example:export http_proxy="proxy.server.com:8080"export ftp_proxy="proxy.server.com:8080"

http://kernel.org/pub/software/scm/git/

http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=tree


Accessing development sources (2)

Choose a git development tree on http://git.kernel.org/

Get a local copy (“clone”) of this tree.Example (Linus tree, the one used for Linux stable releases):gitclone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux2.6.git

Update your copy whenever needed (Linus tree example):cd linux2.6git pull

See our training materials on git:http://freeelectrons.com/docs/git/

http://git.kernel.org/

http://free-electrons.com/docs/git/



Kernel source management tools


Cscope

http://cscope.sourceforge.net/

Tool to browse source code(mainly C, but also C++ or Java)

Supports huge projects like the Linux kernelTakes less than 1 min. to index Linux 2.6.17 sources (fast!)

Can be used from editors like vim and emacs.

In Linux kernel sources, run it with: cscope Rk (see man cscope for details)

Allows searching code for: all references to a symbol global definitions functions called by a function functions calling a function text string regular expression pattern a file files including a file

http://cscope.sourceforge.net/


Cscope screenshot


LXR: Linux Cross Reference

http://sourceforge.net/projects/lxr

Generic source indexing tooland code browser

Web server basedVery easy and fast to use

Identifier or text search available

Very easy to find the declaration, implementation or usages of symbols

Supports C and C++

Supports huge code projectssuch as the Linux kernel(264 M in version 2.6.26).

Takes a little time and patience to setup (configuration, indexing, server configuration).

Indexing a new version is very fast:approximately 20 minutes with LXR 0.3.1(old version, but scales very well).

You don't need to set up LXR by yourself.Use our http://lxr.freeelectrons.com server! Other servers available on the Internet:http://freeelectrons.com/community/kernel/lxr/

This makes LXR the simplest solutionto browse standard kernel sources.

http://sourceforge.net/projects/lxr

http://lxr.free-electrons.com/

http://free-electrons.com/community/kernel/lxr/


LXR screenshot


Practical lab – Kernel sources

Get familiar with the sources

Use a kernel source indexing tool


Embedded Linux kernel usage

Embedded Linuxkernel usage(summary)


Free Electrons

© Copyright 20042009, Free Electrons.Creative Commons BYSA 3.0 licenseLatest update: Mar 5, 2010, Document sources, updates and translations:http://freeelectrons.com/docs/kernelusageCorrections, suggestions, contributions and translations are welcome!

http://free-electrons.com/docs/kernel-usage



Compiling and booting LinuxLinux kernel sources


Getting Linux sources

Full tarballs

Contain the complete kernel sources

Long to download and uncompress, but must be done at least once

Example:http://kernel.org/pub/linux/kernel/v2.6/linux2.6.14.7.tar.bz2

Incremental patches between versions

It assumes you already have a base version and you apply the correct patches in the right order

Quick to download and apply

Exampleshttp://kernel.org/pub/linux/kernel/v2.6/patch2.6.14.bz2 (2.6.13 to 2.6.14)http://kernel.org/pub/linux/kernel/v2.6/patch2.6.14.7.bz2 (2.6.14 to 2.6.14.7)

All previous kernel versions are available in http://kernel.org/pub/linux/kernel/

http://kernel.org/pub/linux/kernel/


You can reversea patch

with the Roption

Using the patch command

The patch command applies changesto files in the current directory:

Making changes to existing files

Creating or deleting files and directories

patch usage examples:

patch p<n> < diff_file

cat diff_file | patch p<n>

bzcat diff_file.bz2 | patch p<n>

zcat diff_file.gz | patch p<n>

n: number of directory levels to skip in the file paths

You can test a patch withthe dryrun

option


Anatomy of a patch file

A patch file is the output of the diff command

diff Nru a/Makefile b/Makefile a/Makefile 20050304 09:27:15 08:00+++ b/Makefile 20050304 09:27:15 08:00@@ 1,7 +1,7 @@ VERSION = 2 PATCHLEVEL = 6 SUBLEVEL = 11EXTRAVERSION =+EXTRAVERSION = .1 NAME=Woozy Numbat

# *DOCUMENTATION*

diff command line

File date info

Line numbers in filesContext info: 3 lines before the changeUseful to apply a patch when line numbers changedRemoved line(s) if anyAdded line(s) if any

Context info: 3 lines after the change


Applying a Linux patch

Linux patches...

Always to apply to the x.y.<z1> versionDownloadable in gzipand bzip2 (much smaller) compressed files.

Always produced for n=1(that's what everybody does... do it too!)

Linux patch command line example:cd linux2.6.13bzcat ../patch2.6.14.bz2 | patch p1bzcat ../patch2.6.14.7.bz2 | patch p1cd ..; mv linux2.6.13 linux2.6.14.7

Keep patch files compressed: useful to check their signature later.You can still view (or even edit) the uncompressed data with vim:vim patch2.6.14.bz2 (on the fly (un)compression)

You can make patch 30%faster by using sp1

instead of p1(silent)

Tested on patch2.6.23.bz2



Compiling and booting LinuxKernel configuration


Kernel configuration (1)

The kernel contains thousands of device drivers, filesystem drivers, network protocols and other configurable items

Thousands of options are available, that are used to selectively compile parts of the kernel source code

The kernel configuration is the process of defining the set of options with which you want your kernel to be compiled

The set of options depends

On your hardware

On the capabilities you would like to give to your kernel


Kernel configuration (2)

The configuration is stored in the .config file at the root of kernel sources

Simple text file, key=value style

As options have dependencies, typically never edited by hand, but through graphical interfaces :

make [xconfig|gconfig|menuconfig|oldconfig]

These are targets from the main kernel Makefile. Run make help to get a list of all available targets.

To modify a kernel in a GNU/Linux distribution:the configuration files are usually released in /boot/, together with kernel images: /boot/config2.6.1711generic


make xconfig

make xconfig

The most common graphical interfaceto configure the kernel.

Make sure you readhelp > introduction: useful options!

File browser: easier to load configuration files

New search interface to look for parameters

Required Debian / Ubuntu packages:libqt3mtdev, g++


make xconfig screenshot


make xconfig search interface

Looks for a keywordin the description string

Allows to selector unselect found parameters.


Kernel configuration options

Compiled as a module (separate file)CONFIG_ISO9660_FS=m

Driver optionsCONFIG_JOLIET=yCONFIG_ZISOFS=y

Compiled statically into the kernelCONFIG_UDF_FS=y


Corresponding .config file excerpt

## CDROM/DVD Filesystems#CONFIG_ISO9660_FS=mCONFIG_JOLIET=yCONFIG_ZISOFS=yCONFIG_UDF_FS=yCONFIG_UDF_NLS=y

## DOS/FAT/NT Filesystems## CONFIG_MSDOS_FS is not set# CONFIG_VFAT_FS is not setCONFIG_NTFS_FS=m# CONFIG_NTFS_DEBUG is not setCONFIG_NTFS_RW=y

Section name(helps to locate settings in the interface)

All parameters are prefixedwith CONFIG_


Kernel option dependencies

There are dependencies between kernel options

For example, enabling a network driver requires the network stack to be enabled

Two types of dependencies

depends on dependencies. In this case, option A that depends on option B is not visible until option B is enabled

select dependencies. In this case, with option A depending on option B, when option A is enabled, option B is automatically enabled

make xconfig allows to see all options, even those that cannot be selected because of missing dependencies. In this case, they are displayed in gray


make menuconfig

make menuconfig

Useful when no graphics are available. Pretty convenient too!

Same interface found in other tools: BusyBox,buildroot...

Required Debian packages: libncursesdev


make oldconfig

make oldconfig

Needed very often!

Useful to upgrade a .config file from an earlier kernel release

Issues warnings for configuration parametersthat no longer exist in the new kernel.

Asks for values for new parameters

If you edit a .config file by hand, it's strongly recommended to run make oldconfig afterwards!



Compiling and installing the kernelfor the host system


Kernel compilation

make

in the main kernel source directory

Remember to run make j 4 if you have multiple CPU cores to speed up the compilation process

No need to run as root !

Generates

vmlinux, the raw uncompressed kernel image, at the ELF format, useful for debugging purposes, but cannot be booted

arch/<arch>/boot/*Image, the final, usually compressed, kernel image that can be booted

bzImage for x86, zImage for ARM, vmImage.gz for Blackfin, etc.

All kernel modules, spread over the kernel source tree, as .ko files.


Kernel installation

make install

Does the installation for the host system by default, so needs to be run as root

Installs

/boot/vmlinuz<version>Compressed kernel image. Same as the one in arch/<arch>/boot

/boot/System.map<version>Stores kernel symbol addresses

/boot/config<version>Kernel configuration for this version

Typically reruns the bootloader configuration utility to take into account the new kernel.


Module installation

make modules_install

Does the installation for the host system by default, so needs to be run as root

Installs all modules in /lib/modules/<version>/

kernel/Module .ko (Kernel Object) files, in the same directory structure as in the sources.

modules.aliasModule aliases for module loading utilities. Example line:alias soundservice?0 snd_mixer_oss

modules.depModule dependencies

modules.symbolsTells which module a given symbol belongs to.


Kernel cleanup targets

Cleanup generated files(to force recompiling drivers):make clean

Remove all generated files. Needed when switching from one architecture to anotherCaution: also removes your .config file!make mrproper

Also remove editor backup and patch reject files:(mainly to generate patches):make distclean



Compiling and booting LinuxLinux device files


Character device files

Accessed through a sequential flow of individual characters

Character devices can be identified by their c type(ls l):

crwrw 1 root uucp 4, 64 Feb 23 2004 /dev/ttyS0crww 1 jdoe tty 136, 1 Feb 23 2004 /dev/pts/1crw 1 root root 13, 32 Feb 23 2004 /dev/input/mouse0crwrwrw 1 root root 1, 3 Feb 23 2004 /dev/null

Example devices: keyboards, mice, parallel port, IrDA, Bluetooth port, consoles, terminals, sound, video...


Block device files

Accessed through data blocks of a given size.Blocks can be accessed in any order.

Block devices can be identified by their b type (ls l):brwrw 1 root disk 3, 1 Feb 23 2004 hda1brwrw 1 jdoe floppy 2, 0 Feb 23 2004 fd0brwrw 1 root disk 7, 0 Feb 23 2004 loop0brwrw 1 root disk 1, 1 Feb 23 2004 ram1brw 1 root root 8, 1 Feb 23 2004 sda1

Example devices: hard or floppy disks, ram disks, loop devices...


Device major and minor numbers

As you could see in the previous examples,device files have 2 numbers associated to them:

First number: major number

Second number: minor number

Major and minor numbers are used by the kernel to bind a driver to the device file. Device file names don't matter to the kernel!

To find out which driver a device file corresponds to,or when the device name is too cryptic,see Documentation/devices.txt.

http://free-electrons.com/kerneldoc/latest/devices.txt


Device file creation

Device files are not created when a driver is loaded.

They have to be created in advance:sudo mknod /dev/<device> [c|b] <major> <minor>

Examples:sudo mknod /dev/ttyS0 c 4 64sudo mknod /dev/hda1 b 3 1



Compiling and booting LinuxOverall system startup


Traditional booting sequence

Bootloader Executed by the hardware at a fixed location in ROM / Flash Initializes support for the device where the kernel image is found (local storage,network, removable media) Loads the kernel image in RAM Executes the kernel image (with a specified command line)

Kernel Uncompresses itself Initializes the kernel core and statically compiled drivers (needed to access the rootfilesystem) Mounts the root filesystem (specified by the root kernel parameter) Executes the first userspace program (specified by the init kernel parameter)

First userspace program Configures userspace and starts up system services


Kernel command line parameters

The Linux kernel can be given parameters at boot time

Kernel command line arguments are part of the bootloader configuration settings.

They are copied to RAM by the bootloader,to a location where the kernel expects them.

Useful to modify the behavior of the kernelat boot time, without having to recompile it.

Useful to perform advanced kernel and driver initialization, without having to use complex userspace scripts.


Kernel command line example

HP iPAQ h2200 PDA booting example:

root=/dev/ram0 \ Root filesystem (first ramdisk)rw \ Root filesystem mounting modeinit=/linuxrc \ First userspace programconsole=ttyS0,115200n8 \ Console (serial)console=tty0 \ Other console (framebuffer)ramdisk_size=8192 \ Misc parameters...cachepolicy=writethrough

Hundreds of command line parameters described on Documentation/kernelparameters.txt

http://free-electrons.com/kerneldoc/latest/kernel-parameters.txt


Drawbacks

Assumption that all device drivers needed to mount the root filesystem (storage and filesystem drivers) are statically compiled inside the kernel.

Assumption can be correct for most embedded systems, where the hardware is known and the kernel can be finetuned for the system.

Assumption is mostly wrong for desktop and servers, since a single kernel image should support a wide range of devices and filesystems

More flexibility was needed

Modules have this flexibility, but they are not available before mounting the root filesystem

Need to handle complex setups (RAID, NFS, etc.)


Solution

A solution is to include a small temporary root filesystem with modules, in the kernel itself. This small filesystem is called the initramfs.

This initramfs is a gzipped cpio archive of this basic root filesystem

A gzipped cpio archive is a kind of zip file, with a much simpler format

The initramfs scripts will detect the hardware, load the corresponding kernel modules, and mount the real root filesystem.

Finally the initramfs scripts will run the init application in the real root filesystem and the system can boot as usual.

The initramfs technique completely replaces init ramdisks (initrds). Initrds were used in Linux 2.4, but are no longer needed.


Booting sequence with initramfs

Bootloader Executed by the hardware at a fixed location in ROM / Flash Initializes support for the device where the images are found (local storage, network, removable media) Loads the kernel image in RAM Executes the kernel image (with a specified command line)

Kernel Uncompresses itself Initializes the kernel core and statically compiled drivers Uncompresses an initramfs cpio archive (if existing, in the kernel image or copied to memory by thebootloader) and extracts it to the kernel file cache (no mounting, no filesystem). If found in the initramfs, executes the first userspace program: /init

Userspace: /init script (what follows is just a typical scenario) Runs userspace commands to configure the device(such as network setup, mounting /proc and /sys...) Mounts a new root filesystem. Switch to it (switch_root) Runs /sbin/init

Userspace: /sbin/init Runs commands to configure the device (if not done yet in the initramfs) Starts up system services (daemons, servers) and user programs

unch

ange

d


Initramfs features and advantages

Root filesystem directly embedded in the kernel image, or copied to RAM by the bootloader, simple solution.

Just a plain compressed cpio archive extracted in the file cache. Neither needs a block nor a filesystem driver.

Simpler to mount complex filesystems from flexible userspace scripts rather than from rigid kernel code. More complexity moved out to userspace!

Possible to add non GPL files (firmware, proprietary drivers)in the filesystem. This is not linking, just file aggregation(not considered as a derived work by the GPL).


How to populate an initramfs

Using CONFIG_INITRAMFS_SOURCEin kernel configuration (General Setup section)

Give a directory containing a root filesystem.

The kernel Makefile will take care of archiving this directoryand including the archive in the compressed kernel image.

Your kernel will then boot automatically on this initramfs(provided it contains a /init file).

See Documentation/filesystems/ramfsrootfsinitramfs.txtand Documentation/earlyuserspace/README in kernel sources.

See also http://www.linuxdevices.com/articles/AT4017834659.html for a nice overview of initramfs (by Rob Landley).

http://free-electrons.com/kerneldoc/latest/filesystems/ramfs-rootfs-initramfs.txt

http://free-electrons.com/kerneldoc/latest/early-userspace/README

http://www.linuxdevices.com/articles/AT4017834659.html


Summary

For embedded systems, two interesting solutions

No initramfs: all needed drivers are included inside the kernel, and the final root filesystem is mounted directly

Everything inside the initramfs


Practical lab – Host kernel compiling

Compile a recent Linux kernel for your GNU/Linux workstation

Update the bootloader configuration

Update the initramfs archive used in the early boot process.



Compiling and booting LinuxRoot filesystem over NFS


Usefulness of rootfs on NFS

Once networking works, your root filesystem could be a directory on your GNU/Linux development host, exported by NFS (Network File System). This is very convenient for system development:

Makes it very easy to update files (driver modules in particular) on the root filesystem, without rebooting. Much faster than through the serial port.

Can have a big root filesystem even if you don't have support for internal or external storage yet.

The root filesystem can be huge. You can even build native compiler tools and build all the tools you need on the target itself (better to crosscompile though).


NFS boot setup (1)

On the host (NFS server)

Install an NFS server (example: Debian, Ubuntu)sudo aptget install nfskernelserver

Add the exported directory to your /etc/exports file:/home/rootfs 192.168.0.202(rw,no_root_squash,no_subtree_check)

Start or restart your NFS server (example: Debian, Ubuntu)sudo /etc/init.d/nfskernelserver restart

client address NFS server options


NFS boot setup (2)

On the target (NFS client)

Compile your kernel with CONFIG_NFS_FS=y,CONFIG_IP_PNP=y (configure IP at boot time)and CONFIG_ROOT_NFS=y

Boot the kernel with the below command line options:root=/dev/nfs

virtual deviceip=192.168.1.111

local IP addressnfsroot=192.168.1.110:/home/nfsroot

NFS server IP address Directory on the NFS server



Compiling and booting LinuxCrosscompiling the kernel


Crosscompiling the kernel

When you compile a Linux kernel for another CPU architecture

Much faster than compiling natively, when the target system is much slower than your GNU/Linux workstation.

Much easier as development tools for your GNU/Linux workstation are much easier to find.

To make the difference with a native compiler, crosscompiler executables are prefixed by the name of the target system, architecture and sometimes library. Examples:mipslinuxgccm68klinuxuclibcgccarmlinuxgnueabigcc


Specifying crosscompilation

The CPU architecture and crosscompiler prefix are defined through the ARCH and CROSS_COMPILE variables in the toplevel Makefile.

The Makefile defines CC = $(CROSS_COMPILE)gccSee comments in Makefile for details

The easiest solution is to modify the Makefile.Example, ARM platform, crosscompiler: armlinuxgccARCH ?= armCROSS_COMPILE ?= armlinux

Other solutions

Pass ARCH and CROSS_COMPILE on the make command line

Define ARCH and CROSS_COMPILE as environment variables

Don't forget to have the values properly set at all steps, otherwise the kernel configuration and build system gets confused


Configuring the kernel

make xconfig

Same as in native compiling

The set of available options will be different

Don't forget to set the right board / machine type!


Readymade config files

assabet_defconfig integrator_defconfig mainstone_defconfigbadge4_defconfig iq31244_defconfig mx1ads_defconfigbast_defconfig iq80321_defconfig neponset_defconfigcerfcube_defconfig iq80331_defconfig netwinder_defconfigclps7500_defconfig iq80332_defconfig omap_h2_1610_defconfigebsa110_defconfig ixdp2400_defconfig omnimeter_defconfigedb7211_defconfig ixdp2401_defconfig pleb_defconfigenp2611_defconfig ixdp2800_defconfig pxa255idp_defconfigep80219_defconfig ixdp2801_defconfig rpc_defconfigepxa10db_defconfig ixp4xx_defconfig s3c2410_defconfigfootbridge_defconfig jornada720_defconfig shannon_defconfigfortunet_defconfig lart_defconfig shark_defconfigh3600_defconfig lpd7a400_defconfig simpad_defconfigh7201_defconfig lpd7a404_defconfig smdk2410_defconfigh7202_defconfig lubbock_defconfig versatile_defconfighackkit_defconfig lusl7200_defconfig

arch/arm/configs example

http://lxr.free-electrons.com/source/arch/arm/configs/


Using readymade config files

Default configuration files available for many boards / machines!Check if one exists in arch/<arch>/configs/ for your target.

Example: if you found an acme_defconfig file, you can run:make acme_defconfig

Using arch/<arch>/configs/ is a very good good way of releasing a default configuration file for a group of users or developers.

Like all make commands, you mustrun make <machine>_defconfigin the toplevel source directory.


Building the kernel

Runmake

Copyarch/<arch>/boot/zImageto the target storage

You can customize arch/<arch>/boot/install.sh so that make install does this automatically for you.

make INSTALL_MOD_PATH=<dir>/ modules_installand copy <dir>/lib/modules/ to /lib/modules/ on the target storage.


Lab – Module development environment

Set up a crosscompiling environment

Crosscompile the kernel for an arm target platform and boot it through NFS


Embedded Linux kernel usage

Using kernel modules


Loadable kernel modules

Modules: add a given functionality to the kernel (drivers, filesystem support, and many others).

Can be loaded and unloaded at any time, only when their functionality is need.

Modules make it easy to develop drivers without rebooting: load, test, unload, rebuild, load...

Useful to keep the kernel image size to the minimum (essential in GNU/Linux distributions for PCs).

Also useful to reduce boot time: you don't spend time initializing devices and kernel features that you only need later.

Caution: once loaded, have full access to the whole kernel address space. No particular protection.


Module dependencies

Some kernel modules can depend on other modules,which need to be loaded first.

Example: the usbstorage module depends on the scsi_mod, libusual and usbcore modules.

Dependencies are describedin /lib/modules/<kernelversion>/modules.depThis file is generated when you run make modules_install.

You can also update the modules.dep fileby yourself, by running (as root):depmod a [<version>]


Kernel log

When a new module is loaded,related information is available in the kernel log.

The kernel keeps its messages in a circular buffer(so that it doesn't consume more memory with many messages)

Kernel log messages are available through the dmesg command. (“diagnostic message”)

Kernel log messages are also displayed in the system console (messages can be filtered by level using /proc/sys/kernel/printk)


Module utilities (1)

modinfo <module_name>modinfo <module_path>.koGets information about a module: parameters, license, description and dependencies.Very useful before deciding to load a module or not.

sudo insmod <module_path>.koTries to load the given module. The full path to the module object file must be given.


Understanding module loading issues

When loading a module fails,insmod often doesn't give you enough details!

Details are often available in the kernel log.

Example:> sudo insmod ./intr_monitor.koinsmod: error inserting './intr_monitor.ko': 1 Device or resource busy> dmesg[17549774.552000] Failed to register handler for irq channel 2



sudo modprobe <module_name>Most common usage of modprobe: tries to load all the modules the given module depends on, and then this module. Lots of other options are available. Modprobe automatically looks in /lib/modules/<version>/ for the object file corresponding to the given module name.

lsmodDisplays the list of loaded modulesCompare its output with the contents of /proc/modules!



sudo rmmod <module_name>Tries to remove the given module.Will only be allowed if the module is no longer in use(for example, no more processes opening a device file)

sudo modprobe r <module_name>Tries to remove the given module and all dependent modules (which are no longer needed afterthe module removal)


Passing parameters to modules

Find available parameters:modinfo sndintel8x0m

Through insmod:sudo insmod ./sndintel8x0m.ko index=2

Through modprobe:Set parameters in /etc/modprobe.conf or in any file in /etc/modprobe.d/:options sndintel8x0m index=2

Through the kernel command line,when the driver is built statically into the kernel:sndintel8x0m.index=2

driver namedriver parameter namedriver parameter value


Useful reading

Linux Kernel in a Nutshell, Dec 2006

By Greg KroahHartman, O'Reillyhttp://www.kroah.com/lkn/

A good reference book and guide on configuring,compiling and managing the Linux kernel sources.

Freely available online!Great companion to the printed bookfor easy electronic searches!Available as single PDF file onhttp://freeelectrons.com/community/kernel/lkn/

http://www.kroah.com/lkn/

http://free-electrons.com/community/kernel/lkn/



Embedded Linuxkernel and driver

development


Free Electrons

© Copyright 20042009, Free Electrons.Creative Commons BYSA 3.0 licenseLatest update: Mar 5, 2010, Document sources, updates and translations:http://freeelectrons.com/docs/kernelCorrections, suggestions, contributions and translations are welcome!

http://free-electrons.com/docs/kernel


Contents

Driver development

Loadable kernel modules

Memory management

I/O memory and ports

Character drivers

Processes and scheduling

Sleeping, Interrupt management

Handling concurrency

Debugging

mmap

Device model, sysfs



Driver developmentLoadable kernel modules


hello module

/* hello.c */#include <linux/init.h>#include <linux/module.h>#include <linux/kernel.h>

static int __init hello_init(void){ printk(KERN_ALERT "Good morrow"); printk(KERN_ALERT "to this fair assembly.\n"); return 0;}

static void __exit hello_exit(void){ printk(KERN_ALERT "Alas, poor world, what treasure"); printk(KERN_ALERT "hast thou lost!\n");}

module_init(hello_init);module_exit(hello_exit);MODULE_LICENSE("GPL");MODULE_DESCRIPTION("Greeting module");MODULE_AUTHOR("William Shakespeare");

__init:removed after initialization(static kernel or module).

__exit: discarded whenmodule compiled staticallyinto the kernel.

Example available on http://freeelectrons.com/doc/c/hello.c

http://lxr.free-electrons.com/source/include/linux/init.h

http://lxr.free-electrons.com/source/include/linux/module.h

http://lxr.free-electrons.com/source/include/linux/kernel.h

http://lxr.free-electrons.com/ident?i=__init


http://lxr.free-electrons.com/ident?i=KERN_ALERT



http://lxr.free-electrons.com/ident?i=__exit





http://lxr.free-electrons.com/ident?i=module_init

http://lxr.free-electrons.com/ident?i=module_exit

http://lxr.free-electrons.com/ident?i=MODULE_LICENSE

http://lxr.free-electrons.com/ident?i=MODULE_DESCRIPTION

http://lxr.free-electrons.com/ident?i=MODULE_AUTHOR



http://free-electrons.com/doc/c/hello.c


Module license

Several usages

Used to restrict the kernel functions that the module can use if it isn't a GPLlicensed module

Difference between EXPORT_SYMBOL() and EXPORT_SYMBOL_GPL()

Used by kernel developers to identify issues coming from proprietary drivers, which they can't do anything about(“Tainted” kernel notice in kernel crashes and oopses).

Useful for users to check that their system is 100% free(check /proc/sys/kernel/tainted)

Values

GPL, GPL v2, GPL and additional rights, Dual MIT/GPL, Dual BSD/GPL, Dual MPL/GPL, Proprietary


Compiling a outoftree module

The below Makefile should be reusable for any outoftree Linux 2.6 module

Intree modules are covered later

Just run make to build the hello.ko file

Caution: make sure there is a [Tab] character at the beginning of the $(MAKE) line (make syntax)

ifneq ($(KERNELRELEASE),)objm := hello.oelseKDIR := /path/to/kernel/sourcesall:

$(MAKE) C $(KDIR) M=`pwd` modulesendif

[Tab]!(no spaces)

Either full kernel source directory (configured and compiled) or just kernel headers directory (minimum needed)


Modules and kernel version

To be compiled, a kernel module needs access to the kernel headers, containing the functions, types and constants definitions

Two solutions

Full kernel sources

Only kernel headers (linuxheaders* packages in Debian/Ubuntu distributions)

The sources or headers must be configured

Many macros or functions depend on the configuration

A kernel module compiled against version X of kernel headers will not load in kernel version Y

modprobe/insmod will say « Invalid module format »


Symbols exported to modules

From a kernel module,only a limited number of kernel functions can be called

Functions and variables have to be explicitly exportedby the kernel to be visible from a kernel module

Two macros are used in the kernelto export functions and variables :

EXPORT_SYMBOL(symbolname), which exports a function or variable to all modules

EXPORT_SYMBOL_GPL(symbolname), which exports a function or variable only to GPL modules

A normal driver should not need any nonexported function.


hello module with parameters

/* hello_param.c */#include <linux/init.h>#include <linux/module.h>#include <linux/moduleparam.h>

MODULE_LICENSE("GPL");

/* A couple of parameters that can be passed in: how many times we say hello, and to whom */

static char *whom = "world";module_param(whom, charp, 0);

static int howmany = 1;module_param(howmany, int, 0);

static int __init hello_init(void){ int i; for (i = 0; i < howmany; i++) printk(KERN_ALERT "(%d) Hello, %s\n", i, whom); return 0;}

static void __exit hello_exit(void){ printk(KERN_ALERT "Goodbye, cruel %s\n", whom);}

module_init(hello_init);module_exit(hello_exit);

Thanks toJonathan Corbetfor the example!

Example available on http://freeelectrons.com/doc/c/hello_param.c

http://lxr.free-electrons.com/source/include/linux/init.h

http://lxr.free-electrons.com/source/include/linux/module.h

http://lxr.free-electrons.com/source/include/linux/moduleparam.h

http://lxr.free-electrons.com/ident?i=MODULE_LICENSE

http://lxr.free-electrons.com/ident?i=module_param



http://lxr.free-electrons.com/ident?i=__printk



http://lxr.free-electrons.com/ident?i=__printk


http://lxr.free-electrons.com/ident?i=module_init

http://lxr.free-electrons.com/ident?i=module_exit

http://free-electrons.com/doc/c/hello_param.c


Declaring a module parameter

#include <linux/moduleparam.h>

module_param(name, /* name of an already defined variable */type, /* either byte, short, ushort, int, uint, long,

ulong, charp, or bool. (checked at compile time!) */

perm /* for /sys/module/<module_name>/parameters/<param> 0: no such module parameter value file */

);

Example

int irq=5;module_param(irq, int, S_IRUGO);



http://lxr.free-electrons.com/ident?i=byte

http://lxr.free-electrons.com/ident?i=short

http://lxr.free-electrons.com/ident?i=ushort

http://lxr.free-electrons.com/ident?i=uint

http://lxr.free-electrons.com/ident?i=ulong

http://lxr.free-electrons.com/ident?i=byte

http://lxr.free-electrons.com/ident?i=bool


http://lxr.free-electrons.com/ident?i=S_IRUGO


Declaring a module parameter array

#include <linux/moduleparam.h>

module_param_array(name, /* name of an already defined array */type, /* same as in module_param */num, /* number of elements in the array, or NULL (no check?) */perm /* same as in module_param */

);

Example

static int base[MAX_DEVICES] = { 0x820, 0x840 };module_param_array(base, int, NULL, 0);


http://lxr.free-electrons.com/ident?i=module_param_array

http://lxr.free-electrons.com/ident?i=module_param_array



Driver developmentAdding sources to the kernel tree


New driver in kernel sources (1)

To add a new driver to the kernel sources:

Add your new source file to the appropriate source directory.Example: drivers/usb/serial/navman.c

Describe the configuration interface for your new driverby adding the following lines to the Kconfig file in this directory:

config USB_SERIAL_NAVMAN tristate "USB Navman GPS device" depends on USB_SERIAL help To compile this driver as a module, choose M here: the module will be called navman.


New driver in kernel sources (2)

Add a line in the Makefile file based on the Kconfig setting:

obj$(CONFIG_USB_SERIAL_NAVMAN) += navman.o

Run make xconfig and see your new options!

Run make and your new files are compiled!

See Documentation/kbuild/ for details

http://free-electrons.com/kerneldoc/latest/kbuild/


How to create Linux patches

Download the latest kernel sources

Make a copy of these sources:cp a linux2.6.9rc2/ linux2.6.9rc2patch/

Apply your changes to the copied sources, and test them.

Run make distclean to keep only source files.

Create a patch file:diff Nur linux2.6.9rc2/ \linux2.6.9rc2patch/ > patchfile

Always compare the whole source structures(suitable for patch p1)

Patch file name: should recall what the patch is about.

If you need to manage a lot of patches, use git or quilt instead

Thanks to Nicolas Rougier (Copyright 2003, http://webloria.loria.fr/~rougier/) for the Tux image

http://webloria.loria.fr/~rougier/


Practical lab – Writing modules

Write a kernel module with several capabilities, including module parameters.

Access kernel internals from your module.

Setup the environment to compile it



Driver developmentMemory management


Physical and virtual memory

0x00000000

0xFFFFFFFF

Physical address space

RAM 0

RAM 1

Flash

I/O memory 1

I/O memory 2

I/O memory 3

MMU

MemoryManagement

Unit

CPU

Virtual address spaces0xFFFFFFFFF

0x00000000

Kernel0xFFFFFFFF

0x00000000

Process1

0xFFFFFFFF

0x00000000

Process2All the processes have their own virtual address space, and run as if they had access to the whole address space.

0xC0000000

0xC0000000Kernel


kmalloc and kfree

Basic allocators, kernel equivalents of glibc's malloc and free.

#include <linux/slab.h>

static inline void *kmalloc(size_t size, int flags);size: number of bytes to allocateflags: priority (explained in a few pages)

void kfree (const void *objp);

Example: (drivers/infiniband/core/cache.c)struct ib_update_work *work;work = kmalloc(sizeof *work, GFP_ATOMIC);...kfree(work);

http://lxr.free-electrons.com/source/include/linux/slab.h


http://lxr.free-electrons.com/ident?i=size_t

http://lxr.free-electrons.com/ident?i=kfree

http://lxr.free-electrons.com/source/drivers/infiniband/core/cache.c


http://lxr.free-electrons.com/ident?i=GFP_ATOMIC

http://lxr.free-electrons.com/ident?i=kfree


kmalloc features

Quick (unless it's blocked waiting for memory to be freed).

Doesn't initialize the allocated area.

The allocated area is contiguous in physical RAM.

Allocates by 2n sizes, and uses a few management bytes.So, don't ask for 1024 when you need 1000! You'd get 2048!

Caution: drivers shouldn't try to kmallocmore than 128 KB (upper limit in some architectures).

Minimum memory consumption:32 or 64 bytes (page size dependent).



Main kmalloc flags

Defined in include/linux/gfp.h (GFP: __get_free_pages)

GFP_KERNELStandard kernel memory allocation. May block. Fine for most needs.

GFP_ATOMICRAM allocated from code which is not allowed to block (interrupt handlers) or which doesn't want to block (critical sections). Never blocks.

GFP_DMAAllocates memory in an area of the physical memory usable for DMA transfers.

http://lxr.free-electrons.com/source/include/linux/gfp.h

http://lxr.free-electrons.com/ident?i=__get_free_pages

http://lxr.free-electrons.com/ident?i=GFP_KERNEL

http://lxr.free-electrons.com/ident?i=GFP_ATOMIC


Related allocation functions

Again, names similar to those of C library functions

static inline void *kzalloc(size_t size, gfp_t flags);

Zeroes the allocated buffer.

static inline void *kcalloc(size_t n, size_t size, gfp_t flags);

Allocates memory for an array of n elements of size size,and zeroes its contents.

void * __must_check krealloc(const void *, size_t, gfp_t);

Changes the size of the given buffer.

http://lxr.free-electrons.com/ident?i=kzalloc

http://lxr.free-electrons.com/ident?i=kcalloc

http://lxr.free-electrons.com/ident?i=krealloc


Slab caches and memory pools

Slab caches: make it possible to allocate multipleobjects of the same size, without wasting RAM.

So far, mainly used in core subsystems,but not much in device drivers(except USB and SCSI drivers)

Memory pools: pools of preallocated objects,to increase the chances of allocations to succeed.Often used with file caches.


Available allocators

Memory is allocated using slabs (groups of one or more continuous pages from which objects are allocated). Several compatible slab allocators are available:

SLAB: original, well proven allocator in Linux 2.6.

SLOB: much simpler. More space efficient but doesn't scale well. Saves a few hundreds of KB in small systems (depends on CONFIG_EMBEDDED)

SLUB: the new default allocator since 2.6.23, simpler than SLAB, scaling much better (in particular for huge systems) and creating less fragmentation.


Allocating by pages

More appropriate when you need big slices of RAM:

A page is usually 4K, but can be made greater in some architectures(sh, mips: 4, 8, 16 or 64K, but not configurable in i386 or arm).

unsigned long get_zeroed_page(int flags);Returns a pointer to a free page and fills it up with zeros

unsigned long __get_free_page(int flags);Same, but doesn't initialize the contents

unsigned long __get_free_pages(int flags, unsigned int order);Returns a pointer on an area of several contiguous pages in physical RAM.order: log

2(<number_of_pages>)

If variable, can be computed from the size with the get_order function.Maximum: 8192 KB (MAX_ORDER=11 in include/linux/mmzone.h),except in a few architectures when overwritten with CONFIG_FORCE_MAX_ZONEORDER.

http://lxr.free-electrons.com/ident?i=get_zeroed_page

http://lxr.free-electrons.com/ident?i=__get_free_page


http://lxr.free-electrons.com/ident?i=get_order

http://lxr.free-electrons.com/source/include/linux/mmzone.h


Freeing pages

void free_page(unsigned long addr);

void free_pages(unsigned long addr, unsigned int order);Need to use the same order as in allocation.

http://lxr.free-electrons.com/ident?i=free_page

http://lxr.free-electrons.com/ident?i=free_pages


vmalloc

vmalloc can be used to obtain contiguous memory zonesin virtual address space (even if pages may not be contiguous in physical memory).

void *vmalloc(unsigned long size);

void vfree(void *addr);

http://lxr.free-electrons.com/ident?i=vmalloc

http://lxr.free-electrons.com/ident?i=vmalloc

http://lxr.free-electrons.com/ident?i=vfree


Memory utilities

void * memset(void * s, int c, size_t count);Fills a region of memory with the given value.

void * memcpy(void * dest, const void *src, size_t count);Copies one area of memory to another.Use memmove with overlapping areas.

Lots of functions equivalent to standard C library ones defined in include/linux/string.hand in include/linux/kernel.h (sprintf, etc.)

http://lxr.free-electrons.com/ident?i=memset


http://lxr.free-electrons.com/ident?i=memcpy


http://lxr.free-electrons.com/ident?i=memmove

http://lxr.free-electrons.com/source/include/linux/string.h


http://lxr.free-electrons.com/ident?i=sprintf


Kernel memory debugging

Debugging features available since 2.6.31

KmemcheckDynamic checker for access to uninitialized memory.Only available on x86 so far, but will help to improve architecture independent code anyway.See Documentation/kmemcheck.txt for details.

KmemleakDynamic checker for memory leaksThis feature is available for all architectures.See Documentation/kmemleak.txt for details.

Both have a significant overhead. Only use them in development!

http://free-electrons.com/kerneldoc/latest/kmemcheck.txt

http://free-electrons.com/kerneldoc/latest/kmemleak.txt



Driver developmentI/O memory and ports


Port I/O vs. MemoryMapped I/O

MMIO

Same address bus to address memory and I/O devices

Access to the I/O devices using regular instructions

Most widely used I/O method across the different architectures supported by Linux

PIO

Different address spaces for memory and I/O devices

Uses a special class of CPU instructions to access I/O devices

Example on x86: IN and OUT instructions


Requesting I/O ports

Tells the kernel which driver is using which I/O ports

Allows to prevent other drivers from using the same I/O ports, but is purely voluntary.

struct resource *request_region( unsigned long start, unsigned long len, char *name);Tries to reserve the given region and returns NULL if unsuccessful.request_region(0x0170, 8, "ide1");

void release_region( unsigned long start, unsigned long len);

/proc/ioports example (x86)

0000001f : dma100200021 : pic100400043 : timer000500053 : timer10060006f : keyboard00700077 : rtc0080008f : dma page reg00a000a1 : pic200c000df : dma200f000ff : fpu0100013f : pcmcia_socket001700177 : ide101f001f7 : ide003760376 : ide10378037a : parport003c003df : vga+03f603f6 : ide003f803ff : serial0800087f : 0000:00:1f.008000803 : PM1a_EVT_BLK08040805 : PM1a_CNT_BLK0808080b : PM_TMR08200820 : PM2_CNT_BLK0828082f : GPE0_BLK...


Accessing I/O ports

Functions to read/write bytes, word and longs to I/O ports:

unsigned in[bwl](unsigned long *addr);void out[bwl](unsigned port, unsigned long *addr);

And the strings variants: often more efficient than the corresponding C loop, if the processor supports such operations!void ins[bwl](unsigned port, void *addr, unsigned long count);void outs[bwl](unsigned port, void *addr, unsigned long count);


Requesting I/O memory

Functions equivalent to request_region() and release_region(), but for I/O memory.struct resource * request_mem_region( unsigned long start, unsigned long len, char *name);

void release_mem_region( unsigned long start, unsigned long len);

/proc/iomem example

000000000009efff : System RAM0009f0000009ffff : reserved000a0000000bffff : Video RAM area000c0000000cffff : Video ROM000f0000000fffff : System ROM001000003ffadfff : System RAM 001000000030afff : Kernel code 0030b000003b4bff : Kernel data3ffae0003fffffff : reserved40000000400003ff : 0000:00:1f.14000100040001fff : 0000:02:01.0 4000100040001fff : yenta_socket4000200040002fff : 0000:02:01.1 4000200040002fff : yenta_socket40400000407fffff : PCI CardBus #034080000040bfffff : PCI CardBus #0340c0000040ffffff : PCI CardBus #0741000000413fffff : PCI CardBus #07a0000000a0000fff : pcmcia_socket0a0001000a0001fff : pcmcia_socket1e0000000e7ffffff : 0000:00:00.0e8000000efffffff : PCI Bus #01 e8000000efffffff : 0000:01:00.0...

http://lxr.free-electrons.com/ident?i=request_mem_region

http://lxr.free-electrons.com/ident?i=release_mem_region


Mapping I/O memory in virtual memory

To access I/O memory, drivers need to have a virtual address that the processor can handle, because I/O memory is not mapped by default in virtual memory.

The ioremap functions satisfy this need:

#include <asm/io.h>;

void *ioremap(unsigned long phys_addr, unsigned long size);void iounmap(void *address);

Caution: check that ioremap doesn't return a NULL address!

http://lxr.free-electrons.com/ident?i=ioremap

http://lxr.free-electrons.com/source/include/asm-i386/io.h


http://lxr.free-electrons.com/ident?i=iounmap



Accessing MMIO devices

Directly reading from or writing to addresses returned by ioremap (“pointer dereferencing”) may not work on some architectures.

To do PCIstyle, littleendian accesses, conversion being done automaticallyunsigned read[bwl](void *addr);void write[bwl](unsigned val, void *addr);

To do raw access, without endianess conversionunsigned __raw_read[bwl](void *addr);void __raw_write[bwl](unsigned val, void *addr);


New API for mixed accesses

A new API allows to write drivers that can work on either devices accessed over PIO or MMIO. Some drivers use it, but there doesn't seem to be a consensus in the kernel community around it.

Mapping

For PIO: ioport_map() and ioport_unmap(). They don't really map, but they return a special cookie.

For MMIO: ioremap() and iounmap(). As usual.

Access, works both on addresses returned by ioport_map() and ioremap()

ioread[8/16/32]() and iowrite[8/16/32] for single access

ioread_rep[8/16/32]() and iowrite_rep[8/16/32]() for repeated accesses


Avoiding I/O access issues

Caching on I/O ports or memory already disabled

Use the macros, they do the right thing for your architecture

The compiler and/or CPU can reorder memory accesses, which might cause troubles for your devices is they expect one register to be read/written before another one.

Memory barriers are available to prevent this reordering

rmb() is a read memory barrier, prevents reads to cross the barrier

wmb() is a write memory barrier

mb() is a readwrite memory barrier


/dev/mem

Used to provide userspace applicationswith direct access to physical addresses.

Usage: open /dev/mem and read or write at given offset.What you read or write is the valueat the corresponding physical address.

Used by applications such as the X serverto write directly to device memory.

Since 2.6.26 (x86 only, 2.6.32 status): only nonRAM can be accessed for security reasons, unless explicitly configured otherwise (CONFIG_STRICT_DEVMEM).


Practical lab – I/O memory and ports

Make a remote connection to your board through ssh.

Access the system console through the network.

Reserve the I/O memory addresses used by the serial port.

Read device registers and write data to them, to send characters on the serial port.



Driver developmentCharacter drivers


Usefulness of character drivers

Except for storage device drivers, most drivers for devices with input and output flows are implemented as character drivers.

So, most drivers you will face will be character driversYou will regret if you sleep during this part!


Creating a character driver

Userspace needs

The name of a device file in /dev to interact with the device driver through regular file operations (open, read, write, close...)

The kernel needs

To know which driver is in charge of device files with a given major / minor number pair

For a given driver, to have handlers (“file operations”) to execute when userspace opens, reads, writes or closes the device file.

/dev/foo

major / minor

Readhandler

Writehandler

Device driver

read write

Readbuffer

Writestring

Cop

y to

use

r

Cop

y fr

o m u

ser

Userspace

Kernel space


Implementing a character driver

Four major steps

Implement operations corresponding to the system calls an application can apply to a file : file operations

Define a file_operations structure associating function pointers to their implementation in your driver

Reserve a set of major and minors for your driver

Tell the kernel to associate the reserved major and minor to your file operations

This is a very common design scheme in the Linux kernel

A common kernel infrastructure defines a set of operations to be implemented by a driver and functions to register your driver

Your driver only needs to implement this set of welldefined operations


File operations

Before registering character devices, you have to define file_operations (called fops) for the device files.

The file_operations structure is generic to all files handled by the Linux kernel. It contains many operations that aren't needed for character drivers.

Here are the most important operations for a character driver. All of them are optional.

struct file_operations { [...] ssize_t (*read) (struct file *, char __user *, size_t, loff_t *); ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); int (*ioctl) (struct inode *, struct file *, unsigned int, unsigned long); int (*mmap) (struct file *, struct vm_area_struct *); int (*open) (struct inode *, struct file *); int (*release) (struct inode *, struct file *); [...]};

http://lxr.free-electrons.com/ident?i=file_operations


open() and release()

int foo_open (struct inode *i, struct file *f)

Called when userspace opens the device file.

inode is a structure that uniquely represent a file in the system (be it a regular file, a directory, a symbolic link, a character or block device)

file is a structure created every time a file is opened. Several file structures can point to the same inode structure.

Contains informations like the current position, the opening mode, etc.Has a void *private_data pointer that one can freely use.A pointer to the file structure is passed to all other operations

int foo_release(struct inode *i, struct file *f)

Called when userspace closes the file.

http://lxr.free-electrons.com/ident?i=inode

http://lxr.free-electrons.com/ident?i=file

http://lxr.free-electrons.com/ident?i=inode



read()

ssize_t foo_read ( struct file *f, __user char *buf,size_t sz, loff_t *off)

Called when userspace uses the read() system call on the device.

Must read data from the device, write at most sz bytes in the userspace buffer buf, and update the current position in the file off. f is a pointer to the same file structure that was passed in the open() operation

Must return the number of bytes read.

On Unix, read() operations typically block when there isn't enough data to read from the device


write()

ssize_t foo_write( struct file *f,__user const char *buf, size_t sz ,loff_t *off)

Called when userspace uses the write() system call on the deviceThe opposite of read, must read at most sz bytes from buf, write it to the device, update off and return the number of bytes written.


Exchanging data with userspace (1)

Kernel code isn't allowed to directly access userspace memory, using memcpy or direct pointer dereferencing

Doing so does not work on some architectures

If the address passed by the application was invalid, the application would segfault

To keep the kernel code portable and have proper error handling, your driver must use special kernel functions to exchange data with userspace


Exchanging data with userspace (2)

A single value

get_user(v, p);The kernel variable v gets the value pointer by the userspace pointer p

put_user(v, p);The value pointed by the userspace pointer p is set to the contents of the kernel variable v.

A buffer

unsigned long copy_to_user(void __user *to,const void *from, unsigned long n);

unsigned long copy_from_user(void *to,const void __user *from, unsigned long n);

The return value must be checked. Zero on success, nonzero on failure. If nonzero, the convention is to return EFAULT.


static ssize_tacme_read(struct file *file, char __user *buf, size_t count, loff_t *ppos){ /* The acme_buf address corresponds to a device I/O memory area */ /* of size acme_bufsize, obtained with ioremap() */ int remaining_size, transfer_size;

remaining_size = acme_bufsize (int) (*ppos); // bytes left to transfer if (remaining_size == 0) { /* All read, returning 0 (End Of File) */

return 0; }

/* Size of this transfer */ transfer_size = min(remaining_size, (int) count); if (copy_to_user(buf /* to */, acme_buf + *ppos /* from */, transfer_size)) { return EFAULT; } else { /* Increase the position in the open file */ *ppos += transfer_size; return transfer_size; }}

read operation example

Read method Piece of code available inhttp://freeelectrons.com/doc/c/acme.c

http://lxr.free-electrons.com/source/include/linux/types.h#071

http://lxr.free-electrons.com/source/include/linux/fs.h#632

http://lxr.free-electrons.com/source/include/linux/compiler.h#007



http://lxr.free-electrons.com/source/include/linux/kernel.h#256

http://lxr.free-electrons.com/ident?i=copy_to_user

http://lxr.free-electrons.com/source/include/asm-generic/errno-base.h#017

http://free-electrons.com/doc/c/acme.c


write operation example

static ssize_tacme_write(struct file *file, const char __user *buf, size_t count, loff_t *ppos){ int remaining_bytes;

/* Number of bytes not written yet in the device */ remaining_bytes = acme_bufsize (*ppos); if (count > remaining_bytes) {

/* Can't write beyond the end of the device */ return EIO;

}

if (copy_from_user(acme_buf + *ppos /* to */, buf /* from */, count)) { return EFAULT; } else {

/* Increase the position in the open file */ *ppos += count; return count; }}

Write method Piece of code available inhttp://freeelectrons.com/doc/c/acme.c



http://lxr.free-electrons.com/source/include/linux/compiler.h#007




http://lxr.free-electrons.com/ident?i=copy_from_user




ioctl()

int ioctl(struct inode *i, struct file *f, unsigned int cmd, unsigned long arg)

Associated to the ioctl() system call

Allows to extend the driver capabilities beyond the limited read/write API

For example: changing the speed of a serial port, setting video output format, querying a device serial number...

cmd is a number identifying the operation to perform

arg is the optional argument passed as third argument of the ioctl() system call. Can be an integer, an address, etc.

The semantic of cmd and arg is driverspecific.


ioctl() example : kernel side

static long phantom_ioctl(struct file *file, unsigned int cmd, unsigned long arg){ struct phm_reg r; void __user *argp = (void __user *)arg;

switch (cmd) { case PHN_SET_REG: if (copy_from_user(&r, argp, sizeof(r))) return EFAULT;

/* Do something */ break;

case PHN_GET_REG: if (copy_from_user(&r, argp, sizeof(r))) return EFAULT;

/* Do something */ break;

default: return ENOTTY;

}

return 0;}


Ioctl() example : application side

int main(void){

int fd, ret;struct phm_reg reg;

fd = open(“/dev/phantom”);assert(fd > 0);

reg.field1 = 42;reg.field2 = 67;

ret = ioctl(fd, PHN_SET_REG, & reg);assert(ret == 0);

return 0;}


file operations definition example (3)

Defining a file_operations structure:

#include <linux/fs.h>

static struct file_operations acme_fops ={

.owner = THIS_MODULE,

.read = acme_read,

.write = acme_write,};

You just need to supply the functions you implemented! Defaults for other functions (such as open, release...) are fine if you do not implement anything special.


http://lxr.free-electrons.com/source/include/linux/fs.h



Information on registered devices

Registered devices are visible in /proc/devices:

Character devices: Block devices: 1 mem 1 ramdisk 4 /dev/vc/0 3 ide0 4 tty 8 sd 4 ttyS 9 md 5 /dev/tty 22 ide1 5 /dev/console 65 sd 5 /dev/ptmx 66 sd 6 lp 67 sd 10 misc 68 sd 13 input 14 sound ... Major

numberRegistered

name


dev_t data type

Kernel data type to represent a major / minor number pair

Also called a device number.

Defined in <linux/kdev_t.h>Linux 2.6: 32 bit size (major: 12 bits, minor: 20 bits)

Macro to compose the device number:MKDEV(int major, int minor);

Macro to extract the minor and major numbers:MAJOR(dev_t dev);MINOR(dev_t dev);

http://lxr.free-electrons.com/source/include/linux/kdev_t.h

http://lxr.free-electrons.com/ident?i=MKDEV

http://lxr.free-electrons.com/ident?i=MAJOR

http://lxr.free-electrons.com/ident?i=dev_t

http://lxr.free-electrons.com/ident?i=MINOR



Registering device numbers (1)

#include <linux/fs.h>

int register_chrdev_region(dev_t from, /* Starting device number */unsigned count, /* Number of device numbers */const char *name); /* Registered name */

Returns 0 if the allocation was successful.

Example

static dev_t acme_dev = MKDEV(202, 128);

if (register_chrdev_region(acme_dev, acme_count, “acme”)) {printk(KERN_ERR “Failed to allocate device number\n”);...

http://lxr.free-electrons.com/source/include/linux/fs.h

http://lxr.free-electrons.com/ident?i=register_chrdev_region





http://lxr.free-electrons.com/ident?i=KERN_ERR


Registering device numbers (2)

If you don't have fixed device numbers assigned to your driver

Better not to choose arbitrary ones.There could be conflicts with other drivers.

The kernel API offers a alloc_chrdev_region functionto have the kernel allocate free ones for you. You can find the allocated major number in /proc/devices.

http://lxr.free-electrons.com/ident?i=alloc_chrdev_region


Character device registration (1)

The kernel represents character drivers with a cdev structure

Declare this structure globally (within your module):#include <linux/cdev.h>static struct cdev acme_cdev;

In the init function, initialize the structure:cdev_init(&acme_cdev, &acme_fops);

http://lxr.free-electrons.com/ident?i=cdev

http://lxr.free-electrons.com/source/include/linux/cdev.h


http://lxr.free-electrons.com/ident?i=cdev_init


Character device registration (2)

Then, now that your structure is ready, add it to the system:int cdev_add(

struct cdev *p, /* Character device structure */dev_t dev, /* Starting device major / minor number

*/unsigned count); /* Number of devices */

Example (continued):if (cdev_add(&acme_cdev, acme_dev, acme_count)) {printk (KERN_ERR “Char driver registration failed\n”);...

http://lxr.free-electrons.com/ident?i=cdev_add







Character device unregistration

First delete your character device:void cdev_del(struct cdev *p);

Then, and only then, free the device number:void unregister_chrdev_region(dev_t from, unsigned count);

Example (continued):cdev_del(&acme_cdev);unregister_chrdev_region(acme_dev, acme_count);

http://lxr.free-electrons.com/ident?i=cdev_del


http://lxr.free-electrons.com/ident?i=unregister_chrdev_region





Linux error codes

The kernel convention for error management is

Return 0 on successreturn 0;

Return a negative error code on failurereturn EFAULT;

Error codes

include/asmgeneric/errnobase.h

include/asm/errno.h


Char driver example summary (1)

static void *acme_buf;static int acme_bufsize=8192;

static int acme_count=1;static dev_t acme_dev = MKDEV(202,128);

static struct cdev acme_cdev;

static ssize_t acme_write(...) {...}

static ssize_t acme_read(...) {...}

static struct file_operations acme_fops ={

.owner = THIS_MODULE,

.read = acme_read,

.write = acme_write};


http://lxr.free-electrons.com/source/include/linux/cdev.h#005





Char driver example summary (2)

static int __init acme_init(void){ int err; acme_buf = ioremap (ACME_PHYS, acme_bufsize);

if (!acme_buf) { err = ENOMEM; goto err_exit; }

if (register_chrdev_region(acme_dev, acme_count, “acme”)) { err=ENODEV; goto err_free_buf; }

cdev_init(&acme_cdev, &acme_fops);

if (cdev_add(&acme_cdev, acme_dev, acme_count)) { err=ENODEV; goto err_dev_unregister; }

return 0;

err_dev_unregister: unregister_chrdev_region( acme_dev, acme_count); err_free_buf: iounmap(acme_buf); err_exit: return err;}

static void __exit acme_exit(void){ cdev_del(&acme_cdev); unregister_chrdev_region(acme_dev, acme_count); iounmap(acme_buf);}

Shows how to handle errors and deallocate resources in the right order!

Complete example code available on http://freeelectrons.com/doc/c/acme.c



http://lxr.free-electrons.com/ident?i=ENOMEM


http://lxr.free-electrons.com/ident?i=ENODEV

http://lxr.free-electrons.com/ident?i=cdev_init


http://lxr.free-electrons.com/ident?i=ENODEV









Character driver summary

Character driver writer Define the file operations callbacks for the device file: read, write, ioctl... In the module init function, reserve major and minor numbers withregister_chrdev_region(), init a cdev structure with your file operations and add it tothe system with cdev_add(). In the module exit function, call cdev_del() and unregister_chrdev_region()

System administration Load the character driver module Create device files with matching major and minor numbers if neededThe device file is ready to use!

System user Open the device file, read, write, or send ioctl's to it.

Kernel Executes the corresponding file operations

Ker

nel

Ker

nel

Use

rsp

ace


Practical lab – Character drivers

Writing a simple character driver, to write data to the serial port.

On your workstation, checking that transmitted data is received correctly.

Exchanging data between userspace and kernel space.

Practicing with the character device driver API.

Using kernel standard error codes.


Embedded Linux Driver Development

Driver developmentProcesses and scheduling


Process, thread ?

Confusion about the terms «process», «thread» and «task»

In Unix, a process is created using fork() and is composed of

An address space, which contains the program code, data, stack, shared libraries, etc.

One thread, that starts executing the main() function.

Upon creation, a process contains one thread

Additional threads can be created inside an existing process, using pthread_create()

They run in the same address space as the initial thread of the process

They start executing a function passed as argument to pthread_create()


Process, thread: kernel point of view

The kernel represents each thread running in the system by a structure of type task_struct

From a scheduling point of view, it makes no difference between the initial thread of a process and all additional threads created dynamically using pthread_create()

Address space

ThreadA

Process after fork()

Address space

ThreadA

ThreadB

Same process after pthread_create()


A thread life

Thread createdby fork() or

pthread_create()

TASK_RUNNINGReady but

not runningTASK_RUNNING

Actually running

TASK_INTERRUPTIBLE TASK_UNINTERRUPTIBLE

or TASK_KILLABLEWaiting

EXIT_ZOMBIETask terminated but its

resources are not freed yet.Waiting for its parent

to acknowledge its death.

Decides to sleepon a wait queue

for a specific event

The event occursor the process receivesa signal. Thread becomesrunnable again

The thread is preemptedby the scheduler to runa higher priority task

The thread is electedby the scheduler

http://lxr.free-electrons.com/ident?i=TASK_RUNNING


http://lxr.free-electrons.com/ident?i=TASK_INTERRUPTIBLE

http://lxr.free-electrons.com/ident?i=TASK_UNINTERRUPTIBLE

http://lxr.free-electrons.com/ident?i=TASK_KILLABLE

http://lxr.free-electrons.com/ident?i=EXIT_ZOMBIE


Execution of system calls

Process executing in user space...(can be preempted)

Kernel code executedon behalf of user space(can be preempted too!)

System callor exception

The execution of system calls takes place in thecontext of the thread requesting them.

Process continuing in user space...(or replaced by a higher priority process)

(can be preempted)

Still has access to processdata (open files...)



Driver developmentSleeping


Sleeping

Sleeping is needed when a process (user space or kernel space)is waiting for data.

User space process...

System call...

read device file

ask fordata

sleep

Otherprocesses

arescheduled

... System call

Interrupthandler

data ready notification

wake up

...User space

return


How to sleep (1)

Must declare a wait queue

A wait queue will be used to store the list of threads waiting for an event.

Static queue declarationuseful to declare as a global variable

DECLARE_WAIT_QUEUE_HEAD (module_queue);

Or dynamic queue declarationuseful to embed the wait queue inside another data structure

wait_queue_head_t queue;init_waitqueue_head(&queue);

http://lxr.free-electrons.com/ident?i=DECLARE_WAIT_QUEUE_HEAD

http://lxr.free-electrons.com/ident?i=wait_queue_head_t

http://lxr.free-electrons.com/ident?i=init_waitqueue_head


How to sleep (2)

Several ways to make a kernel process sleep

wait_event(queue, condition);Sleeps until the task is woken up and the given C expression is true.Caution: can't be interrupted (can't kill the userspace process!)

int wait_event_killable(queue, condition); (Since Linux 2.6.25)Can be interrupted, but only by a “fatal” signal (SIGKILL). Returns ERESTARSYS if interrupted.

int wait_event_interruptible(queue, condition);Can be interrupted by any signal. Returns ERESTARSYS if interrupted.

int wait_event_timeout(queue, condition, timeout);Also stops sleeping when the task is woken up and the timeout expired. Returns 0 if the timeout elapsed, nonzero if the condition was met.

int wait_event_interruptible_timeout(queue, condition, timeout);Same as above, interruptible. Returns 0 if the timeout elapsed, ERESTARTSYS if interrupted, positive value if the condition was met

http://lxr.free-electrons.com/ident?i=wait_event

http://lxr.free-electrons.com/ident?i=wait_event_killable

http://lxr.free-electrons.com/ident?i=wait_event_interruptible

http://lxr.free-electrons.com/ident?i=wait_event_timeout

http://lxr.free-electrons.com/ident?i=wait_event_interruptible_timeout


How to sleep Example

ret = wait_event_interruptible(sonypi_device.fifo_proc_list, kfifo_len(sonypi_device.fifo) != 0);

if (ret)return ret;


Waking up!

Typically done by interrupt handlers when data sleeping processes are waiting for are available.

wake_up(&queue);Wakes up one process in the given wait queue

wake_up_all(& queue);Wakes up all processes in the given wait queue. Not recommended.

wake_up_interruptible(&queue);Wakes up one process waiting in an interruptible sleep on the given queue


Sleeping and waking up implementation

The scheduler doesn't keep evaluating the sleeping condition!

wait_event_interruptible(&queue, condition);The process is put in the TASK_INTERRUPTIBLE state.

wake_up_interruptible(&queue);For all processes waiting in queue, condition is evaluated.When it evaluates to true, the process is put backto the TASK_RUNNING state, and the need_resched flag for the current process is set.

This way, several processes can be woken up at the same time.

http://lxr.free-electrons.com/ident?i=wait_event_interruptible

http://lxr.free-electrons.com/ident?i=TASK_INTERRUPTIBLE

http://lxr.free-electrons.com/ident?i=wake_up_interruptible




Driver developmentInterrupt management


Interrupt handler constraints

Not run from a user context:Can't transfer data to and from user space(need to be done by system call handlers)

Interrupt handler execution is managed by the CPU, not by the scheduler. Handlers can't run actions that may sleep, because there is nothing to resume their execution. In particular, need to allocate memory with GFP_ATOMIC.

Have to complete their job quickly enough:they shouldn't block their interrupt line for too long.


Registering an interrupt handler (1)

Defined in include/linux/interrupt.h

int request_irq( Returns 0 if successful unsigned int irq, Requested irq channel irq_handler_t handler, Interrupt handler unsigned long irq_flags, Option mask (see next page) const char * devname, Registered name void *dev_id); Pointer to some handler data

Cannot be NULL and must be unique for shared irqs!

void free_irq( unsigned int irq, void *dev_id);

dev_id cannot be NULL and must be unique for shared irqs.Otherwise, on a shared interrupt line,free_irq wouldn't know which handler to free.

http://lxr.free-electrons.com/source/include/linux/interrupt.h

http://lxr.free-electrons.com/ident?i=request_irq

http://lxr.free-electrons.com/ident?i=irq_handler_t

http://lxr.free-electrons.com/ident?i=free_irq

http://lxr.free-electrons.com/ident?i=free_irq


Registering an interrupt handler (2)

irq_flags bit values (can be combined, none is fine too)

IRQF_DISABLED"Quick" interrupt handler. Run with all interrupts disabled on the current cpu (instead of just the current line). For latency reasons, should only be used when needed!

IRQF_SHAREDRun with interrupts disabled only on the current irq line and on the local cpu. The interrupt channel can be shared by several devices. Requires a hardware status register telling whether an IRQ was raised or not.

http://lxr.free-electrons.com/ident?i=IRQF_DISABLED

http://lxr.free-electrons.com/ident?i=IRQF_SHARED


Information on installed handlers

/proc/interrupts

CPU00: 5616905 XTPIC timer # Registered name1: 9828 XTPIC i80422: 0 XTPIC cascade3: 1014243 XTPIC orinoco_cs7: 184 XTPIC Intel 82801DBICH48: 1 XTPIC rtc9: 2 XTPIC acpi11: 566583 XTPIC ehci_hcd, uhci_hcd, uhci_hcd, uhci_hcd, yenta, yenta, radeon@PCI:1:0:012: 5466 XTPIC i804214: 121043 XTPIC ide015: 200888 XTPIC ide1NMI: 0 Non Maskable InterruptsERR: 0 Spurious interrupt count


Interrupt handler prototype

irqreturn_t foo_interrupt(int irq, void *dev_id)

Arguments

irq, the IRQ number

dev_id, the opaque pointer passed at request_irq()

Return value

IRQ_HANDLED: recognized and handled interrupt

IRQ_NONE: not on a device managed by the module. Useful to share interrupt channels and/or report spurious interrupts to the kernel.


The interrupt handler's job

Acknowledge the interrupt to the device(otherwise no more interrupts will be generated)

Read/write data from/to the device

Wake up any waiting process waiting for the completion of this read/write operation:wake_up_interruptible(&module_queue);

http://lxr.free-electrons.com/ident?i=wake_up_interruptible


Top half and bottom half processing (1)

Splitting the execution of interrupt handlers in 2 parts

Top half: the interrupt handler must complete as quickly as possible. Once it acknowledged the interrupt, it just schedules the lengthy rest of the job taking care of the data, for a later execution.

Bottom half: completing the rest of the interrupt handler job. Handles data, and then wakes up any waiting user process.Best implemented by tasklets (also called soft irqs).


Top half and bottom half processing (2)

Declare the tasklet in the module source file:DECLARE_TASKLET (module_tasklet, /* name */ module_do_tasklet, /* function */ data /* params */);

Schedule the tasklet in the top half part (interrupt handler):tasklet_schedule(&module_tasklet);

Note that a tasklet_hi_schedule function is available to define high priority tasklets to run before ordinary ones.

By default, tasklets are executed right after all top halves(hard irqs)

http://lxr.free-electrons.com/ident?i=DECLARE_TASKLET

http://lxr.free-electrons.com/ident?i=tasklet_schedule

http://lxr.free-electrons.com/ident?i=tasklet_hi_schedule


Interrupt management fun

In a training lab, somebody forgot to unregister a handler on a shared interrupt line in the module exit function.

? Why did his kernel oops with a segmentation faultat module unload?

Answer...

In a training lab, somebody freed the timer interrupt handler by mistake (using the wrong irq number). The system froze. Remember the kernel is not protected against itself!


Interrupt management summary

Device driver

When the device file is first open, register an interrupt handler for the device's interrupt channel.

Interrupt handler

Called when an interrupt is raised.

Acknowledge the interrupt

If needed, schedule a tasklet taking care of handling data. Otherwise, wake up processes waiting for the data.

Tasklet

Process the data

Wake up processes waiting for the data

Device driver

When the device is no longer opened by any process, unregister the interrupt handler.


Practical lab – Interrupts

Adding read capability to the character driver developed earlier.

Register an interrupt handler.

Waiting for data to be available in the read file operation.

Waking up the code when data is available from the device.



Driver developmentConcurrent access to resources


Sources of concurrency issues

The same resources can be accessed by several kernel processes in parallel, causing potential concurrency issues

Several userspace programs accessing the same device data or hardware. Several kernel processes could execute the same code on behalf of user processes running in parallel.

Multiprocessing: the same driver code can be running on another processor. This can also happen with single CPUs with hyperthreading.

Kernel preemption, interrupts: kernel code can be interrupted at any time (just a few exceptions), and the same data may be access by another process before the execution continues.


Avoiding concurrency issues

Avoid using global variables and shared data whenever possible(cannot be done with hardware resources).

Use techniques to manage concurrent access to resources.

See Rusty Russell's Unreliable Guide To LockingDocumentation/DocBook/kernellocking/in the kernel sources.

http://free-electrons.com/kerneldoc/latest/DocBook/kernel-locking/


Concurrency protection with locks

Shared resource

Critical code section

Acquire lock

Release lock

Process 1 Process 2

Wait lock release

Success

Try again

Failed

Success


Linux mutexes

The main locking primitive since Linux 2.6.16.Better than counting semaphores when binary ones are enough.

Mutex definition:#include <linux/mutex.h>

Initializing a mutex statically:DEFINE_MUTEX(name);

Or initializing a mutex dynamically:void mutex_init(struct mutex *lock);

http://lxr.free-electrons.com/source/include/linux/mutex.h

http://lxr.free-electrons.com/ident?i=DEFINE_MUTEX

http://lxr.free-electrons.com/ident?i=mutex_init

http://lxr.free-electrons.com/ident?i=mutex


locking and unlocking mutexes

void mutex_lock (struct mutex *lock);Tries to lock the mutex, sleeps otherwise.Caution: can't be interrupted, resulting in processes you cannot kill!

int mutex_lock_killable (struct mutex *lock);Same, but can be interrupted by a fatal (SIGKILL) signal. If interrupted, returns a non zero value and doesn't hold the lock. Test the return value!!!

int mutex_lock_interruptible (struct mutex *lock);Same, but can be interrupted by any signal.

int mutex_trylock (struct mutex *lock);Never waits. Returns a non zero value if the mutex is not available.

int mutex_is_locked(struct mutex *lock);Just tells whether the mutex is locked or not.

void mutex_unlock (struct mutex *lock);Releases the lock. Do it as soon as you leave the critical section.

http://lxr.free-electrons.com/ident?i=mutex_lock


http://lxr.free-electrons.com/ident?i=mutex_lock_killable


http://lxr.free-electrons.com/ident?i=mutex_lock_interruptible


http://lxr.free-electrons.com/ident?i=mutex_trylock


http://lxr.free-electrons.com/ident?i=mutex_is_locked


http://lxr.free-electrons.com/ident?i=mutex_unlock



Spinlocks

Locks to be used for code that is not allowed to sleep (interrupt handlers), or that doesn't want to sleep (critical sections). Be very careful not to call functions which can sleep!

Originally intended for multiprocessor systems

Spinlocks never sleep and keep spinningin a loop until the lock is available.

Spinlocks cause kernel preemption to be disabledon the CPU executing them.

Spinlock

Still locked?


Initializing spinlocks

Staticspinlock_t my_lock = SPIN_LOCK_UNLOCKED;

Dynamicvoid spin_lock_init (spinlock_t *lock);

http://lxr.free-electrons.com/ident?i=spinlock_t

http://lxr.free-electrons.com/ident?i=SPIN_LOCK_UNLOCKED

http://lxr.free-electrons.com/ident?i=spin_lock_init



Using spinlocks

Several variants, depending on where the spinlock is called:

void spin_[un]lock (spinlock_t *lock);Doesn't disable interrupts. Used for locking in process context(critical sections in which you do not want to sleep).

void spin_lock_irqsave / spin_unlock_irqrestore (spinlock_t *lock, unsigned long flags);

Disables / restores IRQs on the local CPU.Typically used when the lock can be accessed in both process and interrupt context, to prevent preemption by interrupts.

void spin_[un]lock_bh (spinlock_t *lock);Disables software interrupts, but not hardware ones.Useful to protect shared data accessed in process contextand in a soft interrupt (“bottom half”). No need to disable hardware interrupts in this case.

Note that reader / writer spinlocks also exist.

http://lxr.free-electrons.com/ident?i=spin_lock

http://lxr.free-electrons.com/ident?i=spin_lock_irqsave

http://lxr.free-electrons.com/ident?i=spin_unlock_irqrestore


http://lxr.free-electrons.com/ident?i=spin_lock_bh



Deadlock situations

Don't call a function that can try to get access to the same lock

Holding multiple locks is risky!

They can lock up your system. Make sure they never happen!

Get lock1

Wait for lock1

call

Get lock1

Get lock2

Get lock2

Get lock1

DeadLock!

DeadLock!


Kernel lock validator

From Ingo Molnar and Arjan van de Ven

Adds instrumentation to kernel locking code

Detect violations of locking rules during system life, such as:

Locks acquired in different order(keeps track of locking sequences and compares them).

Spinlocks acquired in interrupt handlers and also in process context when interrupts are enabled.

Not suitable for production systems but acceptable overhead in development.

See Documentation/lockdepdesign.txt for details

http://free-electrons.com/kerneldoc/latest/lockdep-design.txt


Alternatives to locking

As we have just seen, locking can have a strong negative impact on system performance. In some situations, you could do without it.

By using lockfree algorithms like Read Copy Update (RCU).RCU API available in the kernel(See http://en.wikipedia.org/wiki/RCU).

When available, use atomic operations.

http://en.wikipedia.org/wiki/RCU


Atomic variables

Useful when the shared resource is an integer value

Even an instruction like n++ is not guaranteed to be atomic on all processors!

Header

#include <asm/atomic.h>

Type

atomic_tcontains a signed integer (at least 24 bits)

Atomic operations (main ones)

Set or read the counter:atomic_set (atomic_t *v, int i);int atomic_read (atomic_t *v);

Operations without return value:void atomic_inc (atomic_t *v);void atomic_dec (atomic_t *v);void atomic_add (int i, atomic_t *v);void atomic_sub (int i, atomic_t *v);

Simular functions testing the result:int atomic_inc_and_test (...);int atomic_dec_and_test (...);int atomic_sub_and_test (...);

Functions returning the new value:int atomic_inc_and_return (...);int atomic_dec_and_return (...);int atomic_add_and_return (...);int atomic_sub_and_return (...);

http://lxr.free-electrons.com/source/include/asm-i386/atomic.h

http://lxr.free-electrons.com/ident?i=atomic_t

http://lxr.free-electrons.com/ident?i=atomic_set


http://lxr.free-electrons.com/ident?i=atomic_read


http://lxr.free-electrons.com/ident?i=atomic_inc


http://lxr.free-electrons.com/ident?i=atomic_dec


http://lxr.free-electrons.com/ident?i=atomic_add


http://lxr.free-electrons.com/ident?i=atomic_sub


http://lxr.free-electrons.com/ident?i=atomic_inc_and_test

http://lxr.free-electrons.com/ident?i=atomic_dec_and_test

http://lxr.free-electrons.com/ident?i=atomic_sub_and_test

http://lxr.free-electrons.com/ident?i=atomic_inc_and_return

http://lxr.free-electrons.com/ident?i=atomic_dec_and_return

http://lxr.free-electrons.com/ident?i=atomic_add_and_return

http://lxr.free-electrons.com/ident?i=atomic_sub_and_return


Atomic bit operations

Supply very fast, atomic operations

On most platforms, apply to an unsigned long type.Apply to a void type on a few others.

Set, clear, toggle a given bit:void set_bit(int nr, unsigned long * addr);void clear_bit(int nr, unsigned long * addr);void change_bit(int nr, unsigned long * addr);

Test bit value:int test_bit(int nr, unsigned long *addr);

Test and modify (return the previous value):int test_and_set_bit (...);int test_and_clear_bit (...);int test_and_change_bit (...);

http://lxr.free-electrons.com/ident?i=set_bit

http://lxr.free-electrons.com/ident?i=clear_bit

http://lxr.free-electrons.com/ident?i=change_bit

http://lxr.free-electrons.com/ident?i=test_bit

http://lxr.free-electrons.com/ident?i=test_and_set_bit

http://lxr.free-electrons.com/ident?i=test_and_clear_bit

http://lxr.free-electrons.com/ident?i=test_and_change_bit


Practical lab – Locking

Add locking to the driver to prevent concurrent accesses to shared ressources



Driver developmentDebugging and tracing


Debugging with printk

Universal debugging technique used since the beginning of programming (first found in cavemen drawings)

Printed or not in the console or /var/log/messages according to the priority. This is controlled by the loglevel kernel parameter, or through /proc/sys/kernel/printk(see Documentation/sysctl/kernel.txt)

Available priorities (include/linux/kernel.h):#define KERN_EMERG "<0>" /* system is unusable */#define KERN_ALERT "<1>" /* action must be taken immediately */#define KERN_CRIT "<2>" /* critical conditions */#define KERN_ERR "<3>" /* error conditions */#define KERN_WARNING "<4>" /* warning conditions */#define KERN_NOTICE "<5>" /* normal but significant condition */#define KERN_INFO "<6>" /* informational */#define KERN_DEBUG "<7>" /* debuglevel messages */

http://free-electrons.com/kerneldoc/latest/sysctl/kernel.txt


http://lxr.free-electrons.com/ident?i=KERN_EMERG


http://lxr.free-electrons.com/ident?i=KERN_CRIT


http://lxr.free-electrons.com/ident?i=KERN_WARNING

http://lxr.free-electrons.com/ident?i=KERN_NOTICE

http://lxr.free-electrons.com/ident?i=KERN_INFO

http://lxr.free-electrons.com/ident?i=KERN_DEBUG


Debugging with /proc or /sys

Instead of dumping messages in the kernel log, you can have your drivers make information available to user space

Through a file in /proc or /sys, which contents are handled by callbacks defined and registered by your driver.

Can be used to show any piece of informationabout your device or driver.

Can also be used to send data to the driver or to control it.

Caution: anybody can use these files.You should remove your debugging interface in production!

Since the arrival of debugfs, no longer the preferred debugging mechanism


Debugfs

A virtual filesystem to export debugging information to userspace.

Kernel configuration: DEBUG_FSKernel hacking > Debug Filesystem

Much simpler to code than an interface in /proc or /sys.The debugging interface disappears when Debugfs is configured out.

You can mount it as follows:sudo mount t debugfs none /mnt/debugfs

First described on http://lwn.net/Articles/115405/

API documented in the Linux Kernel Filesystem API:http://freeelectrons.com/kerneldoc/latest/DocBook/filesystems/index.html

http://lwn.net/Articles/115405/

http://free-electrons.com/kerneldoc/latest/DocBook/filesystems/index.html


Simple debugfs example

#include <linux/debugfs.h>

static char *acme_buf; // module bufferstatic unsigned long acme_bufsize;static struct debugfs_blob_wrapper acme_blob;static struct dentry *acme_buf_dentry;

static u32 acme_state; // module variablestatic struct dentry *acme_state_dentry;

/* Module init */acme_blob.data = acme_buf;acme_blob.size = acme_bufsize;acme_buf_dentry = debugfs_create_blob("acme_buf", S_IRUGO, // Create

NULL, &acme_blob); // new filesacme_state_dentry = debugfs_create_bool("acme_state", S_IRUGO, // in debugfs

NULL, &acme_state);

/* Module exit */debugfs_remove (acme_buf_dentry); // removing the files from debugfsdebugfs_remove (acme_state_dentry);

http://lxr.free-electrons.com/source/include/linux/debugfs.h

http://lxr.free-electrons.com/ident?i=debugfs_blob_wrapper

http://lxr.free-electrons.com/ident?i=dentry

http://lxr.free-electrons.com/ident?i=dentry

http://lxr.free-electrons.com/ident?i=debugfs_create_blob

http://lxr.free-electrons.com/ident?i=debugfs_create_bool

http://lxr.free-electrons.com/ident?i=debugfs_remove

http://lxr.free-electrons.com/ident?i=debugfs_remove


Debugging with ioctl

Can use the ioctl() system call to query information about your driver (or device) or send commands to it.

This calls the ioctl file operation that you can register in your driver.

Advantage: your debugging interface is not public.You could even leave it when your system (or its driver) is in the hands of its users.


Using Magic SysRq

Allows to run multiple debug / rescue commands even when the kernel seems to be in deep trouble

On PC: Alt + SysRq + <character>

On embedded: break character on the serial line + <character>

. Example commands:

n: makes RT processes niceable.

t: shows the kernel stack of all sleeping processes

w: shows the kernel stack of all running processes

b: reboot the system

You can even register your own!

Detailed in Documentation/sysrq.txt

http://free-electrons.com/kerneldoc/latest/sysrq.txt


kgdb A kernel debugger

The execution of the kernel is fully controlled by gdb from another machine, connected through a serial line.

Can do almost everything, including inserting breakpoints in interrupt handlers.

Feature included in standard Linux since 2.6.26 (x86 and sparc). arm, mips and ppc support merged in 2.6.27.


Using kgdb

Details available in the kernel documentation:http://freeelectrons.com/kerneldoc/latest/DocBook/kgdb/

Recommended to turn on CONFIG_FRAME_POINTER to aid in producing more reliable stack backtraces in gdb.

You must include a kgdb I/O driver. One of them is kgdb over serial console (kgdboc: kgdb over console, enabled by CONFIG_KGDB_SERIAL_CONSOLE)

Configure kgdboc at boot time by passing to the kernel:kgdboc=<ttydevice>,[baud]. For example:kgdboc=ttyS0,115200

http://free-electrons.com/kerneldoc/latest/DocBook/kgdb/


Using kgdb (2)

Then also pass kgdbwait to the kernel:it makes kgdb wait for a debugger connection.

Boot your kernel, and when the console is initialized, interrupt the kernel with [Alt][SyrRq][g].

On your workstation, start gdb as follows:% gdb ./vmlinux(gdb) set remotebaud 115200(gdb) target remote /dev/ttyS0

Once connected, you can debug a kernel the way you would debug an application program.


Debugging with a JTAG interface

Two types of JTAG dongles

Those offering a gdb compatible interface, over a serial port or an Ethernet connexion. Gdb can directly connect to them.

Those not offering a gdb compatible interface are generally supported by OpenOCD (Open On Chip Debugger)

OpenOCD is the bridge between the gdb debugging language and the JTAGdongle specific language

http://openocd.berlios.de/web/

See the very complete documentation: http://openocd.berlios.de/doc/

For each board, you'll need an OpenOCD configuration file (ask your supplier)

See very useful details on using Eclipse / gcc / gdb / OpenOCD on Windows: http://www2.amontec.com/sdk4arm/ext/jlynchtutorial20061124.pdf and http://www.yagarto.de/howto/yagarto2/

http://openocd.berlios.de/web/

http://openocd.berlios.de/doc/

http://www2.amontec.com/sdk4arm/ext/jlynch-tutorial-20061124.pdf

http://www.yagarto.de/howto/yagarto2/


More kernel debugging tips

Enable CONFIG_KALLSYMS_ALL(General Setup > Configure standard kernel features)to get oops messages with symbol names instead of raw addresses(this obsoletes the ksymoops tool).

If your kernel doesn't boot yet or hangs without any message, you can activate Low Level debugging (Kernel Hacking section, only available on arm):CONFIG_DEBUG_LL=y

Techniques to locate the C instruction which caused an oops:http://kerneltrap.org/node/3648

More about kernel debugging in the free Linux Device Drivers book:http://lwn.net/images/pdf/LDD3/ch04.pdf

http://kerneltrap.org/node/3648

http://lwn.net/images/pdf/LDD3/ch04.pdf


Tracing with SystemTap

http://sourceware.org/systemtap/

Infrastructure to add instrumentation to a running kernel:trace functions, read and write variables, follow pointers, gather statistics...

Eliminates the need to modify the kernel sources to add one's own instrumentation to investigated a functional or performance problem.

Uses a simple scripting language.Several example scripts and probe points are available.

Based on the Kprobes instrumentation infrastructure.See Documentation/kprobes.txt in kernel sources.Linux 2.6.26: supported on most popular CPUs (arm included in 2.6.25).However, lack of recent support for mips (2.6.16 only!).

http://sourceware.org/systemtap/

http://free-electrons.com/kerneldoc/latest/kprobes.txt


SystemTap script example (1)

#! /usr/bin/env stap# Using statistics and maps to examine kernel memory allocations

global kmalloc

probe kernel.function("__kmalloc") { kmalloc[execname()] <<< $size

}

# Exit after 10 secondsprobe timer.ms(10000) { exit () }

probe end {foreach ([name] in kmalloc) {

printf("Allocations for %s\n", name)printf("Count: %d allocations\n", @count(kmalloc[name]))printf("Sum: %d Kbytes\n", @sum(kmalloc[name])/1000)printf("Average: %d bytes\n", @avg(kmalloc[name]))printf("Min: %d bytes\n", @min(kmalloc[name]))printf("Max: %d bytes\n", @max(kmalloc[name]))print("\nAllocations by size in bytes\n")print(@hist_log(kmalloc[name]))printf("\n\n");

}}


SystemTap script example (2)

#! /usr/bin/env stap

# Logs each file read performed by each process

probe kernel.function ("vfs_read"){ dev_nr = $file>f_dentry>d_inode>i_sb>s_dev inode_nr = $file>f_dentry>d_inode>i_ino printf ("%s(%d) %s 0x%x/%d\n", execname(), pid(), probefunc(), dev_nr, inode_nr)}

Nice tutorial on http://sources.redhat.com/systemtap/tutorial.pdf

http://sources.redhat.com/systemtap/tutorial.pdf


Kernel crash analysis with kexec/kdump

kexec system call: makes it possible to call a new kernel, without rebooting and going through the BIOS / firmware.

Idea: after a kernel panic, make the kernel automatically execute a new, clean kernel from a reserved location in RAM, to perform postmortem analysis of the memory of the crashed kernel.

See Documentation/kdump/kdump.txtin the kernel sources for details.

1. Copy debugkernel to reservedRAM

Standard kernel

2. kernel panic, kexec debug kernel

3. Analyze crashedkernel RAM

Regular RAM

Debug kernel

http://free-electrons.com/kerneldoc/latest/kdump/kdump.txt


Kernel markers

Capability to add static markers to kernel code,merged in Linux 2.6.24 by Matthieu Desnoyers.

Almost no impact on performance, until the marker is dynamically enabled, by inserting a probe kernel module.

Useful to insert trace points that won't be impacted by changes in the Linux kernel sources.

See marker and probe examplein samples/markers in the kernel sources.

See http://en.wikipedia.org/wiki/Kernel_marker

http://en.wikipedia.org/wiki/Kernel_marker


LTTng

http://lttng.org

The successor of the Linux Trace Toolkit (LTT)

Toolkit allowing to collect and analyze tracing information from the kernel, based on kernel markers and kernel tracepoints.

So far, based on kernel patches, but doing its best to use intree solutions, and to be merged in the future.

Very precise timestamps, very little overhead.

Useful documentation on http://lttng.org/?q=node/2#manuals

http://lttng.org/

http://lttng.org/?q=node/2#manuals


LTTV

Viewer for LTTng traces

Support for huge traces (tested with 15 GB ones)

Can combine multiple tracefiles in a single view.

Graphical or text interface

See http://lttng.org/files/lttvdoc/user_guide/

http://lttng.org/files/lttv-doc/user_guide/


Practical lab – Kernel debugging

Load a broken driver and see it crash

Analyze the error informationdumped by the kernel.

Disassemble the code and locatethe exact C instruction which caused the failure.

Use the JTAG and OpenOCD to remotely control the kernel execution



Driver developmentmmap


mmap (1)

Possibility to have parts of the virtual address space of a program mapped to the contents of a file!> cat /proc/1/maps (init process)start end perm offset major:minor inode mapped file name007710000077f000 rxp 00000000 03:05 1165839 /lib/libselinux.so.10077f00000781000 rwp 0000d000 03:05 1165839 /lib/libselinux.so.10097d00000992000 rxp 00000000 03:05 1158767 /lib/ld2.3.3.so0099200000993000 rp 00014000 03:05 1158767 /lib/ld2.3.3.so0099300000994000 rwp 00015000 03:05 1158767 /lib/ld2.3.3.so0099600000aac000 rxp 00000000 03:05 1158770 /lib/tls/libc2.3.3.so00aac00000aad000 rp 00116000 03:05 1158770 /lib/tls/libc2.3.3.so00aad00000ab0000 rwp 00117000 03:05 1158770 /lib/tls/libc2.3.3.so00ab000000ab2000 rwp 00ab0000 00:00 00804800008050000 rxp 00000000 03:05 571452 /sbin/init (text)0805000008051000 rwp 00008000 03:05 571452 /sbin/init (data, stack)08b4300008b64000 rwp 08b43000 00:00 0f6fdf000f6fe0000 rwp f6fdf000 00:00 0fefd4000ff000000 rwp fefd4000 00:00 0ffffe000fffff000 p 00000000 00:00 0


mmap (2)

Particularly useful when the file is a device file!Allows to access device I/O memory and ports without having to go through (expensive) read, write or ioctl calls!

X server example (maps excerpt)start end perm offset major:minor inode mapped file name08047000081be000 rxp 00000000 03:05 310295 /usr/X11R6/bin/Xorg081be000081f0000 rwp 00176000 03:05 310295 /usr/X11R6/bin/Xorg...f4e08000f4f09000 rws e0000000 03:05 655295 /dev/dri/card0f4f09000f4f0b000 rws 4281a000 03:05 655295 /dev/dri/card0f4f0b000f6f0b000 rws e8000000 03:05 652822 /dev/memf6f0b000f6f8b000 rws fcff0000 03:05 652822 /dev/mem

A more user friendly way to get such information: pmap <pid>


mmap overview

Process virtual address space

ProcessDevice drivermmap fop called

initializes the mapping

mmapsystemcall (once)

Physical address space

accessvirtualaddress

MMUaccessphysicaladdress


How to implement mmap User space

Open the device file

Call the mmap system call (see man mmap for details):void * mmap(

void *start, /* Often 0, preferred starting address */size_t length, /* Length of the mapped area */int prot , /* Permissions: read, write, execute */int flags, /* Options: shared mapping, private copy...

*/int fd, /* Open file descriptor */off_t offset /* Offset in the file */

);

You get a virtual address you can write to or read from.


How to implement mmap Kernel space

Character driver: implement a mmap file operationand add it to the driver file operations:int (*mmap) (

struct file *, /* Open file structure */struct vm_area_struct * /* Kernel VMA structure */

);

Initialize the mapping.Can be done in most cases with the remap_pfn_range() function, which takes care of most of the job.


http://lxr.free-electrons.com/ident?i=vm_area_struct

http://lxr.free-electrons.com/ident?i=remap_fn_range


remap_pfn_range()

pfn: page frame numberThe most significant bits of the page address(without the bits corresponding to the page size).

#include <linux/mm.h>

int remap_pfn_range(struct vm_area_struct *, /* VMA struct */unsigned long virt_addr, /* Starting user virtual address */unsigned long pfn, /* pfn of the starting physical address */unsigned long size, /* Mapping size */pgprot_t /* Page permissions */

);

http://lxr.free-electrons.com/source/include/linux/mm.h

http://lxr.free-electrons.com/ident?i=remap_pfn_range


http://lxr.free-electrons.com/ident?i=pgprot_t


Simple mmap implementation

static int acme_mmap (struct file * file, struct vm_area_struct * vma)

{size = vma>vm_end vma>vm_start;

if (size > ACME_SIZE) return EINVAL;

if (remap_pfn_range(vma,vma>vm_start,ACME_PHYS >> PAGE_SHIFT,size,vma>vm_page_prot))

return EAGAIN;return 0;

}



http://lxr.free-electrons.com/ident?i=EINVAL

http://lxr.free-electrons.com/ident?i=remap_pfn_range

http://lxr.free-electrons.com/ident?i=EAGAIN


devmem2

http://freeelectrons.com/pub/mirror/devmem2.c, by JanDerk Bakker

Very useful tool to directly peek (read) or poke (write) I/O addresses mapped in physical address space from a shell command line!

Very useful for early interaction experiments with a device, without having to code and compile a driver.

Uses mmap to /dev/mem.

Examples (b: byte, h: half, w: word)devmem2 0x000c0004 h (reading)devmem2 0x000c0008 w 0xffffffff (writing)

devmem is now available in BusyBox, making it even easier to use.

http://free-electrons.com/pub/mirror/devmem2.c


mmap summary

The device driver is loaded.It defines an mmap file operation.

A user space process calls the mmap system call.

The mmap file operation is called.It initializes the mapping using the device physical address.

The process gets a starting address to read from and write to (depending on permissions).

The MMU automatically takes care of converting the process virtual addresses into physical ones.

Direct access to the hardware!No expensive read or write system calls!



Driver developmentKernel architecture for device drivers


Kernel and device drivers

Application

System call interface

Framework

Driver

Bus infrastructure

Hardware

Userspace

Kernel


Kernel and device drivers

Many device drivers are not implemented directly as character drivers

They are implemented under a « framework », specific to a given device type (framebuffer, V4L, serial, etc.)

The framework allows to factorize the common parts of drivers for the same type of devices

From userspace, they are still seen as character devices by the applications

The framework allows to provide a coherent userspace interface (ioctl, etc.) for every type of device, regardless of the driver

The device drivers rely on the « bus infrastructure » to enumerate the devices and communicate with them.



Kernel frameworks


« Frameworks »


Application Application Application

Characterdriver

Framebufferdriver

Framebuffercore

V4Lcore

V4Ldriver

TTYcore

Serialcore

Serialdriver

Blockcore

IDEcore

IDEdriver

SCSIcore

USB storagedriver

Bus infrastructure


Example: framebuffer framework

Kernel option CONFIG_FB

Implemented in drivers/video/

fb.c, fbmem.c, fbmon.c, fbcmap.c, fbsysfs.c, modedb.c, fbcvt.c

Implements a single character driver and defines the user/kernel API

First part of include/linux/fb.h

Defines the set of operations a framebuffer driver must implement and helper functions for the drivers

struct fb_ops

Second part of include/linux/fb.h(in #ifdef __KERNEL__)

menuconfig FB tristate "Support for frame buffer devices"

http://lxr.free-electrons.com/source/drivers/video/

http://lxr.free-electrons.com/source/include/linux/fb.h

http://lxr.free-electrons.com/ident?i=fb_ops

http://lxr.free-electrons.com/source/include/linux/fb.h


Framebuffer driver skeleton

Skeleton driver in drivers/video/skeletonfb.c

Implements the set of framebuffer specific operations defined by the struct fb_ops structure

xxxfb_open()

xxxfb_read()

xxxfb_write()

xxxfb_release()

xxxfb_checkvar()

xxxfb_setpar()

xxxfb_setcolreg()

xxxfb_blank()

xxxfb_pan_display()

xxxfb_fillrect()

xxxfb_copyarea()

xxxfb_imageblit()

xxxfb_cursor()

xxxfb_rotate()

xxxfb_sync()

xxxfb_ioctl()

xxxfb_mmap()

http://lxr.free-electrons.com/source/drivers/video/skeletonfb.c



After the implementation of the operations, definition of a struct fb_ops structure

static struct fb_ops xxxfb_ops = { .owner = THIS_MODULE, .fb_open = xxxfb_open, .fb_read = xxxfb_read, .fb_write = xxxfb_write, .fb_release = xxxfb_release, .fb_check_var = xxxfb_check_var, .fb_set_par = xxxfb_set_par, .fb_setcolreg = xxxfb_setcolreg, .fb_blank = xxxfb_blank, .fb_pan_display = xxxfb_pan_display, .fb_fillrect = xxxfb_fillrect, /* Needed !!! */ .fb_copyarea = xxxfb_copyarea, /* Needed !!! */ .fb_imageblit = xxxfb_imageblit, /* Needed !!! */ .fb_cursor = xxxfb_cursor, /* Optional !!! */ .fb_rotate = xxxfb_rotate, .fb_sync = xxxfb_sync, .fb_ioctl = xxxfb_ioctl, .fb_mmap = xxxfb_mmap,};



In the probe() function, registration of the framebuffer device and operations

register_framebuffer() will create the character device that can be used by userspace application with the generic framebuffer API

static int __devinit xxxfb_probe(struct pci_dev *dev,

const struct pci_device_id *ent){ struct fb_info *info;

[...]info = framebuffer_alloc(sizeof(struct xxx_par), device);[...]info>fbops = &xxxfb_ops;[...]if (register_framebuffer(info) < 0)

return EINVAL;[...]

}



Device Model and Bus Infrastructure


Unified device model

The 2.6 kernel included a significant new feature:a unified device model

Instead of having different adhoc mechanisms in the various subsystems, the device model unifies the description of the devices and their topology

Minimizing code duplication

Common facilities (reference counting, event notification, power management, etc.)

Enumerate the devices, view their interconnections, link the devices to their buses and drivers, categorize them by classes.


Bus drivers

The first component of the device model is the bus driver

One bus driver for each type of bus: USB, PCI, SPI, MMC, ISA, etc.

It is responsible for

Registering the bus type

Allowing the registration of adapter drivers (USB controllers, I2C adapters, etc.), able of detecting the connected devices

Allowing the registration of device drivers (USB devices, I2C devices, PCI devices, etc.), managing the devices

Matching the device drivers against the devices detected by the adapter drivers.


List of device identifiers

Depending on the bus type, the method for binding a device to a driver is different. For many buses, it is based on unique identifiers

The device driver defines a table with the list of device identifiers it is able to manage :

static const struct pci_device_id rhine_pci_tbl[] = { { 0x1106, 0x3043, PCI_ANY_ID, PCI_ANY_ID, }, /* VT86C100A */ { 0x1106, 0x3053, PCI_ANY_ID, PCI_ANY_ID, }, /* VT6105M */ { } /* terminate list */};MODULE_DEVICE_TABLE(pci, rhine_pci_tbl);

Code on this slide and on the next slides are takenfrom the viarhine driver in drivers/net/viarhine.c


Defining the driver

The device driver defines a driver structure, usually specialized by the bus infrastructure (pci_driver, usb_driver, etc.)

The structure points to: the device table, a probe function, called when a device is detected and various other callbacks

static struct pci_driver rhine_driver = { .name = DRV_NAME, .id_table = rhine_pci_tbl, .probe = rhine_init_one, .remove = __devexit_p(rhine_remove_one),#ifdef CONFIG_PM .suspend = rhine_suspend, .resume = rhine_resume,#endif /* CONFIG_PM */ .shutdown = rhine_shutdown,};


Registering the driver

In the module initialization function, the driver is registered to the bus infrastructure, in order to let the bus know that the driver is available to handle devices.

If a new PCI device matches one of the identifiers of the table, the probe() method of the PCI driver will get called.

static int __init rhine_init(void){

[...] return pci_register_driver(&rhine_driver);}static void __exit rhine_cleanup(void){ pci_unregister_driver(&rhine_driver);}


Probe method

The probe() method receives as argument a structure describing the device, usually specialized by the bus infrastructure (pci_dev, usb_device, etc.)

This function is responsible for

Initializing the device, mapping I/O memory, registering the interrupt handlers. The bus infrastructure provides methods to get the addresses, interrupts numbers and other devicespecific information.

Registering the device to the proper kernel framework, for example the network infrastructure.


Device driver (5)

static int __devinit rhine_init_one(struct pci_dev *pdev, const struct pci_device_id *ent){ struct net_device *dev;

[...] rc = pci_enable_device(pdev);

[...] pioaddr = pci_resource_start(pdev, 0); memaddr = pci_resource_start(pdev, 1);

[...] dev = alloc_etherdev(sizeof(struct rhine_private));

[...] SET_NETDEV_DEV(dev, &pdev>dev);

[...] rc = pci_request_regions(pdev, DRV_NAME);

[...] ioaddr = pci_iomap(pdev, bar, io_size);

[...] rc = register_netdev(dev);

[...]}


Global architecture

PCI bus driver

viarhinedevice driver

Networkdevice

frameworkThe device driver

registers itself and the

device identifiers to

the bus driver

The bus driver detects a

matching device, and calls the

probe() method of the device

driver.

The probe() method of the device driver initializes the device and

registers a new network interface

1 2

3


sysfs

The bus, device, drivers, etc. structures are internal to the kernel

The sysfs virtual filesystem offers a mechanism to export such information to userspace

Used for example by udev to provide automatic module loading, firmware loading, device file creation, etc.

sysfs is usually mounted in /sys

/sys/bus/ contains the list of buses

/sys/devices/ contains the list of devices

/sys/class enumerates devices by class (net, input, block...), whatever the bus they are connected to. Very useful!

Take your time to explore /sys on your workstation.


Platform devices

On embedded systems, devices are often not connected through a bus allowing enumeration, hotplugging, and providing unique identifiers for devices.

However, we still want the devices to be part of the device model.

The solution to this is the platform driver / platform device infrastructure.

The platform devices are the devices that are directly connected to the CPU, without any kind of bus.


Implementation of the platform driver

static int __init imx_serial_init(void){

[...] ret = platform_driver_register(&serial_imx_driver);

[...]}

The driver implements a platform_driver structure(example taken from drivers/serial/imx.c)

And registers its driver to the platform driver infrastructure

static struct platform_driver serial_imx_driver = { .probe = serial_imx_probe, .remove = serial_imx_remove, .driver = { .name = "imxuart", .owner = THIS_MODULE, },};


Platform device instantiation (1)

In the boardspecific code, the platform devices are instantiated (arch/arm/machimx/mx1ads.c):

The match between the device and the driver is made using the name. It must be unique amongst drivers !

static struct platform_device imx_uart1_device = { .name = "imxuart", .id = 0, .num_resources = ARRAY_SIZE(imx_uart1_resources), .resource = imx_uart1_resources, .dev = { .platform_data = &uart_pdata, }};


Platform device instantiation (2)

The device is part of a list

And the list of devices is added to the systemduring board initialization

static struct platform_device *devices[] __initdata = { &cs89x0_device, &imx_uart1_device, &imx_uart2_device,};

static void __init mx1ads_init(void){

[...]platform_add_devices(devices, ARRAY_SIZE(devices));

}


I/O resources

Each platform device is associated with a set of I/O resources, referenced in the platform_device structure

It allows the driver to be independent of the I/O addresses, IRQ numbers ! See imx_uart2_device for another device using the same platform driver.

static struct resource imx_uart1_resources[] = { [0] = { .start = 0x00206000, .end = 0x002060FF, .flags = IORESOURCE_MEM, }, [1] = { .start = (UART1_MINT_RX), .end = (UART1_MINT_RX), .flags = IORESOURCE_IRQ, },};


Inside the platform driver

When a platform_device is added to the system using platform_add_device(), the probe() method of the platform driver gets called

This method is responsible for initializing the hardware, registering the device to the proper framework (in our case, the serial driver framework)

The platform driver has access to the I/O resources :

res = platform_get_resource(pdev, IORESOURCE_MEM, 0);base = ioremap(res>start, PAGE_SIZE);sport>rxirq = platform_get_irq(pdev, 0);


Framework and bus infrastructure

A typical driver will

Be registered inside a framework

Rely on a bus infrastructure and the device model

Example with the iMX serial driver, drivers/serial/imx.c

At module initialization time, the driver registers itself both to the UART framework and to the platform bus infrastructure

static int __init imx_serial_init(void){ ret = uart_register_driver(&imx_reg);

[...] ret = platform_driver_register(&serial_imx_driver);

[...] return 0;}


iMX serial driver

static struct uart_driver imx_reg = { .owner = THIS_MODULE, .driver_name = DRIVER_NAME, .dev_name = DEV_NAME, .major = SERIAL_IMX_MAJOR, .minor = MINOR_START, .nr = ARRAY_SIZE(imx_ports), .cons = IMX_CONSOLE,};

Definition of the iMX UART driver

Definition of the iMX platform driver

static struct platform_driver serial_imx_driver = { .probe = serial_imx_probe, .remove = serial_imx_remove,

[...] .driver = { .name = "imxuart", .owner = THIS_MODULE, },};


iMX serial driver

When the platform device is instantiated, the probe() method is called

static int serial_imx_probe(struct platform_device *pdev){

struct imx_port *sport;

sport = kzalloc(sizeof(*sport), GFP_KERNEL);res = platform_get_resource(pdev, IORESOURCE_MEM, 0);base = ioremap(res>start, PAGE_SIZE);

/* sport initialization */sport>port.irq = platform_get_irq(pdev, 0);sport>port.ops = &imx_pops;

sport>clk = clk_get(&pdev>dev, "uart_clk");clk_enable(sport>clk);

uart_add_one_port(&imx_reg, &sport>port);}


iMX serial driver

static struct uart_ops imx_pops = { .tx_empty = imx_tx_empty, .set_mctrl = imx_set_mctrl, .get_mctrl = imx_get_mctrl, .stop_tx = imx_stop_tx, .start_tx = imx_start_tx, .stop_rx = imx_stop_rx, .enable_ms = imx_enable_ms, .break_ctl = imx_break_ctl, .startup = imx_startup, .shutdown = imx_shutdown, .set_termios = imx_set_termios, .type = imx_type, .release_port = imx_release_port, .request_port = imx_request_port, .config_port = imx_config_port, .verify_port = imx_verify_port,};

The operation structure uart_ops is associated to each port. The operations are implemented in the iMX driver


References

Kernel documentationDocumentation/drivermodel/Documentation/filesystems/sysfs.txt

Linux 2.6 Device Modelhttp://www.bravegnu.org/devicemodel/devicemodel.html

Linux Device Drivers, chapter 14 «The Linux Device Model»http://lwn.net/images/pdf/LDD3/ch14.pdf

The kernel source codeFull of examples of other drivers!

http://free-electrons.com/kerneldoc/latest/driver-model/

http://free-electrons.com/kerneldoc/latest/filesystems/sysfs.txt

http://www.bravegnu.org/device-model/device-model.html

http://lwn.net/images/pdf/LDD3/ch14.pdf



AnnexesQuiz answers


Quiz answers

Interrupt handlingQ: Why did the kernel segfault at module unload (forgetting to unregister a handler in a shared interrupt line)?A: Kernel memory is allocated at module load time, to host module code. This memory is freed at module unload time. If you forget to unregister a handler and an interrupt comes, the cpu will try to jump to the address of the handler, which is in a freed memory area. Crash!


Linux device driver development

DMA


Free Electrons

© Copyright 20042009, Free Electrons.Creative Commons BYSA 3.0 licenseLatest update: Mar 5, 2010, Document sources, updates and translations:http://freeelectrons.com/docs/dmaCorrections, suggestions, contributions and translations are welcome!

http://free-electrons.com/docs/dma


DMA memory constraints

Need to use contiguous memory in physical space.

Can use any memory allocated by kmalloc (up to 128 KB) or __get_free_pages (up to 8MB).

Can use block I/O and networking buffers,designed to support DMA.

Can not use vmalloc memory(would have to setup DMA on each individual physical page).



Reserving memory for DMA

To make sure you've got enough RAM for big DMA transfers...Example assuming you have 32 MB of RAM, and need 2 MB for DMA:

Boot your kernel with mem=30The kernel will just use the first 30 MB of RAM.

Driver code can now reclaim the 2 MB left:dmabuf = ioremap (

0x1e00000, /* Start: 30 MB */0x200000 /* Size: 2 MB */);



Memory synchronization issues

Memory caching could interfere with DMA

Before DMA to device:Need to make sure that all writes to DMA buffer are committed.

After DMA from device:Before drivers read from DMA buffer, need to make sure that memory caches are flushed.

Bidirectional DMANeed to flush caches before and after the DMA transfer.


Linux DMA API

The kernel DMA utilities can take care of:

Either allocating a buffer in a cache coherent area,

Or making sure caches are flushed when required,

Managing the DMA mappings and IOMMU (if any).

See Documentation/DMAAPI.txtfor details about the Linux DMA generic API.

Most subsystems (such as PCI or USB) supply their own DMA API, derived from the generic one. May be sufficient for most needs.

http://free-electrons.com/kerneldoc/latest/DMA-API.txt


Coherent or streaming DMA mappings

Coherent mappings

The kernel allocates a suitable bufferand sets the mapping for the driver.

Can simultaneously be accessed by the CPU and device.

So, has to be in a cache coherent memory area.

Usually allocated for the whole time the module is loaded.

Can be expensive to setup and use on some platforms.

Streaming mappings

The kernel just sets the mapping for a buffer provided by the driver.

Use a buffer already allocatedby the driver.

Mapping set up for each transfer. Keeps DMA registers free on the hardware.

Some optimizations also available.

The recommended solution.


Allocating coherent mappings

The kernel takes care of both buffer allocation and mapping:

include <asm/dmamapping.h>

void * /* Output: buffer address */dma_alloc_coherent(

struct device *dev, /* device structure */size_t size, /* Needed buffer size in bytes */dma_addr_t *handle, /* Output: DMA bus address */gfp_t gfp /* Standard GFP flags */

);

void dma_free_coherent(struct device *dev,size_t size, void *cpu_addr, dma_addr_t handle);

http://lxr.free-electrons.com/source/include/asm-i386/dma-mapping.h

http://lxr.free-electrons.com/ident?i=dma_alloc_coherent

http://lxr.free-electrons.com/ident?i=device


http://lxr.free-electrons.com/ident?i=dma_addr_t

http://lxr.free-electrons.com/ident?i=gfp_t





Setting up streaming mappings

Works on buffers already allocated by the driver

<include linux/dmapool.h>

dma_addr_t dma_map_single(struct device *, /* device structure */void *, /* input: buffer to use */size_t, /* buffer size */enum dma_data_direction /* Either DMA_BIDIRECTIONAL,

DMA_TO_DEVICE or DMA_FROM_DEVICE */);

void dma_unmap_single(struct device *dev, dma_addr_t handle, size_t size, enum dma_data_direction dir);

http://lxr.free-electrons.com/source/include/linux/dmapool.h


http://lxr.free-electrons.com/ident?i=dma_map_single



http://lxr.free-electrons.com/ident?i=dma_data_direction

http://lxr.free-electrons.com/ident?i=DMA_BIDIRECTIONAL

http://lxr.free-electrons.com/ident?i=DMA_TO_DEVICE

http://lxr.free-electrons.com/ident?i=DMA_FROM_DEVICE

http://lxr.free-electrons.com/ident?i=dma_unmap_single




http://lxr.free-electrons.com/ident?i=dma_data_direction


DMA streaming mapping notes

When the mapping is active: only the device should access the buffer (potential cache issues otherwise).

The CPU can access the buffer only after unmapping!Use locking to prevent CPU access to the buffer.

Another reason: if required, this API can create an intermediate bounce buffer (used if the given buffer is not usable for DMA).

The Linux API also supports scatter / gather DMA streaming mappings.


DMA summary

Coherent mappings

DMA buffer allocated by the kernel

Set up for the whole module life

Can be expensive. Not recommended.

Let both the CPU and deviceaccess the buffer at the same time.

Main functions:dma_alloc_coherentdma_free_coherent

Streaming mappings

DMA buffer allocated by the driver

Set up for each transfer

Cheaper. Saves DMA registers.

Only the device can access the bufferwhen the mapping is active.

Main functions:dma_map_singledma_unmap_single

Most drivers can use the specific API provided by their subsystem: USB, PCI, SCSI... Otherwise they can use the Linux generic API:

http://lxr.free-electrons.com/ident?i=dma_alloc_coherent

http://lxr.free-electrons.com/ident?i=dma_free_coherent

http://lxr.free-electrons.com/ident?i=dma_map_single

http://lxr.free-electrons.com/ident?i=dma_unmap_single


Linux PCI drivers

LinuxPCI drivers

Michael OpdenackerFree Electrons

http://en.wikipedia.org/wiki/Image:PCI_Slots_Digon3.JPG

© Copyright 20042009, Free Electrons.Creative Commons BYSA 3.0 licenseLatest update: Mar 5, 2010, Document sources, updates and translations:http://freeelectrons.com/docs/pcidriversCorrections, suggestions, contributions and translations are welcome!

http://en.wikipedia.org/wiki/Image:PCI_Slots_Digon3.JPG

http://free-electrons.com/docs/pci-drivers


Linux PCI drivers

Understanding PCI


PCI bus family

PCI32 bit bus, 33 or 66 MHz

MiniPCISmaller slot in laptops

CardBusExternal card slot in laptops

PIX Extended (PCIX)Wider slot than PCI, 64 bit, but can accept a standard PCI card

PCI Express (PCIe or PCIE)Current generation of PCI. Serial instead of parallel.

PCI Express Mini CardReplaces MiniPCI in recent laptops

Express CardReplaces CardBus in recent laptops

The following buses belong to the PCI family:

These technologies are compatible and can be handled by the same kernel drivers.The kernel doesn't need to know which exact slot and bus variant is used.


PCI device types

Main types of devices found on the PCI bus

Network cards (wired or wireless)

SCSI adapters

Bus controllers: USB, PCMCIA, I2C, FireWire, IDE

Graphics and video cards

Sound cards


PCI features

For device driver developers

Device resources (I/O addresses, IRQ lines) automatically assigned at boot time, either by the BIOS or by Linux itself (if configured).

The device driver just has to read the corresponding configurations somewhere in the system address space.

Endianism: PCI device configuration information is Little Endian. Remember that in your drivers(conversion taken care of by some kernel functions).


PCI device list (1)

lscpi00:00.0 Host bridge: Intel Corporation Mobile 945GM/PM/GMS, 943/940GML and 945GT Express Memory Controller Hub (rev 03)00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller (rev 03)00:02.1 Display controller: Intel Corporation Mobile 945GM/GMS/GME, 943/940GML Express Integrated Graphics Controller (rev 03)00:1b.0 Audio device: Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller (rev 01)00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1 (rev 01)00:1c.1 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 2 (rev 01)00:1c.2 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 3 (rev 01)00:1d.0 USB Controller [...]00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev e1)00:1f.0 ISA bridge: Intel Corporation 82801GBM (ICH7M) LPC Interface Bridge (rev 01)00:1f.1 IDE interface: Intel Corporation 82801G (ICH7 Family) IDE Controller (rev 01)00:1f.3 SMBus: Intel Corporation 82801G (ICH7 Family) SMBus Controller (rev 01)02:01.0 CardBus bridge: Ricoh Co Ltd RL5c476 II (rev b4)02:01.1 FireWire (IEEE 1394): Ricoh Co Ltd R5C552 IEEE 1394 Controller (rev 09)02:01.2 SD Host controller: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter (rev 18)09:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5752 Gigabit Ethernet PCI Express (rev 02)0c:00.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN Network Connection (rev 61)

lspci tv[0000:00]+00.0 Intel Corporation Mobile 945GM/PM/GMS, 943/940GML and 945GT Express Memory Controller Hub +02.0 Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller +02.1 Intel Corporation Mobile 945GM/GMS/GME, 943/940GML Express Integrated Graphics Controller +1b.0 Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller +1c.0[0000:0b] +1c.1[0000:0c]00.0 Intel Corporation PRO/Wireless 4965 AG or AGN Network Connection +1c.2[0000:09]00.0 Broadcom Corporation NetXtreme BCM5752 Gigabit Ethernet PCI Express +1d.0 Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 [...] +1e.0[0000:0206]+01.0 Ricoh Co Ltd RL5c476 II | +01.1 Ricoh Co Ltd R5C552 IEEE 1394 Controller | \01.2 Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter +1f.0 Intel Corporation 82801GBM (ICH7M) LPC Interface Bridge +1f.1 Intel Corporation 82801G (ICH7 Family) IDE Controller \1f.3 Intel Corporation 82801G (ICH7 Family) SMBus Controller


PCI device list (2)

lspci enumerates all PCI devices02:01.0 CardBus bridge: Ricoh Co Ltd RL5c476 II (rev b4)02:01.1 FireWire (IEEE 1394): Ricoh Co Ltd R5C552 IEEE 1394 Controller02:01.2 SD Host controller: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro

Function numberPCI device number

PCI bus number

lscpi t shows the bus device tree[0000:00]+00.0 Intel Corporation Mobile 945GM/PM/GMS, 943/940GML and 945GT Express Memory Controller Hub +02.0 Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller +02.1 Intel Corporation Mobile 945GM/GMS/GME, 943/940GML Express Integrated Graphics Controller +1b.0 Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller +1c.0[0000:0b] +1c.1[0000:0c]00.0 Intel Corporation PRO/Wireless 4965 AG or AGN Network Connection +1c.2[0000:09]00.0 Broadcom Corporation NetXtreme BCM5752 Gigabit Ethernet PCI Express +1d.0 Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 [...] +1e.0[0000:0206]+01.0 Ricoh Co Ltd RL5c476 II | +01.1 Ricoh Co Ltd R5C552 IEEE 1394 Controller | \01.2 Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter

PCI domain PCI bus 0

PCI bridge PCI bus 2


PCI device list (3)

This tree structure reflects the structure in /sys:/sys/devices/pci0000:00/0000:00:1e.0/0000:02:01.2

Bus 0 Bus 2PCI bridge


PCI device configuration (1)

Each PCI device has a 256 byte address space containing configuration registers.

Device configuration can be displayed with lspci x:0c:00.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN Network Connection (rev 61)00: 86 80 29 42 06 04 10 00 61 00 80 02 10 00 00 0010: 04 e0 df ef 00 00 00 00 00 00 00 00 00 00 00 0020: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 21 1130: 00 00 00 00 c8 00 00 00 00 00 00 00 05 01 00 00


PCI device configuration (2)

Standard information found in PCI configurations:

Offset 0: Vendor Id

Offset 2: Device Id

Offset 10: Class Id (network, display, bridge...)

Offsets 16 to 39: Base Address Registers (BAR) 0 to 6

Offset 44: Subvendor Id

Offset 46: Subdevice Id

Offsets 64 and up: up to the device manufacturer

Kernel sources: these offsets are defined in include/linux/pci_regs.h


Linux PCI drivers

Implementing Linux drivers


Registering supported devices

From drivers/net/ne2kpci.c (Linux 2.6.27):static struct pci_device_id ne2k_pci_tbl[] = { { 0x10ec, 0x8029, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_RealTek_RTL_8029 }, { 0x1050, 0x0940, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_Winbond_89C940 }, { 0x11f6, 0x1401, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_Compex_RL2000 }, { 0x8e2e, 0x3000, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_KTI_ET32P2 }, { 0x4a14, 0x5000, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_NetVin_NV5000SC }, { 0x1106, 0x0926, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_Via_86C926 }, { 0x10bd, 0x0e34, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_SureCom_NE34 }, { 0x1050, 0x5a5a, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_Winbond_W89C940F }, { 0x12c3, 0x0058, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_Holtek_HT80232 }, { 0x12c3, 0x5598, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_Holtek_HT80229 }, { 0x8c4a, 0x1980, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CH_Winbond_89C940_8c4a }, { 0, }};MODULE_DEVICE_TABLE(pci, ne2k_pci_tbl);


Registering a driver (1)

Declaring driver hooks and supported devices table:

static struct pci_driver ne2k_driver = { .name = DRV_NAME, .probe = ne2k_pci_init_one, .remove = __devexit_p(ne2k_pci_remove_one), .id_table = ne2k_pci_tbl,#ifdef CONFIG_PM .suspend = ne2k_pci_suspend, .resume = ne2k_pci_resume,#endif /* CONFIG_PM */};


Registering a driver (2)

static int __init ne2k_pci_init(void){ return pci_register_driver(&ne2k_driver);}

static void __exit ne2k_pci_cleanup(void){ pci_unregister_driver (&ne2k_driver);}

The hooks and supported devices are loaded at module loading time.

The probe() hook gets called by the PCI generic codewhen a matching device is found.

Very similar to USB device drivers!


Marking driver hooks (1)

__init: module init function.Code discarded after driver initialization.

__exit: module exit function.Ignored for statically compiled drivers.

__devinit: probe function and all initialization functionsNormal function if CONFIG_HOTPLUG is set. Identical to __init otherwise.

__devinitconst: for the device id table


Marking driver hooks (2)

__devexit: functions called at remove time.Same case as in __devinit

All references to __devinit function addresses should be declared with __devexit_p(fun). This replaces the function address by NULL if this code is discarded.

Example: same driver:static struct pci_driver ne2k_driver = { .name = DRV_NAME, .probe = ne2k_pci_init_one, .remove = __devexit_p(ne2k_pci_remove_one), .id_table = ne2k_pci_tbl,...};


Device initialization steps

Enable the device

Request I/O port and I/O memory resources

Set the DMA mask size(for both coherent and streaming DMA)

Allocate and initialize shared control data (pci_allocate_coherent())

Initialize device registers (if needed)

Register IRQ handler (request_irq())

Register to other subsystems (network, video, disk, etc.)

Enable DMA/processing engines.


Enabling the device

Before touching any device registers, the driver should first execute pci_enable_device(). This will:

Wake up the device if it was in suspended state

Allocate I/O and memory regions of the device(if not already done by the BIOS)

Assign an IRQ to the device(if not already done by the BIOS)

pci_enable_device() can fail. Check the return value!


pci_enable_device example

From drivers/net/ne2kpci.c (Linux 2.6.27):

static int __devinit ne2k_pci_init_one (struct pci_dev *pdev, const struct pci_device_id *ent){

...i = pci_enable_device (pdev);

if (i) return i;

...}


Enabling the device (2)

Enable DMA by calling pci_set_master(). This will:

Enable DMA by setting the bus master bit in the PCI_COMMAND register. The device will then be able to act as a master on the address bus.

Fix the latency timer value if it's set to something bogus by the BIOS.

If the device can use the PCI MemoryWriteInvalidate transaction (writing entire cache lines), you can also call pci_set_mwi():

This enables the PCI_COMMAND bit for MemoryWriteInvalidate

This also ensures that the cache line size register is set correctly.


Accessing configuration registers (1)

Needed to access I/O memory and port information#include <linux/pci.h>

Reading:int pci_read_config_byte(struct pci_dev *dev,

int where, u8 *val);int pci_read_config_word(struct pci_dev *dev,

int where, u16 *val);int pci_read_config_dword(struct pci_dev *dev,

int where, u32 *val);

Example: drivers/net/cassini.cpci_read_config_word(cp>pdev, PCI_STATUS, &cfg);


Accessing configuration registers (2)

Writing:int pci_write_config_byte(struct pci_dev *dev,

int where, u8 val);int pci_write_config_word(struct pci_dev *dev,

int where, u16 val);int pci_write_config_dword(struct pci_dev *dev,

int where, u32 val);

Example: drivers/net/s2io.c/* Clear "detected parity error" bit pci_write_config_word(sp>pdev, PCI_STATUS, 0x8000);


Accessing I/O registers and memory (1)

Each PCI device can have up to 6 I/O or memory regions,described in BAR0 to BAR5.

Access the base address of the I/O region:#include <linux/pci.h>long iobase = pci_resource_start (pdev, bar);

Access the I/O region size:long iosize = pci_resource_len (pdev, bar);

Reserve the I/O region:request_region(iobase, iosize, “my driver”);or simpler:pci_request_region(pdev, bar, “my driver”);or even simpler (regions for all BARs):pci_request_regions(pdev, “my driver”);


Accessing I/O registers and memory (2)

From drivers/net/ne2kpci.c (Linux 2.6.27):ioaddr = pci_resource_start (pdev, 0);irq = pdev>irq;

if (!ioaddr || ((pci_resource_flags (pdev, 0) & IORESOURCE_IO) == 0)) { dev_err(&pdev>dev, "no I/O resource at PCI BAR #0\n"); return ENODEV;}

if (request_region (ioaddr, NE_IO_EXTENT, DRV_NAME) == NULL) { dev_err(&pdev>dev, "I/O resource 0x%x @ 0x%lx busy\n", NE_IO_EXTENT, ioaddr); return EBUSY;}


Setting the DMA mask size (1)

Use pci_dma_set_mask() to declare any device with more (or less) than 32bit bus master capability

In particular, must be done by drivers for PCIX and PCIe compliant devices, which use 64 bit DMA.

If the device can directly address "consistent memory" in System RAM above 4G physical address, register this by calling pci_set_consistent_dma_mask().


Setting the DMA mask size (2)

Example (drivers/net/wireless/ipw2200.c in Linux 2.6.27):err = pci_set_dma_mask(pdev, DMA_32BIT_MASK);

if (!err)err = pci_set_consistent_dma_mask(pdev, DMA_32BIT_MASK);

if (err) {printk(KERN_WARNING DRV_NAME ": No suitable DMA available.\n");goto out_pci_disable_device;

}


Allocate consistent DMA buffers

Now that the DMA mask size has been allocated...

You can allocate your cache consistent buffersif you plan to use such buffers.

See our DMA presentationand Documentation/DMAAPI.txt for details.

http://free-electrons.com/kerneldoc/latest/DMA-API.txt


Initialize device registers

If needed by the device

Set some “capability” fields

Do some vendor specific initialization or resetExample: clear pending interrupts.


Register interrupt handlers

Need to call request_irq() with the IRQF_SHARED flag, because all PCI IRQ lines can be shared.

Registration also enables interrupts, so at this point

Make sure that the device is fully initialized and ready to service interrupts.

Make sure that the device doesn't have any pending interrupt before calling request_irq().

Where you actually call request_irq() can actually depend on the type of device and the subsystem it could be part of (network, video, storage...).

Your driver will then have to register to this subsystem.


PCI device shutdown (1)

In the remove() function, you typically have to undowhat you did at device initialization (probe() function):

Disable the generation of new interrupts.If you don't, the system will get spurious interrupts, and will eventually disable the IRQ line. Bad for other devices on this line!

Release the IRQ

Stop all DMA activity.Needed to be done after IRQs are disabled(could start new DMAs)

Release DMA buffers: streaming first and then consistent ones.


PCI device shutdown

Unregister from other subsystems

Unmap I/O memory and ports with io_unmap().

Disable the device with pci_disable_device().

Unregister I/O memory and ports.If you don't, you won't be able to reload the driver.


Useful resources

Documentation/PCI/pci.txt in the kernel sourcesAn excellent guide to writing PCI drivers, which helpedus to write these slides.

Book: Essential Linux device drivers (Prentice Hall)A very good and recent book!http://freeelectrons.com/redirect/elddbook.html

Book: Linux Device Drivers (O'Reilly)Available under a free documentation license.http://freeelectrons.com/community/kernel/ldd3/

Wikipedia:http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect

http://free-electrons.com/kerneldoc/latest/PCI/pci.txt

http://free-electrons.com/redirect/eldd-book.html

http://free-electrons.com/community/kernel/ldd3/

http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect


Useful resources (2)

PCI Utilities: http://mj.ucw.cz/pciutils.htmllspci: shows PCI bus and device informationsetpci: allows to manipulate PCI device configuration registers

PCI vendor and device id repository:http://pciids.ucw.cz/read/PC/

http://mj.ucw.cz/pciutils.html

http://pci-ids.ucw.cz/read/PC/


Kernel initialization

Kernelinitialization

Michael OpdenackerFree Electrons

© Copyright 20072009, Free Electrons.Creative Commons BYSA 3.0 licenseLatest update: Mar 5, 2010, Document sources, updates and translations:http://freeelectrons.com/docs/kernelinitCorrections, suggestions, contributions and translations are welcome!

http://free-electrons.com/docs/kernel-init


From bootloader to userspace

Kernelinitialization

BootloaderLow level hardware

initializationFetch and copy

Linux kernelto RAM

init processSystem initialization

from userspace


Kernel bootstrap (1)

... LD vmlinux SYSMAP System.map SYSMAP .tmp_System.map MODPOST vmlinux Kernel: arch/arm/boot/Image is ready AS arch/arm/boot/compressed/head.o GZIP arch/arm/boot/compressed/piggy.gz AS arch/arm/boot/compressed/piggy.o CC arch/arm/boot/compressed/misc.o AS arch/arm/boot/compressed/headxscale.o AS arch/arm/boot/compressed/headsharpsl.o LD arch/arm/boot/compressed/vmlinux OBJCOPY arch/arm/boot/zImage Kernel: arch/arm/boot/zImage is ready ...

How the kernel bootstraps itself appears in kernel building.Sharp Spitz example (arm pxa270 based):


Kernel bootstrap (2)

vmlinuxImage

piggy.gz

piggy.o

head.o

misc.o

headcpu.o

headboard.o

vmlinuxzImage

objcopy gzip as ld objcopy

“Kernel proper”:Raw kernelexecutable

(ELF object)

Strippedkernelbinary(binaryobject)

Compressedkernelbinary

asm wrapperaround piggy.gz

+ bootstrapcode

Compositekernel image(ELF object)

Kernel imagefor bootloader(binary object)

Details found by compilingwith make V=1

piggy.S

(in arch/<arch>boot/compressed)


Bootstrap code

head.o:Architecture specific initialization code.This is what is executed by the bootloader

headcpu.o (here headxscale.o):CPU specific initialization code

headboard.o (here headsharpsl.o):Board specific initialization code

misc.o:Decompression routines


Bootstrap code tasks

Main work done by head.o:

Check the architecture, processor and machine type.

Configure the MMU, create page table entriesand enable virtual memory.

Calls the start_kernel function in init/main.c.Same code for all architectures.Anybody interesting in kernel startup should study this file!

http://lxr.free-electrons.com/ident?i=start_kernel



start_kernel main actions

Calls setup_arch(&command_line)(function defined in arch/<arch>/kernel/setup.c), copying the command line from where the bootloader left it.

On arm, this function calls setup_processor(in which CPU information is displayed) and setup_machine(locating the machine in the list of supported machines).

Initializes the console as early as possible(to get error messages)

Initializes many subsystems (see the code)

Eventually calls rest_init.

http://lxr.free-electrons.com/ident?i=setup_arch

http://lxr.free-electrons.com/source/arch/arm/kernel/setup.c

http://lxr.free-electrons.com/ident?i=setup_processor

http://lxr.free-electrons.com/ident?i=setup_machine

http://lxr.free-electrons.com/ident?i=rest_init


rest_init: starting the init process

Starting a new kernel thread which will later become the init process

static void noinline __init_refok rest_init(void) __releases(kernel_lock) { int pid;

kernel_thread(kernel_init, NULL, CLONE_FS | CLONE_SIGHAND); numa_default_policy(); pid = kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES); kthreadd_task = find_task_by_pid(pid); unlock_kernel();

/* * The boot idle thread must execute schedule() * at least one to get things moving: */ preempt_enable_no_resched(); schedule(); preempt_disable();

/* Call into cpu_idle with preempt disabled */ cpu_idle(); }

Source: Linux 2.6.22


kernel_init

kernel_init does 2 main things:

Call do_basic_setupNow that kernel services are ready, start device initialization:static void __init do_basic_setup(void){ /* drivers will send hotplug events */ init_workqueues(); usermodehelper_init(); driver_init(); init_irq_proc(); do_initcalls();}

Call init_post

http://lxr.free-electrons.com/ident?i=kernel_init

http://lxr.free-electrons.com/ident?i=do_basic_setup

http://lxr.free-electrons.com/ident?i=init_post


do_initcalls

Calls pluggable hooks registered with the macros below.Advantage: the generic code doesn't have to know about them.

/* * A "pure" initcall has no dependencies on anything else, and purely * initializes variables that couldn't be statically initialized. * * This only exists for builtin code, not for modules. */ #define pure_initcall(fn) __define_initcall("0",fn,1)

#define core_initcall(fn) __define_initcall("1",fn,1) #define core_initcall_sync(fn) __define_initcall("1s",fn,1s) #define postcore_initcall(fn) __define_initcall("2",fn,2) #define postcore_initcall_sync(fn) __define_initcall("2s",fn,2s) #define arch_initcall(fn) __define_initcall("3",fn,3) #define arch_initcall_sync(fn) __define_initcall("3s",fn,3s) #define subsys_initcall(fn) __define_initcall("4",fn,4) #define subsys_initcall_sync(fn) __define_initcall("4s",fn,4s) #define fs_initcall(fn) __define_initcall("5",fn,5) #define fs_initcall_sync(fn) __define_initcall("5s",fn,5s) #define rootfs_initcall(fn) __define_initcall("rootfs",fn,rootfs) #define device_initcall(fn) __define_initcall("6",fn,6) #define device_initcall_sync(fn) __define_initcall("6s",fn,6s) #define late_initcall(fn) __define_initcall("7",fn,7) #define late_initcall_sync(fn) __define_initcall("7s",fn,7s)


initcall example

From arch/arm/machpxa/lpd270.c

static int __init lpd270_irq_device_init(void) { int ret = sysdev_class_register(&lpd270_irq_sysclass); if (ret == 0) ret = sysdev_register(&lpd270_irq_device); return ret; } device_initcall(lpd270_irq_device_init);

http://lxr.free-electrons.com/source/arch/arm/mach-pxa/lpd270.c


init_post

The last step of Linux booting

First tries to open a console

Then tries to run the init process,effectively turning the current kernel threadinto the userspace init process.


init_post code

static int noinline init_post(void) { free_initmem(); unlock_kernel(); mark_rodata_ro(); system_state = SYSTEM_RUNNING; numa_default_policy();

if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0) printk(KERN_WARNING "Warning: unable to open an initial console.\n");

(void) sys_dup(0); (void) sys_dup(0);

if (ramdisk_execute_command) { run_init_process(ramdisk_execute_command); printk(KERN_WARNING "Failed to execute %s\n", ramdisk_execute_command); }

/* * We try each of these until one succeeds. * * The Bourne shell can be used instead of init if we are * trying to recover a really broken machine. */ if (execute_command) { run_init_process(execute_command); printk(KERN_WARNING "Failed to execute %s. Attempting " "defaults...\n", execute_command); } run_init_process("/sbin/init"); run_init_process("/etc/init"); run_init_process("/bin/init"); run_init_process("/bin/sh");

panic("No init found. Try passing init= option to kernel."); }

Source:init/main.cin Linux 2.6.22



Kernel initialization graph

Bootloader

head.o(bootstrap code)

start_kernel

rest_init

cpu_idle(idle loop)

kernel_init

init_post init process

System initialization

System operation


Kernel initialization What to remember

The bootloader executesbootstrap code.

Bootstrap code initializes the processor and board, and uncompresses the kernel code to RAM, and calls the kernel's start_kernel function.

Copies the command line from the bootloader.

Identifies the processor and machine.

Initializes the console.

Initializes kernel services (memory allocation, scheduling, file cache...)

Creates a new kernel thread (future init process) and continues in the idle loop.

Initializes devices and execute initcalls.

http://lxr.free-electrons.com/ident?i=start_kernel


Serial drivers

Serial drivers

Thomas PetazzoniFree Electrons

© Copyright 2009, Free Electrons.Creative Commons BYSA 3.0 licenseLatest update: Mar 5, 2010, Document sources, updates and translations:http://freeelectrons.com/docs/serialdriversCorrections, suggestions, contributions and translations are welcome!

http://free-electrons.com/docs/serial-drivers


Architecture (1)


Character driverCharacter driverTTY Core

Linediscipline

TTY driverTTY driverserial_core

Serial driver


Architecture (2)

To be properly integrated in a Linux system, serial ports must be visible as TTY devices from userspace applications

Therefore, the serial driver must be part of the kernel TTY subsystem

Until 2.6, serial drivers were implemented directly behind the TTY core

A lot of complexity was involved

Since 2.6, a specialized TTY driver, serial_core, eases the development of serial drivers

See include/linux/serial_core.h for the main definitions of the serial_core infrastructure

The line discipline that cooks the data exchanged with the tty driver. For normal serial ports, N_TTY is used.


Data structures

A data structure representing a driver : uart_driver

Single instance for each driver

uart_register_driver() and uart_unregister_driver()

A data structure representing a port : uart_port

One instance for each port (several per driver are possible)

uart_add_one_port() and uart_remove_one_port()

A data structure containing the pointers to the operations : uart_ops

Linked from uart_port through the ops field


uart_driver

Usually

Defined statically in the driver

Registered in module_init()

Unregistered in module_cleanup()

Contains

owner, usually set to THIS_MODULE

driver_name

dev_name, the device name prefix, usually “ttyS”

major and minor

Use TTY_MAJOR and 64 to get the normal numbers. But they might conflict with the 8250reserved numbers

nr, the maximum number of ports

cons, pointer to the console device (covered later)


uart_driver code example (1)

static struct uart_driver atmel_uart = { .owner = THIS_MODULE, .driver_name = "atmel_serial", .dev_name = ATMEL_DEVICENAME, .major = SERIAL_ATMEL_MAJOR, .minor = MINOR_START, .nr = ATMEL_MAX_UART, .cons = ATMEL_CONSOLE_DEVICE,};

static struct platform_driver atmel_serial_driver = { .probe = atmel_serial_probe, .remove = __devexit_p(atmel_serial_remove), .suspend = atmel_serial_suspend, .resume = atmel_serial_resume, .driver = { .name = "atmel_usart", .owner = THIS_MODULE, },};


uart_driver code example (2)

static int __init atmel_serial_init(void){

uart_register_driver(&atmel_uart);platform_driver_register(&atmel_serial_driver);return 0;

}

static void __exit atmel_serial_exit(void){ platform_driver_unregister(&atmel_serial_driver); uart_unregister_driver(&atmel_uart);}

module_init(atmel_serial_init);module_exit(atmel_serial_exit);

Warning: error management removed!


uart_port

Can be allocated statically or dynamically

Usually registered at probe() time and unregistered at remove() time

Most important fields

iotype, type of I/O access, usually UPIO_MEM for memorymapped devices

mapbase, physical address of the registers

irq, the IRQ channel number

membase, the virtual address of the registers

uartclk, the clock rate

ops, pointer to the operations

dev, pointer to the device (platform_device or other)


uart_port code example (1)

static int __devinit atmel_serial_probe(struct platform_device *pdev){ struct atmel_uart_port *port;

port = &atmel_ports[pdev>id]; port>backup_imr = 0;

atmel_init_port(port, pdev);

uart_add_one_port(&atmel_uart, &port>uart);

platform_set_drvdata(pdev, port);

return 0;}

static int __devexit atmel_serial_remove(struct platform_device *pdev){ struct uart_port *port = platform_get_drvdata(pdev);

platform_set_drvdata(pdev, NULL); uart_remove_one_port(&atmel_uart, port);

return 0;}



static void __devinit atmel_init_port(struct atmel_uart_port *atmel_port, struct platform_device *pdev){ struct uart_port *port = &atmel_port>uart; struct atmel_uart_data *data = pdev>dev.platform_data;

port>iotype = UPIO_MEM; port>flags = UPF_BOOT_AUTOCONF; port>ops = &atmel_pops; port>fifosize = 1; port>line = pdev>id; port>dev = &pdev>dev;

port>mapbase = pdev>resource[0].start; port>irq = pdev>resource[1].start;

tasklet_init(&atmel_port>tasklet, atmel_tasklet_func, (unsigned long)port);

[... see next page ...]



[... continued from previous page ...]

if (data>regs) /* Already mapped by setup code */ port>membase = data>regs; else { port>flags |= UPF_IOREMAP; port>membase = NULL; }

/* for console, the clock could already be configured */ if (!atmel_port>clk) { atmel_port>clk = clk_get(&pdev>dev, "usart"); clk_enable(atmel_port>clk); port>uartclk = clk_get_rate(atmel_port>clk); clk_disable(atmel_port>clk); /* only enable clock when USART is in use */ }}


uart_ops

Important operations

tx_empty(), tells whether the transmission FIFO is empty or not

set_mctrl() and get_mctrl(), allow to set and get the modem control parameters (RTS, DTR, LOOP, etc.)

start_tx() and stop_tx(), to start and stop the transmission

stop_rx(), to stop the reception

startup() and shutdown(), called when the port is opened/closed

request_port() and release_port(), request/release I/O or memory regions

set_termios(), change port parameters

See the detailed description in Documentation/serial/driver


Implementing transmission

The start_tx() method should start transmitting characters over the serial port

The characters to transmit are stored in a circular buffer, implemented by a struct uart_circ structure. It contains

buf[], the buffer of characters

tail, the index of the next character to transmit. After transmit, tail must be updated using tail = tail & (UART_XMIT_SIZE 1)

Utility functions on uart_circ

uart_circ_empty(), tells whether the circular buffer is empty

uart_circ_chars_pending(), returns the number of characters left to transmit

From an uart_port pointer, this structure can be reached using port>info>xmit


Polledmode transmission

foo_uart_putc(struct uart_port *port, unsigned char c) {while(__raw_readl(port>membase + UART_REG1) & UART_TX_FULL)

cpu_relax();__raw_writel(c, port>membase + UART_REG2);

}

foo_uart_start_tx(struct uart_port *port) {struct circ_buf *xmit = &port>info>xmit;

while (!uart_circ_empty(xmit)) {foo_uart_putc(port, xmit>buf[xmit>tail]);xmit>tail = (xmit>tail + 1) & (UART_XMIT_SIZE 1);port>icount.tx++;

}}


Transmission with interrupts (1)

foo_uart_interrupt(int irq, void *dev_id) {[...]if (interrupt_cause & END_OF_TRANSMISSION)

foo_uart_handle_transmit(port);[...]

}

foo_uart_start_tx(struct uart_port *port) {enable_interrupt_on_txrdy();

}


Transmission with interrupts (2)

foo_uart_handle_transmit(port) {

struct circ_buf *xmit = &port>info>xmit;if (uart_circ_empty(xmit) || uart_tx_stopped(port)) {

disable_interrupt_on_txrdy();return;

}

while (! uart_circ_empty(xmit)) {if (! (__raw_readl(port>membase + UART_REG1) &

UART_TX_FULL))Break;

__raw_writel(xmit>buf[xmit>tail],port>membase + UART_REG2);

xmit>tail = (xmit>tail + 1) & (UART_XMIT_SIZE 1);port>icount.tx++;

}

if (uart_circ_chars_pending(xmit) < WAKEUP_CHARS)uart_write_wakeup(port);

}


Reception

On reception, usually in an interrupt handler, the driver must

Increment port>icount.rx

Call uart_handle_break() if a BRK has been received, and if it returns TRUE, skip to the next character

If an error occurred, increment port>icount.parity, port>icount.frame, port>icount.overrun depending on the error type

Call uart_handle_sysrq_char() with the received character, and if it returns TRUE, skip to the next character

Call uart_insert_char() with the received character and a status

Status is TTY_NORMAL is everything is OK, or TTY_BREAK, TTY_PARITY, TTY_FRAME in case of error

Call tty_flip_buffer_push() to push data to the TTY later


Understanding Sysrq

Part of the reception work is dedicated to handling Sysrq

Sysrq are special commands that can be sent to the kernel to make it reboot, unmount filesystems, dump the task state, nice realtime tasks, etc.

These commands are implemented at the lowest possible level so that even if the system is locked, you can recover it.

Through serial port: send a BRK character, send the character of the Sysrq command

See Documentation/sysrq.txt

In the driver

uart_handle_break() saves the current time + 5 seconds in a variable

uart_handle_sysrq_char() will test if the current time is below the saved time, and if so, will trigger the execution of the Sysrq command


Reception code sample (1)

foo_receive_chars(struct uart_port *port) {int limit = 256;

while (limit > 0) {status = __raw_readl(port>membase + REG_STATUS);ch = __raw_readl(port>membase + REG_DATA);flag = TTY_NORMAL;

if (status & BREAK) {port>icount.break++;if (uart_handle_break(port))

Continue;}else if (status & PARITY)

port>icount.parity++;else if (status & FRAME)

port>icount.frame++;else if (status & OVERRUN)

port>icount.overrun++;

[...]


Reception code sample (2)

[...]status &= port>read_status_mask;

if (status & BREAK)flag = TTY_BREAK;

else if (status & PARITY)flag = TTY_PARITY;

else if (status & FRAME)flag = TTY_FRAME;

if (uart_handle_sysrq_char(port, ch))continue;

uart_insert_char(port, status, OVERRUN, ch, flag);}

spin_unlock(& port>lock);tty_flip_buffer_push(port>info>port.tty);spin_lock(& port>lock);

}


Modem control lines

Set using the set_mctrl() operation

The mctrl argument can be a mask of TIOCM_RTS (request to send), TIOCM_DTR (Data Terminal Ready), TIOCM_OUT1, TIOCM_OUT2, TIOCM_LOOP (enable loop mode)

If a bit is set in mctrl, the signal must be driven active, if the bit is cleared, the signal must be driven inactive

Status using the get_mctrl() operation

Must return read hardware status and return a combination of TIOCM_CD (Carrier Detect), TIOCM_CTS (Clear to Send), TIOCM_DSR (Data Set Ready) and TIOCM_RI (Ring Indicator)


set_mctrl() example

foo_set_mctrl(struct uart_port *uart, u_int mctrl) {unsigned int control = 0, mode = 0;

if (mctrl & TIOCM_RTS)control |= ATMEL_US_RTSEN;

elsecontrol |= ATMEL_US_RTSDIS;

if (mctrl & TIOCM_DTS)control |= ATMEL_US_DTREN;

elsecontrol |= ATMEL_US_DTRDIS;

__raw_writel(port>membase + REG_CTRL, control);

if (mctrl & TIOCM_LOOP)mode |= ATMEL_US_CHMODE_LOC_LOOP;

elsemode |= ATMEL_US_CHMODE_NORMAL;

__raw_writel(port>membase + REG_MODE, mode);}


get_mctrl() example

foo_get_mctrl(struct uart_port *uart, u_int mctrl) { unsigned int status, ret = 0;

status = __raw_readl(port>membase + REG_STATUS);

/* * The control signals are active low. */ if (!(status & ATMEL_US_DCD)) ret |= TIOCM_CD; if (!(status & ATMEL_US_CTS)) ret |= TIOCM_CTS; if (!(status & ATMEL_US_DSR)) ret |= TIOCM_DSR; if (!(status & ATMEL_US_RI)) ret |= TIOCM_RI;

return ret;}


termios

“The termios functions describe a general terminal interface that is provided to control asynchronous communications ports”

A mechanism to control from userspace serial port parameters such as

Speed

Parity

Byte size

Stop bit

Hardware handshake

Etc.

See termios(3) for details


set_termios()

The set_termios() operation must

apply configuration changes according to the arguments

update port>read_config_mask and port>ignore_config_mask to indicate the events we are interested in receiving

static void set_termios(struct uart_port *port, struct ktermios *termios, struct ktermios *old)

port, the port, termios, the new values and old, the old values

Relevant ktermios structure fields are

c_cflag with word size, stop bits, parity, reception enable, CTS status change reporting, enable modem status change reporting

c_iflag with frame and parity errors reporting, break event reporting


set_termios() example (1)

static void atmel_set_termios(struct uart_port *port, struct ktermios *termios, struct ktermios *old){ unsigned long flags; unsigned int mode, imr, quot, baud;

mode = __raw_readl(port>membase + REG_MODE); baud = uart_get_baud_rate(port, termios, old, 0, port>uartclk / 16); quot = uart_get_divisor(port, baud);

switch (termios>c_cflag & CSIZE) { case CS5: mode |= ATMEL_US_CHRL_5; break; case CS6: mode |= ATMEL_US_CHRL_6; break; case CS7: mode |= ATMEL_US_CHRL_7; break; default: mode |= ATMEL_US_CHRL_8; break; }

[...]

Read current configuration

Compute the mode modification for the byte size parameter



[...]

if (termios>c_cflag & CSTOPB) mode |= ATMEL_US_NBSTOP_2;

if (termios>c_cflag & PARENB) { /* Mark or Space parity */ if (termios>c_cflag & CMSPAR) { if (termios>c_cflag & PARODD) mode |= ATMEL_US_PAR_MARK; else mode |= ATMEL_US_PAR_SPACE; } else if (termios>c_cflag & PARODD) mode |= ATMEL_US_PAR_ODD; else mode |= ATMEL_US_PAR_EVEN; } else mode |= ATMEL_US_PAR_NONE;

if (termios>c_cflag & CRTSCTS) mode |= ATMEL_US_USMODE_HWHS; else mode |= ATMEL_US_USMODE_NORMAL;

[...]

Compute the mode modification for● the stop bit● Parity● CTS reporting



[...]

port>read_status_mask = ATMEL_US_OVRE; if (termios>c_iflag & INPCK) port>read_status_mask |= (ATMEL_US_FRAME | ATMEL_US_PARE); if (termios>c_iflag & (BRKINT | PARMRK)) port>read_status_mask |= ATMEL_US_RXBRK;

port>ignore_status_mask = 0; if (termios>c_iflag & IGNPAR) port>ignore_status_mask |= (ATMEL_US_FRAME | ATMEL_US_PARE); if (termios>c_iflag & IGNBRK) { port>ignore_status_mask |= ATMEL_US_RXBRK; if (termios>c_iflag & IGNPAR) port>ignore_status_mask |= ATMEL_US_OVRE; }

uart_update_timeout(port, termios>c_cflag, baud);

[...]

Compute the read_status_mask and ignore_status_mask according to the events we're interested in. These values are used in the interrupt handler.

The serial_core maintains a timeout that corresponds to the duration it takes to send the full transmit FIFO. This timeout has to be updated.



[...]

/* Save and disable interupts */ imr = UART_GET_IMR(port); UART_PUT_IDR(port, 1);

/* disable receiver and transmitter */ UART_PUT_CR(port, ATMEL_US_TXDIS | ATMEL_US_RXDIS);

/* set the parity, stop bits and data size */ UART_PUT_MR(port, mode);

/* set the baud rate */ UART_PUT_BRGR(port, quot); UART_PUT_CR(port, ATMEL_US_RSTSTA | ATMEL_US_RSTRX); UART_PUT_CR(port, ATMEL_US_TXEN | ATMEL_US_RXEN);

/* restore interrupts */ UART_PUT_IER(port, imr);

/* CTS flowcontrol and modemstatus interrupts */ if (UART_ENABLE_MS(port, termios>c_cflag)) port>ops>enable_ms(port);}

Finally, apply the mode and baud rate modifications. Interrupts, transmission and reception are disabled when the modifications are made.


Console

To allows early boot messages to be printed, the kernel provides a separate but related facility: console

This console can be enabled using the console= kernel argument

The driver developer must

Implement a console_write() operation, called to print characters on the console

Implement a console_setup() operation, called to parse the console= argument

Declare a struct console structure

Register the console using a console_initcall() function


Console: registration

static struct console serial_txx9_console = { .name = TXX9_TTY_NAME, .write = serial_txx9_console_write, .device = uart_console_device, .setup = serial_txx9_console_setup, .flags = CON_PRINTBUFFER, .index = 1, .data = &serial_txx9_reg,};

static int __init serial_txx9_console_init(void){ register_console(&serial_txx9_console); return 0;}console_initcall(serial_txx9_console_init);

Helper function from the serial_core layer

Ask for the kernel messages buffered during boot to be printed to the console when activated

This will make sure the function is called early during the boot process.

start_kernel() calls console_init() that calls our function


Console : setup

static int __init serial_txx9_console_setup(struct console *co, char *options){ struct uart_port *port; struct uart_txx9_port *up; int baud = 9600; int bits = 8; int parity = 'n'; int flow = 'n';

if (co>index >= UART_NR) co>index = 0; up = &serial_txx9_ports[co>index]; port = &up>port; if (!port>ops) return ENODEV;

serial_txx9_initialize(&up>port);

if (options) uart_parse_options(options, &baud, &parity, &bits, &flow);

return uart_set_options(port, co, baud, parity, bits, flow);}

Function shared with the normal serial driver

Helper function from serial_core that parses the console= string

Helper function from serial_core that calls the >set_termios() operation with the proper arguments to configure the port


Console : write

static void serial_txx9_console_putchar(struct uart_port *port, int ch){ struct uart_txx9_port *up = (struct uart_txx9_port *)port; wait_for_xmitr(up); sio_out(up, TXX9_SITFIFO, ch);}static void serial_txx9_console_write( struct console *co,

const char *s, unsigned int count){ struct uart_txx9_port *up = &serial_txx9_ports[co>index]; unsigned int ier, flcr;

/* Disable interrupts ier = sio_in(up, TXX9_SIDICR); sio_out(up, TXX9_SIDICR, 0);

/* Disable flow control */ flcr = sio_in(up, TXX9_SIFLCR); if (!(up>port.flags & UPF_CONS_FLOW) && (flcr & TXX9_SIFLCR_TES)) sio_out(up, TXX9_SIFLCR, flcr & ~TXX9_SIFLCR_TES);

uart_console_write(&up>port, s, count, serial_txx9_console_putchar);

/* Reenable interrupts */ wait_for_xmitr(up); sio_out(up, TXX9_SIFLCR, flcr); sio_out(up, TXX9_SIDICR, ier);}

Busywait for transmitter ready and output a single character.

Helper function from serial_core that repeatedly calls the given putchar() callback


Practical lab – Serial drivers

Improve the character driver of the previous labs to make it a real serial driver


Network drivers

Network drivers


© Copyright 2009, Free Electrons.Creative Commons BYSA 3.0 licenseLatest update: Mar 5, 2010, Document sources, updates and translations:http://freeelectrons.com/docs/networkdriversCorrections, suggestions, contributions and translations are welcome!

http://free-electrons.com/docs/network-drivers


Architecture

Network stack

Network hardware driver

Bus infrastructure(platform, pci, usb, etc.)

sk_buff net_device


sk_buff

The struct sk_buff is the structure representing a network packet

Designed to easily support encapsulation/decapsulation of data through the protocol layers

In addition to the data itself, an sk_buff maintains

head, the start of the packet

data, the start of the packet payload

tail, the end of the packet payload

end, the end of the packet

len, the amount of data of the packet

These fields are updated when the packet goes through the protocol layers

http://lxr.free-electrons.com/ident?i=sk_buff

http://lxr.free-electrons.com/ident?i=sk_buff


Allocating a SKB

skb = dev_alloc_skb(length + NET_IP_ALIGN);

data, head, taillength +

NET_IP_ALIGN

Function dev_alloc_skb() allows to allocate an SKB

Can be called from an interrupt handler.Usually the case on reception.

On Ethernet, the size allocated is usually the length of the packet + 2, so that the IP header is wordaligned (the Ethernet header is 14 bytes)

http://lxr.free-electrons.com/ident?i=dev_alloc_skb

http://lxr.free-electrons.com/ident?i=NET_IP_ALIGN


http://lxr.free-electrons.com/ident?i=dev_alloc_skb


Reserving space in a SKB

Need to skip NET_IP_ALIGN bytes at the beginning of the SKB

Done with skb_reserve()

skb_reserve(skb, NET_IP_ALIGN);

head

length

NET_IP_ALIGNdata, tail


http://lxr.free-electrons.com/ident?i=skb_reserve

http://lxr.free-electrons.com/ident?i=skb_reserve




Copy the received data

The packet payload must be copied from the DMA buffer to the SKB, using

static inline void skb_copy_to_linear_data(struct sk_buff *skb, const void *from, const unsigned int len);

static inline void skb_copy_to_linear_data_offset(struct sk_buff *skb, const int offset, const void *from, const unsigned int len);

skb_copy_to_linear_data(skb, dmabuffer, length);

head

length

NET_IP_ALIGNdata, tailpacketdata

http://lxr.free-electrons.com/ident?i=skb_copy_to_linear_data

http://lxr.free-electrons.com/ident?i=skb_copy_to_linear_data_offset

http://lxr.free-electrons.com/ident?i=skb_copy_to_linear_data



Update pointers in SKB

skb_put() is used to update the SKB pointers after copying the payload

skb_put(skb, length);

head

length

NET_IP_ALIGNdatapacketdata

tail

http://lxr.free-electrons.com/ident?i=skb_put

http://lxr.free-electrons.com/ident?i=skb_put


struct net_device

This structure represents a single network interface

Allocation takes place with alloc_etherdev()

The size of private data must be passed as argument. The pointer to these private data can be read in net_device>priv

alloc_etherdev() is a specialization of alloc_netdev() for Ethernet interfaces

Registration with register_netdev()

Unregistration with unregister_netdev()

Liberation with free_netdev()


struct net_device_ops

The methods of a network interface. The most important ones:

ndo_open(), called when the network interface is up'ed

ndo_close(), called when the network interface is down'ed

ndo_start_xmit(), to start the transmission of a packet

And others:

ndo_get_stats(), to get statistics

ndo_do_ioctl(), to implement device specific operations

ndo_set_rx_mode(), to select promiscuous, multicast, etc.

ndo_set_mac_address(), to set the MAC address

ndo_set_multicast_list(), to set multicast filters

Set the netdev_ops field in the struct net_device structure to point to the struct net_device_ops structure.


Utility functions

netif_start_queue()

Tells the kernel that the driver is ready to send packets

netif_stop_queue()

Tells the kernel to stop sending packets. Useful at driver cleanup of course, but also when all transmission buffers are full.

netif_queue_stopped()

Tells whether the queue is currently stopped or not

netif_wake_queue()

Wakeup a queue after a netif_stop_queue().The kernel will resume sending packets


Transmission

The driver implements the ndo_start_xmit() operation

The kernel calls this operation with a SKB as argument

The driver sets up DMA buffers and other hardwaredependent mechanisms and starts the transmission

Depending on the number of free DMA buffers available, the driver can also stop the queue with netif_stop_queue()

When the packet has been sent, an interrupt is raised. The driver is responsible for

Acknowledging the interrupt

Freeing the used DMA buffers

Freeing the SKB with dev_kfree_skb_irq()

If the queue was stopped, start it again

Returns NETDEV_TX_OK or NETDEV_TX_BUSY


Reception: original mode

Reception is notified by an interrupt. The interrupt handler should

Allocate an SKB with dev_alloc_skb()

Reserve the 2 bytes offset with skb_reserve()

Copy the packet data from the DMA buffers to the SKB skb_copy_to_linear_data() or skb_copy_to_linear_data_offset()

Update the SKB pointers with skb_put()

Update the skb>protocol field with eth_type_trans(skb, netdevice)

Give the SKB to the kernel network stack with netif_rx(skb)


Reception: NAPI mode (1)The original mode is nice and simple, but when the network traffic is high, the interrupt rate is high. The NAPI mode allows to switch to polled mode when the interrupt rate is too high.

In the network interface private structure, add a struct napi_struct

At driver initialization, register the NAPI poll operation:netif_napi_add(dev, &bp>napi, macb_poll, 64);

dev is the network interface

&bp>napi is the struct napi_struct

macb_poll is the NAPI poll operation

64 is the «weight» that represents the importance of the network interface. It limits the number of packets each interface can feed to the networking core in each polling cycle. If this quota is not met, the driver will return back to interrupt mode. Don't send this quota to a value greater than the number of packets the interface can store.


Reception: NAPI mode (2)

In the interrupt handler, when a packet has been received:if (napi_schedule_prep(&bp>napi)) {

/* Disable reception interrupts */__napi_schedule(& bp>napi);

}

The kernel will call our poll() operation regularly

The poll() operation has the following prototypestatic int macb_poll(struct napi_struct *napi, int budget)

It must receive at most budget packets and push them to the network stack using netif_receive_skb().

If less than budget packets have been received, switch back to interrupt mode using napi_complete(& bp>napi) and reenable interrupts

Must return the number of packets received


Communication with the PHY (1)

Usually, on embedded platforms, the SoC contains the Ethernet controller, that takes care of layer 2 (MAC) communication.

An external PHY is responsible for layer 1 communication.

The MAC and the PHY are connected using a MII or RMII interface

MII = Media Independent Interface

RMII = Reduced Media Independent Interface

This interface contains two wires used for the MDIO bus (Management Data Input/Output)

The Ethernet driver needs to communicate with the PHY to get information about the link (up, down, speed, full or half duplex) and configure the MAC accordingly


Communication with the PHY (2)

SoC

MAC PHYEthernet

connector

(R)MII

MDIO


PHY in the kernel

The kernel provides a framework that

Exposes an API to communicate with the PHY

Allows to implement PHY drivers

Implements a basic generic PHY driver that works with all PHY

Implemented in drivers/net/phy/

Documented in Documentation/networking/phy.txt

http://lxr.free-electrons.com/source/drivers/net/phy/

http://free-electrons.com/kerneldoc/latest/networking/phy.txt


MDIO bus initialization

The driver must create a MDIO bus structure that tells the PHY infrastructure how to communicate with the PHY.

Allocate a MDIO bus structurestruct mii_bus *mii_bus = mdiobus_alloc();

Fill the MDIO bus structuremii_bus>name = “foo”mii_bus>read = foo_mii_bus_read,mii_bus>write = foo_mii_bus_write,snprintf(mii_bus>id, MII_BUS_ID_SIZE, "%x", pdev>id);mii_bus>parent = struct net_device *

The foo_mii_bus_read() and foo_mii_bus_write() are operations to read and write a value to the MDIO bus. They are hardware specific and must be implemented by the driver.


MDIO bus initialization (2)

The >irq[] array must be allocated and initialized. To use polling, set the values to PHY_POLL.mii_bus>irq = kmalloc(sizeof(int)*PHY_MAX_ADDR, GFP_KERNEL);for (i = 0; i < PHY_MAX_ADDR; i++)

bp>mii_bus>irq[i] = PHY_POLL;

Finally, register the MDIO bus. This will scan the bus for PHYs and fill the mii_bus>phy_map[] array with the result.mdiobus_register(bp>mii_bus)


Connection to the PHY

The mdiobus_register() function filled the mii_bus>phy_map[] array with struct phy_device * pointers

The appropriate PHY (usually, only one is detected) must be selected

Then, connecting to the PHY allows to register a callback that will be called when the link changes :

int phy_connect_direct(struct net_device *dev,struct phy_device *phydev,void (*handler)(struct net_device *),u32 flags,phy_interface_t interface

)

interface is usually PHY_INTERFACE_MODE_MII or PHY_INTERFACE_MODE_RMII


Updating MAC capabilities

The MAC and the PHY might have different capabilities. Like a PHY handling Gigabit speed, but not the MAC

The driver is responsible for updating phydev>advertise and phydev>supported to remove any PHY capability that the MAC doesn't support

A typical solution for a 10/100 controller is

phydev>supported &= PHY_BASIC_FEATURES

phydev>advertising = phydev>supported


Handling link changes

The callback that handle link changes should have the following prototypevoid foo_handle_link_change(struct net_device *dev)

It must check the duplex, speed and link fields of the struct phy_device structure, and update the Ethernet controller configuration accordingly

duplex is either DUPLEX_HALF or DUPLEX_FULL

speed is either SPEED_10, SPEED_100, SPEED_1000, SPEED_2500 or SPEED_10000

link is a boolean


Starting and stopping the PHY

After set up, the PHY driver doesn't operate. To make it poll regularly the PHY hardware, one must start it with

phy_start(phydev)

And when the network is stopped, the PHY must also be stopped, using

phy_stop(phydev)


ethtool

ethtool is a userspace tool that allows to query lowlevel information from an Ethernet interface and to modify its configuration

On the kernel side, at the driver level, a struct ethtool_ops structure can be declared and connected to the struct net_device using the ethtool_ops field.

List of operations: get_settings(), set_settings(), get_drvinfo(), get_wol(), set_wol(), get_link(), get_eeprom(), set_eeprom(), get_tso(), set_tso(), get_flags(), set_flags(), etc.

Some of these operations can be implemented using the PHY interface (phy_ethtool_gset(), phy_ethtool_sset()) or using generic operations (ethtool_op_get_link() for example)


Statistics

The network driver is also responsible for keeping statistics up to date about the number of packets/bytes received/transmitted, the number of errors, of collisions, etc.

Collecting these informations is left to the driver

To expose these information, the driver must implement a get_stats() operation, with the following prototypestruct net_device_stats *foo_get_stats

(struct net_device *dev);

The net_device_stats structure must be filled with the driver. It contains fields such as rx_packets, tx_packets, rx_bytes, tx_bytes, rx_errors, tx_errors, rx_dropped, tx_dropped, multicast, collisions, etc.


Power management

To support suspend and resume, the network driver must implement the suspend() and resume() operations

These operations are referenced by the xxx_driver structure corresponding to the bus on which the Ethernet controller is

The suspend() operation should

Call netif_device_detach()

Do the hardwaredependent operations to suspend the devices (like disable the clocks)

The resume() operation should

Do the hardwaredependent operations (like enable the clocks)

Call netif_device_attach()


References

«Essential Linux Device Drivers», chapter 15

«Linux Device Drivers», chapter 17 (a little bit old)

Documentation/networking/netdevices.txt

Documentation/networking/phy.txt

include/linux/netdevice.h, include/linux/ethtool.h, include/linux/phy.h, include/linux/sk_buff.h

And of course, drivers/net/ for several examples of drivers

Driver code templates in the kernel sources:drivers/usb/usbskeleton.cdrivers/net/isaskeleton.cdrivers/net/pciskeleton.cdrivers/pci/hotplug/pcihp_skeleton.c


Practical lab – Network drivers

Implement a working network driver for the MACB Ethernet controller of the AT91SAM9263 CPU


Porting the Linux kernel to an ARM board

Porting the Linux kernel

to an ARM board


© Copyright 20092010, Free Electrons.Creative Commons BYSA 3.0 licenseLatest update: Mar 5, 2010, Document sources, updates and translations:http://freeelectrons.com/docs/kernelportingCorrections, suggestions, contributions and translations are welcome!

http://free-electrons.com/docs/kernel-porting


Porting the Linux kernel

The Linux kernel supports a lot of different CPU architectures

Each of them is maintained by a different group of contributors

See the MAINTAINERS file for details

The organization of the source code and the methods to port the Linux kernel to a new board are therefore very architecturedependent

For example, PowerPC and ARM are very different

PowerPC relies on device trees to describe hardware details

ARM relies on source code only

This presentation is focused on the ARM architecture only


Architecture, CPU and machine

In the source tree, each architecture has its own directoryarch/arm for the ARM architecture

This directory contains generic ARM code

boot, common, configs, kernel, lib, mm, nwfpe, vfp, oprofile, tools

And many directories for different CPU families

mach* directories : machpxa for PXA CPUs, machimx for Freescale iMX CPUs, etc.

Each of these directories contain

Support for the CPU

Support for several boards using this CPU

Some CPU types share some code, in an entity called a platform

platomap contains common code from machomap1 and machomap2


Source code for Calao USB A9263

Taking the case of the Calao USB A9263 board, which uses a AT91SAM9263 CPU.

arch/

arm/

machat91/

AT91 generic codeclock.c, leds.c, irq.c, pm.c

CPUspecific code for the AT91SAM9263at91sam9263.c, at91sam926x_time.c, at91sam9263_devices.c

Board specific codeboardusba9263.c

For the rest of this presentation, we will focus on board support only


Configuration

A configuration option must be defined for the board, in arch/arm/machat91/Kconfig

This option must depend on the CPU type option corresponding to the CPU used in the board

Here the option is ARCH_AT91SAM9263, defined in the same file

A default configuration file for the board can optionally be stored in arch/arm/configs/. For our board, it's usba9263_defconfig

config MACH_USB_A9263 bool "CALAO USBA9263" depends on ARCH_AT91SAM9263 help Select this if you are using a Calao Systems USBA9263. <http://www.calaosystems.com>


Compilation

The source files corresponding to the board support must be associated with the configuration option of the board

This is done in arch/arm/machat91/Makefile

The Makefile also tells which files are compiled for every AT91 CPU

And which files for our particular CPU, the AT91SAM9263

obj$(CONFIG_MACH_USB_A9263) += boardusba9263.o

objy := irq.o gpio.oobj$(CONFIG_AT91_PMC_UNIT) += clock.oobjy += leds.oobj$(CONFIG_PM) += pm.oobj$(CONFIG_AT91_SLOW_CLOCK) += pm_slowclock.o

obj$(CONFIG_ARCH_AT91SAM9263) += at91sam9263.o at91sam926x_time.o at91sam9263_devices.o sam9_smc.o


Machine structure

Each board is defined by a machine structure

The word « machine » is quite confusing since every mach* directory contains several machine definitions, one for each board using a given CPU type

For the Calao board, at the end of arch/arm/machat91/boardusba9263.c

MACHINE_START(USB_A9263, "CALAO USB_A9263") /* Maintainer: calaosystems */ .phys_io = AT91_BASE_SYS, .io_pg_offst = (AT91_VA_BASE_SYS >> 18) & 0xfffc, .boot_params = AT91_SDRAM_BASE + 0x100, .timer = &at91sam926x_timer, .map_io = ek_map_io, .init_irq = ek_init_irq, .init_machine = ek_board_init,MACHINE_END


Machine structure macros

MACHINE_START and MACHINE_END

Macros defined in arch/arm/include/asm/mach/arch.h

They are helpers to define a struct machine_desc structure stored in a specific ELF section

Several machine_desc structures can be defined in a kernel, which means that the kernel can support several boards.

The right structure is chosen at boot time


Machine type number

In the ARM architecture, each board type is identified by a machine type number

The latest machine type numbers list can be found athttp://www.arm.linux.org.uk/developer/machines/download.php

A copy of it exists in the kernel tree in arch/arm/tools/machtypes

For the Calao boardusb_a9263 MACH_USB_A9263 USB_A9263 1710

At compile time, this file is processed to generate a header file, include/asmarm/machtypes.h

For the Calao board#define MACH_TYPE_USB_A9263 1710

And a few other macros in the same file

http://www.arm.linux.org.uk/developer/machines/download.php


Machine type number

The machine type number is set in the MACHINE_START() definitionMACHINE_START(USB_A9263, "CALAO USB_A9263")

At run time, the machine type number of the board on which the kernel is running is passed by the bootloader in register r1

Very early in the boot process (arch/arm/kernel/head.S), the kernel calls __lookup_machine_type in arch/arm/kernel/headcommon.S

__lookup_machine_type looks at all the machine_desc structures of the special ELF section

If it doesn't find the requested number, prints a message and stops

If found, it knows the machine descriptions and continues the boot process


Early debugging and boot parameters

Early debugging

phys_io is the physical address of the I/O space

io_pg_offset is the offset in the page table to remap the I/O space

These are used when CONFIG_DEBUG_LL is enabled to provide very early debugging messages on the serial port

Boot parameters

boot_params is the location where the bootloader has left the boot parameters (the kernel command line)

The bootloader can override this address in register r2

See also Documentation/arm/Booting for the details of the environment expected by the kernel when booted


System timer

The timer field point to a struct sys_timer structure, that describes the system timer

Used to generate the periodic tick at HZ frequency to call the scheduler periodically

On the Calao board, the system timer is defined by the at91sam926x_timer structure in at91sam926x_time.c

It contains the interrupt handler called at HZ frequency

It is integrated with the clockevents and the clocksource infrastructures

See include/linux/clocksource.h and include/linux/clockchips.h for details


map_io()

The map_io() function points to ek_map_io(), which

Initializes the CPU using at91sam9263_initialize()

Map I/O space

Register and initialize the clocks

Configures the debug serial port and set the console to be on this serial port

Called at the very beginning of the C code execution

init/main.c: start_kernel()

arch/arm/kernel/setup.c: setup_arch()

arch/arm/mm/mmu.c: paging_init()

arch/arm/mm/mmu.c: devicemaps_init()

mdesc>map_io()


init_irq()

init_irq() to initialize the IRQ hardware specific details

Implemented by ek_init_irq(), which calls at91sam9263_init_interrupts() in at91sam9263.c, which mainly calls at91_aic_init() in irq.c

Initialize the interrupt controller, assign the priorities

Register the IRQ chip (irq_chip structure) to the kernel generic IRQ infrastructure, so that the kernel knows how to ack, mask, unmask the IRQs

Called a little bit later than map_io()

init/main.c: start_kernel()

arch/arm/kernel/irq.c: init_IRQ()

init_arch_irq() (equal to mdesc>init_irq)


init_machine()

init_machine() completes the initialization of the board by registering all platform devices

Called by customize_machines() in arch/arm/kernel/setup.c

This function is an arch_initcall (list of functions whose address is stored in a specific ELF section, by levels)

At the end of kernel initialization, just before running the first userspace program init:

init/main.c: kernel_init()

init/main.c: do_basic_setup()

init/main.c: do_initcalls()

Calls all initcalls, level by level


init_machine() for Calao

For the Calao board, implement in ek_board_init()

Registers serial ports, USB host, USB device, SPI, Ethernet, NAND flash, 2IC, buttons and LEDs

Uses at91_add_device_*() helpers, defined in at91sam9263_devices.c

These helpers call platform_device_register() to register the different platform_device structures defined in the same file

For some devices, the board specific code does the registration itself (buttons) or passes boardspecific data to the registration helper (USB host and device, NAND, Ethernet, etc.)


Drivers

The at91sam9263_devices.c file doesn't implement the drivers for the platform devices

The drivers are implemented at different places of the kernel tree

For the Calao board

USB host, driver at91_ohci, drivers/usb/host/ohciat91.c

USB device, driver at91_udc, drivers/usb/gadget/at91_udc.c

Ethernet, driver macb, drivers/net/macb.c

NAND, driver atmel_nand, drivers/mtd/nand/atmel_nand.c

I2C on GPIO, driver i2cgpio, drivers/i2c/busses/i2cgpio.c

SPI, driver atmel_spi, drivers/spi/atmel_spi.c

Buttons, driver gpiokeys, drivers/input/keyboard/gpio_keys.c

All these drivers are selected by the readymade configuration file


Power management

PowerManagement


Free Electrons

© Copyright 20072009, Free Electrons.Creative Commons BYSA 3.0 licenseLatest update: Mar 5, 2010, Document sources, updates and translations:http://freeelectrons.com/docs/powerCorrections, suggestions, contributions and translations are welcome!

http://free-electrons.com/docs/power


General observations

Power management development efforts in the Linux kerneloriginally very PC centric (using ACPI or APM).

However, there is now substantial work to improve power management in the embedded Linux world as well.

For example, it is now possible to implement suspend to disk and even to flash (“hibernating”) wherever suspend to RAM is available.


PM building blocks

Several power management « building blocks »

Suspend and resume

CPUidle

Frequency and voltage scaling

Applications

Independent « building blocks » that can be improved gradually during development

Presentation and « building blocks » term inspired from Kevin Hilman talks on power management:http://www.celinux.org/elc08_presentations/PM_Building_Blocks1.pdf

http://www.celinux.org/elc08_presentations/PM_Building_Blocks1.pdf


Clock framework (1)

Generic framework to manage clocks used by devices in the system

Allows to reference count clock usersand to shutdown the unused clocks to save power

Simple API described inhttp://freeelectrons.com/kerneldoc/latest/DocBook/kernelapi/ch17.html

clk_get() to get a reference to a clock

clk_enable() to start the clock

clk_disable() to stop the clock

clk_put() to free the clock source

clk_get_rate() to get the current rate

http://free-electrons.com/kerneldoc/latest/DocBook/kernel-api/ch17.html


Clock framework (2)

The clock framework API and the clk structure are usually implemented by each architecture

See arch/arm/machat91/clock.c for an example

Clocks are identified by a name string specific to a given platform

Drivers can then use the clock API.Example from drivers/net/macb.c:

clk_get() called when device is probed,clk_put() when removed

clk_enable() and clk_disable()on power management notifications

http://lxr.free-electrons.com/source/arch/arm/mach-at91/clock.c

http://lxr.free-electrons.com/source/drivers/net/macb.c


Clock disable implementation

From arch/arm/machat91/clock.c: (2.6.32)static void __clk_disable(struct clk *clk){ BUG_ON(clk>users == 0); if (clk>users == 0 && clk>mode) clk>mode(clk, 0); if (clk>parent) __clk_disable(clk>parent);}

Example mode function (same file):

static void pmc_sys_mode(struct clk *clk, int is_on){ if (is_on) at91_sys_write(AT91_PMC_SCER, clk>pmc_mask); else at91_sys_write(AT91_PMC_SCDR, clk>pmc_mask);}

Call the hardware function switching off this clock

http://lxr.free-electrons.com/source/arch/arm/mach-at91/clock.c


Suspend and resume

Infrastructure in the kernel to support suspend and resume

Platform hooks

prepare(), enter(), finish(), valid()in a platform_suspend_ops structure

Registered using the suspend_set_ops() function

See arch/arm/machat91/pm.c

Device drivers

suspend() and resume() hooks in the *_driver structures (platform_driver, usb_driver, etc.)

See drivers/net/macb.c

http://lxr.free-electrons.com/ident?i=platform_suspend_ops

http://lxr.free-electrons.com/ident?i=suspend_set_ops

http://lxr.free-electrons.com/source/arch/arm/mach-at91/pm.c

http://lxr.free-electrons.com/source/drivers/net/macb.c


Board specific power management

Typically takes care of battery and charging management.

Also defines presuspend and postsuspend handlers.

Example:arch/arm/machpxa/spitz_pm.c

http://lxr.free-electrons.com/source/arch/arm/mach-pxa/spitz_pm.c


arch/arm/mach<cpu>/sleep.S

Assembly code implementing CPU specificsuspend and resume code. Note: only found on arm, just 3 other occurrences in other architectures, with other paths.

First scenario: only a suspend function. The code goes in sleep state (after enabling DRAM selfrefresh), and continues with resume code.

Second scenario: suspend and resume functions.Resume functions called by the bootloader.

Examples to look at:arch/arm/machomap2/sleep.S (1st case)arch/arm/machpxa/sleep.S (2nd case)

http://lxr.free-electrons.com/source/arch/arm/mach-pxa/sleep.S


Triggering suspend

Whatever the power management implementation, CPU specific suspend_ops functions are called by the enter_state function.

enter_state also takes care of executing the suspend and resume functions for your devices.

The execution of this function can be triggered from userspace. To suspend to RAM:echo mem > /sys/power/state

Can also use the s2ram programfrom http://suspend.sourceforge.net/

Read kernel/power/suspend.c

http://lxr.free-electrons.com/ident?i=enter_state

http://lxr.free-electrons.com/ident?i=enter_state

http://suspend.sourceforge.net/

http://lxr.free-electrons.com/source/kernel/power/suspend.c


Saving power in the idle loop

The idle loop is what you runwhen there's nothing left to run in the system.

Implemented in all architecturesin arch/<arch>/kernel/process.c

Example to read: look for cpu_idle inarch/arm/kernel/process.c

Each ARM cpu defines its own arch_idle function.

The CPU can run power saving HLT instructions, enter NAP mode, and even disable the timers (tickless systems).

See also http://en.wikipedia.org/wiki/Idle_loop

http://lxr.free-electrons.com/ident?i=arch_idle

http://en.wikipedia.org/wiki/Idle_loop


Managing idle

Adding support for multiple idle levels

Modern CPUs have several sleep states offering different power savings with associated wake up latencies

Since 2.6.21, the dynamic tick feature allows to remove the periodic tick to save power, and to know when the next event is scheduled, for smarter sleeps.

CPUidle infrastructure to change sleep states

Platformspecific driver defining sleep states and transition operations

Platformindependent governors (ladder and menu)

Added in 2.6.24 for x86/ACPI, work in progress on various ARM cpus.(look for cpuidle* files under arch/arm/)

See Documentation/cpuidle/ in kernel sources.

http://lxr.free-electrons.com/source/arch/arm/

http://free-electrons.com/kerneldoc/latest/cpuidle/


PowerTOP

http://www.lesswatts.org/projects/powertop/

With dynamic tick, new interest in fixing the kernel code and applications that wake up the system too often

PowerTOP application allows to track the worst offender

Experimental availability on arm (OMAP cpus)

http://www.lesswatts.org/projects/powertop/


Frequency and voltage scaling (1)

Frequency and voltage scaling possible through the cpufreq kernel infrastructure.

Generic infrastructuredrivers/cpufreq/cpufreq.cinclude/linux/cpufreq.h

Generic governors, responsible for deciding frequency and voltage transitions

performance: maximum frequency

powersave: minimum frequency

ondemand: measures CPU consumption to adjust frequency

conservative: often better than ondemand.Only increases frequency gradually when the CPU gets loaded.

userspace: leaves the decision to an userspace daemon.

This infrastructure can be controlled from/sys/devices/system/cpu/cpu<n>/cpufreq/

http://lxr.free-electrons.com/source/drivers/cpufreq/cpufreq.c

http://lxr.free-electrons.com/source/include/linux/cpufreq.h


Frequency and voltage scaling (2)

CPU support code in architecture dependent files.Example to read: arch/arm/platomap/cpuomap.c

Must implement the operations of the cpufreq_driver structureand register them using cpufreq_register_driver()

init() for initialization

exit() for cleanup

verify() to verify the userchosen policy

setpolicy() or target()to actually perform the frequency change

etc.

See Documentation/cpufreq/ for useful explanations

http://lxr.free-electrons.com/source/arch/arm/plat-omap/cpu-omap.c

http://lxr.free-electrons.com/ident?i=cpufreq_register_driver

http://free-electrons.com/kerneldoc/latest/cpu-freq/


PM QoS

PM QoS is a framework developed by Intel introduced in 2.6.25

It allows kernel code and applications to set their requirements in terms of

CPU DMA latency

Network latency

Network throughput

According to these requirements, PM QoS allows kernel drivers to adjust their power management

See Documentation/power/pm_qos_interface.txtand Mark Gross' presentation at ELC 2008

Still in very early deployment (only 4 drivers in 2.6.32).

http://free-electrons.com/kerneldoc/latest/power/pm_qos_interface.txt


Regulator framework

Modern embedded hardware have hardwareresponsible for voltage and current regulation

The regulator framework allows to take advantage of this hardware to save power when parts of the system are unused

A consumer interface for device drivers (i.e users)

Regulator driver interface for regulator drivers

Machine interface for board configuration

sysfs interface for userspace

Merged in Linux 2.6.27.See Documentation/power/regulator/ in kernel sources.

See Liam Girdwood's presentation at ELC 2008http://freeelectrons.com/blog/elc2008report#girdwood

http://free-electrons.com/kerneldoc/latest/power/regulator/

http://free-electrons.com/blog/elc-2008-report#girdwood


BSP work for a new board

In case you just need to create a BSP for your board, and your CPU already has full PM support, you should just need to:

Create clock definitions and bind your devices to them.

Implement PM handlers (suspend, resume) in the drivers for your board specific devices.

Implement board specific power management if needed(mainly battery management)

Implement regulator framework hooks for your board if needed.

All other parts of the PM infrastructure should be already there:suspend / resume, cpuidle, cpu frequency and voltage scaling.


Useful resources

Documentation/power/ in the Linux kernel sources.Will give you many useful details.

http://lesswatts.orgIntel effort trying to create a Linux power saving community.Mainly targets Intel processors.Lots of useful resources.

http://freeelectrons.com/redirect/elc2007suspend.htmlELC 2007, Toshiba: indepth review of their Suspend to RAM implementation on Freescale 74xx, without PMU.

http://free-electrons.com/kerneldoc/latest/power/

http://lesswatts.org/

http://free-electrons.com/redirect/elc2007-suspend.html


Practical lab – Power management

Suspend and resume your Linux system

Change the CPU frequency of your system


Free Electrons

Kerneladvice and resources


Free Electrons

© Copyright 2009, Free Electrons.Creative Commons BYSA 3.0 licenseLatest update: Mar 5, 2010, Document sources, updates and translations:http://freeelectrons.com/docs/kernelresourcesCorrections, suggestions, contributions and translations are welcome!

http://free-electrons.com/docs/kernel-resources


Solving issues

If you face an issue, and it doesn't look specific to your work but rather to the tools you are using, it is very likely that someone else already faced it.

Search the Internet for similar error reports.

You have great chances of finding a solution or workaround, or at least an explanation for your issue.

Otherwise, reporting the issue is up to you!


Getting help

If you have a support contract, ask your vendor.

Otherwise, don't hesitate to share your questions and issues

Either contact the Linux mailing list for your architecture (like linuxarmkernel or linuxshdev...).

Or contact the mailing list for the subsystem you're dealing with (linuxusbdevel, linuxmtd...). Don't ask the maintainer directly!

Most mailing lists come with a FAQ page. Make sure you read it before contacting the mailing list.

Useful IRC resources are available too(for example on http://kernelnewbies.org).

Refrain from contacting the Linux Kernel mailing list, unless you're an experienced developer and need advice.

http://kernelnewbies.org/


Getting contributions

Applies if your project can interest other people:developing a driver or filesystem, porting Linux on a new processor, board or device available on the market...

External contributors can help you a lot by

Testing

Writing documentation

Making suggestions

Even writing code


Encouraging contributions

Open your development process: mailing list, Wiki, public CVS read access

Let everyone contribute according to their skills and interests.

Release early, release often

Take feedback and suggestions into account

Recognize contributions

Make sure status and documentation are up to date

Publicize your work and progress to broader audiences


Reporting Linux bugs

First make sure you're using the latest version

Make sure you investigate the issue as much as you can:see Documentation/BUGHUNTING

Make sure the bug has not been reported yet. A bug tracking system(http://bugzilla.kernel.org/) exists but very few kernel developers use it. Best to use web search engines (accessing public mailing list archives)

If the subsystem you report a bug on has a mailing list, use it. Otherwise, contact the official maintainer (see the MAINTAINERS file). Always give as many useful details as possible.

http://free-electrons.com/kerneldoc/latest/BUG-HUNTING

http://bugzilla.kernel.org/

http://lxr.free-electrons.com/source/MAINTAINERS


How to submit patches or drivers

Don't merge patches addressing different issues

You should identify and contact the official maintainerfor the files to patch.

See Documentation/SubmittingPatches for details. For trivial patches, you can copy the Trivial Patch Monkey.

See also http://kernelnewbies.org/UpstreamMerge for very helpful advice to have your code merged upstream (by Rik van Riel).

Patches related to embedded systems can be submitted on the linuxembedded mailing list: http://vger.kernel.org/vgerlists.html#linuxembedded

ARM platform: it's best to submit your ARM patches to Russell King's patch system: http://www.arm.linux.org.uk/developer/patches/

http://free-electrons.com/kerneldoc/latest/SubmittingPatches

http://kernelnewbies.org/UpstreamMerge

http://vger.kernel.org/vger-lists.html#linux-embedded

http://www.arm.linux.org.uk/developer/patches/


How to become a kernel developer?

Greg KroahHartman wrote useful guidelines in the kernel documentation:Documentation/HOWTO

How to Participate in the Linux Community (by Jonathan Corbet)A Guide To The Kernel Development Processhttp://ldn.linuxfoundation.org/documentation/howparticipatelinuxcommunity

Read this last document first.It is probably sufficient!

http://ldn.linuxfoundation.org/documentation/how-participate-linux-community



Advice and resourcesReferences


Specific training materials

More training materials available on http://freeelectrons.com/docs:

New features in 2.6

Kernel initialization

Porting Linux to new hardware

Power management

USB device drivers

Block drivers

More will be available in the next months.

Don't hesitate to ask us to create the ones you need for a training session!

http://free-electrons.com/docs


Information sites (1)

Linux Weekly Newshttp://lwn.net/

The weekly digest off all Linux and free software information sources

In depth technical discussions about the kernel

Subscribe to finance the editors ($5 / month)

Articles available for non subscribers after 1 week.

http://lwn.net/


Information sites (2)

Kernel Podcasthttp://www.kernelpodcast.org/

Daily podcast about Linux kernel development

Load it on your media player!

KernelTraphttp://kerneltrap.org/

Forum website for kernel developers

News, articles, whitepapers, discussions, polls, interviews

Perfect if a digest is not enough!

http://www.kernelpodcast.org/

http://kerneltrap.org/


Useful reading (1)

Essential Linux Device Drivers, April 2008http://freeelectrons.com/redirect/elddbook.html

By Sreekrishnan Venkateswaran, an embedded IBM engineer with more than 10 years of experience

Covers a wide range of topics not covered by LDD : serial drivers, input drivers, I2C, PCMCIA and Compact Flash, PCI, USB, video drivers, audio drivers, block drivers, network drivers, Bluetooth, IrDA, MTD, drivers in userspace, kernel debugging, etc.

« Probably the most wide ranging and complete Linux device driver book I've read » Alan Cox

http://free-electrons.com/redirect/eldd-book.html


Useful reading (2)

Writing Linux Device drivers, September 2009http://www.coopj.com/

Self published by Jerry CoopersteinAvailable like any other book (Amazon and others)

Though not as thorough as the previous book on specific drivers, still a good complement on multiple aspects of kernel and device driver development.

Based on Linux 2.6.31

Multiple exercises. Solutions available.

http://www.coopj.com/


Useful reading (3)

Linux Device Drivers, 3rd edition, Feb 2005

By Jonathan Corbet, Alessandro Rubini,Greg KroahHartman, O'Reillyhttp://www.oreilly.com/catalog/linuxdrive3/

Freely available online!Great companion to the printed bookfor easy electronic searches!http://lwn.net/Kernel/LDD3/ (1 PDF file per chapter)http://freeelectrons.com/community/kernel/ldd3/ (single PDF file)

Getting outdatedBut still useful for Linux device driver writers!

http://www.oreilly.com/catalog/linuxdrive3/

http://lwn.net/Kernel/LDD3/

http://free-electrons.com/community/kernel/ldd3/


Useful reading (4)

Linux Kernel Development, 2nd Edition, Jan 2005Robert Love, Novell Presshttp://freeelectrons.com/redirect/lkd2book.htmlA very synthetic and pleasant way to learn about kernelsubsystems (beyond the needs of device driver writers)

Understanding the Linux Kernel, 3rd edition, Nov 2005Daniel P. Bovet, Marco Cesati, O'Reillyhttp://oreilly.com/catalog/understandlk/An extensive review of Linux kernel internals,covering Linux 2.6 at last.Unfortunately, only covers the PC architecture.

http://free-electrons.com/redirect/lkd2-book.html

http://oreilly.com/catalog/understandlk/


Useful online resources

Linux kernel mailing list FAQhttp://www.tux.org/lkml/Complete Linux kernel FAQRead this before asking a question to the mailing list

Kernel Newbieshttp://kernelnewbies.org/Glossary, articles, presentations, HOWTOs,recommended reading, useful tools for peoplegetting familiar with Linux kernel or driver development.

Kernel glossary:http://kernelnewbies.org/KernelGlossary

http://www.tux.org/lkml/

http://kernelnewbies.org/

http://kernelnewbies.org/KernelGlossary


International conferences

Japan Linux Symposium: Tokyo, Oct 2123, 2009 http://events.linuxfoundation.org/events/japanlinuxsymposiumA conference right after the kernel summit. Many key kernel hackers should still be there!

linux.conf.au: http://linux.org.au/conf/ (Australia / New Zealand)Features a few presentations by key kernel hackers.

Don't miss our free conference videos onhttp://freeelectrons.com/community/videos/conferences/!

http://events.linuxfoundation.org/events/japan-linux-symposium

http://linux.org.au/conf/

http://free-electrons.com/community/videos/conferences/


ARM resources

ARM Linux project: http://www.arm.linux.org.uk/

Developer documentation: http://www.arm.linux.org.uk/developer/

armlinuxkernel mailing list:http://lists.infradead.org/mailman/listinfo/linuxarmkernel

FAQ: http://www.arm.linux.org.uk/armlinux/mlfaq.php

How to post kernel fixes:http://www.arm.uk.linux.org/developer/patches/

ARMLinux @ Simtec: http://armlinux.simtec.co.uk/A few useful resources: FAQ, documentation and Who's who!

ARM Limited: http://www.linuxarm.com/Wiki with links to useful developer resources

http://www.arm.linux.org.uk/

http://www.arm.linux.org.uk/developer/

http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

http://www.arm.linux.org.uk/armlinux/mlfaq.php

http://www.arm.uk.linux.org/developer/patches/

http://armlinux.simtec.co.uk/

http://www.linux-arm.com/



Advice and resourcesLast advice


Use the Source, Luke!

Many resources and tricks on the Internet find you will, but solutions to all technical issues only in the Source lie.

Thanks to LucasArts


Introduction to Git

Introduction to Git


© Copyright 2009, Free Electrons.Creative Commons BYSA 3.0 licenseLatest update: Mar 5, 2010, Document sources, updates and translations:http://freeelectrons.com/docs/gitCorrections, suggestions, contributions and translations are welcome!

http://free-electrons.com/docs/git


What is Git?

A version control system, like CVS, SVN, Perforce or ClearCase

Originally developed for the Linux kernel development, now used by a large number of projects, including UBoot, GNOME, Buildroot, uClibc and many more

Contrary to CVS or SVN, Git is a distributed version control system

No central repository

Everybody has a local repository

Local branches are possible, and very important

Easy exchange of code between developers

Wellsuited to the collaborative development model used in opensource projects


Install and setup

Git is available as a package in your distributionsudo aptget install gitcore

Everything is available through the git command

git has many commands, called using git <command>, where <command> can be clone, checkout, branch, etc.

Help can be found for a given command using git help <command>

Setup your name and email address

They will be referenced in each of your commits

git config global user.name 'My Name'

git config global user.email [email protected]


Clone a repository

To start working on a project, you use Git's clone operation.

With CVS or SVN, you would have used the checkout operation, to get a working copy of the project (latest version)

With Git, you get a full copy of the repository, including the history, which allows to perform most of the operations offline.

Cloning Linus Torvalds' Linux kernel repositorygit clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux2.6.git

git:// is a special Git protocol. Most repositories can also be accessed using http://, but this is slower.

After cloning, in linux2.6/, you have the repository and a working copy of the master branch.


Explore the history

git log will list all the commits. The latest commit is the first.

git log p will list the commits with the corresponding diff

The history in Git is not linear like in CVS or SVN, but it is a graph of commits

Makes it a little bit more complicated to understand at the beginning

But this is what allows the powerful features of Git (distributed, branching, merging)

commit 4371ee353c3fc41aad9458b8e8e627eb508bc9a3Author: Florian Fainelli <[email protected]>Date: Mon Jun 1 02:43:17 2009 0700

MAINTAINERS: take maintainership of the cpmac Ethernet driver This patch adds me as the maintainer of the CPMAC (AR7) Ethernet driver. Signedoffby: Florian Fainelli <[email protected]> Signedoffby: David S. Miller <[email protected]>


Visualize the history

gitk is a graphical tool that represents the history of the current Git repository

Can be installed from the gitk package


Visualize the history

Another great tool is the Web interface to Git. For the kernel, it is available at http://git.kernel.org/

http://git.kernel.org/


Update your repository

The repository that has been cloned at the beginning will change over time

Updating your local repository to reflect the changes of the remote repository will be necessary from time to time

git pull

Internally, does two things

Fetch the new changes from the remote repository (git fetch)

Merge them in the current branch (git merge)


Tags

The list of existing tags can be found usinggit tag l

To check out a working copy of the repository at a given taggit checkout <tagname>

To get the list of changes between a given tag and the latest available versiongit log v2.6.30..master

List of changes with diff on a given file between two tagsgit log v2.6.29..v2.6.30 MAINTAINERS

With gitkgitk v2.6.30..master


Branches

To start working on something, the best is to make a branch

It is localonly, nobody except you sees the branch

It is fast

It allows to split your work on different topics, try something and throw it away

It is cheap, so even if you think you're doing something small and quick, do a branch

Unlike other version control systems, Git encourages the use of branches. Don't hesitate to use them.


Branches

Create a branchgit branch <branchname>

Move to this branchgit checkout <branchname>

Both at once (create and switch to branch)git checkout b <branchname>

List of local branchesgit branch l

List of all branches, including remote branchesgit branch a


Making changes

Edit a file with your favorite text editor

Get the status of your working copygit status

Git has a feature called the index, which allows you to stage your commits before committing them. It allows to commit only part of your modifications, by file or even by chunk.

On each modified filegit add <filename>

Then commit. No need to be online or connected to commit.git commit

If all modified files should be part of the commitgit commit a


Sharing changes by email

The simplest way of sharing a few changes is to send patches by email

The first step is to generate the patchesgit formatpatch n master..<yourbranch>

Will generate one patch for each of the commits done on <yourbranch>

The patch files will be 0001...., 0002...., etc.

The second step is to send these patchesgit sendemail compose to [email protected] 00*.patch

Assumes that the local mail system is properly configuredNeeds the gitemail package in Ubuntu.

Or git config allows to set the SMTP server, port, user and password if needed


Sharing changes: your own repository

If you do a lot of changes and want to ease collaboration with others, the best is to have your own repository

Create a bare version of your repositorycd /tmpgit clone bare ~/project project.gittouch project.git/gitdaemonexportok

Transfer the contents of project.git to a publiclyvisible place (reachable readonly by HTTP for everybody, and readwrite by you through SSH)

Tell people to clone http://yourhost.com/path/to/project.git

Push your changes usinggit push ssh://yourhost.com/path/toproject.git srcbranch:destbranch

http://yourhost.com/path/to/project.git


Tracking remote trees

In addition to the official Linus Torvalds tree, you might want to use other development or experimental trees

The OMAP tree at git://git.kernel.org/pub/scm/linux/kernel/git/tmlind/linuxomap2.6.git

The realtime tree at git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux2.6rt.git

The git remote command allows to manage remote treesgit remote add rt git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux2.6rt.git

Get the contents of the treegit fetch rt

Switch to one of the branchesgit checkout rt/master


About Git

We have just seen the very basic features of Git.A lot more interesting features are available (rebasing, bisection, merging and more)

References

Git Manualhttp://www.kernel.org/pub/software/scm/git/docs/usermanual.html

Git Bookhttp://book.gitscm.com/

Git official websitehttp://gitscm.com/

James Bottomley's tutorial on using Githttp://freeelectrons.com/pub/video/2008/ols/ols2008jamesbottomleygit.ogg

http://www.kernel.org/pub/software/scm/git/docs/user-manual.html

http://book.git-scm.com/

http://git-scm.com/

http://free-electrons.com/pub/video/2008/ols/ols2008-james-bottomley-git.ogg


Practical lab – Git

Clone a Git repository and explore history

Make and share changes to a project managed in Git


Archive your lab directory

Clean up files that are easy to retrieve, remove downloads.

Generate an archive of your lab directory.


Related documents

All our technical presentationson http://freeelectrons.com/docs

Linux kernelDevice driversArchitecture specificsEmbedded Linux system development

http://free-electrons.com/docs


Life after training

Here are things we could do to support you in your embedded Linux and kernel projects:

BSP development for your hardware(drivers, bootloader, toolchain)

Make the official Linux sources support your hardware

System development and integration

System optimization

Hunting and fixing nasty bugs

More training: see http://freeelectrons.com/training/. Your colleagues who missed this class could go to our public sessions.

See http://freeelectrons.com/developmentand http://freeelectrons.com/services

http://free-electrons.com/training/

http://free-electrons.com/development

http://free-electrons.com/services


Last slides

Thank you! And may the Source be with you

slides

Documents

legalcode corrections

comdocskernelintro corrections

license latestupdate

tqueue init

mbflash 2usb2

c kernelsourcecode

thefasterwe progressasagroup

mbitethernetport poweredbyusb