Thu 6 March, 4:10pm, Victor Kamensky

LCA14-416: Big Endian - Where are we?

• Mandatory read:Big and Little Endian Inside Out – Ben Herrenschmidt, LPC12

• ARM CPU architecture is endian agnostic - can do either• ARM Linux ecosystem traditionally is Little Endian• Surge of interest in ARM Big Endian support

What is Big and Little Endian?

• Very big, 100M+ LOC, legacy code, Big Endian only• Runs on MIPS and PPC CPUs, looking forward to ARM

CPU adoption• Not because Big Endian is better for networking

applications

Why is Big Endian?

• Last few years ARM Linux runs in Little Endian mode• XScale CPU runs in BE32 mode• Big Endian BE32 ecosystem for XScale CPUs still exists

in distros like OpenWRT• Some vendors offered very minimal V7 BE BSP support

ARM Big Endian Linux History

• Build reasonable, supported, Linux Big Endian ecosystem for ARM V7 and V8 CPUs. Support as “second class citizen” is good enough

• Mixed mode support where Little Endian and Big Endian code could coexist on the same box

• Tools and methodologies that could help transition Big Endian Only code to Endian Agnostic

LNG Endian Team was Set to Do

• Truly great open source effort• Effort got help from many companies and individuals• Few to mention, but not limited:

• Codethink• ARM• Linaro

• Thank you!

We Got Help … a Lot!

• V7 ARM BE8 gcc support existed for a while• V8 ARM BE gcc support was verified/fixed by Linaro

toolchain group• Linaro toolchain releases for gcc 4.8 include Big Endian

variants:• V7 - armeb• V8 - aarch64_be

• Some issues may remain for Neon vector intrinsics

ARM V7 & V8 Big Endian Toolchain

• Bootloader, u-boot or uefi, remains Little Endian• All typical SoCs peripheral devices are Little Endian

ARM Big Endian Kernel Context

• BE8 ARM V7 core support was driven by Ben Dooks from Codethink

• LNG Endian team collaborated with Ben: testing, fixes, review

• BE8 V7 core support series appeared in 3.13

ARM V7 Big Endian Kernel Core Support

• Some just worked … with very minor changes• TC2• Mvebu• Highbank

• LNG Endian team use TI Pandaboard and Samsung Arndale as reference and development platform for Big Endian work (a lot of regular driver fixes)• TI Pandaboard changes are in process of being upstreamed• Samsung Arndale changes no intention to upstream

• LNG Endian team helped, in different degrees, to enable Big Endian support on• TI Keystone• LSI Amarillo

ARM V7 Big Endian Kernel BSP

• Supported through Linaro OE• Changes are in meta-linaro/meta-bigendian• Very few user-land packages are affected• Few examples:

• strace• python libffi

ARM V7 Big Endian User-land

• LNG Endian team tested and fixed different kernel functional areas which could be affected by endianity change

• Just to mention few:• BE kexec/kdump - minor fixes• BE kprobes - upstream pull request posted• BE uprobes - folded into regular Linaro uprobes work• BE KVM host support - patches posted for upstream review

ARM V7 Big Endian Kernel Advanced Features

• Done by ARM folks: Matthew Leach, Will Deacon• Was very nice surprise• Included in 3.13• Works/tested on fastmodels and real V8 h/w

ARM V8 Big Endian Kernel Core Support

• Big Endian V8 KVM host support patches posted for upstream review

• We will test other advanced features on V8 as soon as they are available on V8

ARM V8 Kernel Big Endian Advanced Features

• Supported through Linaro OE• aarch64_be target support was added to

openembedded-core with help of Linaro release team• Ran into few minor issues

ARM V8 Big Endian User-land

• Java• Valgrind• Gcc neon vector intrinsics

Anything else? What did we miss?

Known Big Endian ARM Linux Features that don’t Work

• “Special” Big Endian user-land process on top of Little Endian Kernel along side with regular Little Endian processes

• Byte-swaps on system call and ioctl level in kernel• Cisco conducted some experiments in this area:

• converted around 30 system calls and 20 ioctls• messy, big, intrusive change• non-trivial code in futex area

• Byte-swaps before calling Little endian function:• did not actually try• but seems even more harder• i.e how to debug process that may have Big Endian and Little Endian

code coexisting

Mixed Mode: Big and Little Endian

• Done by Marc Zyngier• Looks nice and clean• See Crossing the endianness bridge - Marc Zyngier KVM Forum '13• Marc showed Big Endian guest running on top of Little

Endian KVM host• Require changes in emulator - patches for kvmtool by

Marc are available• Guest need to treat virtio config as LE; not committed

yet; may need to wait for virtio-2 support• Once Big Endian KVM support appeared it could run

Little Endian guests (tested on V7 and V8)• If we have time will make a demo of it

Mixed Mode: through KVM

• Cisco runs big project in this area, attempting to convert 100M+ Big Endian only LOC into Endian agnostic code

• Effort is much larger compared to anything LNG Endian team did

• Approach outline:• sparse with a lot of changes used to annotated code and detect

endian issues• sparse tool was changed to do source to source transformation• python plugin added that allows write external rules how to transform

code• Considered very risky and big, may take long time• Cisco most likely will be able to share and present

details about the effort by next Linaro Connect (tentative agreement reached)

Big Endian to Endian Agnostic Tools

• ARM V7 and V8 Big Endian basic ecosystem looks good• Mixed mode support should be addressed through KVM

virtualization• Big Endian porting mainly will be based on Cisco internal

project experience

Summary

More about Linaro Connect: http://connect.linaro.orgMore about Linaro: http://www.linaro.org/about/

More about Linaro engineering: http://www.linaro.org/engineering/Linaro members: www.linaro.org/members