KTAU: Kernel TAU

Aroon Nataraj

Suravee Suthikulpanit

Agenda

KTAU Infrastructure

KTAU - TAU Integration

KTAU Source Distribution

Generic Kernel Profiling API (BG/L compute-node)

KTAU Infrastructure

Modular Design:Goal: Separate into 3 phases:

Source Instr. -- KTAU -- User Interface

KTAU Kernel InstrumentationKTAU Profiling Infrastructure

KTAU Proc InterfaceKTAU User-API

KTAU INSTRUMENTATIONS

Motivations

Minimal addition to existing kernel source Instrumentation-based profiling Inclusive / Exclusive time High resolution time (rdtsc) Flexible and extensible design Light-weight and Efficient Configurable

Inst. Point Design GOAL: Minimal overhead (perturbation)

Address-Symbol mapping (System.map, /proc/kallsyms)

Run-time function profile registration(Dynamic function ID mapping)

Types- Timer (start, stop)- Event Counter, Miscellaneous Counter

Dynamic function ID

GOAL: Fast access to profile data Each function has a unique ID used for the

indexing the profile table Register itself at run-time ID:

- 0-299 : System Calls- 300-400 : Static Index - 400-1023 : Dynamic Index

Inst. Placement

Inside kernel routine

Randezvous entry/exit points- System calls entry/exit (/linux/arch/i386/entry.S)- bottom-half entry/exit routine

(/linux/kernel/softirq.c)

#ifdef CONFIG_KTAU

#include <linux/ktau/ktau_inst.h>

#endif /*CONFIG_KTAU*/

Int foo( ){

#ifdef CONFIG_KTAU

GET_KTAU_INDEX();

ktau_start_timer(&foo);

#endif /*CONFIG_KTAU*/……………………….

……………………….

……………………….

#ifdef CONFIG_KTAU

ktau_stop_timer(&foo);

#endif /*CONFIG_KTAU*/

}

ENTRY(system_call) pushl %eax SAVE_ALL GET_THREAD_INFO(%ebp) testb $(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT),

TI_flags(%ebp) jnz syscall_trace_entry cmpl $(nr_syscalls), %eax jae syscall_badsyssyscall_call:

#ifdef CONFIG_KTAU_SYSCALL <START_KTAU_PROF CODE >

#endif /* CONFIG_KTAU */

call *sys_call_table(,%eax,4)

#ifdef CONFIG_KTAU_SYSCALL

<STOP_KTAU_PROF CODE >

#endif /* CONFIG_KTAU */

movl %eax,EAX(%esp)

syscall_exit: cli movl TI_flags(%ebp), %ecx testw $_TIF_ALLWORK_MASK, %cx jne syscall_exit_workrestore_all: RESTORE_ALL

Kernel Control FlowScheduling

KTAU Profiling Infrastructure

Motivations

Context-Of-Execution-based profiling Easily extensible w/ different type of inst. Small Memory Footprint No running daemon SMP-Support

COE-based profiling

Maintain profile data in task_struct which allows task level profiling as well as whole system

Consistant with TAU framework

No lookup is required

KTAU-PROC Interface

Requirements

Bi-Directional Communication

Extensible, allowing different commands

Scales with load

Able to support per-process & System profile

As independent of KTAU as possible

High-Level Design / Architecture

Kernel Space

User Space

KTAU

KTA

U -

AP

I

Lib-KTAU

Profiled Apps Utilities

IOCTL on /proc/ktau

Exposes /proc/ktau

KTAU - PROC

Con

trol -

AP

I

SYS_IOCTL

KTAU - TAU Integration

Motivations

Providing integration of kernel and user profile

Interface with TAU Profiler through libKtau

Use Paraprof as visualization tool

TAU configuration

./configure -cc=icc -c++=icpc -ktau-ktau_merge -ktauinc=/usr/src/linux-2.6.10-ktau-1.2/include/ -fortran=intel -mpi -mpiinc=/software/mpich2-install/include/ -mpilib=/software/mpich2-install/lib/

InterfaceTauKtau.h

public:

TauKtau::StartKProfile();TauKtau::StopKProfile();

private:

TauKtau::MapKallsyms();

TauKtau::ReadKallsyms();

TauKtau::DiffKProfile();

TauKtau::DumpKProfile();

Ktau_proc_interface.h

read_size(); read_data(); purge_data();unpack_bindata();

/proc/kallsyms

c01589c0 T sys_openc0158a50 T sys_creatc0158a80 T filp_closec0158b10 T sys_closec0158ba0 T sys_vhangupc0158be0 T generic_file_openc0158c50 T nonseekable_openc0158c62 t .text.lock.openc0158d20 T generic_file_llseekc0158e10 T remote_llseekc0158f20 T no_llseekc0158f30 T default_llseekc0159000 T vfs_llseekc0159070 T sys_lseekc0159110 T sys_llseekc01591e0 T do_sync_readc01592d0 T vfs_readc0159400 T do_sync_writec01594f0 T vfs_writec0159620 T sys_readc01596a0 T sys_writec0159720 T sys_pread64c01597b0 T sys_pwrite64c0159840 T iov_shortenc0159880 t do_readv_writevc0159b00 T vfs_readvc0159b70 T vfs_writevc0159be0 T sys_readvc0159c60 T sys_writev

/proc/ktau/read

Profiler::StartProfiler::StoreData

Output Format

Kernel Profile (User and Interrupt Context)

- System-wide- Individual process

Merged Kernel-User Profile (User Context)

System-wide Profile (Lu.W.4)

Individual Process Profile

- Lu.W.4- Node 0- Excl time- usec

Merged Kernel-User Profile

- Lu.W.4- Node 0

Source Distribution

KTAU suite- Kernel patches- Kernel extension source - User-space library/tools- Documentations

Generic Kernel Profiling

API Suite

BG/L Compute-node

Motivations

Goal:

Providing a generic API for kernel profiling on BG/L compute-node

BG/L Key Characteristics:- Common address space- Single Process

Strategies

Dynamic ID mapping to kernel routines

( Based on KTAU prototype )

Dynamic registration of profiling handler routines

Instrumentation API

Provides kernel intercept points

Kernel routine taxonomy

Dynamic function ID mapping to routine

Instrumentation API (cont’d)unsigned int get_fid ( unsigned int group )

void start_timer ( unsigned int funcId , unsigned int group,

char* funcName)

void stop_timer ( unsigned int funcId , unsigned int group, char* funcName)

void start_prof ( unsigned int funcId , unsigned int group,

char* funcName,unsigned int data )

void stop_prof ( unsigned int funcId , unsigned int group,

char* funcName, unsigned int data )

void event_prof ( unsigned int funcId , unsigned group,

char* funcName, unsigned int data)

Registration API

Profiling tools selection- Combination of tools according to the tool

strength

Group selection- Focus on the group of kernel routines of

interest at a particular section of application

Registration API (cont’d)

int register_handler ( unsigned int group_mask,

void ( *handler )

( unsigned int,

unsigned int,

char*,

void* )

)

int unregister_handler ( unsigned int group_mask )

Handler API

Tool specific routines for processing data

void handler ( unsigned int funcId,

unsigned int type,

char * funcName,

void * data )

Merged Profile

Trace