© 2001 covalent technologies – commercial in confidence – 29 january 2001 - 1 1 writing apache...

© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 11

Writing Apache 2.0 Modules andporting 1.3 modules to 2.0

Dirk-Willem van GulikVP of Research, Covalent Technologies

[email protected]

O'Reilly Open Source Conference 2002v1.09


Overview

What are modules– Apache fundamentals

Module anatomy 2.0 - what is new 1.3 -> 2.0 checklist Writing an authentication module (PAM) Sample 1.3 -> 2.0 Migration Acronym Module - filter sample (time permitting) Q&A

http://www.apache.org/~dirkx/oscon2002/


Fundamentals

Apache webserver– NCSA roots; The 'Shambala' rewrite– Modular architecture

Other Apache projects– Apr, Apr-utils, Proxy, docs– Tomcat,– Xml, java, Tcl, …



History

Apache 0.9 Apache 1.2 Apache 1.3.29

shambalaApache 2.0

APRAPR Utils

Apache 2.0New Proxy

patches

ncsa

….

modulesmodules

?

PerlPHPJava


Apache: Basic facts

Webserver is written in ‘C’ Runs on virtually all platforms

– Unix, QNX, BeOS, Win, OS/390 Very open modular infrastructure

– Sequence of handlers act on each stage of the (http) request

Server core handles ‘child care’ and ensures protocol compliance.


Changes in 2.0

Protocol engine abstracted out– http, https– POP3, Commercial FTP module

Filters and daisy chaining now possible

Abstracted out ‘child’ management– (pre)forked, threaded– Or a hybrid

More infrastructure for portability


Modules tasks:

Modules are small and simple:– Auth: verify a username and

password– Access: verify if a username is on a list

or an IP address matches– Rewrite: change a URL– Logging: log to a file or backend– Headers: Modify or add a header– Content: Simple content substitution


When - or when NOT

Simple specific task

Task maps to a single handler

Relatively light weight task for each request

Security close to the wire needed

Complex interaction with other sources required

Complex to secure.

Complex rules

Large footprint


Module Environment

Modules have– One known entry point– Set of registered callbacks– Set of known directives it can handle

Modules live a shielded live– Memory management– Log files, UID, socket access– Private memory for

– Configuration information– Per request data


Anatomy - the short version

At startup– Load the modules– Set up basic infrastructure (logs, uids)– Read the configuration file

– Hand each module its registered configuration directives

– Ask each module to initialize Runtime

– Wait for requests to process– (Re)Initialize new children

Shutdown– Ask each module to shut down


Anatomy - Requests

Request comes in– File name translation– Check user id– Check access permissions– Check type– Last change fixups– Log when all is done


The API

Module block– Master router

Command block– Registry of configuration commands

Hooks– Registry of functions called at each stage

Handlers– Registry of functions called on condition

APR, APR utils– The backside; system abstraction


Configuration file

Apache configuration file– Httpd.conf, srm.conf, access.conf– .htaccess– Include 'file.conf'

Three types of directives– Processing and core

– Includes, directory– UID, LoadModule, errorlog– Hierarchy

– Module specific– Everything else


Anatomy - Configuration

Configuration file is read– Hierarchy is build

– <Directory, <Server, ..– Modules are instructed to create/merge

matching configurations– Server– Directory

Directive is read– Registered module is called with


Anatomy - your inputs

Pointer to your private memory segment.

Specific Arguments for– Parsed Directive details– Request you are expected to handle– I/O and Log streams


Anatomy - your outputs

Very few– DECLINED: 'aint me gov'– OK: 'Yup - I've done my job'– SERVER_ERROR: 'Scream murder'– Protocol/http status codes

But..– You can modify data in the structures

passed on– url, mime type, headers


Modules: 1.3 versus 2.0

No fundamental Changes

More APR and APR utils support– Module authors can rely on a portability layer;

existing code often reduced or simplified significantly.

Extra features– Filters; layered IO– More hooks, fewer pointer structs

Process management– Threaded, Process or hybrid


2.0 Checklist

No global variables– Your environment may be Threading– Use thread-safe libraries

Use Request record for passing state– No assumptions as to which child,

process or thread will handle successive requests (from the client) or phase (within a request).

Potential longer lifetimes– Do your cleanups.


Examples

Mod_auth_pam– Popular existing 1.3 module– Authenticates against PAM

– Pluggable authentication layer– Solaris, FreeBSD, Linux, others– /etc/passwd, SecurID, ldap, others

Mod_acronym (time permitting)– Contrived example– Uses filters



Writing the module

Don’t write modules !– You sure it is not already in the base

install ? Don't write modules !!

– You sure there is not a module already on modules.apache.org ?

Don’t' write modules !!!– There must be a module already which

comes close and which you can simply adapt to suit your purpose.



Essentials

ANSI -C

(binary) version of apache

C compiler (gcc), make, linker

Documentation and Samples:– http://dev.apache.org/apidoc/– http://modules.apache.org/



Initial Setup

Fetch a copy of apache– http://httpd.apache.org/dist

Compile and install– ./configure --enable-maintainer-mode– make && make install

And build your own module– cd ~/myModule– apxs -c mod_auth_pam2.c



Initial testing

Install the module– apxs -I mod_auth_pam2.c– Check httpd.conf

– Add directives as needed (Re)start the server

– apachectl stop– apachectl start

Always keep an eye on the log !– tail -f ../logs/errorlog



Common gotcha's

The 'apxs' in your path is -not- the one of the installed apache

'--enable-maintainer-mode' or -O3, -Wall not present.

You should restart your server each time. gdb ./httpd

– run -X (1.2.x and above)– run -D ONE_PROCESS (2.0.x)– Use forking MPM if possible



Mod PAM - Requirements

Requirements:– Use 'Basic Auth'– Authenticate a user against PAM– Access control on username– Access control on group membership

PAM API provides the basics– Pam_start(), Pam_authenticate()..– In: Username, password– Out: valid, has account– Thread safe.



MaP: Outline

Configuration:– Accept Configuration directives– Store our configuration

Authenticate a username and a password– Respond with a yes/no/duh– Flag errors as appropriate.



MaP: Headers

#include "ap_config.h"

#include "httpd.h"

#include "http_config.h"

#include "http_core.h"

#include "http_log.h"

#include "http_protocol.h"

#include "http_request.h"

#include <security/pam_appl.h>

@144


MaP: Configuration.

Pick your directives with care– Auth: we authenticate– PAM: against the PAM system

AuthPAM_Enabled <yes|no>– General on/off switch

AuthPAM_FailDelay <int - mSec>– PAM specific timeout

AuthPAM_FallThrough <yes|no>– Am I authoritative ?


MaP: configuration (cont)

Your Private Configuration

typedef struct {

int fail_delay;

int fall_through;

int enabled;

} auth_pam_dir_config;

@191


MaP: Initialize

Few modules actually need this.

Note: static and different pools.

static int auth_pam_init(

apr_pool_t *p, *plog,*ptemp,

server_rec *s

) {

ap_log_error(APLOG_MARK, APLOG_INFO, 0, s,"PAM: mod_auth_pam/" VERSION);

return OK;

}

@201


MaP: Real initialize 1/4

Happens piece by piece– Within the hierarchy– Passed the correct pool– Return is opaque pointer.


MaP: Real Initialize 2/4

static

void* create_auth_pam_dir_config(apr_pool_t *p, char *dummy)

{

auth_pam_dir_config *new =

(auth_pam_dir_config*) apr_palloc (p,

sizeof(auth_pam_dir_config));

new->fail_delay = 0; /* 0 ms */

new->fall_through = 0; /* off */

new->enabled = 1; /* on */

return new;

}

@214



Create your own memory segment.

Use the right pool!

Do not even think to use malloc()

auth_pam_dir_config *new =

(auth_pam_dir_config*) apr_palloc (p, sizeof(auth_pam_dir_config));

@217



Always initialize– -O5– predictable stale pools – Security assumptions.

new->fail_delay = 0; /* 0 ms */

new->fall_through = 0; /* off */

new->enabled = 1; /* on */

And return the initialized memory segment:

return new;

@220


MaP: Populating 1/4

static command_rec auth_pam_cmds[] =

{

AP_INIT_TAKE1("AuthPAM_FailDelay", …

AP_INIT_FLAG("AuthPAM_FallThrough", ..

AP_INIT_FLAG("AuthPAM_Enabled", ..

{NULL}

}

@245


MaP: Populating 2/4

Calls 'parse_function(args)' when directive is seen.

Lots of pre defined parse functions Be sensibly strict Always tell the default.

AP_INIT_TAKE1("MyDirective",

parse_function, extra_argument,

allowed,

"helptext shown with httpd -V" ),

@226


MaP: Populating 3/4

AP_INIT_TAKE1("AuthPAM_FailDelay",

ap_set_int_slot,

(void *)APR_OFFSETOF(

auth_pam_dir_config, fail_delay),

OR_AUTHCFG,

"number of micro seconds to wait after failed authentication attempt. (default is 0.)"

),

@247


MaP: Populating: 4/4

AP_INIT_FLAG("AuthPAM_Enabled",

ap_set_flag_slot,

(void *)APR_OFFSETOF(auth_pam_dir_config, enabled),

OR_AUTHCFG,

"on|off - determines if PAM authentication is enabled; "

"(default is on.)"

),

@259


MaP: recap

Configuration:

<directory /www/internaldocs>AuthType BasicAuthName "Your Intranet

passwd"AuthPAM_Enable yesrequire valid-user

<directory>


MaP: Check User 1/3

On entry– We get our configuration back.– The protocol (http) and core can supply more

information.

static int pam_auth_basic_user (request_rec *r)

{auth_pam_dir_config *conf =

ap_get_module_config(r->per_dir_config,&pam_auth_module);

@341


MaP: Check User 2/3

We're always called. Make sure we are configured (This is why initialization was so important):if (!conf->enabled)

return DECLINED;

And find out if a username and password was supplied: if ((res = ap_get_basic_auth_pw (r,

(const char**)&(userinfo.pw))))

return res;

userinfo.name = r->user;

@357

@361


MaP: recap

At this point– We have access to the right

configuration at the right place in the configuration hierarchy.

– The username and password are available to us.

– As is everything else in 'r'.

Decision time !


MaP: Check User 3/3

Log!

ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r,

"PAM: user '%s' - not authenticated: %s",r->user,compat_pam_strerror(pamh, res));

On Error

return HTTP_INTERNAL_SERVER_ERROR;

On success

return OK;

On unkown

if (fall_through)

return DECLINED;

Otherwise

ap_note_basic_auth_failure(r);

return HTTP_UNAUTHORIZED;

@374

@377

@423

@404

@410


MaP: Wiring it all up 1/3

Advertising– Our configuration entry points– Our initialization callbacks– Registration of our hooks

– Post_config (init message)– Check_user (verification)A

Apache 1.3/2.0 - biggest difference.


MaP: Wiring it all up 2/3

Apache 1.3 /2.0 quite similar

module AP_MODULE_DECLARE_DATA

pam_auth_module = {

STANDARD20_MODULE_STUFF,

create_auth_pam_dir_config, /* dir config creater */

NULL, /* dir merger */

NULL, /* server config */

NULL, /* merge server config */

auth_pam_cmds, /* command table */

……

@517


MaP: Wiring it all up 3a/3

The Apache 1.3 method ….

auth_pam_init, /* initialize */

….

NULL, /* handlers */

NULL, /* filename translation */

pam_auth_basic_user, /* check_user_id */

pam_check_auth, /* check auth */

NULL, /* check access */

NULL, /* type_checker */

NULL, /* fixups */

NULL /* logger */

};

@519

@525


MaP: Wiring it all up 3b/3

OR The Apache 2.0 way:

…

pam_register_hooks, /* register hooks */

};

static void pam_register_hooks(apr_pool_t *p)

{

ap_hook_post_config(auth_pam_init,

NULL, NULL, APR_HOOK_MIDDLE);

ap_hook_check_user_id(pam_auth_basic_user,

NULL,NULL,APR_HOOK_MIDDLE);

}

@525

@511


MaP: 1.3 versus 2.0 1/3

Easy– Thread safe, no global variables

#include "ap_config.h" (114) Closed security niggle in _init. (209) Removed 'legacy' (2.0: good excuse) Removed 'require' checking. (430). Changed: ap_palloc() -> apr_palloc(). All directive functions removed (226)

– Stock string/int/boolean parsers used. Rewrote command_rec with macro's. (245)


MaP: 1.3 to 2.0 changes 2/3

server_rec; (336)– r->connection->user changed location to r-

>user Logging functions (374)

– 'r' variant (security)– PAM: prefic for clarity.– Use of log levels.

AUTH_REQUIRED became (419)– HTTP_UNAUTHORIZED

Module callback list shorter (511)– And callbacks actively registered instead.


MaP: Conclusion

Very typical module

Change to 2.0– Made the module simpler.– Made it more portable– Was simple and we could borrow from

'modules/aaa/mod_auth_*.c


Example deficiencies

PAM library assumed to be thread safe– If not - needs to be 'mutex'-ed.

Use of /etc/group rather than a AuthGroupFile– Use mod_access_etc_group.c


Note:

APXS is not the only way to build a module. Do not use it if:– You need 'autotconf'.– Complex linking or dynamic loading.

Other methods– ./configure pointing and Config.m4– --with-module=type:name


Fine !

Questions


Acronym Expansion - filters

Filters– The true cool feature of 2.0– Daisy chain modules– On Input and Output

– Some header/body understanding– Not carved in stone

– Buckets and Brigades



Acronyms

Outline and requirements– Add a footnote to each page containing

acronyms (e.g. ASF*).– Detects acronyms a˜utomatically– Be reasonable fast

Solution:– An Output filter.– Filter each APR‡ buckets and replace or

insert where needed

*) ASF: Apache Software Foundation

‡) APR: Apache Portable Runtime


Rough Design

Configuration– Enable/Disable per directory/file or

similar resource blocks.– Standard unix dictionary format.

– E.g. /usr/share/misc/airports.txt– Not use too much memory.

– Needs to work on only the HTML files.

Sample– Lots of deficiencies.


Outline 1/2

Configuration– Per directory– For the server as a whole

Read in a dictionary – And store in a hash

Plug the filter into the stream– When enabled and for html only



Outline 2/2

Output Filter– Detect an acronym– Add a ‘dagger’ and a link.– Add some in situ javascript/ALT– At the bottom of the page;

– add the expansion.



Bucket Brigades

Brigade

Bucket Bucket Bucket

Brigade

Bucket Bucket

EOS

EOS


“Hello World of Apache”

Brigade

“Hell” “o Worl” “d ”

Brigade

“of Ap” “ache”

EOS

EOS


Configuration

In memory structures

Create/Store in our private context.

Configure Directives

Read in the specified acronym file(s).

@103

@126

@492

@413


Parsing (1/8)

Let’s ignore the header - focus on body.

File read or generated.– Verified to be HTML, enabled.– And chained into the stream output

stream set up to the client.

I am flying out of SFO to night.

@147

@168


Parsing (2/8)

Passed in several brigades– Each brigade is a single call


@171


Parsing (3/8)

Passed in several brigades– Consisting in several buckets!


@517

@517

@517


Parsing (4/8)

Prepare a brigade with the output

First brigade can be parsed until the end

and passed down the chain.


I am flying

@200

@201

@397


Parsing (5/8)

Second brigade– Buckets parsed until we hit the acronym– Pass ‘S’ state on to the next brigade


I am flying out ofS

@307


Parsing (6/8)

Using ‘S’ passed– parse until end of Acronym

Add <html> around acronym.


I am flying out of SFO‡SFO‡

@319


Parsing (7/8)

And parse out the final part of the string


I am flying out of SFO‡ To night.


Parsing (8/8)

Once done - add the footnote with the acronyms.


I am flying out of SFO‡ To night.

SFO: San Francisco International.

@352,208


Demo

<h2><hr>Travel Plan<hr></h2>

Travel for next week:

<ul>

<li> Tuesday: From IAD to SFO

<li> Thursday: back from IAD to SLC

<lI> Sunday: out from SLC to OAK.

</ul>


In the browser:

QuickTime™ and aTIFF (LZW) decompressorare needed to see this picture.


Summary

Filtering module– Brigades with buckets– Be prepared– Keep state

Simple and Fast– One specific function

Imagination– Combine with anything apache does !


Reminder

No need to write modules!– It is probably there in core.

No need to write modules!– modules.apache.org may have it.

No need to write modules– Something must be close enough to be

subverted for your purpose. But if you do

– Let the world know !


Fine !

Questions


© 2001 covalent technologies – commercial in confidence – 29 january 2001 - 1 1 writing apache...

Documents