Download - Selenium sandwich-3: Being where you aren't
Selenium Sandwich Part 3: What you aren't
Steven LembarkWorkhorse [email protected]
What is a Selenium Sandwich?
Tasty!!!
No really...
What is a Selenium Sandwich?
Last time we saw how to combine Selenium and Plack.
Selenium calls a page.
Plack returns a specific response.
Catch: You can' get there from here.
What is a Selenium Sandwich?
Last time we saw how to combine Selenium and Plack.
Selenium calls a page.
Plack returns a specific response.
Catch: You can' get there from here.
Or you can, which is the problem.
Getting to the server
Q: How do we get a specific page loaded?
Say a Google map, Yelp search, or *aaS dashboard?
A: Load the page from a server?
Getting to the server
Q: How do we get a specific page loaded?
Say a Google map, Yelp search, or *aaS dashboard?
A: Load the page from a server?
What about our static content?
Locally sourced
You want to test a Google page.
How?
Save it locally?
Only if you want to save all of it.
Trucked in
Q: How many URL's does it take to screw in a...
Trucked in
Q: How many URL's does it take to make a Google page?
A: Lots.
Banners, logos, JS lib's, Java lib's, ads...
Trucked in
Q: How many URL's does it take to make a Google page?
A: Lots.
Banners, logos, JS lib's, Java lib's, ads...
Many are dynamic: they cannot be saved.
Werefore art thou?
Many URL's are relative.
They re-cycle the schema+host+port.
Relative paths
Many URL's are relative.
They re-cycle the schema+host+port:
http://localhost:24680/foobar.
http://localhost:24680/<everything else>
Relative paths
Need to ask locally for a remote page.
With the browser having no idea where it came from.
In other words: We need a proxy.
HTTP Proxying
Normally for security or content filtering.
Or avoiding security and content filtering.
How?
Explicit proxy
Configure browser.
It asks the proxy for everything.
Proxy pulls content, returns it.
Proxy decides which content goes to test server.
HTTP::Proxy
Run as a daemon.
User filters.
LWP as back-end for fetching.
Slow but reliable...
Basic proxy setup
Grab a port...
and go!
use HTTP::Proxy;
my $proxy = HTTP::Proxy->new( port => 24680 );
# or...
my $proxy = HTTP::Proxy->new;
$proxy->port( 24680 );
# loop forever
$proxy->start;
Initializing HTTP::Proxy
Base class supplies “new”.
Derived class provides its own “init”.
package Mine;use parent qw( HTTP::Proxy );my $src_dir = '';sub init{ # @args == whatever was passed to new # in this case a path.
my ( undef, %argz ) = @_;
$src_dir = $argz{ src_dir } || '.'or die 'Missing “work_dir” in MyPath';
...}
Adding filters
HTTP::Proxy supports request and response filters.
Requests modify outgoing content.
Response filters hack what comes back.
Our trick is to only filter some of it.
Four ways to filter content
request-headers request-body
response-headers response-body
Filters go onto a stack:
$proxy->push_filter( response => $filter # or request => ...);
Massage your bodypackage MyFilter;use base qw( HTTP::Proxy::BodyFilter );sub filter{ # modify content in the reply
my ( $self, $dataref, $message, $protocol, $buffer ) = @_; $$dataref =~ s/PERL/Perl/g;}1__END__
Fix your head
package MyFilter;use base qw( HTTP::Proxy::HeaderFilter );
# change User-Agent header in all requests
sub filter{ my ( $self, $headers, $message ) = @_; $message->headers->header ( User_Agent => 'MyFilter/1.0' ); ...}
Have to hack the request
Change:
https://whatever
to:
http://localhost:test_port/...
Or pass through to remote server.
Timing is everything
Modifying the response is too late.
That leaves the request or agent.
Timing is everything
Modifying the response is too late.
That leaves the request or agent.
Request can easily modify headers or body.
Not the request.
Timing is everything
Modifying the response is too late.
That leaves the request or agent.
Request can easily modify headers or body.
Not the request.
That leaves the agent.
Secret Agents
Choice is a new HTTP::Proxy class (is-a).
Or replacing the agent (has-a).
For now let's try the agent.
Wrapping LWP::UserAgent
Anything LWP does, we check first.
Any path we know goes to test.
Any we don't goes to LWP.
Wrapping LWP::UserAgent
Anything LWP does, we check first.
Any path we know goes to test.
Any we don't goes to LWP.
Intercept all methods with AUTOLOAD.
Requires we have none of our own.
Generic wrapperpackage Wrap::LWP;use parent qw( LWP::UserAgent );
use Exporter::Proxy qw( wrap_lwp install_known );
our $wrap_lwp= sub{ my $lwp = shift or die ... ;
my $wrapper = bless \$lwp, __PACKAGE __;
$wrapper};
Generic wrapperuse Exporter::Proxy qw( wrap_lwp handle_locally );use List::MoreUtils qw( uniq );
our @localz = ();
our $handle_locally= sub{
# list of URL's is on the stack.# could be literals, regexen, objects.# lacking smart match, use if-blocks.
@localz = uniq @localz, @_;
return};
Generic wrapperour $AUTOLOAD = '';AUTOLOAD{
my ( $wrapper, $request ) = @_;
my $url = $request->url;my $path = $url->path;
if( exists $known{ $path } ){
# redirect this to the test server$url->scheme( 'http' );$url->host ( 'localhost' );$url->port ( 24680 );
}...
Generic wrapper
# now re-dispatch this to the LWP object.# this is the same for any wrapper.# goto preserves the call order (e.g., croak works).
my $i = rindex $AUTOLOAD, ':';my $name = substr $AUTOLOAD, 1+$i;my $agent = $$wrapper;
my $handler = $agent->can( $name )or die ... ;
splice @_, 0, 1, $agent;
goto $handler}
Using the wrapperuse Wrap::LWP;use HTTP::Proxy;
$handle_locally->( 'https://foo/bar', 'http://bletch/blort?bim="bam"');
my $proxy = HTTP::Proxy->new( ... );
my $wrapper = $wrap_lwp->( $proxy->agent );$proxy->agent( $wrapper );
$proxy->start;
TMTOWDTI
AUTOLOAD can handle known sites.
Instead of modifying the URL: just deal with it.
Upside: Skip LWP for local content.
Downside: Proxy gets more complicated.
Result
Known pages are handled locally.
Others are passed to the cloud.
Server & client have repeatable sequence.
The test loop is closed.
So...
When you need to be who you're not: Use a proxy.
HTTP::Proxy gives control of request, reply, & agent.
Handling LWP is easy enough.
Which gives us a nice, wrapped sandwich.