App::highlighta simple grep-like highlighter app

Alex Balhatchet @ London.pm Technical Meeting, July 2013

Intro

alex@masaki:~$ cat words.txt

fredbarneybettywilmadinopebblesbamm-bamm

Intro

alex@masaki:~$ cat words.txt | grep abarneywilmabamm-bamm

Intro

alex@masaki:~$ cat words.txt | highlight afredbarneybettywilmadinopebblesbamm-bamm

Why?

Real World Example 1

alex@masaki:~$ servicectl status

servicectl status: checking arbyte-job-buffer-probe... running.servicectl status: checking arbyte-job-runner-cleanup-cron... running.servicectl status: checking arbyte-job-runners... running.servicectl status: checking arbyte-manager... running.servicectl status: checking arbyte-status-accepter... running.servicectl status: checking imagesys-managerd... running.servicectl status: checking launch-target-cron... running.servicectl status: checking listings-db... running.servicectl status: checking ops-cron... not running.servicectl status: checking search-db... running.servicectl status: checking searchindex-cron... running.servicectl status: checking updatesem-cron... running.

Real World Example 1

alex@masaki:~$ servicectl status | highlight -l 'not running'servicectl status: checking arbyte-job-buffer-probe... running.servicectl status: checking arbyte-job-runner-cleanup-cron... running.servicectl status: checking arbyte-job-runners... running.servicectl status: checking arbyte-manager... running.servicectl status: checking arbyte-status-accepter... running.servicectl status: checking imagesys-managerd... running.servicectl status: checking launch-target-cron... running.servicectl status: checking listings-db... running.servicectl status: checking ops-cron... not running.servicectl status: checking search-db... running.servicectl status: checking searchindex-cron... running.servicectl status: checking updatesem-cron... running.

Real World Example 2

alex@masaki:~$ zcat listings.txt.gz | grep "'id' => '8244118'"

$VAR1 = {'debug' => {'remote_used' => 1},'etl' => {'agent_database_key' => undef,'feed_version' => '1.7','feed_timestamp' => '20130717100846','id' => '8244118','feed_segment' => undef,'feed_name' => 'domain','partner_database_key' => undef},'listing_attributes' => {'agent_description' => undef,'gross_size' => undef,'floor' => '1','num_baths' => 1,'gross_size_unit' => undef,'construction_year' => undef,'num_rooms' => undef,'num_beds' => 3,'commission' => undef,'property_type' => 'house','datasource_custom_two' => undef,'land_size' => undef,'datasource_custom_five' => undef,'net_size_unit' => undef,'energyrating' => '0','net_size' => undef,'agent_logo_url' => undef,'rental_deposit' => undef,'energyindex' => '0','car_spaces' => '1','land_size_unit' => undef,'datasource_custom_three' => undef,'auction_date' => undef,'datasource_custom_four' => undef,'datasource_custom_one' => undef},'listers' => {'name' => 'Private Advertiser','type' => undef,'company' => 'Private Advertiser'},'listings' => {'price_currency' => 'AUD','full_description' => 'Alarm System, Balcony, Garden, Internal Laundry, Polished Timber Floors, Built-In Wardrobes, Fireplaces, Renovated - 6 metres wide, grand and spacious, light-filled terrace Restoration by award-winning architect 5 minute stroll to Woollahra village, Centennial Park and Oxford Street Enormous family/dining/kitchen area opening to rear courtyard garden Double Living area with marble fireplaces and mirrors 3 large bedrooms - two with marble fireplaces, all with quality built-ins and plantation shutters Rear lane car access via remote control roll-a-door Luxury marble bathroom with bath and separate frameless glass shower Built-in laundry Downstairs powder room Large, private rear courtyard with established garden Entrance courtyard garden Garden maintenance included Foxtel connections to living areas and two bedrooms Alarm system Telephone 0412 758 772 to arrange an appointment for inspection','placenames' => '{"street":"Windsor Street","building":"157","province":"NSW","town":"Paddington"}','listing_type' => 'rent','title_keywords' => 'paddington,windsor street','keywords' => ['backyard','balcony','garden','kitchen','terrace'],'actually_the_original_address' => '157 Windsor Street, Paddington, NSW, 2021','latitude' => '-33.886527','display_address' => undef,'raw_content_digest' => 'c9080b2827fbf612aec4369dd6af1931','debug' => undef,'address' => '157 Windsor Street, Paddington, NSW, 2021','price_type' => 'weekly','longitude' => '151.235183','teaser' => 'Alarm System, Balcony, Garden, Internal Laundry, Polished Timber Floors, Built-In Wardrobes, Fireplaces, Renovated 6 metres wide, grand and light-filled terrace Restoration by award-winning architect 5 minute stroll to Woollahra village, Centennial Park and Oxford Street Enormous...','geocode_method' => 'remote','geocode_accuracy' => 9,'photo_url' => 'http://images.domain.com.au/img/2013320/0/158ffe47-3469-466b-9727-a51c4def7e1b_FS.jpg','datasource_dbkey' => '8244118','site_url' => 'http://www.domain.com.au/Public/PropertyDetails.aspx?adid=8244118','photo_count' => '8','ext_categories' => '','price' => 1850,'coldrent' => undef,'title' => 'Windsor Street, Paddington - Balcony','datasource_name' => 'domain','price_low' => 1850,'photos' => {'urls' => ['http://images.domain.com.au/img/2013320/0/158ffe47-3469-466b-9727-a51c4def7e1b_FS.jpg'],'total' => '8'},'monetized' => '0','status_type' => 'active','datasource_campaign_id' => undef,'site_mobile_url' => undef,'display_brand' => undef,'attribute_hash' => '8244118','postcode' => '2021','country' => undef,'orig_address' => '157 Windsor Street, Paddington, NSW, 2021','geocode_matched' => '2021, 157 Windsor Street, Paddington, New South Wales, Australia','price_high' => 1850,'display_price' => '$1,850'}};

Real World Example 2

alex@masaki:~$ zcat listings.txt.gz | grep "'id' => '8244118'"| highlight 'postcode'$VAR1 = {'debug' => {'remote_used' => 1},'etl' => {'agent_database_key' => undef,'feed_version' => '1.7','feed_timestamp' => '20130717100846','id' => '8244118','feed_segment' => undef,'feed_name' => 'domain','partner_database_key' => undef},'listing_attributes' => {'agent_description' => undef,'gross_size' => undef,'floor' => '1','num_baths' => 1,'gross_size_unit' => undef,'construction_year' => undef,'num_rooms' => undef,'num_beds' => 3,'commission' => undef,'property_type' => 'house','datasource_custom_two' => undef,'land_size' => undef,'datasource_custom_five' => undef,'net_size_unit' => undef,'energyrating' => '0','net_size' => undef,'agent_logo_url' => undef,'rental_deposit' => undef,'energyindex' => '0','car_spaces' => '1','land_size_unit' => undef,'datasource_custom_three' => undef,'auction_date' => undef,'datasource_custom_four' => undef,'datasource_custom_one' => undef},'listers' => {'name' => 'Private Advertiser','type' => undef,'company' => 'Private Advertiser'},'listings' => {'price_currency' => 'AUD','full_description' => 'Alarm System, Balcony, Garden, Internal Laundry, Polished Timber Floors, Built-In Wardrobes, Fireplaces, Renovated - 6 metres wide, grand and spacious, light-filled terrace Restoration by award-winning architect 5 minute stroll to Woollahra village, Centennial Park and Oxford Street Enormous family/dining/kitchen area opening to rear courtyard garden Double Living area with marble fireplaces and mirrors 3 large bedrooms - two with marble fireplaces, all with quality built-ins and plantation shutters Rear lane car access via remote control roll-a-door Luxury marble bathroom with bath and separate frameless glass shower Built-in laundry Downstairs powder room Large, private rear courtyard with established garden Entrance courtyard garden Garden maintenance included Foxtel connections to living areas and two bedrooms Alarm system Telephone 0412 758 772 to arrange an appointment for inspection','placenames' => '{"street":"Windsor Street","building":"157","province":"NSW","town":"Paddington"}','listing_type' => 'rent','title_keywords' => 'paddington,windsor street','keywords' => ['backyard','balcony','garden','kitchen','terrace'],'actually_the_original_address' => '157 Windsor Street, Paddington, NSW, 2021','latitude' => '-33.886527','display_address' => undef,'raw_content_digest' => 'c9080b2827fbf612aec4369dd6af1931','debug' => undef,'address' => '157 Windsor Street, Paddington, NSW, 2021','price_type' => 'weekly','longitude' => '151.235183','teaser' => 'Alarm System, Balcony, Garden, Internal Laundry, Polished Timber Floors, Built-In Wardrobes, Fireplaces, Renovated 6 metres wide, grand and light-filled terrace Restoration by award-winning architect 5 minute stroll to Woollahra village, Centennial Park and Oxford Street Enormous...','geocode_method' => 'remote','geocode_accuracy' => 9,'photo_url' => 'http://images.domain.com.au/img/2013320/0/158ffe47-3469-466b-9727-a51c4def7e1b_FS.jpg','datasource_dbkey' => '8244118','site_url' => 'http://www.domain.com.au/Public/PropertyDetails.aspx?adid=8244118','photo_count' => '8','ext_categories' => '','price' => 1850,'coldrent' => undef,'title' => 'Windsor Street, Paddington - Balcony','datasource_name' => 'domain','price_low' => 1850,'photos' => {'urls' => ['http://images.domain.com.au/img/2013320/0/158ffe47-3469-466b-9727-a51c4def7e1b_FS.jpg'],'total' => '8'},'monetized' => '0','status_type' => 'active','datasource_campaign_id' => undef,'site_mobile_url' => undef,'display_brand' => undef,'attribute_hash' => '8244118','postcode' => '2021','country' => undef,'orig_address' => '157 Windsor Street, Paddington, NSW, 2021','geocode_matched' => '2021, 157 Windsor Street, Paddington, New South Wales, Australia','price_high' => 1850,'display_price' => '$1,850'}};

Features

Features

alex@masaki:~$ highlight --help

highlight [-bCcehlnor] [long options...]-c --color use terminal color for highlighting-C --no-color don't use terminal color-e --escape auto-escape input (default)-r -n --regex --no-escape don't auto-escape input (regex mode)

-i --ignore-case ignore case for matches-l --full-line highlight the whole matched line-o --one-color use only one color for all matches-b --show-bad-spaces highlight spaces at the end of lines

-v --version show version number-h --help display a usage message

Color

alex@masaki:~$ cat words.txt | highlight fred barney betty wilma dino pebbles bamm-bammfredbarneybettywilmadinopebblesbamm-bamm

No Color

alex@masaki:~$ cat words.txt | highlight fred barney betty wilma dino pebbles bamm-bamm --no-color<<fred>>[[barney]]((betty)){{wilma}}**dino**__pebbles__<<bamm-bamm>>

Escape

alex@masaki:~$ cat words.txt | highlight 'b[ea]t{2}.'fredbarneybettywilmadinopebblesbamm-bamm

No Escape (aka regex mode)

alex@masaki:~$ cat words.txt | highlight --regex 'b[ea]t{2}.'fredbarneybettywilmadinopebblesbamm-bamm

Ignore Case

alex@masaki:~$ cat words.txt | highlight --ignore-case 'MM'fredbarneybettywilmadinopebblesbamm-bamm

Full Line

alex@masaki:~$ cat words.txt | highlight --full-line 'a'fredbarneybettywilmadinopebblesbamm-bamm

One Color

alex@masaki:~$ cat words.txt | highlight --one-color 'f' 'r' 'e' 'b'fredbarneybettywilmadinopebblesbamm-bamm

Bad Spaces (inspired by git diff)

alex@masaki:~$ cat words.txt | highlight --show-bad-spacesfredbarneybettyxxxxwilmadinopebblesxbamm-bamm

Cool Tricks

SVN Diff

alias svndiff=" svn diff | highlight --show-bad-spaces | highlight --regex --full-line '^-[^-]' '^[+][^+]' '^[+-]{2}' | less -R"

SVN Diffalex@masaki:~$ svndiff===================================================================--- SERP.pm (revision 62240)+++ SERP.pm (working copy)@@ -1,13 +1,14 @@+use utf8;+xxxx package Lokku::URL::SERP; use base 'Lokku::URL'; -use Clone 'clone';-use List::MoreUtils 'uniq';+use Clone qw(clone);+use List::MoreUtils qw(uniq); use Lokku::Base::Config;

Tailing Access Logsalex@masaki:~$ tail -f ~/common/logs/apache/frontend/access_log.2013-07-17-00_00_00 | highlight -l -r 'HTTP\/1\.1" 50\d ' 'HTTP\/1\.1" 200 ' 'HTTP\/1\.1" 30\d ' 'HTTP\/1\.1" 40\d 'x.x.x.x - - [17/Jul/2013:14:13:51 +0000] "GET /abode-sale-house-in-friday-hill HTTP/1.1" 200 9549 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" 125669 www

x.x.x.x - - [17/Jul/2013:14:13:51 +0000] "GET /paige-petrook-rent-flat-in-the-avenue-ha5 HTTP/1.1" 200 9586 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" 97016 www

x.x.x.x - - [17/Jul/2013:14:13:51 +0000] "GET /gunness/property/buy HTTP/1.1" 200 16501 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 109951 www

x.x.x.x - - [17/Jul/2013:14:13:52 +0000] "GET /rd?l=UbeTFSQlTSlUJUMFlk&url=2-ePz73aTBU4LVGFByDI484Ner8aIMqOimmnO-vZos3KDcnfaeeMPe4rGzMAHEMEMIx5MZsli8WOfUu6sPU8-oGyl-oBxFgBrDY1PNVVZlNerg11bkCYzKZwBlYrKuUnoqgEDyvQYT0AWjNiAXvEvzby0n8wKI_4SYr67NnK50C3WeiSB3ATzAdHRuPx89ZkLfV7zVXG2QDbeS6DmC6o17lS3zg4TQbgURP9KPebQfsYvRw-KZZSJe5bz67v9wC0YOb6G85BbGUnJzoOHf6gadGwCRAqFSnkZv4Fqu84Vq_WqIfy_oPVJlWQ%3D%3D&v=2&s=Vbelwusrlloqtlnpusmosplslplu&itype=1 HTTP/1.1" 302 - "http://www.nestoria.co.uk/victoria-road-ct14/house/sale" "Mozilla/5.0 (Linux; Android 4.1.2; GT-P3100 Build/JZO54K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.64 Safari/537.36" 10465 rd

x.x.x.x - - [17/Jul/2013:14:13:52 +0000] "GET /help/unknown-location HTTP/1.1" 404 9279 "-" "Mozilla/5.0 (Linux; U; en-gb; KFOT Build/IML74K) AppleWebKit/535.19 (KHTML, like Gecko) Silk/3.1 Safari/535.19 Silk-Accelerated=true" 5932 www

x.x.x.x - - [17/Jul/2013:14:13:53 +0000] "GET /api?action=search_listings&encoding=json&sort=price_lowhigh&price_min=150001&price_max=200000&page=1&radius=52.360992,-2.039034,1mi HTTP/1.1" 200 23729 "-" "-" 65233 api

Using /usr/bin/watch with highlight

alex@masaki:~$ watch --color 'du -hsc * | highlight -l G K M'1.1G alex32M davidl16K savio1.3G total

Tech Details

Perl Modules Used

For the app

App::Cmd::SimpleTerm::ANSIColorGetopt::Long::Descriptive

For testing

App::Cmd::TesterTest::Without::Module

Bug Reports / Contributions

https://metacpan.org/module/App::highlightVersion 0.13 released July 18th 2013 :-)

https://github.com/kaoru/App-highlight

https://travis-ci.org/kaoru/App-highlight

Nestoria

is H

iring!

http://l

okku.co

m/jobs

Thanks!