The  Open  Source  Journey  at  OS  

Hillary  Corney  Principal  Architect  

Ordnance  Survey    

An  introduc9on  to  Ordnance  Survey  

•  Ordnance  Survey  over  220  years  old.  Founded  in  1791  •  Started  as  a  military  mapping  unit  now  the  UK’s  na9onal  mapping  agency.  

•  No  longer  military,  now  a  UK  Government  owned  company  a  “GovCo”  

•  Famous  for  the  best  paper  maps  in  the  world  •  What  you  may  not  know  is  that  OS  is  now  a  mostly  digital  organisa9on  and  that  it  hosts  the  largest  Geospa9al  database  in  the  world  called  “MasterMap”.  

•  We  also  hold  the  Guinness  Book  of  Records  world  record  for  the  largest  MinecraT  map  ever  made  with  an  astonishing  83  billion  blocks.  (And  you  thought  big    Government  compu9ng  projects  always  failed!)  

Digital  Maps  of  the  UK  accurate  to  a  1cm  updated  10,000  9mes  a  day    

Produc9on  IT  soTware  at  Ordnance  Survey  •  Ini9al  experiments  using  IT  to  support  map  making  started  in  late  1960s  •  First  digital  product  made  available  in  1971  (1:125,000  coastline)  •  Increasing  ramping  up  of  digi9sa9on  (1970s  –  1990s)  •  Move  from  automa9ng  map  produc9on  to  digital  data  (1990s  onwards)  •  In  the  early  years  much  of  the  Mapping    soTware  we  used  was  bespoke  •  Over  9me  increasing  use  made  of  proprietary  off-­‐the-­‐shelf  soTware  products  (as  

capability  aligned  with  our  needs)  •  In  the  last  10  years,  open-­‐source  op9ons  have  developed  and  matured  (as  

capability  has  improved)  

Bespoke  Proprietary  

Open  source  

1970   Mid-­‐1980s   Mid-­‐2000s   Today  1970   Mid-­‐1980s   Mid-­‐2000s   Today  

The  state  of  Open  Source  SoTware  adop9on  at  OS  Pre-­‐2011  

•  Managers  and  Architects  shied  away  from  open  source.  

•  Its  use  was  organic,  developer  led,  no  enterprise  level  involvement  or  strategy  

•  Open  Source  soTware  used  in  ad-­‐hoc  way  •  OTen  used  “under  the  radar”  •  SoTware  repositories  and  version  control  available  but  seldom  used.  

•  Very  licle  maintenance,  Patching,  upgrades  etc.  •  Separate  discrete  code  bases  for  the  same  soTware  

•  Almost  no  hand-­‐over  into  produc9on  and  licle  documenta9on    

A  short  history  of  open  source  at  Ordnance  Survey  Apache, Tomcat, J2EE & MySQL for initial web systems

2002

Linux operating system for web-facing applications

2004

Linux operating system for internal production systems

2006

Using Open Layers for OS OpenSpace

2008

PostGIS, GeoNetwork, GeoServer, INSPIRE

2010

PostGIS, Web Services Consolidation

2012 2013

Magento, Solr for Map Shop, Apache Jena for linked data

Open source

First

Push  

The  Open  Source  in  the  Enterprise  “revolu9on”  at  OS  

•  In  2010  the  outgoing  Treasury  Minister  Liam  Burne,  leT  a  now  infamous  note,  “there's  no  cash  le.”.  He  has  recently  said  it  was  a  foolish  thing  to  do  and  not  a  day  goes  by  without  him  feeling  guilty  for  wri9ng  it.  

•  But  don't  despair  Liam,  there  was  an  unexpected  upside......  

•  The  incoming  government,  worried,  and  looking  for  savings,  sent  out  the  “Open  Source  first”  direc9ve.  

The key points of the Government’s new directive were:

1)  The  Government  will  ac=vely  and  fairly  consider  open  source  solu=ons  alongside  proprietary  ones  in  making  procurement  decisions,    

2)  Procurement  decisions  will  be  made  on  the  basis  on  the  best  value  for  money  solu9on  to  the  business  requirement,  taking  account  of  total  life9me  cost  of  ownership  of  the  solu9on,  including  exit  and  transi9on  costs,  aTer  ensuring  that  solu9ons  fulfill  minimum  and  essen9al  capability,  security,  scalability,  transferability,  support  and  manageability  requirements.  

3)  The  Government  will  expect  those  pujng  forward  IT  solu9ons  to  develop  where  necessary  a  suitable  mix  of  open  source  and  proprietary  products  to  ensure  that  the  best  possible  overall  solu9on  can  be  considered.    

4)  Where  there  is  no  significant  overall  cost  difference  between  open  and  non-­‐open  source  products,  open  source  will  be  selected  on  the  basis  of  its  addi=onal  inherent  flexibility.    

How  do  you  “ac=vely  and  fairly  consider  open  source  solu=ons  alongside  proprietary  ones”?  

•  What  are  the  barriers  to  the  adop=on  of  Open  Source  So.ware?  

•  The  lack  of  a  Vendor  is  a  factor  in  low  OSS  uptake.  

•  Because  there  is  no  Vendor:  •  No  one  promotes  and  markets    the  soTware  •  No  one  pays  for  reviews  of  the  soTware  •  No  one  champions  the  soTware  •  No  one  sponsors  the  soTware  

•   So  OS  so.ware  is  o.en  invisible,  even  good  OS  so.ware  is  o.en  a  well  kept  secret.    

 

Barriers  to  the  adop9on  of  Open  Source  Software:

•  Myths  and  FUD  taken  as  truth.  •  MicrosoT  and  others  spent  large  amounts  of  effort  in  the  first  half  of  the  last  decade  debunking  Linux/OSS  by  spreading  fear  and  rumours  which  they  now  admit  actually  backfired  and  increased  awareness  and  adop9on  of  Linux/OSS  but  much  of  the  misinforma9on  from  those  days  s9ll  lingers  on.  

•   If  you  are  interested  Google  “MicrosoT  The  Halloween  document”  

•  Lack  of  a  “Vendor”  driving  any  tender  response  •  Lack  of  external  support  Lack  of  documenta9on  •  Lack  of  appropriate  in-­‐house  coding  skills  •  Lack  of  fair  and  honest  evalua=ons  comparisons  and  reviews  

More  Myths  -­‐  Open  Source  SoTware  is  “poor  quality”  

•  A  Coverity  2011  survey,  found  Open  source  was  as  good  as  proprietary  soTware  

•  “Both  open  source  code  quality  and  proprietary  code  quality,  as  measured  by  defect  density,  is  becer  than  the  average  for  the  soTware  industry,  which  is  a  defect  density  of  1.0.”  

•  “Linux  2.6,  PHP  5.3,  and  PostgreSQL  9.1  are  recognized  as  open  source  projects  with  superior  code  quality  and  can  be  used  as  industry  benchmarks,  achieving  defect  densi9es  of  .62,  .20,  and  .21  respec9vely.”  

•  Open  source  code  quality  is  on  par  with  proprietary  code  quality,  par9cularly  in  cases  where  codebases  are  of  similar  size.  For  instance,  Linux  2.6,  a  project  with  nearly  7  million  lines  of  code,  has  a  defect  density  of  .62  which  is  roughly  iden9cal  to  that  of  its  proprietary  codebase  counterparts.  

Coverity's  2013  Scan  concluded  that  the  defect  density  of  Open  Source  code  was  now    be-er  than  proprietary  code!  

•  In  2013,  the  defect  density  rate  of  open  source  code  was  lower  than  that  of  proprietary  code.  According  to  recent  reports  sponsored  by  Black  Duck  SoTware  and  North  Bridge  Venture  Partners,  “quality  concerns  are  no  longer  a  barrier  to  open  source  adop9on  in  the  enterprise.  In  fact,  the  quality  of  the  open  source  code  for  Coverity  Scan  par9cipants  can  be  higher  than  the  proprietary  code  included  in  an  enterprise  product.”  

So  if  quality  is  not  a  problem  how  do  you  “ac=vely  and  fairly  consider  open  source  solu=ons  alongside  proprietary  ones”?    

•  Appoint  an  internal  Open  Source  Champion.  Someone  who  will  take  the  place  of  the  missing  vendor  and  “market”  the  product  to  the  stakeholders.  

•  Use  proof  of  concept  builds  to  evaluate  the  suitability  of  OSS.  

•  Think  crea9vely  about  how  you  can  find  equivalents  to  the  normal  Vendor  procurement  process.  

Evaluate OSS like any COTS package but….

•  Where  you  would  evaluate  the  vendor  subs9tute  the  User  community.  

•  Where  you  would  evaluate  help  desk  support  look  at  the  availability  of  of  OSS  Specialist  Books,  the  community  and  internal  knowledge.    

•  Where  you  would  review  the  release  roadmap  Look  at  the  release  frequency  and  access  to  nightly  builds.  

•  Ask  the  user  community  for  reference  cases  and  recommenda9ons.  

Understand  Licensing  

•  There  are  over  70  licenses  approved  by  the  open  source  ini=a=ve,  so  it  is  definitely  not  an  easy  task  to  understand  all  of  the  licenses  out  there.  

•   Fortunately  the  10  most  popular  licenses  make  up  about  80%  of  the  open  source  available,  so  understanding  the  first  10  licenses  is  a  preXy  reasonable  start.    

•  Create  an  open  source  review  board  consis9ng  of  both  legal  counsel  and  SoTware  engineering  governance,  who  can  understand  the  implica9ons  of  the  various  licenses.    

•  Create  a  summary  of  current  OS  licences  keep  it  up  to  date.  

•  Lead  developers  and  Architects  will  need  to  review  the  summary  and  be  aware  of  the  legal  traps.  

Working  with  Open  Source  

•  As Laurie Wurster of the Gartner Group said: “Just because something is free, doesn’t mean it has no cost.” “Companies must have a policy for procuring OSS, deciding which applications will be supported by OSS, and identifying the intellectual property risk or supportability risk associated with using OSS. Once a policy is in place, then there must be a governance process to enforce it.”

•  There must be a policy and enforcing governance for procuring Open source.

•  There must also be a policy and governance for implementing, developing and supporting Open Source Software.

Recommenda9ons  

•  We  incorporate  the  Government  Open  Source  Policy  into  our  procurement  policy  

•  Ensure  that  when  Evalua9ng  OSS  there  is  a  quick  route  to  financing  proof  of  Concept  Evalua9on  Environments.  

•  We  adopt  a  structured  approach  to  evalua9ng  Open  Source  SoTware.  

•  We  educate  developers  to  be  aware  of  the  risks  inherent  in  Open  Source  licences.  We  encourage  open  source  skills  in  house  and  we  create  a  library  of  open  source  books  for  developers  to  use.  

•  We  appoint  OSS  Champions  to  each  Procurement  ini9a9ve  to  make  sure  that  OSS  is  considered  fairly  and  is  “Marketed”  to  the  Procurement  Stakeholders  in  a  proac9ve  way.  

•  We  ensure  good  legal  and  technical  governance  of  our  use  of  Open  Source.  

•  We  create  our  own  Ordnance  Survey  Open  Source  Licence  

Devise  a  “Procurement  decision  chart”  

Open  Source  first  

Based  on  our  experiences,  we  have  adopted    an  “open  source  first”  enterprise  level  policy  for  new  soTware    When  looking  for  new  soTware  we  will  evaluate  open  source  packages  ahead  of  proprietary  ones  We  will  adopt  an  open  source  solu9on  if:  

•  The  licence  is  suitable  for  our  needs  •  The  project  is  well  supported  with  a  lively  and  responsive  user  

base  •  The  documenta9on  is  good  •  We  have  the  appropriate  skills  in-­‐house  or  training  is  readily  

available  •  We  can  support  it  (or  we  can  buy  support)  •  It  works  equally  well  as  available  proprietary  soTware.    

If  not,  we  will  go  through  standard  COTS  selec9on  /  procurement  processes  

 

What  have  we  learned  

•  Start  small,  download  the  soTware  and  try  stuff  out.  

•  Involve  and  listen  to  your  developers  •  Broaden  technical  knowledge  and  skills  •  invest  in  training  •  Change  the  COTS  only  culture  •  “Open  Source”  is  free  is  a  myth  the  costs  are  just  in  different  places.  

•  Be  rigorous  about  evalua9ng  licenses    

What  about  support?  

•  Just  because  it’s  Open  Source  does  not  necessarily  mean  you  are  on  your  own,  but  the  support  model  is  different.  •  You  can  do  it  in  house  (but  be  aware  of  the  effort  involved  and  don’t  scrimp)  

•  Or  you  can  buy  support  from  vendors  such  as  WSO2  which  is  an  increasing  sign  that  Open  Source  is  entering  the  mainstream.  

How  did  we  apply  what  we  have  learned  to  our  API  delivery  plarorm  project?  

•  We  reviewed  the  Open  Source  packages  first.  •  We  evaluated  WSO2  against  other  Open  Source  and  COTS  solu9ons    

•  We  loaded  it  onto  our  developer  sandboxes  and  “kicked  the  9res”  

•  We  engaged  WSO2  and  ini9ated  a  very  successful  proof  of  concept….  

POC  Solu9on  hosted  on  two  servers  in  AWS  and  built  with  a  Joint  OS/WSO2  team  in  just  two  weeks  

Integra9on  with  Magento  

•  Another  benefit  of  the  “Open  Source  First”  approach  is  that  we  are  able  to  closely  couple  and  integrate  another  Open  Source  program  “Magento  Enterprise”  a  leading  eCommerce  package  with  the  WSO2  SoTware  

•  As  we  have  access  to  the  source  code  of  both  plarorms  we  can  9ghtly  integrate  the  two  in  a  way  that  would  not  be  possible,  or  very  much  more  difficult  with  COTTS  soTware.    

Logical  Architecture  PRODUCTION  environment  

Authen9ca9on  Architecture  Integra9on  between  Magento  and  WSO2  Iden9ty  Server  via  SAML  and  SCIM  

The  outcome  of  the  Open  Source  First  process:  

•  We  purchased  support  and  finally  having  sa9sfied  ourselves  that  WSO2  can  meet  our  needs….  

•  We  engaged  WSO2  and  are  building  not  one  but  two  API  delivery  produc9on  environments  with  WSO2  soTware.  U9lising:  

•  WSO2  API  manager  •  WSO2  Iden9ty  Manager  •  WSO2  Business  ac9vity  Monitor  •  WSO2  Complex  event  processor  •  Magento  Enterprise  Open  Source  Shopping  cart.  

The  best  of  both  worlds?  

•  It  is  my  opinion  that  organisa9ons  like  WSO2  and  Magento  offer  the  best  of  both  worlds.  

•  A  vibrant  user  community  feeding  ideas  and  fixes  back  to  the  user  base  

•  With  Enterprise  level  support  and  SoTware  roadmap  and  strategy  from  the  Parent  organisa9on.  

•  WSO2  are  the  “Vendor”  providing  Enterprise  level  support  oTen  missing  from  tradi9onal  Open  Source  products,  eBay  Enterprise  provide  similar  Support  for  Magento.  

Ian  James  OS  Chief  Architect  said:  

“Open  Source  allows  us  to  deliver  more  effec9vely”    

“Open  Source  is  mainstream”    

“Open  Source  is  here  to  stay  at  Ordnance  Survey”  

Thank  You  

Ques9ons?