harnessing the power of python in arcgis using the conda ...xlrd, netcdf4, requests, pypdf, pytz...
TRANSCRIPT
-
HarnessingthePowerofPythoninArcGISUsingtheCondaDistributionShaunWalbridge
-
https://github.com/scw/conda-uc-2016-demo
HighQualityPDF(2MB)
https://github.com/scw/conda-uc-2016-demohttps://4326.us/esri/conda-uc/uc-2016-conda-conda-demo-full.pdf
-
Python
-
WhyPython?Accessiblefornew-comers,andthe
Extensivepackagecollection(56thousandon ),broaduser-baseStronggluelanguageusedtobindtogethermanyenvironments,bothopensourceandcommercialOpensourcewithliberallicense—dowhatyouwant
mosttaughtfirstlanguageinUSuniversites
PyPI
http://cacm.acm.org/blogs/blog-cacm/176450-python-is-now-the-most-popular-introductory-teaching-language-at-top-us-universities/fulltexthttps://pypi.python.org/pypi
-
WhyPython?Inthebox:
TheSciPyStack(NumPy,SciPy,Pandas,matplotlib,sympy)
xlrd,netCDF4,requests,PyPDF,pytz
ScientificProgrammingwiththeSciPyStack
https://4326.us/esri/scipy/#/
-
WhyPython?Beyondthebox:
IntegratingOpen-SourceStatisticalPackageswithArcGISusingPythonandR—Tomorrowat10:15am,Ballroom6D
Python:ExtendingwithOtherLibraries—Todayat4:00pm,TechTheater16
DeeperDiveintoCondainDevSummitTechSessionVideo
http://video.esri.com/watch/5072/harnessing-the-power-of-python-in-arcgis-using-the-conda-distribution
-
PackageManagementforPython
Whynotpip,wheels,virtualenvs?Don’thandletheharderproblemofsystemdependencies,consideredoutofscopebyPythonpackagers—doesitendupinsite-packages?Packagedevs:OnOSXandLinux,‘easy’togetthedeps!Useasystempackagemanager(e.g.apt,brew,yum)andtheincludedcompiler(e.g.clang,gcc).It’sstillnoteasytomakereproduciblebuilds,andwhataboutWindows?
-
WhataboutWindows?WeareparticularlystuckonWindowswhichlacksbroadlyusedpackagemanagementOnlydevelopershaveaCcompilerontheirmachineAhardproblem
-
WhataboutWindows?WeareparticularlystuckonWindowswhichlacksbroadlyusedpackagemanagementOnlydevelopershaveaCcompilerontheirmachineAhardproblem
EnterConda
-
WhyConda?
ScientificPythoncommunityidentifiedthattherewasagapnotbeingaddressedbythecorePythoninfrastructure,limitingtheirabilitytogetpackagesintothehandsofusers
Industrystandardbuiltbypeoplewhocareaboutthisspace—ContinuumAnalytics
-
WhyConda?
Itsolvesahardproblem:
Handlesdependenciesformanylanguages(C,C++,RandofcoursePython)
BuiltforPythonfirst,butitreallysolvesamuchbroaderinfrastructuralissue.
-
Conda
-
CondaCross-platform:simplydeveloprecipesforbuildingandinstallingsoftwareonLinux,OSXandWindows.Opensource:Esriisusingit,youcanuseitinyourownprojectsforothercontexts
Whatcanitinstall?Notjustscientificpackages.Itcanhelpwith:
GUItoolkits(PyQt,TKinter)C++Libraries(Boost)IDEs(Spyder,Juptyer)
-
CondaEnvironments:CanisolateaPythonenvironment,flexiblymakechangeswithotaffectinginstalledsoftware.Requirements—includeexplicitstateinformation,notjustthepackagename.Namesaren’tenough!AlsohandlesplatformsandJupyternotebooks
-
HowDoesitWork?Condapackagescancomefromavarietyoflocations:
Ondisk(file://)Publicrepositories(AnacondaCloud,self-hosted)Privaterepositoriesanaconda.org
https://anaconda.org/
-
CondaBasicsCommandlineinterface,fornow
CondaCheatsheet
http://conda.pydata.org/docs/_downloads/conda-cheatsheet.pdf
-
CondaBasicsDemo
-
CondaBasicsActivatingenvironments,acoupleways:
UsetheshortcutsManuallyactivatetheenvironment:
cd C:\ArcGIS\bin\Python\Scripts activate arcgispro-py3
-
CondaBasicsTostart:
AcollectionofpackagesandPythoninstalliscalledanenvironmentorenv,thebuildingblockformanagingPythonwithCondaCanhavemultipleenvironmentsandseamlesslyswitchbetweenthem
conda --help
-
CondaBasicsOnceyou’reinanenvironmentgetdetailswithinfo:
Condainfoisthestartingpoint—ittellsyouthestateoftheenvironment.
conda info
-
CondaBasicsconda info
Current conda install:
platform : win-64 conda version : 4.0.6 conda-build version : not installed python version : 3.5.1.final.0 requests version : 2.9.1 root environment : C:\ArcGIS\bin\Python (writable) default environment : C:\ArcGIS\bin\Python\envs\arcgispro-py3 envs directories : C:\ArcGIS\bin\Python\envs package cache : C:\ArcGIS\bin\Python\pkgs channel URLs : https://conda.anaconda.org/esri/win-64/ https://conda.anaconda.org/esri/noarch/ https://repo.continuum.io/pkgs/free/win-64/ https://repo.continuum.io/pkgs/free/noarch/ config file : C:\ArcGIS\bin\Python\.condarc
-
CondaBasicsconda list
# packages in environment at C:\ArcGIS\bin\Python\envs\arcgispro-py3:#arcgispro 1.3 0 esricolorama 0.3.6 py34_0 defaultsfuture 0.15.2 py34_0 defaultsmatplotlib 1.4.3 np19py34_0 defaultsmsvc_runtime 1.0.1 vc10_0 [vc10] defaultsnose 1.3.7 py34_0 defaultsnumpy 1.9.3 py34_0e [arcgispro] esriopenssl 1.0.2h vc10_0 [vc10] defaultspandas 0.17.1 np19py34_0 esripip 8.1.1 py34_1 defaultspy 1.4.31 py34_0 defaultspyparsing 2.1.1 py34_0 defaultspypdf2 1.25.1 py_0 esripytest 2.9.1 py34_0 defaultspython 3.4.4 4 defaultspython-dateutil 2.5.3 py34_0 defaultspytz 2016.4 py34_0 defaultsrequests 2.9.1 py34_0 defaultsscipy 0.16.1 np19py34_0e [arcgispro] esrisetuptools 20.7.0 py34_0 defaultssix 1.10.0 py34_0 defaultssympy 0.7.6.1 py34_0 defaultsvs2010_runtime 10.00.40219.1 0 defaults
-
CondaBasicsCreatingnewenvironments:
Afewdifferentways.Canmanuallyspecifythedependencies:
Canalsouseafilewhichincludesallthedependencies:
Thesecancontainexplcitinformationaboutchannels,toensurethatthenewenvironmentpreciselymatchestherequirements.
conda create --name my_env python=3.4 numpy flask dask
conda create --name my_env --file my_sweet_depends.txt
-
Condavs…Name Means Will
Ship?
Conda Thecommanditself ✓Miniconda AminimumsetofPythonpackagesto
buildandrunConda.✓
Anaconda Adistribution200+packagesbuiltwithConda
AnacondaServer
Hostthefullinfrastructureinternally
-
SkikitLearnDemoHavetweetsaboutaniOSappreleasedat#SXSWWhattopeoplethinkofit?Useanaivebayesclassifiertodeterminesentiment
-
SkikitLearnDemo# Scikit learn model based Lukas Biewald's Scikit Learn class# https://github.com/lukas/scikit-class
import arcpyimport pandas as pdfrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn.naive_bayes import MultinomialNB
input_csv = arcpy.GetParameterAsText(0)test_string = arcpy.GetParameterAsText(1)
df = pd.read_csv(input_csv)target = df['is_there_an_emotion_directed_at_a_brand_or_product']text = df['tweet_text']
-
SkikitLearnDemofixed_text = text[pd.notnull(text)]fixed_target = target[pd.notnull(text)]
count_vect = CountVectorizer()count_vect.fit(fixed_text)counts = count_vect.transform(fixed_text)
nb = MultinomialNB()nb.fit(counts, fixed_target)
# print out our predictionarcpy.AddMessage(nb.predict(count_vect.transform([test_string][0])))
-
DeeperDive
-
MultiplePythonsCurrently:
Platform Pythonversion
Desktop Python2.7.x(2.7.10)
Pro Python3.4.x(3.4.3)
-
MultiplePythonsUpgradecode?
Doit!Youcansupport2+3withoutthatmuchworkStillneedtochangearcpy.mappingtoarcpy.mpwhenmovingfromDesktoptoPro,butnoPythonlanguagelevelchangesneeded....
But…thiscanbecostly.Formanyorganizations,asignificantburden,evenifthelanguagechangesarerelativelysmall.MultiplePythonsis
asolutiontothis.
PythonmigrationforArcGISPro
http://pro.arcgis.com/en/pro-app/arcpy/get-started/python-migration-for-arcgis-pro.htm
-
ChallengesHavetomakesureyou’rerunningtherightPython(whathappens
whenyoutypepythonatthecommandline?)WorkingtomakethiseasyaspossibleIt’llbeeasytotellinappIsolatedinstallationfixesavarietyofissues
Requiressomeusereducationoverthe“onlyonePythononthebox”model
-
WhatDoIGetOutoftheBox?
CondacommandandaCondarootPythoninstallNewmodules(e.g.requests)CondaenvironmentwithalloftheArcGISProdependenciesasCondapackages
-
HowcanIusethis?WealreadyshipyoutheSciPystack—powerfulandoutofthebox,canusetoday(Proand10.4)Canstartusingcondatoday.Minicondaisfullystand-alone,won’taffectyourglobalPython(unlessyoutellitto)Packageyourwork:thisisanopportunitytodistributeit,possiblyincludingcommercialsideaswell.
-
WhereCanIRunThis?
ArcGISPro1.3WillbethePythoninstall.
Future:UIforinteractionTakeadvantageofmorefeaturesIntegrationwithplatform
-
fromfutureimport*EffectivelymanagecomplexsoftwaredependencieswithConda.Thousandsofpackagesexisttoday,canintegrateitintoyour
organization’sneeds.
-
Closing
-
ThanksEsriCondaTeam:
ContinuumAnalyticsforcreatingandopensourcingConda
-
RateThisSessionPleasetakeoursurvey,findsessioninappandprovidereview