import this, that, and the other thing: custom importers
DESCRIPTION
Mr. Brett Cannonin PyCon2010-USA-Atlanta45min ◊◊◊ AdvancedFriday 11:45am, Centennial Icategories: coreSince Python 2.3, the ability has existed to customize the import process so that one can support code stored in alternative formats, e.g. zipimport and its support of importing code from a zip file. The aim of this talk is to make sure you understand how the custom import mechanism works and how to write your own custom importers using importlib from Python 3.1.TRANSCRIPT
![Page 1: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/1.jpg)
If you do not know what __path__ is,
this talk is NOT for you.
Sorry.
![Page 2: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/2.jpg)
import this, that, and the other thingCustom importers in Python
Brett Cannonwww.DrBrett.ca
Slides are sparse, so do listen to what I say.
![Page 3: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/3.jpg)
Thanks ...
• Python Software Foundation• PyCon Financial Aid committee
• Nasuni• Jesse Noller
![Page 4: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/4.jpg)
What the heck is an importer?
Relevant since Python 2.3
![Page 5: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/5.jpg)
importer=
finder + loader
![Page 6: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/6.jpg)
A finder finds modules.
![Page 7: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/7.jpg)
A loader loads modules.
![Page 8: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/8.jpg)
“Why do I want one?”
Customization/control, easier to work w/ than __import__
![Page 9: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/9.jpg)
How are custom importers used by
import?
Simplified view; ignoring implicit importers
![Page 10: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/10.jpg)
Meta pathsys.meta_path
![Page 11: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/11.jpg)
Start
for finder in sys.meta_path:
loader = finder.find_module(name, path)
return loader.load_module(name)
...
True
False
What ‘path’ arg is
![Page 12: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/12.jpg)
Pathsys.path or __path__, sys.path_hooks,
& sys.path_importer_cache
![Page 13: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/13.jpg)
... Parent module
has __path__
search = parent's __path__
search = sys.path
search
True
False
![Page 14: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/14.jpg)
for entry in search:
finder = sys.path_importer_cache[entry]
loader = finder.find_module(name)
return loader.load_module(name)
Search
path hookFalse
True
False finder
raise ImportError
True
![Page 15: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/15.jpg)
for hook in sys.path_hooks:
finder = hook(entry)
sys.path_importer_cache[entry] = finder
sys.path_importer_cache[entry] = dummy
path hook
finder
False
True
True/False = ImportError (not) raised
![Page 16: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/16.jpg)
how do I write my own
importer?Only masochists need apply.
![Page 17: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/17.jpg)
Option 1:Painfully from
scratchRead PEP 302 for the gory details.
![Page 18: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/18.jpg)
Option 2:Use importlib
Available since Python 3.1.I have suffered so you don’t have to.
![Page 19: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/19.jpg)
Option 3: importers
http://packages.python.org/importers/
File path abstraction on top of importlib.Treating as purgatory for importlib
inclusion.
If a lesson here, then it is to use option 2 or 3 depending on your needs.Rest of talk is about lessons that led to ‘importers’.
![Page 20: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/20.jpg)
Using azipfile importeras an example
Assuming use of importlib.Talking from perspective of using an archive.
![Page 21: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/21.jpg)
we need a hookFor sys.path_hooks.
![Page 22: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/22.jpg)
Refresher:Hooks look for a finder for a path
Path either from sys.path or __path__
![Page 23: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/23.jpg)
Hooks can get funky paths
E.g. /path/to/file/code.zip/some/pkg
Search backwards looking for a file; find a directory then you have gone too far.
![Page 24: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/24.jpg)
Consider caching archive file objects
No need to keep 3 connection objects open for the same sqlite3 file
![Page 25: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/25.jpg)
Pass your finder the “location”:
1)the path/object &2) the package path
Import assumes you are looking in a part of a package.
![Page 26: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/26.jpg)
Raise ImportError if you got nuthin’
![Page 27: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/27.jpg)
Have finder, will look for code
![Page 28: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/28.jpg)
Don’t treat modules as code but as files
Just trust me. Too many people/code make this assumption already for stuff like __file__, __path__, etc.
![Page 29: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/29.jpg)
You did remember where in the
package you are looking, RIGHT?!?
Needed because of __path__ manipulation by user code.
![Page 30: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/30.jpg)
fullname.rpartition(‘.’)[-1]
![Page 31: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/31.jpg)
Need to care about packages &
modulessome/pkg/name/__init__.py
andsome/pkg/name.py
Care about bytecode if you want.Notice how many stat calls this takes?
![Page 32: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/32.jpg)
Avoid caching within a finder
Blame sys.path_importer_cache
![Page 33: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/33.jpg)
Tell the loader if package &path to code
Don’t Repeat Yourself ... within reason.
![Page 34: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/34.jpg)
Nuthin’?Give back None
![Page 35: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/35.jpg)
Now it gets tricky
Writing a loader.
![Page 36: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/36.jpg)
Are you still thinking in terms of
file paths?
![Page 37: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/37.jpg)
importlib.abc.PyLoader
• source_path()• Might be changing...
• is_package()• get_data()
Everything in terms of exactly what it takes to import source
![Page 38: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/38.jpg)
importlib.abc.PyPycLoader
• source_path()• is_package()• get_data()• source_mtime()• bytecode_path()
• Might be changing...
This is what is needed to get source w/ bytecode right
![Page 39: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/39.jpg)
Reasons to ignore .pyc
• Jython, IronPython couldn’t care less.• Safe to support, though.
• Another thing to code up.• Bytecode is just an optimization.• If you only ship .pyc for code
protection, stop it.
![Page 40: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/40.jpg)
What to do when using
importlib ABCs
![Page 41: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/41.jpg)
Require anchor point for paths
somewhere/mod.py is too ambiguous
Too hazy as to where a relative path is anchored; archive? Package location?
![Page 42: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/42.jpg)
Consider cachingstat calls
Only for stand-alone loaders!Also consider caching if package or not.
Consider whether storage is read-only, append-only, or read-write.
![Page 43: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/43.jpg)
Don’t overdo error checking
EAFP is your friend.
![Page 44: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/44.jpg)
Perk of importers is the abstraction
![Page 45: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/45.jpg)
Lazy loader mix-inwritten in19 lines
![Page 46: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/46.jpg)
class Module(types.ModuleType): pass
class Mixin: def load_module(self, name): if name in sys.modules: return super().load_module(name) # Create a lazy module that will type check. module = LazyModule(name) # Set the loader on the module as ModuleType will not. module.__loader__ = self # Insert the module into sys.modules. sys.modules[name] = module return module
class LazyModule(types.ModuleType): def __getattribute__(self, attr): # Remove this __getattribute__ method by re-assigning. self.__class__ = Module # Fetch the real loader. self.__loader__ = super(Mixin, self.__loader__) # Actually load the module. self.__loader__.load_module(self.__name__) # Return the requested attribute. return getattr(self, attr)
![Page 47: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/47.jpg)
... or you could use the importers package
http://packages.python.org/importers/
![Page 48: Import this, that, and the other thing: custom importers](https://reader035.vdocument.in/reader035/viewer/2022062511/54b79e774a79593c588b45c7/html5/thumbnails/48.jpg)
Fin