path dependent-development (pycon india)
DESCRIPTION
(Image on page 3: it's the traditional fast/good/cheap trade-off. Something glitched in the conversion))TRANSCRIPT
![Page 1: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/1.jpg)
Path Dependent Development
Nick Coghlan@ncoghlan_dev
Red Hat ToolsmithCPython Core Developer
![Page 2: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/2.jpg)
Usefully Wrong
“All models are wrong. Some models are useful.”
“... the practical question is: How wrong do they have to be to not be useful?”
George E. P. Box (statistician) “Empirical Model-Building”
![Page 3: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/3.jpg)
Choose Any Two?
![Page 4: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/4.jpg)
Path Dependence
● “good enough to be useful” -> ship it● The decisions we make leave their mark on
the software we ship● These marks remain long after the scope of
the software expands to other use cases
![Page 5: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/5.jpg)
What is “Good Enough”?
● Depends on your priorities and resources– What are you building?
– Why are you building it?
– Who are you building it for?
– Who is building it?
– What are you building it with?
– How much risk can you tolerate?
![Page 6: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/6.jpg)
Context Matters
● Building an intranet web service– Trusted network
– Enforced user base
● Building a web startup– Hostile network
– Business lives or dies by user choice
● Building hardware control and management systems– Usage driven by hardware
– Software as a necessary evil
![Page 7: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/7.jpg)
Trade-Offs Needed:Inquire Within
![Page 8: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/8.jpg)
Functionality
● Doing one (or a few) things well is often better than doing a lot of things badly
● Adding functionality later is usually easier to sell than taking it away (no matter how broken it turns out to be)
![Page 9: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/9.jpg)
Flexibility
● Don't make things configurable● Configurability = testing and maintenance pain● Do separate concerns (if you make it configurable
later, only one place needs to change)● Do use flexible support tools
– SQL Alchemy makes it easy to change database
– Django locks in some major decisions (like ORM and templating language) but provides a rich ecosystem of prebuilt components that work well together
![Page 10: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/10.jpg)
Security
● A lot of software is still insecure by default– Unhashed (or poorly hashed) passwords
– Unencrypted communications channels
● Multiple layers of defence can hide this● Try to make the “easy option” and the “secure
option” one and same● Can be very hard to fix poor security choices
![Page 11: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/11.jpg)
Reinventing Wheels
● Reuse means dependency management● Often simpler to roll your own to start● With good modularity, easy to replace later● Watch for increasing complexity
![Page 12: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/12.jpg)
Documentation
● How sophisticated are users expected to be?– Installed by developers? Admins? End users?
– Intended for domain experts only?
● Is it stable enough to document?● Documentation can highlight design flaws
![Page 13: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/13.jpg)
Test Quality
● Fine grained tests pinpoint failures easily
● Coarse grained tests are often easier to write
● Can easily start with coarse grained tests, then add more fine grained tests to narrow down failures
● Slow tests are better than no tests
● External dependencies are better than no tests
● Regression tests are great, but don't let them block fixes for problems that can't be reproduced reliably
![Page 14: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/14.jpg)
Code Reviews
● Code is written to:– Tell the computer what to do
– Tell future maintainers what it does
● Tests cover the first, reviews the second● Debatable value for small teams● Highly valuable for large teams● Needs appropriate tools
![Page 15: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/15.jpg)
Performance & Scalability
● Don't stress about it if you don't need to● Start with measurement infrastructure● If simple is fast enough, stick with simple
![Page 16: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/16.jpg)
Reliability
● Not all software is mission critical● Pay attention to failure modes● Error quality matters
![Page 17: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/17.jpg)
Usability
● Humans are still a lot smarter than computers● If users have no choice, they'll usually cope● Hence, awful UX in most “enterprise” software
![Page 18: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/18.jpg)
Maintainability & Business Risks
● The Bus Factor– Most startups = 1
– Large companies want it to be higher
● Developer docs (including comments)● Legal risks (copyrights, patents, trademarks)
![Page 19: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/19.jpg)
Automation
● Critical to speeding up release cycles● Is a process stable enough to automate?
![Page 20: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/20.jpg)
Managing Path Dependence
![Page 21: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/21.jpg)
Exit Strategies
● Know what you're not doing● Have a vague idea how to fix it when needed● Actual fixes will depend on future needs● Sometimes, the only right answer is “No”
![Page 22: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/22.jpg)
Patterns and Processes
● Keep your options open● Minimise current complexity● This is not easy
– Software architecture and design patterns
– Software processes and methodologies
● “interim” solutions may last a long time● If you don't have a test suite, start there
![Page 23: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/23.jpg)
Prototyping vs Implementation
● Two very different modes of development● Prototyping
– Exploration
– Trying to figure out what is feasible
● Implementation– Already known to be feasible
– Making it happen to a known specification
● Big difference in priorities!
![Page 24: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/24.jpg)
Social Implications
● Design decisions are context dependent● Easy to criticise in hindsight● Design trade-offs can influence community● Actually getting better at building software● Ambitions are (more than?) keeping pace
![Page 25: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/25.jpg)
Path Dependence in Action
![Page 26: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/26.jpg)
An Innocent Start
● PulpDist: Mirroring network based on rsync ● Simple job definitions{ "remote_server": "localhost", "remote_path": "/demo/simple/", "local_path": "/var/www/pub/sync_demo_raw/", ...}
● Simple custom validator for JSON data
– Checks on individual values
– Overall sanity checks on full jobs
![Page 27: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/27.jpg)
Don't Repeat Yourself
● Simple format turned out to be too simple– Hard to modify given multiple jobs from same source
● Enhanced format with reusable elements{ "mirror_id": "local_copy", "tree_id": "simple_sync", "site_id": "bne", ...}
● Simple validator was no longer adequate
![Page 28: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/28.jpg)
What To Do?
● Upgrade the existing validator– Possible, but tedious to test properly
– Not a good wheel to reinvent
● JSON validation library– Research would be starting from scratch
– Hard to assess quality quickly
● Relational database– Enforces the constraints by its very nature
– Error quality would likely be poor
![Page 29: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/29.jpg)
Two Birds...
● For validation, I needed to:– Ensure identifiers were unique
– Ensure cross references were valid
● For UI purposes I also needed:– To filter by component identifiers
– To sorting by various fields
● Sound familiar?
![Page 30: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/30.jpg)
Two Birds...
● For validation, I needed to:– Ensure identifiers were unique
– Ensure cross references were valid
● For UI purposes I also needed:– To filter by component identifiers
– To sorting by various fields
● Sound familiar?
![Page 31: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/31.jpg)
...One Stone
● An in-memory SQLite database was perfect● But writing SQL by hand is still horrible● SQL Alchemy in target environment● Problem solved!
– Config loaded into DB after simple field validation
– If the DB accepts it, references are also valid
![Page 32: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/32.jpg)
How Does The Story End?
● Still some very rough edges– Sqlite error messages are quite user hostile
– Schema changes are triple-keyed
● Future changes?– Master in database, JSON only as export?
– Improved error messages?
– Switch to an actual schema engine?
● Other priorities!
![Page 33: Path dependent-development (PyCon India)](https://reader033.vdocument.in/reader033/viewer/2022060200/559876af1a28ab39058b4581/html5/thumbnails/33.jpg)
Q & A
Pulp:
http://pulpproject.org/
PulpDist:
https://fedorahosted.org/pulpdist/
CPython Sprints
Monday & Tuesday