calcifer - read the docs

calciferRelease 0.3.0

May 05, 2017

Contents

1 Installation 3

2 Development 52.1 TL;DR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3 Release History 73.1 Next Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.2 Test-case Usage Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.3 License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.4 Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.5 Indices and tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

i

calcifer, Release 0.3.0

Calcifer is designed to provide interfaces for describing the evaluation and processing of nested higher-order datastructures by nested definitions of policy rules.

Policies may be used to evaluate some source object both for validation, and to generate template descriptions of a“complete” version of that object. This evaluation is done at runtime and can hook into arbitrary functions, e.g. forchoosing policies based on some current system state. (Hypermedia style)

Policies may be defined with implicit non-determinism, allowing the specification of multiple policy choices withminimal boilerplate for handling the aggregation of results. (Prolog style)

Calcifer also provides a system by which application-layer code can annotate specific policy rules, making the point-in-time context of a policy computation into a first-class value. This allows for rich error handling, by being aware ofspecific points of policy failure and allowing annotated policy rules to control the formatting of their own errors.

This library was written to facilitate the development of a hypermedia subscription management API. This library’sdesign is informed by that API’s goals of business logic cohesion and adaptability to changing policy rules. A majorgoal for that project has been to alleviate client integrations of their need to perform any policy determination locally;Calcifer has stemmed largely from this effort.

Contents 1


2 Contents

CHAPTER 1

Installation

pip install calcifer

3


4 Chapter 1. Installation

CHAPTER 2

Development

1. Create a new virtual environment

2. Install development requirements from dev-requirements.txt

3. Run tests nosetests

4. detox is installed and will run the test suite across all supported python platforms

5. python setup.py build_sphinx will generate documentation into build/sphinx/html

TL;DR

$ virtualenv env$ ./env/bin/pip install -qr dev-requirements.txt$ source env/bin/activate(env) $ nosetests(env) $ python setup.py build_sphinx(env) $ detox

5

https://testrun.org/tox/


6 Chapter 2. Development

CHAPTER 3

Release History

Next Release

• Implement greatness.

Test-case Usage Examples

# from tests/test_contexts.py

def test_apply_alchemy(self):# for our test today, we will be doing some basic alchemyinventory = [

"aqua fortis","glauber's salt","lunar caustic","mosaic gold","plumbago","salt","tin salt","butter of tin","stibnite","naples yellow",

]

# backstory:# ~~~~~~~~~~## falling asleep last night, you finally figured out how to complete# your life's work: discovering the elusive *elixir of life*!## and it's only two ingredients! and you have them on hand!#

7


# ...## unfortunately this morning you can't remember which two# ingredients it was.## you'll know it once you've gotten it, just have to try out# all possible mixtures. (should be safe enough, right?)

forgotten_elixir_of_life = set(random.sample(inventory, 2))

discoveries_today = set(["frantic worry", "breakfast"])

# ok time to go do some arbitrary alchemy!## game plan:alchemy_ctx = Context()

# you'll grab one ingredient,selected_first_ctx = alchemy_ctx.select("/inventory").each()first_substance = selected_first_ctx.value

# and another,selected_second_ctx = selected_first_ctx.select("/inventory").each()second_substance = selected_second_ctx.value

# take them to your advanced scientific mixing equipment,workstation_ctx = selected_second_ctx.select("/workstation")

# (btw this is your advanced scientific procedure that you are# 100% certain will tell you what some mixture is)def mix(first, second):

"""takes two ingredients and returns the resulting substance"""if set([first, second]) == forgotten_elixir_of_life:

return "elixir of life"return "some kind of brown goo"

# then you'll mix your ingredients...mixed_ctx = workstation_ctx.apply(

mix,first_substance, second_substance

)resulting_mixture = mixed_ctx.value

# ... and! in today's modern age, scientists now know to record their# results!mixed_ctx.select("/discoveries").append_value(resulting_mixture)

# got it? good!result = run_policy(

alchemy_ctx.finalize(),{"inventory": inventory, "discoveries": discoveries_today}

)

# in a flurry of excitement, i bet you didn't even stop to# look at your discoveries as you made them!## well, let's see...

8 Chapter 3. Release History


self.assertIn("elixir of life", result["discoveries"])

def test_apply_dangerous_alchemy(self):# nice job! and you even finished in time to go foraging for# more ingredients!inventory = [

"aqua fortis","glauber's salt","lunar caustic","mosaic gold","plumbago","salt","tin salt","butter of tin","stibnite","naples yellow",

# nice find"anti-plumbago"

]

# but unfortunately, it's the next day, and the same thing# has happened to you! except this time it was for your# other life's goal: discover the ~elixir of discord~!## well, since it was so easy...

whatever_concoction = set(['some ingredients'])

discoveries_today = set([])should_be_fine = 'overconfidence' not in discoveries_todayassert should_be_fine

# doing alchemy la la laalchemy_ctx = Context()

# grabbin' things off shelvesselected_first_ctx = alchemy_ctx.select("/inventory").each()first_substance = selected_first_ctx.value

selected_second_ctx = selected_first_ctx.select("/inventory").each()second_substance = selected_second_ctx.value

# got our ingredientsgot_ingredients_ctx = selected_second_ctx

workstation_ctx = got_ingredients_ctx.select("/workstation")

# mixin' - don't stop to thinkdef mix(first, second):

mixture = set([first, second])if mixture == whatever_concoction:

return 'missing elixir'if mixture == set(['plumbago', 'anti-plumbago']):

return 'concentrated danger'return 'more brown goo'

mixed_ctx = workstation_ctx.apply(

3.2. Test-case Usage Examples 9


mix,first_substance, second_substance

)resulting_mixture = mixed_ctx.value

mixed_ctx.select("/discoveries").append_value(resulting_mixture)

# wait wait wait!!def danger(mixture):

if mixture == 'concentrated danger':return True

return False

# we can't have that.danger_ctx = mixed_ctx.check(

danger,resulting_mixture

)danger_ctx.forbid()

# moral:## a strong understanding of policies and processes facilitates a# hazard-free lab environment.result = run_policy(

alchemy_ctx.finalize(),{"inventory": inventory, "discoveries": discoveries_today}

)

self.assertIn("errors", result)self.assertTrue(len(result['errors']))

License

The Calcifer library is distributed under the MIT License

Documentation

Concepts

Overview

Calcifer aims to provide a computing model for describing data processing procedures that closely match policies in asource domain.

Ultimately, Calcifer seeks to provide a means to build systems that not only validate input, but also “fill in the gaps”for incomplete or incorrect input. Calcifer offers systems the ability to generate template descriptions of valid input,as well as automatically indicate, with source domain semantics, what makes a given input invalid.

Calcifer hopes to offer this functionality with minimal impedance, to the end that code using the library can stillresemble traditional imperative and/or functional paradigms.


https://github.com/DramaFever/calcifer/blob/master/LICENSE


Policy

The term policy is used to refer specifically to Calcifer “policies”, or computations in the Calcifer model. Policies aredesigned to allow analogous description of various kinds of real-world policies, e.g. software business logic, requestvalidation, and data pipeline operation.

Policy computation is stateful and matches imperative programming styles in that values are mutable and statementsare ordered. It is often more natural to describe procedures from a source domain in a non-pure1, state-driven fashion.

An example policy expression:

from calcifer import Policy

@Policydef allowed_favorites_policy(ctx):

# sorry greenctx.select("favorite_color").whitelist_values(["purple", "orange"])

This specifies that "purple" and "orange" are the only two valid choices for favorite_color.

Usage is as follows:

>>> allowed_favorites_policy.run({"favorite_color": "purple"})[{'favorite_color': 'purple'}]

>>> allowed_favorites_policy.run({"favorite_color": "red"}) # not valid[{'favorite_color': 'red',

'errors': [{u'code': 'INVALID_VALUE_SELECTION',u'context': [<policy 'allowed_favorites_policy'>,<policy 'select("favorite_color")'>,<policy 'whitelist_values'>],

u'scope': u'/favorite_color',u'value': 'red',u'values': ['purple', 'orange']}]}]

>>> allowed_favorites_policy.run({}) # anything goes![{'favorite_color': 'purple'}, {'favorite_color': 'orange'}]

Computing Model

Calcifer employs the following concepts to create its computing model:

Higher-Order Trees The data maintained in the computation of policies is structured as a tree where each node mayhave a value, and/or have a template description of values, or have neither of these.

Templates afford the ability to describe constraints on acceptable values without requiring producing actualinstances of values.

Nodes can have neither a value or a template and be completely unknown, the system attempts to make assump-tions as nodes get referenced through usage. This allows defining a node very precisely several levels deep,without committing to defining all the parent nodes upfront.

Scoping and Policy Partials In addition to the underlying tree, which can be thought of merely as a reference toits own root, the computation model also maintains a pointer to a given sub-tree or node. This is similar tojsonpointer or CSS selectors.

1 Future versions of Calcifer may attempt to model state less implicitly in order to encourage safer programming.

3.4. Documentation 11


This allows modular computations - individual sub-policies can be defined that operate on a scoped subset ofthe rest of the data.

At any given step in a particular computation, the whole of the computing model’s “data memory” is known asa partial. Partials are isomorphic to tuples of the form (𝑡𝑟𝑒𝑒, 𝑠𝑐𝑜𝑝𝑒),

Free Computation, First-class Context-awareness Calcifer operates in a free2 fashion, meaning that policies arebuilt as first-class values: operations for a given policy are specified and defined as data.

At a high level, contexts define nested collections of operating conditions for individual computations. Theparticular context at any step in a policy computation is analagous to the usual point-in-time instruction-pointerstack.

The difference is - the “code as data” structure gives certain affordances.

Namely, contexts themselves are first-class and can be passed around as values, providing certain “convenient”capabilities, such as customized error handling and the composition/nesting of contexts as “regular” Pythonvariables. Calcifer Context objects are individually strange little Turing toys, able to behave in different ways,depending how they are put together.

Non-Determinism Mimicking the common logic programming paradigm, Calcifer’s computing model allows for theforking and pruning of operations. Policy determination may return 0 results, 1 result exactly, or any number ofvalid results.

The over-arching mechanism is akin to the parallel computation of different policy alternatives, removing failingpolicies along the way, and producing some list of results.

Usage Examples of Features

Deferred Values - Rudimentary Role Permissions

Given some data access for permission lookup based on role:

def fetch_allowed_permissions(role='nobody'):"""Given some system role, return allowed permissions"""# ...# connect to db, fetch from an API, load from settings, whatever.# ...return list(permissions)

The policy for an incoming request can be expressed as follows, where "role" and "permission" are propertiesof the incoming request.

from calcifer import Policy

@Policydef allowed_permissions_policy(ctx):

role_ctx = ctx.select("role") # available when the policy is computedfetched_permissions_ctx = ctx.apply(

fetch_permitted_values, # apply this functionrole_ctx # over this value

)

ctx.select("permission").whitelist_values(

2 The concept is that of a Free Monad, giving access to the underlying AST of the computation at definition runtime as well as execution runtime.



fetched_permissions_ctx)

The _ctx suffix is used to indicate that the variables do not hold actual values. role_ctx can be verbalized as “thecontext where the value of the role is known”, or fetched_permissions_ctx can similarly be thought of as“the context where the permissions have been fetched.”

(Sometimes these contexts are never reached, a topic for another section, but N.B. that these are first-class values andsubject to control flow)

In this example, Context values are used in three capacities:

1. As deferreds: role_ctx is used as a stand-in for the value of the role, whenever, say, a request comes in, andthe system is calculating what actions to allow.

2. As function applications over deferred values: ctx.apply() takes args function_or_function_ctx,*values_or_value_ctxes, and connects the plumbing to ensure that the function is called correctly whenvalues are available.

3. As stateful operators: ctx.select("permission").whitelist_values(...) indicates that thepolicy computation may have more than one valid result. This is the forking operation described above: oncethe policy knows the fetched permissions, the policy specifies some number of valid alternatives.

API

Context API

Overview and Purpose

Contexts are ordered containers of policy rules, functions that generate policy rules, and/or sub-contexts. Contextsprovide semantic grouping of policy rules and policy rule generation through means of deferred value resolution.

The primary goal for Context is to allow the semantic expression of request- processing policies, so that application-level policy code can be written with minimal regard for the common underlying non-determinism and error-propagation behaviors.

General Structure / Scope

Structurally, Context comprises the following parts:

• An ordered list of items contained in the context

• A wrapper, within which run the context’s contained items

• Additional semantic annotations such as a context name

A context object may be used for either or both of the following purposes:

• Contexts may be used to represent a distinct semantic grouping of policy rules, as might be expressedby business logic or security requirements. This may be granular or broad. This semantic grouping mayspecify some control flow for the contained items, or it may define the way in which errors are represented.

Typically, the semantic meaning of each context is along the lines of either: “with regard to some field orparameter”, or “with regard to some condition being the case”.

Using contexts in this fashion follows the builder pattern - applying chained method calls to the contextobject.



• Or, the context may represent a value that will exist when policy is run on a given request. For instance,ctx.select(“merchant_type”) would represent the value for that node.

A context may be used as this deferred value by passing it as an argument to any exposed method on anyparent context object.

Contexts may be used as values in either a context free or a context specific manner. The former is moretypical, just passing the context to some method. Context-specific values can be accessed as a propertysomectx.value, and can only be used as arguments for methods in contexts descending from the valueprovider (somectx)

Context

class calcifer.contexts.Context(wrapper=None, *ctx_args, **kwargs)Context provides a high-level interface for building policies.

Policies are built by performing stateful operations on Context objects.

Method calls/property retrievals modify the Context to potentially include additional policy rules, as appropriate.

Implementation details can be found in BaseContext

add_error()Create a blank error

append_value(value)Appends value to the current node, assuming the node to be a list if not defined

children()Return the context with the list of scopes that are direct children of the current node

each(**kwargs)Create and return a context that operates on each child of the current node.

Parameters ref – An injectable reference object that has matching children nodes (same struc-ture dict or list)

err()Trigger error handling

error_ctx()Retrieves, possibly creating, a specialized error handler policy for the Context

fail_early()Returns a new context that checks node “/errors” and short-circuits if any errors exist.

forbid(*args)Opposite of require() - errors when value is defined

last_errorReturns the context selecting the most recently defined error

or_error()If context fails, inject error instead. Error has the following properties:

value Value found at node

scope The current scope at the time of error

context The contextual traceback

require(*args)Requires that a value is defined and truthy.



Parameters value – if not provided, uses value for current node

set_value(value)Sets the value for the current node

whitelist_values(values)Forks computation, erring if value is provided already and does not match

Low-level Policy Operators

Premium Command Policy StateT Operators.

These are provided as building blocks for specifying Premium Command Policies for the purposes of template gener-ation and command validation.

Partial Operators

calcifer.operators.scope()Returns the current scope for the partial

Returns PolicyRule string json pointer

calcifer.operators.select(scope, set_path=False)Retrieves the policy node at a given selector and optionally sets the scope to that selector. Recursively definesUnknownPolicyNodes in the partial.

Parameters

• scope (json pointer string) – Scope to select

• set_path (bool) – Sets the scope

Returns PolicyRule (Node v)

calcifer.operators.get_node()Retrieves the node at the current scope


calcifer.operators.define_as(node)Define the node at the current scope

Parameters node – Node v


calcifer.operators.get_value()Retrieves the value for the node at the current pointer. Equivalent to get_node() >> unit_value

Returns PolicyRule v

calcifer.operators.set_value(value)Sets the value for the currently scoped policy node. Overwrites the node with a LeafPolicyNode

Parameters value (v) – new value

Returns PolicyRule v

calcifer.operators.append_value(value)Gets the value at the current node and appends value. The current node value should be either a set or a list, orundefined.

Parameters value – value to append


https://docs.python.org/2/library/functions.html#bool


calcifer.operators.children()For DictPolicyNodes or ListPolicyNodes, returns all scopes that are direct children.

Returns PolicyRule [scope]

Control-flow Operators

calcifer.operators.unit(value)Returns a value inside the monad

Parameters value – the value returned inside the PolicyRule monad

calcifer.operators.unit_value(node)Given a node (often returned as monadic result), return the value for the node.

Parameters node (Node _v_) – the node whose value is to be returned inside the PolicyRulemonad

Returns PolicyRule _v_

calcifer.operators.collect(*rule_funcs)Given a list of policy rule functions, returns a single policy rule func that accepts some value, provides that toeach function, resetting the scope each time.

calcifer.operators.policies()Given a list of policy rules, returns a single policy rule that applies each in turn, keeping scope constant for each.(By resetting the path each time)

calcifer.operators.regarding()Given a selector and a list of functions that generate policy rules, returns a single policy rule that, for each rulefunction:

1. sets the scope to the selector / retrieves the node there 3. passes the node to the rule_func to generate a policyrule 4. applies the policy rule at the new scope

In addition, regarding checks the current scope and restores it when it’s done.

calcifer.operators.check()Given a function that takes no arguments, returns a policy rule that runs the function and returns the result andan unchanged partial

calcifer.operators.each(*rule_funcs, **kwargs)each(rule_func) is a policy rule function that accepts a dictionary and calls rule_func(value) successively, withthe partial scope set to the key.

each optionally takes a named argument ref=dict() to provide a built-in lookup for some reference dictionary. Ifref is provided, rule_func(ref[key]) is called instead.

Non-Determinism

calcifer.operators.match()Given an expected value, selects the currently scoped node and ensures it matches expected. If the match resultsin a new node definition, the partial is updated accordingly.

For non-matches, returns a monadic zero (e.g. if we’re building a list of policies, this would collapse from[partial] to [])

calcifer.operators.require_value()Returns an mzero (empty list, e.g.) if the provided node is missing a value



Examples

>>> select("/does/not/exist") >> require_value[]

calcifer.operators.forbid_value()Returns an mzero (empty list, e.g.) if the provided node is missing a value

For instance: select(“/does/not/exist”) >> forbid_value

returns []

calcifer.operators.permit_values()Given a list of allowed values, matches the current partial against each, forking the non-deterministic computa-tion.

calcifer.operators.fail()

Error-Handling

calcifer.operators.attempt(*rules)Keeping track of the value and partial it receives, if the result of *rules on the partial is mzero, then attemptreturns unit( (initial_value, initial_policy) ) otherwise, attempt returns the result of the rules.

calcifer.operators.trace()Collates the current scope, the current node’s value, and the current policy context and returns it as a dict

calcifer.operators.unless_errors()

Context Annotation

calcifer.operators.push_context()Add an additional context to the stack for the partial

calcifer.operators.pop_context()Pop the partial’s context stack, returning whatever value it was called with.

calcifer.operators.wrap_context()Run some operator inside some context

Release History

Next Release

• Implement greatness.

Contributing

If you want to help make this project better you are officially an awesome person. Any and all contributions, whetherit’s patches, documentation, or bug reports, are very much welcome and appreciated.

Pull requests or Github issues are always welcome. If you want to contribute a patch please do the following.

1. Fork this repo and create a new branch

2. Do work


https://github.com/DramaFever/calcifer/pulls

https://github.com/DramaFever/calcifer/issues


3. Add tests for your work (Mandatory)

4. Submit a pull request

5. Wait for Coveralls and Travis-CI to run through your PR

6. It’ll be code reviewed and merged

As a note, code without sufficient tests will not be merged.

Indices and tables

• genindex

• modindex

• search


Index

Aadd_error() (calcifer.contexts.Context method), 14append_value() (calcifer.contexts.Context method), 14append_value() (in module calcifer.operators), 15attempt() (in module calcifer.operators), 17

Ccheck() (in module calcifer.operators), 16children() (calcifer.contexts.Context method), 14children() (in module calcifer.operators), 15collect() (in module calcifer.operators), 16Context (class in calcifer.contexts), 14

Ddefine_as() (in module calcifer.operators), 15

Eeach() (calcifer.contexts.Context method), 14each() (in module calcifer.operators), 16err() (calcifer.contexts.Context method), 14error_ctx() (calcifer.contexts.Context method), 14

Ffail() (in module calcifer.operators), 17fail_early() (calcifer.contexts.Context method), 14forbid() (calcifer.contexts.Context method), 14forbid_value() (in module calcifer.operators), 17

Gget_node() (in module calcifer.operators), 15get_value() (in module calcifer.operators), 15

Llast_error (calcifer.contexts.Context attribute), 14

Mmatch() (in module calcifer.operators), 16

Oor_error() (calcifer.contexts.Context method), 14

Ppermit_values() (in module calcifer.operators), 17policies() (in module calcifer.operators), 16pop_context() (in module calcifer.operators), 17push_context() (in module calcifer.operators), 17

Rregarding() (in module calcifer.operators), 16require() (calcifer.contexts.Context method), 14require_value() (in module calcifer.operators), 16

Sscope() (in module calcifer.operators), 15select() (in module calcifer.operators), 15set_value() (calcifer.contexts.Context method), 15set_value() (in module calcifer.operators), 15

Ttrace() (in module calcifer.operators), 17

Uunit() (in module calcifer.operators), 16unit_value() (in module calcifer.operators), 16unless_errors() (in module calcifer.operators), 17

Wwhitelist_values() (calcifer.contexts.Context method), 15wrap_context() (in module calcifer.operators), 17

19

calcifer - read the docs

Documents