gruvi documentation · excellent cross-platform support including windows. •i wanted to use a pep...

Gruvi DocumentationRelease 0.10.3

Geert Jansen

Jun 14, 2017

Contents

1 Documentation 3

2 Project files 45

Python Module Index 49

i

Gruvi Documentation, Release 0.10.3

Improved ergonomics for Python programmers wanting to use asynchronous IO.

Gruvi is an asynchronous IO library for Python. It focuses on the following desirable properties:

• Simple. Async IO code should look just like normal code, with simple, sequential control flow and regularfunctions.

• Efficient. An IO library should have a very low memory and CPU overhead, so that it can scale to small systemsor to many concurrent connections.

• Powerful. Common protocols like SSL/TLS and HTTP should be part of every IO library.

Contents 1


2 Contents

CHAPTER 1

Documentation

Rationale

Gruvi is an asynchronous I/O library for Python, just like asyncio, gevent and eventlet. It is therefore a fair to ask whyI decided to create a new library. The short and simple answer is that I didn’t agree with some of the decision decisionthat went into these other projects, and thought it would be good to have a library based on different assumptions.

These are the design requirements that I had when creating Gruvi:

• I wanted to use green threads. Compared to Posix threads, green threads are very light weight and scalable.Compared to a generator-based yield from approach, they provide better ergonomics where there is just onetype of function, rather than two that cannot easily call into each other.

• I wanted to use libuv, which is a very high performance asynchronous I/O library famous from node.js, and hasexcellent cross-platform support including Windows.

• I wanted to use a PEP 3156 style internal API based on transports and protocols. This model matches verywell with libuv’s completion based model. Compared to a socket based interface, a transport/completion basedinterface forces a strict separation between “network facing” and “user facing” code. In a socket based interfaceit is easy to make mistakes such as putting a socket read deep in some protocol code, and it incentives hackssuch as monkey patching.

Comparison to other frameworks

The table below compares some of the design decisions and features of Gruvi against asyncio, gevent and eventlet.

Note: tables like these necessarily compress a more complex reality into a much more limited set of answers, and theyalso a snapshot in time. Please assume good faith. If you spot an error in the table, please let me know and I willchange it.

3

http://docs.python.org/3.4/library/asyncio.html

http://gevent.org/

http://eventlet.net/

https://github.com/joyent/libuv

http://nodejs.org/

https://www.python.org/dev/peps/pep-3156


http://gevent.org/


mailto:[email protected]


Feature Gruvi Asyncio Gevent EventletIO library libuv stdlib libev stdlib / libeventIO abstraction Transports /

ProtocolsTransports / Protocols Green sockets Green sockets

Threading fibers yield from greenlet greenletResolver threadpool threadpool threadpool /

c-aresblocking /dnspython

Python: 2.x YES (2.7) YES (2.6+, viaTrollius)

YES YES

Python: 3.x YES (3.3+) YES NO NOPython: PyPy NO NO YES YESPlatform: Linux FAST FAST FAST FASTPlatform: MacOSX

FAST FAST FAST FAST

Platform:Windows

FAST (IOCP) FAST (IOCP) SLOW (select) SLOW (select)

SSL: Posix FAST FAST FAST FASTSSL: Windows FAST (IOCP) FAST (IOCP 3.5+) SLOW (select) SLOW (select)SSL: Contexts YES (also Py2.7) YES (also Py2.6+) NO NOHTTP FAST (via

http-parser)NO (external) SLOW (stdlib) SLOW (stdlib)

Monkey Patching NO NO YES YES

Motivations for choosing green threads

Green threads have a very low memory overhead, and can therefore support a high level of concurrently. But moreimportantly, green threads are cooperatively scheduled. This means that a thread switch happens only when specificallyinstructed by a switch() call, and with a bit of care, we can write concurrent code that does not require locking.

When combining a thread implementation with an IO framework, one of the key design decisions is whether toimplement explicit or implicit switching.

Green thread switching in Gruvi is implicit. This means that whenever a function would block, for example to wait fordata from the network, it will switch to a central scheduler called the hub. The hub will then switch to another greenthread if one is ready, or it will run the event loop. A function that can block is just like any other function. It is calledas a regular function, and can call other (blocking and not) functions.

A common criticism of the implicit switching approach is that the locations where these switches happen, the so-calledswitch points, are not clearly marked. As a programmer you could call into a function that 3 levels down in the callstack, causes a thread switch. The drawback of this, according to the criticism, is that the switch points could happenessentially anywhere, and that therefore it’s like pre-emptive multi-threading where you need full and careful locking.

The alternative to implicit switching is explicit switching. This is the approach taken for example by the asyncioframework. In this approach, every switch point is made explicit, in the case of asyncio because it is not called as anormal function but instead used with the yield from construct.

In my view, a big disadvantages of the explicit approach is that the explicit behavior needs to percolate all the way upthe call chain: any function that calls a switch point also needs to be called as a switch point. In my view this requirestoo much up-front planning, and it reduces composability.

Rather than implementing explicit switching, Gruvi sticks with implicit switching but tries to address the “unknownswitch points” problem, in the following way:

• First, it makes explicit that a switch can only happen when you call a function. If you need to change someglobal state atomically, it is sufficient to do this without making function calls, or by doing it in a leaf function.In this case, it is guaranteed that no switch will happen.

4 Chapter 1. Documentation


http://libev.schmorp.de/

http://libevent.org/

https://pypi.python.org/pypi/fibers

https://pypi.python.org/pypi/greenlet

https://pypi.python.org/pypi/greenlet

http://c-ares.haxx.se/

http://www.dnspython.org/

https://bitbucket.org/enovance/trollius

https://github.com/joyent/http-parser



• Secondly, in case you do need to call out to multiple functions to change a shared global state, then as a pro-grammer you need to make sure, by reading the documentation of the functions, that they do not cause a switch.Gruvi assists you here by marking all switch points with a special decorator. This puts a clear notice in the func-tion’s docstring. In addition, Gruvi also provides a gruvi.assert_no_switchpoints context managerthat will trigger an assertion if a switch point does get called within its body. You can use this to be sure that noswitch will happen inside a block.

• Thirdly, the “traditional” option, Gruvi provides a full set of synchronization primitives including locks that youcan use if the two other approaches don’t work.

In the end, the difference between implicit and explicit switching is a trade-off. In my view, with the safeguardsof Gruvi e.g. the marking of switch points and the gruvi.assert_no_switchpoints context manager, thebalance tips in favor of implicit switching. Some people have come to the same conclusion, others to a different one.Both approached are valid and as an programmer you should pick the approach you like most.

Motivations for not doing Monkey patching

One other important design decision in Gruvi that I decided early on is not to implement monkey patching. Monkeypatching is an approach employed by e.g. gevent and eventlet where they make the Python standard library cooperativeby replacing blocking functions with cooperative functions using runtime patching.

In my experience, monkey patching is error prone and fragile. You end up distributing parts of the standard libraryyourself, bugs included. This is a maintenance burden that I’m not willing to take on. Also the approach is verysusceptible to dependency loading order problems, and it only works for code that calls into the blocking functions viaPython. Extension modules using e.g. the C-API don’t work, as well as extension modules that use an external libraryfor IO (e.g. psycopg).

Finally, monkey patching does not work well with libuv because libuv provides a completion based interface while thestandard library assumes a ready-based interface.

The solution that Gruvi offers is two-fold:

• Either, use Gruvi’s own API if available. For example, Gruvi includes classes to work with streams and pro-cesses, and it also provides an excellent HTTP client and server implementation. This is the preferred option.

• When integrating with third-party blocking code, run it in the Gruvi maintained thread pool. The easiest way isto call this code via the gruvi.blocking() function.

Installation

Gruvi uses the setuptools, so installation is relatively straightforward. The following Python / OS combinations aresupported:

OS Python versions NotesPosix 2.7, 3.3+ Only Linux is regularly testedMac OSX 2.7, 3.3+ PPC is not testedWindows 2.7, 3.3+ No SSL backports on 2.7

Gruvi and some of its dependencies contain C extensions. This means that you need to have a C compiler, and alsothe Python development files. Gruvi also uses CFFI so you need to have that installed as well.

Installation using pip

If you have the “pip” package manager, then you can install Gruvi directly from the Python Package Index:

1.2. Installation 5

http://initd.org/psycopg

https://pypi.python.org/pypi/cffi


$ pip install cffi$ pip install gruvi

You need to install CFFI first because the Gruvi setup script depends on it.

Installation from source

The following instructions install Gruvi in a virtualenv development environment:

$ pyvenv gruvi-dev # "virtualenv gruvi-dev" with Python <= 3.3$ . gruvi-dev/bin/activate$ git clone https://github.com/geertj/gruvi$ cd gruvi$ pip install -r requirements.txt$ python setup.py develop

Linux Specific Notes

Most mainstream Linux versions have a supported version of Python installed by default, with the notable exceptionof RHEL/CentOS version 6. For this OS you can use Software Collections to install a supported Python version.

To install the required dependencies on Debian/Ubuntu systems:

$ sudo apt-get update$ sudo apt-get install -y gcc python-dev python-virtualenv libffi-dev

On Red Hat/Fedora type systems:

$ sudo yum install -y gcc python-devel python-virtualenv libffi-devel

Mac OSX Specific Notes

Recent versions of OSX have a supported Python version available.

The easiest way to install a C compiler is to install Xcode from the Mac App Store. This is a free download.

Since OSX Mavericks, the required libffi dependency has become part of OSX itself, so there’s no need to install itseparately. If you are on an earlier version of OSX, it’s recommended to install libffi via Homebrew.

Windows Specific Notes

Windows does not come with a system provided version of Python, so you have to install one from python.org. It isrecommended to use version 3.3 or 3.4.

For the C compiler, I recommend to use Microsoft Visual C++ 2010 Express. It is a free download, and it is the samecompiler used to create the official Python builds for Windows since version 3.3.

You can also use MinGW. In that case make sure you use either Python 2.7.6, 3.3.3, 3.4 or later. These version containa fix for issue12641.


http://softwarecollections.org/

https://www.microsoft.com/visualstudio/eng/downloads#d-2010-express

http://bugs.python.org/issue12641


Reference

Front Matter

Package Namespace

The API of Gruvi is exposed through the gruvi package. The individual pieces that make up the API are actuallydefined in submodules, but those are pulled into the package scope by the package’s __init__.py.

The symbol names in the gruvi package are designed to be unique and descriptive within the scope of Gruvi, but arenot globally unique. Therefore you should use Gruvi as follows:

# Import the packageimport gruvigruvi.sleep(1)

# Or.. import individual symbolsfrom gruvi import Processproc = Process()

But never like this:

# Don't use this. It won't work because __all__ is set to []from gruvi import *

Gruvi includes a few important protocol implementations that are provided under its “batteries includes” philosophy.These are exposed as submodules under the Gruvi namespace. For example:

from gruvi.http import HttpClientclient = HttpClient()

External protocols for Gruvi are recommended to install themselves into the gruvi.ext namespace package.

Exceptions

Errors are reported through exceptions. The following base exceptions are defined:

exception ErrorBase class for Gruvi exceptions.

exception TimeoutA timeout has occurred.

exception CancelledA fiber or calback was cancelled.

More exceptions are defined by subsystems and they are documented with their respective subsystem.

Functions that are a direct mapping of a pyuv function can also raise a pyuv exception. Gruvi does not try to capturethese and transform them into a Gruvi exception, so in these cases you might want to capture the pyuv exceptionseparately. The relevant pyuv base exception is:

exception pyuv.error.UVErrorBase exception for all pyuv errors.

1.3. Reference 7


Timeouts

Many parts of the Gruvi API use timeouts. Timeouts follow the Python convention that they are expressed in seconds,and can be either passed as an integer or a floating point argument. The granularity of timeouts in Gruvi is approxi-mately 1 millisecond due to the fact that the libuv API represents timeouts as an integer number of milliseconds.

In addition the above, there are two other values for timeout arguments:

• None means “no timeout”.

• -1 means a default timeout. This arguments is accepted for methods of objects where the constructor allowsyou to set a default timeout.

When a timeout occurs in a function or method, a Timeout exception is raised.

Reference Counting

Memory management and the freeing of unused resources in Gruvi requires a little cooperation from the programmer.In summary, if a object has a close() method, you should call it before you are done with the object. You shouldnever expect it to be called automatically for you by a destructor. If you do not call a close() method, then thememory associated with the object will likely not be fully reclaimed.

The reason for this behavior is as follows. Gruvi, as an event-based IO library, uses a central event loop. The event loopcontains a reference to all objects for which Gruvi interested in getting events. These events are registered as callbacksback into Gruvi objects. The event loop is provided by libuv, through the pyuv bindings. The libuv framework callsthese objects “handles”.

Gruvi provides many objects that wrap a pyuv handle. For example, there are objects for a TCP socket or a process.These objects contain a reference to a pyuv handle, and while the handle is active, it has a callback back into the Gruviobject itself. This forms a cyclic reference in which destructors are not called. This means destructors cannot be usedto call close() methods automatically.

Some magic could be done by using weak references. However, the fact remains that a libuv handle will not bereleased until it is deactivated (and there’s good reasons for that behavior). So even with weak references, you wouldstill be required to call close(), making the whole point of them moot. Don’t be too concerned though, you will seethat in practice it is not a big deal.

Fibers and the Hub

Fibers are light weight execution contexts that are cooperatively scheduled. They form the basis for concurrency inGruvi. Comparing fibers to threads:

• Like threads, fibers represent an independent flow of control in a program.

• Fibers are more light weight than threads, meaning you can run more of them.

• Unlike threads, there can only ever be one active fiber (per thread, see below). This means that fibers give youconcurrency but not parallelism.

• Unlike threads, fibers are cooperatively scheduled. They are never preempted and have to yield control otherfibers explicitly (called switching).

Gruvi uses the python-fibers package as its fiber library.

Fibers logically exist within threads. Each fiber has its own Python and C-level stack. When a Python programstarts, there will be one thread called 'Main', and this thread will have one fiber called 'Root'. Both are createdautomatically; the main thread by Python and the root fiber by the python-fibers package.


https://github.com/saghul/pyuv/issues/63

https://github.com/saghul/python-fibers


Fibers are organized in a tree. Each fiber except the root fiber has a parent. When a fiber exits, either because its mainfunction exits or because of an uncaught exception, control is passed to its parent. If the root fiber exits, the thread willexit.

The hub and fiber scheduling

The Gruvi framework maintains a special fiber called the Hub. The hub runs an event loop, and acts as a central fiberscheduler. The event loop in Gruvi is provided by libuv through the pyuv bindings.

From a high-level, the flow of a Gruvi program is as follows:

1. Run the current fiber (initially the root fiber).

2. When the fiber needs to wait for something, for example network data, the event to wait for is registered withthe event loop, together with a switch back callback.

3. The fiber switches to the hub, which run the event loop.

4. When the event loop detects an event for which a fiber registered interest, it will will call the callback. Thiscauses a switch back to the fiber that installed the event.

The hub instance is automatically created when in is first needed. It can be retrieved using the following function:

get_hub()Return the instance of the hub.

The event loop is available as the .loop property:

Hub.loopThe event loop used by this hub instance. This is an instance of pyuv.Loop.

Creating fibers and switching

To create a new fiber, instantiate the Fiber class and pass it a main function. The Fiber class is a thin wrapper ontop of fibers.Fiber, to which it adds a few behaviors:

• The fiber is created as a child of the hub.

• Only the hub is allowed to switch to this fiber. This prevent complex interactions where any fiber can switch toany other fiber. In other words, the hub is a real hub.

class Fiber(target, args=(), kwargs={}, name=None, hub=None)An cooperatively scheduled execution context aka green thread aka co-routine.

The target argument is the main function of the fiber. It must be a Python callable. The args and kwargs specifyits arguments and keyword arguments, respectively.

The name argument specifies the fiber name. This is purely a diagnositic tool is used e.g. in log messages.

The hub argument can be used to override the hub that will be used to schedule this fiber. This argument is usedby the unit tests and should not by needed.

nameThe fiber’s name.

aliveWhether the fiber is alive.

start()Schedule the fiber to be started in the next iteration of the event loop.

1.3. Reference 9


https://pypi.python.org/pypi/pyuv

https://pyuv.readthedocs.io/en/latest/loop.html#pyuv.Loop

https://python-fibers.readthedocs.io/en/latest/index.html#fibers.Fiber


cancel(message=None)Schedule the fiber to be cancelled in the next iteration of the event loop.

Cancellation works by throwing a Cancelled exception into the fiber. If message is provided, it will beset as the value of the exception.

switchpoint join(timeout=None)Wait until the fiber completes.

The only two fibers in a Gruvi program that use fibers.Fiber directly are the hub and the root fiber. All otherfibers should be created as instances of Fiber.

When a fiber is created it doesn’t run yet. To switch to the fiber, call its start() method and call a function that willswitch to the hub:

def say_hello():print('Hello, there!')

fiber = Fiber(say_hello)fiber.start()

print('Starting fiber')sleep(0)print('Back in root fiber')

The output of this will be:

Starting fiberHello, there!Back in root fiber

Working with the event loop

To register interest in a certain event, you need to create the appropriate pyuv.Handle instance and add it to theloop. The callback to the handle should cause a switch back to the current fiber. You also want to make sure youimplement a timeout on the event, and that you clean up the handle in case it times out. Because this logic can berelatively tricky to get right, Gruvi provides the switch_back context manager for this:

class switch_back(timeout=None, hub=None, lock=None)A context manager to facilitate switching back to the current fiber.

Instances of this class are callable, and are intended to be used as the callback argument for an asynchronousoperation. When called, the switchback object causes Hub.switch() to return in the origin fiber (the fiberthat created the switchback object). The return value in the origin fiber will be an (args, kwargs) tuplecontaining positional and keyword arguments passed to the callback.

When the context manager exits it will be deactivated. If it is called after that then no switch will happen. Alsothe cleanup callbacks are run when the context manager exits.

In the example below, a switchback object is used to wait for at most 10 seconds for a SIGHUP signal:

hub = get_hub()with switch_back(timeout=10) as switcher:

sigh = pyuv.Signal(hub.loop)sigh.start(switcher, signal.SIGHUP)switcher.add_cleanup(sigh.close)hub.switch()



https://pyuv.readthedocs.io/en/latest/handle.html#pyuv.Handle


The timeout argument can be used to force a timeout after this many seconds. It can be an int or a float. If atimeout happens, Hub.switch() will raise a Timeout exception in the origin fiber. The default is None,meaning there is no timeout.

The hub argument can be used to specify an alternate hub to use. This argument is used by the unit tests andshould normally not be needed.

fiberThe origin fiber.

timeoutThe Timeout exception if a timeout has occurred. Otherwise the timeout parameter provided to theconstructor.

activeWhether the switchback object is currently active.

switch(value=None)Switch back to the origin fiber. The fiber is switch in next time the event loop runs.

throw(typ, val=None, tb=None)Throw an exception into the origin fiber. The exception is thrown the next time the event loop runs.

add_cleanup(callback, *args)Add a cleanup action. The callback is run with the provided positional arguments when the context man-ager exists.

Lockless operation and switchpoints

Functions that may cause a switch to happen are called switch points. In Gruvi these functions are marked by a specialdecorator:

switchpoint(func)Mark func as a switchpoint.

In Gruvi, all methods and functions that call Hub.switch() directly, and all public APIs that can cause anindirect switch, are marked as a switchpoint. It is recommended that you mark your own methods and functionsin the same way. Example:

@switchpointdef myfunc():

# may call Hub.switch() here

Knowing where switches may happen is important if you need to modify global state in a non-atomic way. If you canbe sure that during the modification no switch points are called, then you don’t need any locks. This lockless operationis one of the main benefits of green threads. Gruvi offers a context manager that can help with this:

class assert_no_switchpoints(hub=None)Context manager that defines a block in which no switches may happen, and in which no switchpoints may becalled.

Use it as follows:

with assert_no_switchpoints():do_something()do_something_else()

If the context manager detects a switch or a call into a switchpoint it raises an AssertionError.

1.3. Reference 11

https://docs.python.org/3.5/library/exceptions.html#AssertionError


The assert_no_switchpoints context manager should not be overused. Instead it is recommended to try andconfine non-atomic changes to a global state to single functions.

Utility functions

current_fiber()Return the current fiber.

Note: The root and hub fiber are “bare” fibers.Fiber instances. Calling this method there returns the bareinstance, not a gruvi.Fiber instance.

spawn(func, *args, **kwargs)Spawn a new fiber.

A new Fiber is created with main function func and positional arguments args. The keyword arguments arepassed to the Fiber constructor, not to the main function. The fiber is then scheduled to start by calling itsstart() method.

The fiber instance is returned.

switchpoint sleep(secs)Sleep for secs seconds. The secs argument can be an int or a float.

Fiber local data

class localFiber-local data.

To manage fiber-local data, instantiate this class and store attributes on it:

mydata = local()mydata.x = 10

Attributes have a value or are unset independently for each fiber.

Hub reference

class HubThe central fiber scheduler and event loop manager.

ignore_interrupt = FalseBy default the hub will raise a KeyboardInterrupt in the root fiber when a SIGINT (CTRL-C) is received.Set this to True to ignore SIGINT instead.

nameThe name of the Hub, which is 'Hub'.

loopThe event loop used by this hub instance. This is an instance of pyuv.Loop.

dataA per-hub dictionary that can be used by applications to store data.

Keys starting with 'gruvi:' are reserved for internal use.

pollA centrally managed poller that can be used install callbacks for file descriptor readiness events.



https://pyuv.readthedocs.io/en/latest/loop.html#pyuv.Loop


switchpoint close()Close the hub and wait for it to be closed.

This may only be called in the root fiber. After this call returned, Gruvi cannot be used anymore in thecurrent thread. The main use case for calling this method is to clean up resources in a multi-threadedprogram where you want to exit a thead but not yet the entire process.

switch()Switch to the hub.

This method pauses the current fiber and runs the event loop. The caller should ensure that it has set upappropriate callbacks so that it will get scheduled again, preferably using switch_back. In this casethen return value of this method will be an (args, kwargs) tuple containing the arguments passed tothe switch back instance.

If this method is called from the root fiber then there are two additional cases. If the hub exited due to acall to close(), then this method returns None. And if the hub exited due to a exception, that exceptionis re-raised here.

run_callback(callback, *args)Queue a callback.

The callback will be called with positional arguments args in the next iteration of the event loop. If youadd multiple callbacks, they will be called in the order that you added them. The callback will run in theHub’s fiber.

This method is thread-safe: it is allowed to queue a callback from a different thread than the one runningthe Hub.

Mixing threads and fibers

There are two common situations where you might want to mix threads and fibers:

• When running CPU intensive code. In this case, you should run the code the CPU thread pool.

• When running third party code that performs blocking IO. In this case, run the code in the IO thread pool.

In both cases, running the code in a thread pool allows the hub to continue servicing IO for fibers. All other cases ofmixing threads and fibers are generally a bad idea.

The following code is Gruvi is thread safe:

• The Hub.run_callback() method.

• All synchronization primitives in Synchronization primitives.

All other code is not thread safe. If, for whatever reason, you must access this code from multiple threads, use locksto mediate access.

Synchronization primitives

Gruvi contains a set of primitives that can be used for synchronization between multiple fibers and threads. Allprimitives documented below are thread safe. They are modeled after the primitives in the Python threading andqueue modules.

class LockA lock.

The lock can be locked and unlocked explicitly using acquire() and release(), and it can also be usedas a context manager.

1.3. Reference 13

https://docs.python.org/3.5/library/threading.html#module-threading

https://docs.python.org/3.5/library/queue.html#module-queue


switchpoint acquire(blocking=True, timeout=None)Acquire the lock.

If blocking is true (the default), then this will block until the lock can be acquired. The timeout parameterspecifies an optional timeout in seconds.

The return value is a boolean indicating whether the lock was acquired.

locked()Whether the lock is currently locked.

release()Release the lock.

class RLockA reentrant lock.

A reentrant lock has the notion of a “lock owner” and a “lock count”. If a reentrant lock is acquired, and it wasalready acquired by the current fiber, then the lock count is increased and the acquire call will be successful.Unlocking a reentrant lock may only be done by the lock owner. The lock becomes unlocked only after it isreleased as many times as it was acquired.

switchpoint acquire(blocking=True, timeout=None)Acquire the lock.

If blocking is true (the default), then this will block until the lock can be acquired. The timeout parameterspecifies an optional timeout in seconds.

The return value is a boolean indicating whether the lock was acquired.

locked()Whether the lock is currently locked.

release()Release the lock.

class EventAn event.

An event contains an internal flag that is initially False. The flag can be set using the set() method and clearedusing the clear() method. Fibers can wait for the flag to become set using wait().

Events are level triggered, meaning that the condition set by set() is “sticky”. Setting the event will unblockall current waiters and will cause future calls to wait() not to block, until clear() is called again.

set()Set the internal flag, and wake up any fibers blocked on wait().

clear()Clear the internal flag.

switchpoint wait(timeout=None)If the internal flag is set, return immediately. Otherwise block until the flag gets set by another fiber callingset().

class Condition(lock=None)A condition.

A condition is always associated with a lock. The state of the condition may only change when the caller hasacquired the lock. While the lock is held, a condition can be waited for using wait(). The wait method willrelease the lock just before blocking itself, so that another fiber can call notify() to notify the condition.

The difference between a condition and an Event is that a condition is edge-trigerred. This means that whena condition is notified, only fibers that are waiting at the time of notification are unblocked. Any fiber that calls



wait() after the notification, will block until the next notification. This also explains why a lock is needed.Without the lock there would be a race condition between notification and waiting.

The lock argument can be used to share a lock between multiple conditions. It must be a Lock or RLockinstance. If no lock is provided, a RLock is allocated.

notify(n=1)Raise the condition and wake up fibers waiting on it.

The optional n parameter specifies how many fibers will be notified. By default, one fiber is notified.

notify_all()Raise the condition and wake up all fibers waiting on it.

switchpoint wait(timeout=None)Wait for the condition to be notified.

The return value is True, unless a timeout occurred in which case the return value is False.

The lock must be held before calling this method. This method will release the lock just before blockingitself, and it will re-acquire it before returning.

switchpoint wait_for(predicate, timeout=None)Like wait() but additionally for predicate to be true.

The predicate argument must be a callable that takes no arguments. Its result is interpreted as a booleanvalue.

exception QueueEmptyQueue is empty.

exception QueueFullQueue is full.

class Queue(maxsize=0)A synchronized FIFO queue.

The maxsize argument specifies the maximum queue size. If it is less than or equal to zero, the queue size isinfinite.

qsize()Return the size of the queue, which is the sum of the size of all its elements.

switchpoint put(item, block=True, timeout=None, size=None)Put item into the queue.

If the queue is currently full and block is True (the default), then wait up to timeout seconds for space tobecome available. If no timeout is specified, then wait indefinitely.

If the queue is full and block is False or a timeout occurs, then raise a QueueFull exception.

The optional size argument may be used to specify a custom size for the item. The total qsize() of thequeue is the sum of the sizes of all the items. The default size for an item is 1.

put_nowait(item, size=None)“Equivalent of put(item, False).

switchpoint get(block=True, timeout=None)Pop an item from the queue.

If the queue is not empty, an item is returned immediately. Otherwise, if block is True (the default), waitup to timeout seconds for an item to become available. If not timeout is provided, then wait indefinitely.

If the queue is empty and block is false or a timeout occurs, then raise a QueueEmpty exception.

1.3. Reference 15


get_nowait()“Equivalent of get(False).

task_done()Mark a task as done.

join()Wait until all tasks are done.

class LifoQueue(maxsize=0)A queue with LIFO behavior.

See Queue for a description of the API.


class PriorityQueue(maxsize=0)A priority queue.

Items that are added via put() are typically (priority, item) tuples. Lower values for priority indicatea higher priority.

See Queue for a description of the API.


Asynchronous function calls

Gruvi provides functionality to execute functions asynchronously outside the current flow of control. The API is afutures based interface, modeled after the concurrent.futures and asyncio packages in the Python standardlibrary.

class FutureThe state of an asynchronous function call.

A future captures the state and the future result of an asynchronous function call. Futures are not instantiateddirectly but are created by a PoolBase implementation. The pool accepts work in the form of a Pythonfunction via its submit() method. This method returns a future representing the state and the result of thefunction that will be executed asynchronously in the pool.

A future progresses through different states during its lifecyle:

•Initially the future is in the pending state.

•Once the pool has capacity to run the function, it moves to the running state. In this state, running()will return True.

•Once the function completes or raises an exception, the future moves to the done state. In this state,done() will return True.

The future and the asynchronous function it represents are two distinct entities but for brevity the terms are oftenused interchangeably, including in this manual. For example, if the asynchronous function completes it is saidthat the future has completed, and cancelling the future really means cancelling the asynchronous function it iscapturing the state of.

running()Return whether this future is running.

cancelled()Return whether this future was successfully cancelled.


https://docs.python.org/3.5/library/concurrent.futures.html#module-concurrent.futures

https://docs.python.org/3.5/library/asyncio.html#module-asyncio


done()Return whether this future is done.

cancel()Cancel the execution of the async function, if possible.

This method marks the future as done and sets the Cancelled exception.

A future that is not running can always be cancelled. However when a future is running, the ability tocancel it depends on the pool implementation. For example, a fiber pool can cancel running fibers but athread pool cannot.

Return True if the future could be cancelled, False otherwise.

switchpoint result(timeout=None)Wait for the future to complete and return its result.

If the function returned normally, its return value is returned here. If the function raised an exception, theexception is re-raised here.

switchpoint exception(timeout=None)Wait for the async function to complete and return its exception.

If the function did not raise an exception this returns None.

add_done_callback(callback, *args)Add a callback that gets called when the future completes.

The callback will be called in the context of the fiber that sets the future’s result. The callback is calledwith the positional arguments args provided to this method.

The return value is an opaque handle that can be used with remove_done_callback() to remove thecallback.

If the future has already completed, then the callback is called immediately from this method and the returnvalue will be None.

remove_done_callback(handle)Remove a callback that was added by add_done_callback().

It is not an error to remove a callback that was already removed.

class PoolBase(maxsize=None, minsize=0, name=None)Base class for pools.

A pool contains a set of workers that can execute function calls asynchronously.

The maxsize argument specifies the maximum numbers of workers that will be created in the pool. If maxsize isNone then the pool can grow without bound.

Normally the pool starts with zero workers and grows up to maxsize on demand. The minsize parameter can beused to change this behavior an make sure that there will always be at least this many workers in the pool.

The name parameter gives a name for this pool. The pool name will show up in log messages related to the pool.

maxsizeThe maximum size of this pool.

minsizeThe minimum size of this pool.

nameThe pool name.

1.3. Reference 17


submit(func, *args)Run func asynchronously.

The function is run in the pool which will run it asynchrously. The function is called with positionalargument args.

The return value is a Future that captures the state and the future result of the asynchronous functioncall.

switchpoint map(func, *iterables, **kwargs)Apply func to the elements of the sequences in iterables.

All invocations of func are run in the pool. If multiple iterables are provided, then func must take this manyarguments, and is applied with one element from each iterable. All iterables must yield the same numberof elements.

An optional timeout keyword argument may be provided to specify a timeout.

This returns a generator yielding the results.

switchpoint join()Wait until all jobs in the pool have completed.

New submissions are not blocked. This means that if you continue adding work via submit() or map()then this method might never finish.

switchpoint close()Close the pool and wait for all workers to exit.

New submissions will be blocked. Workers will exit once their current job is finished. This method willreturn after all workers have exited.

class FiberPool(*args, **kwargs)A pool that uses fiber workers.

class ThreadPool(*args, **kwargs)A pool that uses thread workers.

get_io_pool()Return the thread pool for IO tasks.

By default there is one IO thread pool per application, which is shared with all threads.

get_cpu_pool()Return the thread pool for CPU intenstive tasks.

By default there is one CPU thread pool per application, which it is shared with all threads.

blocking(func, *args, **kwargs)Run a function that uses blocking IO.

The function is run in the IO thread pool.

switchpoint wait(objects, count=None, timeout=None)Wait for one or more waitable objects.

This method waits until count elements from the sequence of waitable objects objects have become ready. Ifcount is None (the default), then wait for all objects to become ready.

What “ready” is means depends on the object type. A waitable object is a objects that implements theadd_done_callback() and remove_done_callback methods. This currently includes:

•Event - an event is ready when its internal flag is set.

•Future - a future is ready when its result is set.



•Fiber - a fiber is ready when has terminated.

•Process - a process is ready when the child has exited.

switchpoint as_completed(objects, count=None, timeout=None)Wait for one or more waitable objects, yielding them as they become ready.

This is the iterator/generator version of wait().

Addresses

Addresses in Gruvi are a location to connect to or listen on. Addresses can be local (e.g. when using a pipe), or remote(when using TCP or UDP).

The following convention is used throughput Gruvi:

• Pipe addresses are represented as str or bytes instances. On Unix these represent paths in the file system buton Windows there is no such correspondence. On Linux there is a special class of abstract socket addresses thatstart with a null byte ('\x00').

• IP addresses are represented as (node, service) tuples. The format is identical for IPv4 and IPv6, andboth protocols are supported transparently. The host component of the tuple is either DNS name or a stringformat numerical IPv4/IPv6 address, while the port component is either an integer port number or a servicename.

In addition, some APIs also support connecting to a file descriptor or to a pyuv.Stream instance.

Name Resolution

IP (node, service) tuples are resolved to numerical host addresses and port numbers using thegetaddrinfo() function. The resulting, resolved address is called a socket address, and should be distinguishedfrom a non-resolved addressed which we simply call “address”.

switchpoint getaddrinfo(node, service=0, family=0, socktype=0, protocol=0, flags=0, timeout=30)Resolve an Internet node name and service into a socket address.

The family, socktype and protocol are optional arguments that specify the address family, socket type and proto-col, respectively. The flags argument allows you to pass flags to further modify the resolution process. See thesocket.getaddrinfo() function for a detailed description of these arguments.

The return value is a list of (family, socktype, proto, canonname, sockaddr) tuples. Thefifth element (sockaddr) is the socket address. It will be a 2-tuple (addr, port) for an IPv4 address, anda 4-tuple (addr, port, flowinfo, scopeid) for an IPv6 address.

The address resolution is performed in the libuv thread pool.

Note that the distinction between addresses and socket addresses is only present for IP addresses. Pipes don’t useaddress resolution and their addresses and socket addresses are the same.

A socket address can be resolved back to an address using getnameinfo():

switchpoint getnameinfo(sockaddr, flags=0, timeout=30)Resolve a socket address sockaddr back to a (node, service) tuple.

The flags argument can be used to modify the resolution process. See the socket.getnameinfo() functionfor more information.

The address resolution is performed in the libuv thread pool.

1.3. Reference 19

https://docs.python.org/3.5/library/socket.html#socket.getaddrinfo

https://docs.python.org/3.5/library/socket.html#socket.getnameinfo


Gruvi API function always accept addresses, and address resolution is performed automatically. The only placewhere you will work with raw socket addresses is when you query the address of an existing socket, using e.g. theget_extra_info() method of a transport.

Note that the first two elements of an IP socket address are always the numerical-as-string address and the port number(for both IPv4 and IPv6). Because getaddrinfo() also accepts numerical-as-string addresses and port numbers,they also form a valid address and could be passed back to getaddrinfo() again.

Some operating systems support a special convention to specify the scope ID for an IPv6 addressing using the'host%ifname' notation where 'ifname' is the name of a network interface. Support for this is OS specificand neither libuv nor Gruvi try to present a portable interface for this.

String Addresses

Gruvi contains two utility functions to transform addresses to and from a string representation. This is useful whenlogging or when accepting addresses from the command line.

saddr(address)Return a string representation for an address.

The address paramater can be a pipe name, an IP address tuple, or a socket address.

The return value is always a str instance.

paddr(address)Parse a string representation of an address.

This function is the inverse of saddr().

Transports and protocols (low-level API)

Gruvi is built around the transport/protocol abstraction layers that are documented in PEP 3156 and implemented byasyncio.

A transport is a standard interface to a communications channel. Different types of channels implement differentinterfaces. Since Gruvi uses libuv / pyuv, its transports are mostly wrappers around the various pyuv.Handleclasses. Transport methods are always non-blocking.

A protocol is a callback based interface that is connected to a transport. The transport calls specific callbacks onthe protocol when specific events occur. The callbacks are always non-blocking, but a protocol may expose certainprotocol operations are part of their API that are switch points.

As a programmer you will use transports and protocols occasionally, but you will mostly use the higher-level interfaceprovided by Gruvi. The exception is when adding support for a new protocol.

Transports

The following transport classes are available:

exception TransportErrorA transport error.

class BaseTransportBase class for pyuv based transports. There is no public constructor.

start(protocol)Bind to protocol and start calling callbacks on it.





https://pypi.python.org/pypi/pyuv


https://pyuv.readthedocs.io/en/latest/pyuv.html#module-pyuv


get_write_buffer_size()Return the total number of bytes in the write buffer.

get_write_buffer_limits()Return the write buffer limits as a (low, high) tuple.

set_write_buffer_limits(high=None, low=None)Set the low and high watermark for the write buffer.

resume_reading()Resume calling callbacks on the protocol.

As a relaxation from the requirements in Python asyncio, this method may be called even if reading wasalready paused, and also if a close is pending. This simplifies protocol design, especially in combinationwith SSL where reading may need to be enabled by the transport itself, without knowledge of the protocol,to complete handshakes.

pause_reading()Pause calling callbacks on the protocol.

This method may be called even if reading was already paused or a close is pending. See the note inresume_reading().

close()Close the transport after all oustanding data has been written.

abort()Close the transport immediately.

get_extra_info(name, default=None)Get transport specific data.

The following information is available for all transports:

Name Description'handle' The pyuv handle that is being wrapped.

class Transport(handle, server_hostname=None, mode=’rw’)A connection oriented transport.

The handle argument is the pyuv handle for which to create the transport. It must be a pyuv.Stream instance,so either a pyuv.TCP, pyuv.Pipe or a pyuv.TTY.

The server_hostname argument specifies the host name of the remote peer, if available and applicable for thehandle.

The mode argument specifies if this is transport is read-only ('r'), write-only ('w') or read-write ('rw').






1.3. Reference 21

https://pyuv.readthedocs.io/en/latest/tcp.html#pyuv.TCP

https://pyuv.readthedocs.io/en/latest/pipe.html#pyuv.Pipe

https://pyuv.readthedocs.io/en/latest/tty.html#pyuv.TTY


write(data, handle=None)Write data to the transport.

writelines(seq)Write all elements from seq to the transport.

write_eof()Shut down the write direction of the transport.

can_write_eof()Whether this transport can close the write direction.

get_extra_info(name, default=None)Get transport specific data.

In addition to the fields from BaseTransport.get_extra_info(), the following information isalso available:

Name Description'sockname' The socket name i.e. the result of the getsockname() system call.'peername' The peer name i.e. the result of the getpeername() system call.'winsize' The terminal window size as a (cols, rows) tuple. Only available for

pyuv.TTY handles.'unix_creds' The Unix credentials of the peer as a (pid, uid, gid) tuple. Only available

for pyuv.Pipe handles on Unix.'server_hostname'The host name of the remote peer prior to address resolution, if applicable.

class DatagramTransport(handle, mode=’rw’)A datagram transport.

The handle argument is the pyuv handle for which to create the transport. It must be a pyuv.UDP instance.

The mode argument specifies if this is transport is read-only ('r'), write-only ('w') or read-write ('rw').






sendto(data, addr=None)Send a datagram containing data to addr.

The addr argument may be omitted only if the handle was bound to a default remote address.

SSL/TLS support

SSL and TLS support is available by means of a special SslTransport transport:




https://pyuv.readthedocs.io/en/latest/udp.html#pyuv.UDP


class SslTransport(handle, context, server_side, server_hostname=None,do_handshake_on_connect=True, close_on_unwrap=True)

An SSL/TLS transport.

The handle argument is the pyuv handle on top of which to layer the SSL transport. It must be a pyuv.Streaminstance, so either a pyuv.TCP, pyuv.Pipe or a pyuv.TTY.

SSL transports are always read-write, so the handle provided needs to support that.

The context argument specifies the ssl.SSLContext to use. You can usecreate_default_context() to create a new context which also works on Python 2.x where ssl.SSLContext does not exist.

The server_side argument specifies whether this is a server side or a client side transport.

The optional server_hostname argument can be used to specify the hostname you are connecting to. You mayonly specify this parameter if your Python version supports SNI.

The optional do_handshake_on_connect argument specifies whether to to start the SSL handshake immediately.If False, then the connection will be unencrypted until do_handshake() is called. The default is to start thehandshake immediately.

The optional close_on_unwrap argument specifies whether you want the ability to continue using the connectionafter you call unwrap() of when an SSL “close_notify” is received from the remote peer. The default is toclose the connection.

get_extra_info(name, default=None)Return transport specific data.

The following fields are available, in addition to the information exposed by Transport.get_extra_info().

Name Description'ssl' The internal ssl.SSLObject instance used by this transport.'sslctx' The ssl.SSLContext instance used to create the SSL object.

pause_reading()Stop reading data.

Flow control for SSL is a little bit more complicated than for a regular Transport because SSL hand-shakes can occur at any time during a connection. These handshakes require reading to be enabled, evenif the application called pause_reading() before.

The approach taken by Gruvi is that when a handshake occurs, reading is always enabled even ifpause_reading() was called before. I believe this is the best way to prevent complex read sidebuffering that could also result in read buffers of arbitrary size.

The consequence is that if you are implementing your own protocol and you want to support SSL, thenyour protocol should be able to handle a callback even if it called pause_reading() before.

resume_reading()Resume reading data.

See the note in pause_reading() for special considerations on flow control with SSL.

do_handshake()Start the SSL handshake.

This method only needs to be called if this transport was created with do_handshake_on_connect set toFalse (the default is True).

The handshake needs to be synchronized between the both endpoints, so that SSL record level data isnot incidentially interpreted as plaintext. Usually this is done by starting the handshake directly after aconnection is established, but you can also use an application level protocol.

1.3. Reference 23




https://docs.python.org/3.5/library/ssl.html#ssl.SSLContext




unwrap()Remove the security layer.

Use this method only if you want to send plaintext data on the connection after the security layer has beenremoved. In all other cases, use close().

If the unwrap is initiated by us, then any data sent after it will be buffered until the correspondingclose_notify response is received from our peer.

If the unwrap is initiated by the remote peer, then this method will acknowledge it. You need an applicationlevel protocol to determine when to do this because the receipt of a close_notify is not communicated tothe application.

close()Cleanly shut down the SSL protocol and close the transport.

create_default_context(purpose=None, **kwargs)Create a new SSL context in the most secure way available on the current Python version. See ssl.create_default_context().

Protocols

A Protocols is a collection of named callbacks. A protocol is are attached to a transport, and its callbacks are calledby the transport when certain events happen.

The following protocol classes are available:

exception ProtocolErrorA protocol error.

class BaseProtocol(timeout=None)Base class for all protocols.

The timeout argument specifies a default timeout for various protocol operations.

connection_made(transport)Called when a connection is made.

connection_lost(exc)Called when a connection is lost.

pause_writing()Called when the write buffer in the transport has exceeded the high water mark. The protocol should stopwriting new data.

resume_writing()Called when the write buffer in the transport has fallen below the low water mark. The protocol can startwriting data again.

class Protocol(timeout=None)Base class for connection oriented protocols.


data_received(data)Called when a new chunk of data is received.

eof_received()Called when an EOF is received.

class DatagramProtocol(timeout=None)Base classs for datagram oriented protocols.


https://docs.python.org/3.5/library/ssl.html#ssl.create_default_context

https://docs.python.org/3.5/library/ssl.html#ssl.create_default_context



datagram_received(data, addr)Called when a new datagram is received.

error_received(exc)Called when an error has occurred.

The following class does not exist in PEP 3156 but is a useful base class for most protocols:

class MessageProtocol(message_handler=None, timeout=None)Base class for message oriented protocols.

Creating transports and protocols

Transports and protocols operate in pairs; there is always exactly one protocol for each transport. A new trans-port/protocol pair can be created using the factory functions below:

switchpoint create_connection(protocol_factory, address, ssl=False, server_hostname=None, lo-cal_address=None, family=0, flags=0, ipc=False, timeout=None,mode=’rw’)

Create a new client connection.

This method creates a new pyuv.Handle, connects it to address, and then waits for the connection to beestablished. When the connection is established, the handle is wrapped in a transport, and a new protocolinstance is created by calling protocol_factory. The protocol is then connected to the transport by calling thetransport’s start() method which in turn calls connection_made() on the protocol. Finally the resultsare returned as a (transport, protocol) tuple.

The address may be either be a string, a (host, port) tuple, a pyuv.Stream handle or a file descriptor:

•If the address is a string, this method connects to a named pipe using a pyuv.Pipe handle. The addressspecifies the pipe name.

•If the address is a tuple, this method connects to a TCP/IP service using a pyuv.TCP handle. The firstelement of the tuple specifies the IP address or DNS name, and the second element specifies the portnumber or service name.

•If the address is a pyuv.Stream instance, it must be an already connected stream.

•If the address is a file descriptor, then it is attached to a pyuv.TTY stream if os.isatty() returns true,or to a pyuv.Pipe instance otherwise.

The ssl parameter indicates whether an SSL/TLS connection is desired. If so then an ssl.SSLContextinstance is used to wrap the connection using the ssl module’s asynchronous SSL support. The context iscreated as follows. If ssl is an SSLContext instance, it is used directly. If it is a function, it is called with theconnection handle as an argument and it must return a context . If it is True then a default context is createdusing gruvi.create_default_context(). To disable SSL (the default), pass False. If SSL is active,the return transport will be an SslTransport instance, otherwise it will be a Transport instance.

The server_hostname parameter is only relevant for SSL connections, and specifies the server hostname to usewith SNI (Server Name Indication). If no server hostname is provided, the hostname specified in address isused, if available.

The local_address keyword argument is relevant only for TCP transports. If provided, it specifies the localaddress to bind to.

The family and flags keyword arguments are used to customize address resolution for TCP handles as describedin socket.getaddrinfo().

1.3. Reference 25






https://docs.python.org/3.5/library/os.html#os.isatty



https://docs.python.org/3.5/library/ssl.html#module-ssl




The mode parameter specifies if the transport should be put in read-only ('r'), write-only ('w') or read-write('rw') mode. For TTY transports, the mode must be either read-only or write-only. For all other transport themode should usually be read-write.

switchpoint create_server(protocol_factory, address=None, ssl=False, family=0, flags=0, ipc=False,backlog=128)

Create a new network server.

This creates one or more pyuv.Handle instances bound to address, puts them in listen mode and startsaccepting new connections. For each accepted connection, a new transport is created which is connected to anew protocol instance obtained by calling protocol_factory.

The address argument may be either be a string, a (host, port) tuple, or a pyuv.Stream handle:

•If the address is a string, this method creates a new pyuv.Pipe instance and binds it to address.

•If the address is a tuple, this method creates one or more pyuv.TCP handles. The first element of thetuple specifies the IP address or DNS name, and the second element specifies the port number or servicename. A transport is created for each resolved address.

•If the address is a pyuv.Stream handle, it must already be bound to an address.

The ssl parameter indicates whether SSL should be used for accepted connections. Seecreate_connection() for a description.

The family and flags keyword arguments are used to customize address resolution for TCP handles as describedin socket.getaddrinfo().

The ipc parameter indicates whether this server will accept new connections via file descriptor passing. Thisworks for pyuv.Pipe handles only, and the user is required to call Server.accept_connection() when-ever a new connection is pending.

The backlog parameter specifies the listen backlog i.e the maximum number of not yet accepted active opens toqueue. To disable listening for new connections (useful when ipc was set), set the backlog to None.

The return value is a Server instance.

Endpoints

Endpoints wrap transports and protocols for client and server side connections, and provide a more object-orientedway of working with them.

class Endpoint(protocol_factory, timeout=None)A communications endpoint.

The protocol_factory argument constructs a new protocol instance. Normally you would pass a Protocolsubclass.

The timeout argument specifies a timeout for various network operations.

timeoutThe network timeout.

close()Close the endpoint.

class Client(protocol_factory, timeout=None)A client endpoint.

transportReturn the transport, or None if not connected.







protocolReturn the protocol, or None if not connected.

switchpoint connect(address, **kwargs)Connect to address and wait for the connection to be established.

See create_connection() for a description of address and the supported keyword arguments.

switchpoint close()Close the connection.

class Server(protocol_factory, timeout=None)A server endpoint.

addressesThe addresses this server is listening on.

connectionsAn iterator yielding the (transport, protocol) pairs for each connection.

accept_connection(handle, ssl=False)Accept a new connection on handle. This method needs to be called when a connection was passed viafile descriptor passing.

handle_connection(client, ssl)Handle a new connection with handle client.

This method exists so that it can be overridden in subclass. It is not intended to be called directly.

connection_made(transport, protocol)Called when a new connection is made.

connection_lost(transport, protocol, exc=None)Called when a connection is lost.

switchpoint listen(address, ssl=False, family=0, flags=0, ipc=False, backlog=128)Create a new transport, bind it to address, and start listening for new connections.

See create_server() for a description of address and the supported keyword arguments.

switchpoint close()Close the listening sockets and all accepted connections.

switchpoint run()Run the event loop and start serving requests.

This method stops serving when a CTRL-C/SIGINT is received. It is useful for top-level scripts that runonly one server instance. In more complicated application you would call Hub.switch() directly, orwait on some kind of “request to shutdown” event.

Streams (generic high-level API)

Streams provide a generic, high-level API to work with connection-oriented transports using file-like, blocking andflow-controlled API.

Utility classes

class StreamBuffer(transport=None, timeout=None)A stream buffer.

This is a utility class that is used to by Stream to implement the read side buffering.

1.3. Reference 27


get_buffer_size()Return the size of the buffer.

set_buffer_limits(high=None, low=None)Set the low and high watermarks for the read buffer.

feed(data)Add data to the buffer.

feed_eof()Set the EOF condition.

feed_error(exc)Set the error condition to exc.

class Stream(transport, mode=’rw’, autoclose=False, timeout=None)A byte stream.

This class implements buffered, flow controlled and blocking read/write access on top of a transport.

A stream works with bytes instances. To create a stream that works with unicode strings, you can wrap it witha io.TextIOWrapper.

The transport argument specifies the transport on top of which to create the stream.

The mode argument can be 'r' for a read-only stream, 'w' for a write-only stream, or 'rw' for a read-writestream.

The autoclose argument controls whether the underlying transport will be closed in the close() method. Becareful with this as the close method is called from the io.BufferedIOBase destructor, which may lead tounexpected closing of the transport when the stream goes out of scope.

transportReturn the transport wrapped by this stream.

wrap(encoding, **textio_args)Return a io.TextIOWrapper that wraps the stream.

The wrapper provides text IO on top of the byte stream, using the specified encoding. The textio_args key-word arguments are additional keyword arguments passed to the TextIOWrapper constructor. Unlessanother buffering scheme is specified, the write_through option is enabled.

readable()Return whether the stream is readable.

writable()Return whether the stream is writable.

closedReturn whether the stream is closed.

bufferReturn the StreamBuffer for this stream.

switchpoint read(size=-1)Read up to size bytes.

This function reads from the buffer multiple times until the requested number of bytes can be satisfied.This means that this function may block to wait for more data, even if some data is available. The onlytime a short read is returned, is on EOF or error.

If size is not specified or negative, read until EOF.


https://docs.python.org/3.5/library/io.html#io.TextIOWrapper

https://docs.python.org/3.5/library/io.html#io.BufferedIOBase




switchpoint read1(size=-1)Read up to size bytes.

This function reads from the buffer only once. It is useful in case you need to read a large input, and wantto do so efficiently. If size is big enough, then this method will return the chunks passed into the memorybuffer verbatim without any copying or slicing.

switchpoint readline(limit=-1, delim=b’\n’)Read a single line.

If EOF is reached before a full line can be read, a partial line is returned. If limit is specified, at most thismany bytes will be read.

switchpoint readlines(hint=-1)Read lines until EOF, and return them as a list.

If hint is specified, then stop reading lines as soon as the total size of all lines exceeds hint.

switchpoint write(data)Write data to the transport.

This method will block if the transport’s write buffer is at capacity.

switchpoint writelines(seq)Write the elements of the sequence seq to the transport.


switchpoint write_eof()Close the write direction of the transport.


switchpoint close()Close the stream.

If autoclose was passed to the constructor then the underlying transport will be closed as well.

Stream Protocol

class StreamProtocol(timeout=None)Byte stream protocol.

The timeout argument specifies a default timeout for protocol operations.

streamA Stream instance, providing blocking, flow controlled read and write access to the underlying transport.

Stream Client and Server

class StreamClient(timeout=None)A stream client.

streamAn alias for self.protocol.reader

switchpoint read(size=-1)A shorthand for self.stream.read().

switchpoint read1(size=-1)A shorthand for self.stream.read1().

1.3. Reference 29


switchpoint readline(limit=-1, delim=b’\n’)A shorthand for self.stream.readline().

switchpoint readlines(hint=-1)A shorthand for self.stream.readlines().

switchpoint write(data)A shorthand for self.stream.write().

switchpoint write_eof()A shorthand for self.stream.write_eof().

switchpoint writelines(seq)A shorthand for self.stream.writelines().

class StreamServer(stream_handler, timeout=None)A stream server.

The stream_handler argument is a handler function to handle client connections. The handler will be called asstream_handler(stream, transport, protocol). The handler for each connection will run in aseparate fiber so it can use blocking I/O on the stream. When the handler returns, the stream is closed.

See Example 2: echo server, using a StreamServer for an example.

Working with Processes

gruvi.PIPE = ...

gruvi.DEVNULL = ...

class Process(encoding=None, timeout=None)A child process.

This class allows you to start up a child process, communicate with it and control it. The API is modeled afterthe subprocess.Popen class.

The encoding argument specifies the encoding to use for communications with the child. If an encoding isspecified, the standard input and output handles provide text-based IO, otherwise they provide bytes-based IO.

stdinThe child’s standard input, or None.

stdoutThe child’s standard output, or None.

stderrThe child’s standard error, or None.

returncodeThe child’s exit status, or None if it has not exited yet.

On Unix, if the child was terminated by a signal, return -SIGNUM.

pidThe child’s process ID, or None if there is no child.

spawn(args, executable=None, stdin=None, stdout=None, stderr=None, shell=False, cwd=None,env=None, flags=0, extra_handles=None)

Spawn a new child process.

The executable to spawn and its arguments are determined by args, executable and shell.


https://docs.python.org/3.5/library/subprocess.html#subprocess.Popen


When shell is set to False (the default), args is normally a sequence and it contains both the program toexecute (at index 0), and its arguments.

When shell is set to True, then args is normally a string and it indicates the command to execute throughthe shell.

The executable argument can be used to override the executable to execute. If shell is False, it overridesargs[0]. This is sometimes used on Unix to implement “fat” executables that behave differently basedon argv[0]. If shell is True, it overrides the shell to use. The default shell is '/bin/sh' on Unix, andthe value of $COMSPEC (or 'cmd.exe' if it is unset) on Windows.

The stdin, stdout and stderr arguments specify how to handle standard input, output, and error, respectively.If set to None, then the child will inherit our respective stdio handle. If set to the special constant PIPEthen a pipe is created. The pipe will be connected to a gruvi.StreamProtocol which you can use toread or write from it. The stream protocol instance is available under either stdin, stdout or stderr.All 3 stdio arguments can also be a file descriptor, a file-like object, or a pyuv Stream instance.

The extra_handles specifies any extra handles to pass to the client. It must be a sequence where eachelement is either a file descriptor, a file-like objects, or a pyuv.Stream instance. The position in thesequence determines the file descriptor in the client. The first position corresponds to FD 3, the second to4, etc. This places these file descriptors directly after the stdio handles.

The cwd argument specifies the directory to change to before executing the child. If not provided, thecurrent directory is used.

The env argument specifies the environment to use when executing the child. If provided, it must be adictionary. By default, the current environment is used.

The flags argument can be used to specify optional libuv uv_process_flags. The only relevant flagsare pyuv.UV_PROCESS_DETACHED and pyuv.UV_PROCESS_WINDOWS_HIDE. Both are Windowsspecific and are silently ignored on Unix.

child_exited(exit_status, term_signal)Callback that is called when the child has exited.

switchpoint close()Close the process and frees its associated resources.

This method waits for the resources to be freed by the event loop.

send_signal(signum)Send the signal signum to the child.

On Windows, SIGTERM, SIGKILL and SIGINT are emulated using TerminateProcess(). This will causethe child to exit unconditionally with status 1. No other signals can be sent on Windows.

terminate()Terminate the child process.

It is not an error to call this method when the child has already exited.

switchpoint wait(timeout=-1)Wait for the child to exit.

Wait for at most timeout seconds, or indefinitely if timeout is None. Return the value of the returncodeattribute.

switchpoint communicate(input=None, timeout=-1)Communicate with the child and return its output.

If input is provided, it is sent to the client. Concurrent with sending the input, the child’s standard outputand standard error are read, until the child exits.

1.3. Reference 31


The return value is a tuple (stdout_data, stderr_data) containing the data read from standardoutput and standard error.

Included Protocols (specific high-level API)

gruvi.http – HTTP Client and Server

The gruvi.http module implements a HTTP client and server.

The client and server are relatively complete implementations of the HTTP protocol. Some of the supported featuresare keepalive, pipelining, chunked transfers and trailers.

This implementation supports both HTTP/1.0 and HTTP/1.1. The default for the client is 1.1, and the server willrespond with the same version as the client.

Connections are kept alive by default. This means that you need to make sure you close connections when they are nolonger needed, by calling the appropriate close() method.

It is important to clarify how the API exposed by this module uses text and binary data. Data that is read from orwritten to the HTTP header, such as the version string, method, and headers, are text strings (str on Python 3, str orunicode on Python 2). However, if the string type is unicode aware (str on Python 3, unicode on Python 2), youmust make sure that it only contains code points that are part of ISO-8859-1, which is the default encoding specifiedin RFC 2606. Data that is read from or written to HTTP bodies is always binary. This is done in alignment with theWSGI spec that requires this.

This module provides a number of APIs. Client-side there is one:

• A gruvi.Client based API. You will use connect() to connect to a server, and then use request()and getresponse() to interact with it.

The following server-side APIs are available:

• A gruvi.Server based API. Incoming HTTP messages are passed to a message handler that needs to takecare of all aspects of HTTP other than parsing.

• A WSGI API, as described in PEP 333.

The server-side API is selected through the adapter argument to HttpServer constructor. The default adapter isWsgiAdapter, which implements the WSGI protocol. To use the raw server interface, pass the identity function(lambda x: x).

REQUESTConstant indicating a HTTP request.

RESPONSEConstant indicating a HTTP response.

exception HttpErrorException that is raised in case of HTTP protocol errors.

class ParsedUrlA namedtuple() with the following fields: scheme, host, port, path, query, fragment anduserinfo.

In addition to the tuple fields the following properties are defined:

addrAddress tuple that can be used with create_connection().

sslWhether the scheme requires SSL/TLS.


https://tools.ietf.org/html/rfc2606.html


https://docs.python.org/3.5/library/collections.html#collections.namedtuple


targetThe “target” i.e. local part of the URL, consisting of the path and query.

parse_url(url, default_scheme=’http’, is_connect=False)Parse an URL and return its components.

The default_scheme argument specifies the scheme in case URL is an otherwise valid absolute URL but with amissing scheme.

The is_connect argument must be set to True if the URL was requested with the HTTP CONNECT method.These URLs have a different form and need to be parsed differently.

The result is a ParsedUrl containing the URL components.

class HttpMessageHTTP message.

Instances of this class are returned by HttpClient.getresponse() and passed as an argument toHttpServer message handlers.

message_typeThe message type, either REQUEST or RESPONSE.

versionThe HTTP version as a string, either '1.0' or '1.1'.

status_codeThe HTTP status code as an integer. Only for response messages.

methodThe HTTP method as a string. Only for request messages.

urlThe URL as a string. Only for request messages.

parsed_urlThe parsed URL as a ParsedUrl instance.

headersThe headers as a list of (name, value) tuples.

charsetThe character set as parsed from the “Content-Type” header, if available.

bodyThe message body, as a Stream instance.

get_header(headers, name, default=None)A shorthand for get_header(headers, ...).

class HttpRequest(protocol)HTTP client request.

Usually you do not instantiate this class directly, but use the instance returned by HttpProtocol.request(). You can however start new request yourself by instantiating this class and passing it a protocolinstance.

switchpoint start_request(method, url, headers=None, bodylen=None)Start a new HTTP request.

The optional headers argument contains the headers to send. It must be a sequence of (name, value)tuples.

1.3. Reference 33


The optional bodylen parameter is a hint that specifies the length of the body that will follow. A length of-1 indicates no body, 0 means an empty body, and a positive number indicates the body size in bytes. Thisparameter helps determine whether to use the chunked transfer encoding. Normally when the body size isknown chunked encoding is not used.

switchpoint write(buf)Write buf to the request body.

switchpoint end_request()End the request body.

class WsgiAdapter(application)WSGI Adapter

This class adapts the WSGI callable application so that instances of this class can be used as a message handlerin HttpProtocol.

class HttpProtocol(handler=None, server_side=False, server_name=None, version=None, time-out=None)

HTTP protocol implementation.

The handler argument specifies a message handler to handle incoming HTTP requests. It must be a callablewith the signature handler(message, transport, protocol).

The server_side argument specifies whether this is a client or server side protocol.

For client-side protocols, the server_name argument specifies the remote server name, which is used as theHost: header. It is normally not required to set this as it is taken from the unresolved hostname passed toconnect().

For server-side protocols, server_name will be used as the value for the SERVER_NAME WSGI environmentvariable.

default_version = ‘1.1’Default HTTP version.

max_header_size = 65536Max header size. The parser keeps the header in memory during parsing.

max_buffer_size = 65536Max number of body bytes to buffer. Bodies larger than this will cause the transport to be paused until thebuffer is below the threshold again.

max_pipeline_size = 10Max number of pipelined requests to keep before pausing the transport.

writerA Stream instance for writing directly to the underlying transport.

switchpoint request(method, url, headers=None, body=None)Make a new HTTP request.

The method argument is the HTTP method as a string, for example 'GET' or 'POST'. The url argumentspecifies the URL.

The optional headers argument specifies extra HTTP headers to use in the request. It must be a sequenceof (name, value) tuples.

The optional body argument may be used to include a body in the request. It must be a bytes instance,a file-like object opened in binary mode, or an iterable producing bytes instances. To send potentiallylarge bodies, use the file or iterator interfaces. This has the benefit that only a single chunk is kept inmemory at a time.



The response to the request can be obtained by calling the getresponse() method. You may makemultiple requests before reading a response. For every request that you make however, you must callgetresponse() exactly once. The remote HTTP implementation will send by the responses in thesame order as the requests.

This method will use the “chunked” transfer encoding if here is a body and the body size is unknown aheadof time. This happens when the file or interator interface is used in the abence of a “Content-Length”header.

switchpoint getresponse()Wait for and return a HTTP response.

The return value will be a HttpMessage. When this method returns only the response header has beenread. The response body can be read using read() and similar methods on the message body .

Note that if you use persistent connections (the default), it is required that you read the entire body of eachresponse. If you don’t then deadlocks may occur.

class HttpClient(version=None, timeout=None)HTTP client.

The optional version argument specifies the HTTP version to use. The default is HttpProtocol.default_version.

The optional timeout argument specifies the timeout for various network and protocol operations.


switchpoint getresponse()A shorthand for self.protocol.getresponse().

switchpoint request(method, url, headers=None, body=None)A shorthand for self.protocol.request().

class HttpServer(application, server_name=None, adapter=None, timeout=None)HTTP server.

The application argument is the web application to expose on this server. The application is wrapped in adapterto create a message handler as required by HttpProtocol. By default the adapter in default_adapteris used.

The optional server_name argument specifies the server name. The server name is made available to WSGIapplications as the $SERVER_NAME environment variable.

The optional timeout argument specifies the timeout for various network and protocol operations.

default_adapterThe default adapter to use.

alias of WsgiAdapter

Example

# HTTP client and server example.

from gruvi.http import HttpClient, HttpServer

def handler(env, start_response):headers = [('Content-Type', 'text/plain; charset=UTF-8')]status = '200 OK'

1.3. Reference 35


body = 'pong'start_response(status, headers)yield body.encode('utf-8')

server = HttpServer(handler)server.listen(('localhost', 0))addr = server.addresses[0]

client = HttpClient()client.connect(addr)client.request('GET', '/ping')

resp = client.getresponse()assert resp.status_code == 200

body = resp.body.read()print('result = {}'.format(body.decode(resp.charset)))

gruvi.jsonrpc – JSON-RPC Client and Server

The gruvi.jsonrpc module implements a JSON-RPC client and server.

There are two version of JSON-RPC: version 1.0 and version 2.0. While these versions are similar, they are notmutually compatible. The classes exposed by this module implement both versions. The default version to use isstored in JsonRpcProtocol.default_version, and most constructors take an argument that can be used tooverride this.

The “batch” feature of version 2.0 is not supported. It more relevant for JSON-RPC over HTTP rather for that clientsand servers that operate directly on top of a connection.

The two main classes in this module are JsonRpcClient and JsonRpcServer. The difference is only whoinitiates the connection at the transport level. The JSON-RPC protocol itself does not distinguish between clients andservers.

Both the client and the server can receive and respond to incoming messages. These may be method calls (morecommon for servers), or notifications (common for both client and servers). These incoming messages are handled bya message handler, which is mandatory for a server and optional for a client. Note that for simple blocking methodcallls from client to a server no message handler is needed.

Message handlers run in a separate dispatcher fiber, one per connection. This means that a client will have at mostone dispatcher fiber, while a server will have exactly one fiber per connection. Because they run in a separate fiber,message handlers can call into switchpoints themselves.

REQUESTConstant indicating a JSON-RPC request message.

RESPONSEConstant indicating a JSON-RPC response message.

errorcode {...}Dictionary mapping error codes to error names.

strerror(code)Return an error message for error code.

The error code is the error.code field of a JSON-RPC response message. The JSON-RPC version 2 speccontains a list of standard error codes.



exception JsonRpcErrorException that is raised in case of JSON-RPC errors.

errorThe error field from the JSON-RPC response, if available.

class JsonRpcVersion(name)Class that encapsulates the differences between JSON-RPC vesion 1.0 and 2.0.

nameReturn the JSON-RPC version as a string, e.g. ‘1.0’.

next_id()Return a unique message ID.

static create(version)Return a new instance for version, which can be either ‘1.0’ or ‘2.0’.

check_message(message)Check message and return the message type. Raise a ValueError in case of an invalid message.

create_request(method, args=None, notification=False)Create a new request message for method with arguments args.

The optional notification argument, if nonzero, creates a notification and no response to this message isexpected.

create_response(request, result=None, error=None)Create a new response message, as a response to request.

A successful response is created when the optional error argument is not provided. In this case resultspecifies the result, which may be any value including None. If error is provided, an error response iscreated, and error should be a dict as specified by the JSON-RPC specifications.

The request argument may be explicitly set to None, in which case this is an error response that is notspecific to any one request.

class JsonRpcProtocol(handler=None, version=None, timeout=None)A JSON-RPC client and server Protocol implementation.

default_timeout = 30Class level attribute that specifies the default timeout.

default_version = ‘2.0’Class level attribute that specifies the default JSON-RPC version.

versionReturn the JsonRpcVersion instance for this protocol.

switchpoint send_message(message)Send a raw JSON-RPC message.

The message argument must be a dictionary containing a valid JSON-RPC message according to the ver-sion passed into the constructor.

switchpoint call_method(method, *args)Call a JSON-RPC method and wait for its result.

The method is called with positional arguments args.

On success, the result field from the JSON-RPC response is returned. On error, a JsonRpcError israised, which you can use to access the error field of the JSON-RPC response.

1.3. Reference 37


switchpoint send_notification(method, *args)Send a JSON-RPC notification.

The notification method is sent with positional arguments args.

switchpoint send_response(request, result=None, error=None)Respond to a JSON-RPC method call.

This is a response to the message in request. If error is not provided, then this is a succesful response, andthe value in result, which may be None, is passed back to the client. if error is provided and not Nonethen an error is sent back. In this case error must be a dictionary as specified by the JSON-RPC spec.

class JsonRpcClient(handler=None, version=None, timeout=None)A JSON-RPC Client.

The handler argument specifies an optional JSON-RPC message handler. You need to supply a message handlerif you want to listen to notifications or you want to respond to server-to-client method calls. If provided, themessage handler it must be a callable with signature handler(message, transport, protocol).

The version and timeout argument can be used to override the default protocol version and timeout, respectively.


switchpoint call_method(method, *args)A shorthand for self.protocol.call_method().

switchpoint send_message(message)A shorthand for self.protocol.send_message().

switchpoint send_notification(method, *args)A shorthand for self.protocol.send_notification().

class JsonRpcServer(handler, version=None, timeout=None)A JSON-RPC Server.

The handler argument specifies the JSON-RPC message handler. It must be a callable with signaturehandler(message, transport, protocol). The message handler is called in a separate dispatcherfiber (one per connection).

The version and timeout argument can be used to override the default protocol version and timeout, respectively.

Example

# Ping-pong between JSON-RPC client and server.

from gruvi.jsonrpc import JsonRpcClient, JsonRpcServer

def handler(message, transport, protocol):method = message.get('method')if method == 'ping':

protocol.send_response(message, 'pong')

server = JsonRpcServer(handler)server.listen(('localhost', 0))addr = server.addresses[0]

client = JsonRpcClient()client.connect(addr)



result = client.call_method('ping')

print('result = {}'.format(result))

gruvi.dbus – D-BUS Client and Server

The gruvi.dbus module implements a D-BUS client and server.

The implementation uses parts of the txdbus project. A cut down copy of txdbus, containing only those parts neededby Gruvi, is available as gruvi.txdbus. You need this if you are providing a message handler (see below).

Both a client and a server/bus-side implementation are provided. The bus-side implementation is very bare bones andapart from the “Hello” message it does not implement any of the “org.freedestkop.DBus” interface. It also does notimplement any message routing. The server side is provided mostly for testing purposes (but it could serve as the basisfor a real D-BUS server).

The client side of a D-BUS connection is implemented by DbusClient and the server/bus-side by DbusServer.Both implement a procedural interface. Messages can be send using e.g. DbusClient.send_message() orDbusClient.call_method(). An object-oriented interface that represents D-BUS objects as Python objects,like the one txdbus provides, is currently not available. The procedural interface can be used as a basis for your ownobject-oriented interface though.

To receive notifications or to respond to method calls, you need to provide a message handler to the client or theserver constructor. The signature of the message handler is: message_handler(message, protocol).Here, the message argument is an instance of gruvi.txdbus.DbusMessages, and the protocol will be theDbusProtocol instance for the current connection.

Message handlers runs in their own fiber, which allows them to call into switchpoints. There is one fiber for everyconnection.

Usage example:

client = gruvi.DbusClient()client.connect('session')result = client.call_method('org.freedesktop.DBus', '/org/freedesktop/DBus',

'org.freedesktop.DBus', 'ListNames')for name in result[0]:

print('Name: {}'.format(name))

exception DbusErrorException that is raised in case of D-BUS protocol errors.

exception DbusMethodCallError(method, reply)Exception that is raised when a error reply is received for a D-BUS method call.

class DbusProtocol(message_handler=None, server_side=False, server_guid=None, timeout=None)D-BUS Protocol.

switchpoint get_unique_name()Return the unique name of the D-BUS connection.

switchpoint send_message(message)Send a D-BUS message.

The message argument must be gruvi.txdbus.DbusMessage instance.

switchpoint call_method(service, path, interface, method, signature=None, args=None,no_reply=False, auto_start=False, timeout=-1)

Call a D-BUS method and wait for its reply.

1.3. Reference 39

https://github.com/cocagne/txdbus


This method calls the D-BUS method with name method that resides on the object at bus address service,at path path, on interface interface.

The signature and args are optional arguments that can be used to add parameters to the method call. Thesignature is a D-BUS signature string, while args must be a sequence of python types that can be convertedinto the types specified by the signature. See the D-BUS specification for a reference on signature strings.

The flags no_reply and auto_start control the NO_REPLY_EXPECTED and NO_AUTO_START flags onthe D-BUS message.

The return value is the result of the D-BUS method call. This will be a possibly empty sequence of values.

class DbusClient(message_handler=None, timeout=30)A D-BUS client.

The message_handler argument specifies an optional message handler.

The optional timeout argument specifies a default timeout for protocol operations in seconds.

switchpoint connect(address=’session’)Connect to address and wait until the connection is established.

The address argument must be a D-BUS server address, in the format described in the D-BUS specification.It may also be one of the special addresses 'session' or 'system', to connect to the D-BUS sessionand system bus, respectively.


switchpoint call_method(service, path, interface, method, signature=None, args=None,no_reply=False, auto_start=False, timeout=-1)

A shorthand for self.protocol.call_method().

switchpoint get_unique_name()A shorthand for self.protocol.get_unique_name().

switchpoint send_message(message)A shorthand for self.protocol.send_message().

class DbusServer(message_handler, timeout=30)A D-BUS server.

The message_handler argument specifies the message handler.

The optional timeout argument specifies a default timeout for protocol operations in seconds.

switchpoint listen(address=’session’)Start listening on address for new connection.

The address argument must be a D-BUS server address, in the format described in the D-BUS specification.It may also be one of the special addresses 'session' or 'system', to connect to the D-BUS sessionand system bus, respectively.

Examples

Example 1: curl like URL downloader

1 # Gruvi example program: a cURL like URL downloader2

3 import sys


http://dbus.freedesktop.org/doc/dbus-specification.html


4 import argparse5 from gruvi.http import HttpClient, parse_url6

7 parser = argparse.ArgumentParser()8 parser.add_argument('url')9 args = parser.parse_args()

10

11 url = parse_url(args.url)12

13 client = HttpClient()14 client.connect(url.addr, ssl=url.ssl)15 client.request('GET', url.target)16

17 resp = client.getresponse()18 if not 200 <= resp.status_code <= 399:19 sys.stderr.write('Error: got status {}\n'.format(resp.status_code))20 sys.exit(1)21

22 stdout = getattr(sys.stdout, 'buffer', sys.stdout)23

24 while True:25 buf = resp.body.read(4096)26 if not buf:27 break28 stdout.write(buf)

Example 2: echo server, using a StreamServer

1 # Gruvi example program: echo server, using StreamServer2

3 import gruvi4

5 def echo_handler(stream, transport, protocol):6 peer = transport.get_extra_info('peername')7 print('New connection from {0}'.format(gruvi.saddr(peer)))8 while True:9 buf = stream.read1()

10 if not buf:11 break12 stream.write(buf)13 print('Connection lost')14

15 server = gruvi.StreamServer(echo_handler)16 server.listen(('localhost', 0))17 for addr in server.addresses:18 print('Listen on {0}'.format(gruvi.saddr(addr)))19

20 server.run()

Example 3: echo server, using a custom Protocol

1 # Gruvi example program: an echo server, using a Protocol2

3 import gruvi

1.4. Examples 41


4

5 class EchoProtocol(gruvi.Protocol):6

7 def connection_made(self, transport):8 self._transport = transport9 peer = transport.get_extra_info('peername')

10 print('New connection from {0}'.format(gruvi.saddr(peer)))11

12 def data_received(self, data):13 self._transport.write(data)14

15 def eof_received(self):16 print('Connection lost')17

18 server = gruvi.create_server(EchoProtocol, ('localhost', 0))19 for addr in server.addresses:20 print('Listen on {0}'.format(gruvi.saddr(addr)))21

22 server.run()

Example 4: netcat client

1 # Gruvi example program: a nc(1) like client, using StreamClient2

3 import sys4 import argparse5 import gruvi6

7 parser = argparse.ArgumentParser()8 parser.add_argument('hostname')9 parser.add_argument('port', type=int)

10 parser.add_argument('--ssl', action='store_true')11 args = parser.parse_args()12

13 remote = gruvi.StreamClient()14 remote.connect((args.hostname, args.port), ssl=args.ssl)15

16 stdin = gruvi.StreamClient()17 stdin.connect(sys.stdin.fileno(), mode='r')18 stdout = gruvi.StreamClient()19 stdout.connect(sys.stdout.fileno(), mode='w')20

21 done = gruvi.Event()22

23 def stdin_reader():24 while True:25 buf = stdin.read1()26 if not buf:27 print('Got EOF on stdin')28 break29 remote.write(buf)30 done.set()31

32 def remote_reader():33 while True:34 buf = remote.read1()



35 if not buf:36 print('Got EOF from remote')37 break38 stdout.write(buf)39 done.set()40

41 gruvi.spawn(stdin_reader)42 gruvi.spawn(remote_reader)43

44 done.wait()

Example 5: fortune web service

1 # Gruvi example program: a "fortune" web service2

3 import locale4 import gruvi5 import contextlib6

7 def fortune_app(environ, start_response):8 print('New connection from {0}'.format(environ['REMOTE_ADDR']))9 proc = gruvi.Process(encoding=locale.getpreferredencoding())

10 proc.spawn('fortune', stdout=gruvi.PIPE)11 with contextlib.closing(proc):12 fortune = proc.stdout.read()13 proc.wait()14 start_response('200 OK', [('Content-Type', 'text/plain; charset=utf-8')])15 return [fortune.encode('utf-8')]16

17 server = gruvi.HttpServer(fortune_app)18 server.listen(('localhost', 0))19 for addr in server.addresses:20 print('Listen on {0}'.format(gruvi.saddr(addr)))21

22 server.run()

1.4. Examples 43

CHAPTER 2

Project files

README

Improved ergonomics for Python programmers wanting to use asynchronous IO.

Gruvi is an asynchronous IO library for Python. It focuses on the following desirable properties:

• Simple. Async IO code should look just like normal code, with simple, sequential control flow and regularfunctions.

• Efficient. An IO library should have a very low memory and CPU overhead, so that it can scale to small systemsor to many concurrent connections.

• Powerful. Common protocols like SSL/TLS and HTTP should be part of every IO library.

Gruvi uses libuv (via pyuv) as the underlying high-performance event-based I/O layer, and coroutines based on fibersto create a traditional sequential programming model on top of the libuv completion/callback model.

Gruvi is similar in concept existing async IO frameworks like asyncio, gevent, and eventlet. For a comparison, seeRationale.

Features

Gruvi has the following features:

• Excellent platform support (mostly thanks to libuv). Linux, Mac OSX and Windows are all first-class citizens.

• PEP-3156 compatible transport/protocol interface.

• A traditional, sequential programming model based on green threads, where there is no distinction betweenasynchronous and normal functions.

• Great SSL/TLS support, also on Windows. The asynchronous SSL support in the Python standard library camefrom Gruvi before it was included into the stdlib.

• Small core and focus on low memory usage and fast performance. This makes Gruvi very suitable for mobileapplications and embedded web servers.

45

https://github.com/libuv/libuv

http://pyuv.readthedocs.org/en/latest

http://python-fibers.readthedocs.org/en/latest

https://github.com/libuv/libuv


http://gevent.org/


http://gruvi.readthedocs.org/en/latest/rationale.html

https://docs.python.org/3/library/ssl.html#ssl.SSLObject


• A full suite of synchronization primitives including locks, conditions and queues.

• Thread and fiber pools with a concurrent.futures interface.

• Batteries includes: built-in client/server support for HTTP, JSON-RPC and D-BUS.

• Support for Python 2.7.x and 3.3+.

Example

An simple echo server, using a StreamServer:

import gruvi

def echo_handler(stream, transport, protocol):while True:

buf = stream.read1()if not buf:

breakstream.write(buf)

server = gruvi.StreamServer(echo_handler)server.listen(('localhost', 7777))server.run()

Requirements

You need Python 2.7 or 3.3+.

The following operating systems are currently supported:

• Linux (might work on other Posix OSs)

• macOS (not currently tested via CI)

• Windows (not currently tested via CI)

Installation

Development install in a virtualenv:

$ git clone https://github.com/geertj/gruvi$ cd gruvi$ pip install -r requirements.txt$ python setup.py build$ python setup.py install

To run the test suite:

$ python runtests.py unit

For other installation options, see the Installation section in the manual.

Documentation

The documentation is available on readthedocs.

46 Chapter 2. Project files

http://gruvi.readthedocs.org/en/latest/streams.html

http://gruvi.readthedocs.org/en/latest/install.html

https://gruvi.readthedocs.org/


License

Gruvi is free software, provided under the MIT license.

Contact

Feel free to contact the author at [email protected]. You can also submit tickets or suggestions for improvements onGithub.

CHANGELOG

Changes since latest version:

• Verify SSL certificates by default.

• New function: sslcompat.create_default_context()

• Limit SSL support for Python <= 2.7.8.

Changes in version 0.10.3:

• Refactoring and cleanups of many places in the code.

• Refactored and simplified flow control.

• Made @switchpoint return a closure.

• Documentation improvements.


• New infrastructure for polling FDs.

• Add Process.close() to explicitly close the process and stdio handles.

• Remove redundant Process.kill() and Process.poll()

• Use Memory BIO for faster and more compatible SSL implementation.

• Add support for Python 3.5.

• Move to new pyuv upstream (1.0.0-dev1, using libuv 1.0.0-rc2).

• Improve test coverage.

• Remove non-standard method Queue.clear().

• Drop support for Python 2.6.

• Make futures thread-safe and add .cancel() method.

• Make Event() follow threading API for timeouts.

• New functions: wait(), as_completed()

• Store callbacks in a linked list for fast removal.

• Fix threading issues with timeouts for sync primitves.

• Add test coverage via coveralls.

• Add .alive property to Fiber.

• http: communicate connection_lost() to waiters.

2.2. CHANGELOG 47


https://github.com/geertj/gruvi


• create_connection(): raise on callback error.

• Workaround for Windows build failure (link error 1104).


• Re-release of 0.10.0 with correct README file.


• PEP-3156 style transports and protocols.

• Incorporate SSL backports from Bluepass.

• Replace Signal with stdlib-like Event and Condition.

• Add support for working with child processes.

• Add support for connecting protocols to a file descriptor.

• Lots of small bug fixes and improvements.

• Improved documentation.


• Add support for JSON-RPC v2.0

• Logger supports Python 2.7+ style docstrings on 2.6

• Lots of small bug fixes and some cleanups


• Add thread-safe synchronization primitives.

• Add gruvi.futures.

• Test suite now uses unittest.

• Lots of smaller refactorings to make the API cleaner.

• Remove dependency on six.

• Switch from greenlet to python-fibers.

• Fixed Windows support.


• Almost complete rewrite.

• First release of beta quality.

AUTHORS

Main author:

• Geert Jansen <[email protected]>

Gruvi includes some third party code:

• The gruvi/txdbus package contains code from the “txdbus” project by Tom Cocagne ([email protected]).MIT licensed. Source code is at http://github.com/cocagne/txdbus.

48 Chapter 2. Project files



http://github.com/cocagne/txdbus

Python Module Index

ggruvi, 7gruvi.dbus, 39gruvi.http, 32gruvi.jsonrpc, 36

49


50 Python Module Index

Index

Aabort() (BaseTransport method), 21accept_connection() (Server method), 27active (switch_back attribute), 11add_cleanup() (switch_back method), 11add_done_callback() (Future method), 17addr (ParsedUrl attribute), 32addresses (Server attribute), 27alive (Fiber attribute), 9assert_no_switchpoints (class in gruvi), 11

BBaseProtocol (class in gruvi), 24BaseTransport (class in gruvi), 20blocking() (in module gruvi), 18body (HttpMessage attribute), 33buffer (Stream attribute), 28

Ccan_write_eof() (Transport method), 22cancel() (Fiber method), 9cancel() (Future method), 17Cancelled, 7cancelled() (Future method), 16charset (HttpMessage attribute), 33check_message() (JsonRpcVersion method), 37child_exited() (Process method), 31clear() (Event method), 14Client (class in gruvi), 26close() (BaseTransport method), 21close() (Endpoint method), 26close() (SslTransport method), 24closed (Stream attribute), 28Condition (class in gruvi), 14connection_lost() (BaseProtocol method), 24connection_lost() (Server method), 27connection_made() (BaseProtocol method), 24connection_made() (Server method), 27connections (Server attribute), 27

create() (JsonRpcVersion static method), 37create_default_context() (in module gruvi), 24create_request() (JsonRpcVersion method), 37create_response() (JsonRpcVersion method), 37current_fiber() (in module gruvi), 12

Ddata (Hub attribute), 12data_received() (Protocol method), 24datagram_received() (DatagramProtocol method), 25DatagramProtocol (class in gruvi), 24DatagramTransport (class in gruvi), 22DbusClient (class in gruvi.dbus), 40DbusError, 39DbusMethodCallError, 39DbusProtocol (class in gruvi.dbus), 39DbusServer (class in gruvi.dbus), 40default_adapter (HttpServer attribute), 35default_timeout (JsonRpcProtocol attribute), 37default_version (HttpProtocol attribute), 34default_version (JsonRpcProtocol attribute), 37do_handshake() (SslTransport method), 23done() (Future method), 16

EEndpoint (class in gruvi), 26eof_received() (Protocol method), 24Error, 7error (JsonRpcError attribute), 37error_received() (DatagramProtocol method), 25errorcode (in module gruvi.jsonrpc), 36Event (class in gruvi), 14

Ffeed() (StreamBuffer method), 28feed_eof() (StreamBuffer method), 28feed_error() (StreamBuffer method), 28Fiber (class in gruvi), 9fiber (switch_back attribute), 11

51


FiberPool (class in gruvi), 18Future (class in gruvi), 16

Gget_buffer_size() (StreamBuffer method), 27get_cpu_pool() (in module gruvi), 18get_extra_info() (BaseTransport method), 21get_extra_info() (SslTransport method), 23get_extra_info() (Transport method), 22get_header() (HttpMessage method), 33get_hub() (in module gruvi), 9get_io_pool() (in module gruvi), 18get_nowait() (Queue method), 15get_write_buffer_limits() (BaseTransport method), 21get_write_buffer_size() (BaseTransport method), 20get_write_buffer_size() (DatagramTransport method), 22get_write_buffer_size() (Transport method), 21gruvi (module), 7gruvi.dbus (module), 39gruvi.http (module), 32gruvi.jsonrpc (module), 36

Hhandle_connection() (Server method), 27headers (HttpMessage attribute), 33HttpClient (class in gruvi.http), 35HttpError, 32HttpMessage (class in gruvi.http), 33HttpProtocol (class in gruvi.http), 34HttpRequest (class in gruvi.http), 33HttpServer (class in gruvi.http), 35Hub (class in gruvi), 12

Iignore_interrupt (Hub attribute), 12

Jjoin() (Queue method), 16JsonRpcClient (class in gruvi.jsonrpc), 38JsonRpcError, 36JsonRpcProtocol (class in gruvi.jsonrpc), 37JsonRpcServer (class in gruvi.jsonrpc), 38JsonRpcVersion (class in gruvi.jsonrpc), 37

LLifoQueue (class in gruvi), 16local (class in gruvi), 12Lock (class in gruvi), 13locked() (Lock method), 14locked() (RLock method), 14loop (Hub attribute), 9, 12

Mmax_buffer_size (HttpProtocol attribute), 34

max_header_size (HttpProtocol attribute), 34max_pipeline_size (HttpProtocol attribute), 34maxsize (PoolBase attribute), 17message_type (HttpMessage attribute), 33MessageProtocol (class in gruvi), 25method (HttpMessage attribute), 33minsize (PoolBase attribute), 17

Nname (Fiber attribute), 9name (Hub attribute), 12name (JsonRpcVersion attribute), 37name (PoolBase attribute), 17next_id() (JsonRpcVersion method), 37notify() (Condition method), 15notify_all() (Condition method), 15

Ppaddr() (in module gruvi), 20parse_url() (in module gruvi.http), 33parsed_url (HttpMessage attribute), 33ParsedUrl (class in gruvi.http), 32pause_reading() (BaseTransport method), 21pause_reading() (DatagramTransport method), 22pause_reading() (SslTransport method), 23pause_reading() (Transport method), 21pause_writing() (BaseProtocol method), 24pid (Process attribute), 30poll (Hub attribute), 12PoolBase (class in gruvi), 17PriorityQueue (class in gruvi), 16Process (class in gruvi), 30Protocol (class in gruvi), 24protocol (Client attribute), 26protocol (DbusClient attribute), 40protocol (HttpClient attribute), 35protocol (JsonRpcClient attribute), 38ProtocolError, 24put_nowait() (Queue method), 15Python Enhancement Proposals

PEP 3156, 3, 20, 25PEP 333, 32

Qqsize() (Queue method), 15Queue (class in gruvi), 15QueueEmpty, 15QueueFull, 15

Rreadable() (Stream method), 28release() (Lock method), 14release() (RLock method), 14

52 Index


remove_done_callback() (Future method), 17REQUEST (in module gruvi.http), 32REQUEST (in module gruvi.jsonrpc), 36RESPONSE (in module gruvi.http), 32RESPONSE (in module gruvi.jsonrpc), 36resume_reading() (BaseTransport method), 21resume_reading() (DatagramTransport method), 22resume_reading() (SslTransport method), 23resume_reading() (Transport method), 21resume_writing() (BaseProtocol method), 24returncode (Process attribute), 30RFC

RFC 2606, 32RLock (class in gruvi), 14run_callback() (Hub method), 13running() (Future method), 16

Ssaddr() (in module gruvi), 20send_signal() (Process method), 31sendto() (DatagramTransport method), 22Server (class in gruvi), 27set() (Event method), 14set_buffer_limits() (StreamBuffer method), 28set_write_buffer_limits() (BaseTransport method), 21spawn() (in module gruvi), 12spawn() (Process method), 30ssl (ParsedUrl attribute), 32SslTransport (class in gruvi), 22start() (BaseTransport method), 20start() (Fiber method), 9status_code (HttpMessage attribute), 33stderr (Process attribute), 30stdin (Process attribute), 30stdout (Process attribute), 30Stream (class in gruvi), 28stream (StreamClient attribute), 29stream (StreamProtocol attribute), 29StreamBuffer (class in gruvi), 27StreamClient (class in gruvi), 29StreamProtocol (class in gruvi), 29StreamServer (class in gruvi), 30strerror() (in module gruvi.jsonrpc), 36submit() (PoolBase method), 17switch() (Hub method), 13switch() (switch_back method), 11switch_back (class in gruvi), 10switchpoint() (in module gruvi), 11

Ttarget (ParsedUrl attribute), 32task_done() (Queue method), 16terminate() (Process method), 31ThreadPool (class in gruvi), 18

throw() (switch_back method), 11Timeout, 7timeout (Endpoint attribute), 26timeout (switch_back attribute), 11Transport (class in gruvi), 21transport (Client attribute), 26transport (Stream attribute), 28TransportError, 20

Uunwrap() (SslTransport method), 24url (HttpMessage attribute), 33

Vversion (HttpMessage attribute), 33version (JsonRpcProtocol attribute), 37

Wwrap() (Stream method), 28writable() (Stream method), 28write() (Transport method), 21write_eof() (Transport method), 22writelines() (Transport method), 22writer (HttpProtocol attribute), 34WsgiAdapter (class in gruvi.http), 34

Index 53

gruvi documentation · excellent cross-platform support including windows. •i wanted to use a pep...

Documents