Reprogrammable Electro-Optic Nonlinear Activation Functionsfor Optical Neural Networks

Ian A. D. Williamson,1, ∗ Tyler W. Hughes,2 Momchil Minkov,1 Ben Bartlett,2 Sunil Pai,1 and Shanhui Fan1, †

1Department of Electrical Engineering and Ginzton Laboratory, Stanford University, Stanford, CA 94305, USA2Department of Applied Physics and Ginzton Laboratory, Stanford University, Stanford, CA 94305, USA

We introduce an electro-optic hardware platform for nonlinear activation functions in optical neu-ral networks. The optical-to-optical nonlinearity operates by converting a small portion of the inputoptical signal into an analog electric signal, which is used to intensity-modulate the original opti-cal signal with no reduction in processing speed. Our scheme allows for complete nonlinear on-offcontrast in transmission at relatively low optical power thresholds and eliminates the requirementof having additional optical sources between each layer of the network. Moreover, the activationfunction is reconfigurable via electrical bias, allowing it to be programmed or trained to synthesizea variety of nonlinear responses. Using numerical simulations, we demonstrate that this activationfunction significantly improves the expressiveness of optical neural networks, allowing them to per-form well on two benchmark machine learning tasks: learning a multi-input exclusive-OR (XOR)logic function and classification of images of handwritten numbers from the MNIST dataset. Theaddition of the nonlinear activation function improves test accuracy on the MNIST task from 85%to 94%.

I. INTRODUCTION

In recent years, there has been significant interest inalternative computing platforms specialized for high per-formance and efficiency on machine learning tasks. Forexample, graphical processing units (GPUs) have demon-strated peak performance with trillions of floating pointoperations per second (TFLOPS) when performing ma-trix multiplication, which is several orders of magnitudelarger than general-purpose digital processors such asCPUs [1]. Moreover, analog computing has been ex-plored for achieving high performance because it is notlimited by the bottlenecks of sequential instruction exe-cution and memory access [2–6].

Optical hardware platforms are particularly appealingfor computing and signal processing due to their ultra-large signal bandwidths, low latencies, and reconfigura-bility [7–9]. They have also gathered significant interestin machine learning applications, such as artificial neu-ral networks (ANNs). Nearly three decades ago, the firstoptical neural networks (ONNs) were proposed based onfree-space optical lens and holography setups [10, 11].More recently, ONNs have been implemented in chip-integrated photonic platforms [12] using programmablewaveguide interferometer meshes which perform matrix-vector multiplications [13]. In theory, the performanceof such systems is competitive with digital computingplatforms because they may perform matrix-vector mul-tiplications in constant time with respect to the matrixdimension. In contrast, matrix-vector multiplication hasa quadratic time complexity on a digital processor. Otherapproaches to performing matrix-vector multiplicationsin chip-integrated ONNs, such as microring weight banksand photodiodes, have also been proposed [14].

∗ iwill@stanford.edu† shanhui@stanford.edu

Nonlinear activation functions play a key role in ANNsby enabling them to learn complex mappings betweentheir inputs and outputs. Whereas digital processorshave the expressiveness to trivially apply nonlinearitiessuch as the widely-used sigmoid, ReLU, and tanh func-tions, the realization of nonlinearities in optical hardwareplatforms is more challenging. One reason for this is thatoptical nonlinearities are relatively weak, necessitating acombination of large interaction lengths and high sig-nal powers, which impose lower bounds on the physicalfootprint and the energy consumption, respectively. Al-though it is possible to resonantly enhance optical non-linearities, this comes with an unavoidable trade-off inreducing the operating bandwidth, thereby limiting theinformation processing capacity of an ONN. Additionally,maintaining uniform resonant responses across many ele-ments of an optical circuit necessitates additional controlcircuitry for calibrating each element [15].

A more fundamental limitation of optical nonlinearitiesis that their responses tend to be fixed during device fab-rication. This limited tunability of the nonlinear opticalresponse prevents an ONN from being reprogrammed torealize different forms of nonlinear activation functions,which may be important for tailoring ONNs for differentmachine learning tasks. Similarly, a fixed nonlinear re-sponse may also limit the performance of very deep ONNswith many layers of activation functions since the opticalsignal power drops below the activation threshold, wherenonlinearity is strongest, in later layers due to loss inprevious layers. For example, with optical saturable ab-sorption from 2D materials in waveguides, the activationthreshold is on the order of 1-10 mW [16–18], meaningthat the strength of the nonlinearity in each subsequentlayer will be successively weaker as the transmitted powerfalls below the threshold.

In light of these challenges, the ONN demonstratedin Ref. 12 implemented its activation functions by de-tecting each optical signal, feeding them through a con-

arX

iv:1

903.

0457

9v2

[ee

ss.S

P] 2

3 Ju

l 201

9

2

ventional digital computer to apply the nonlinearity, andthen modulating new optical signals for the subsequentlayer. Although this approach benefits from the flexi-bility of digital signal processing, conventional proces-sors have a limited number of input and output chan-nels, which make it challenging to scale this approachto very large matrix dimensions, which corresponds toa large number of optical inputs. Moreover, digitallyapplied nonlinearities add latency from the analog-to-digital conversion process and constrain the computa-tional speed of the neural network to the same GHz-scale clock rates which ONNs seek to overcome. Thus, ahardware nonlinear optical activation, which doesn’t re-quire repeated bidirectional optical-electronic signal con-version, is of fundamental interest for making integratedONNs a viable machine learning platform.

In this article, we propose an electro-optic architecturefor synthesizing optical-to-optical nonlinearities which al-leviates the issues discussed above. Our architecture fea-tures complete on-off contrast in signal transmission,a variety of nonlinear response curves, and a low ac-tivation threshold. Rather than using traditional opti-cal nonlinearities, our scheme operates by measuring asmall portion of the incoming optical signal power andusing electro-optic modulators to modulate the originaloptical signal, without any reduction in operating band-width or computational speed. Additionally, our schemeallows for the possibility of performing additional non-linear transformations on the signal using analog elec-trical components. Related electro-optical architecturesfor generating optical nonlinearities have been previouslyconsidered [19–21]. In this work, we focus on the applica-tion of our architecture as an element-wise activation in afeedforward ONN, but the synthesis of low-threshold op-tical nonlinearities could be of broader interest to opticalcomputing and information processing.

The remainder of this paper is organized as follows.First, we review the basic operating principles of ANNsand their integrated optical implementations in waveg-uide interferometer meshes. We then introduce ourelectro-optical activation function architecture, showingthat it can be reprogrammed to synthesize a variety ofnonlinear responses. Next, we discuss the performanceof an ONN using this architecture by analyzing the scal-ing of power consumption, latency, processing speed, andfootprint. We then draw an analogy between our pro-posed activation function and the optical Kerr effect. Fi-nally, using numerical simulations, we demonstrate thatour architecture leads to improved performance on twodifferent machine learning tasks: (1) learning an N-inputexclusive OR (XOR) logic function; (2) classifying imagesof handwritten numbers from the MNIST dataset.

II. FEEDFORWARD OPTICAL NEURALNETWORKS

In this section, we briefly review the basics of feedfor-ward artificial neural networks (ANNs) and describe theirimplementation in a reconfigurable optical circuit, as pro-posed in Ref. 12. As outlined in Fig. 1(a), an ANN isa function which accepts an input vector, x0 and returnsan output vector, xL. This is accomplished in a layer-by-layer fashion, with each layer consisting of a linearmatrix-vector multiplication followed by the applicationof an element-wise nonlinear function, or activation, onthe result. For a layer with index i, containing a weightmatrix Wi and activation function fi(·), its operation isdescribed mathematically as

xi = fi

(Wi · xi−1

)(1)

for i from 1 to L.

Before they are able to perform a given machine learn-ing task, ANNs must be trained. The training process istypically accomplished by minimizing the prediction er-ror of the ANN on a set of training examples, which comein the form of input and target output pairs. For a givenANN, a loss function is defined to quantify the differencebetween the target output and output predicted by thenetwork. During training, this loss function is minimizedwith respect to tunable degrees of freedom, namely theelements of the weight matrix Wi within each layer. Ingeneral, although less common, it is also possible to trainthe parameters of the activation functions [22].

Optical hardware implementations of ANNs have beenproposed in various forms over the past few decades.In this work, we focus on a recent demonstration inwhich the linear operations are implemented using anintegrated optical circuit [12]. In this scheme, the in-formation being processed by the network, xi, is en-coded into the modal amplitudes of the waveguides feed-ing the device and the matrix-vector multiplications areaccomplished using meshes of integrated optical inter-ferometers. In this case, training the network requiresfinding the optimal settings for the integrated opticalphase shifters controlling the inteferometers, which maybe found using an analytical model of the chip, or usingin-situ backpropagation techniques [23].

In the next section, we present an approach for re-alizing the activation function, fi(·), on-chip with a hy-brid electro-optic circuit feeding an inteferometer. In Fig.1(b), we show how this activation scheme fits into a singlelayer of an ONN and show the specific form of the activa-tion in Fig. 1(c). We also give the specific mathematicalform of this activation and analyze its performance inpractical operation.

3

Figure 1. (a) Block diagram of a feedforward neural network of L layers. Each layer consists of a Wi block representing a linearmatrix which multiplies vector inputs xi−1. The fi block in each layer represents an element-wise nonlinear activation functionoperating on vectors zi to produce outputs xi. (b) Schematic of the optical interferometer mesh implementation of a singlelayer of the feedforward neural network. (c) Schematic of the proposed optical-to-optical activation function which achieves anonlinear response by converting a small portion of the optical input, z into an electrical signal, and then intensity modulatingthe remaining portion of the original optical signal as it passes through an interferometer.

III. NONLINEAR ACTIVATION FUNCTIONARCHITECTURE

In this section, we describe our proposed nonlinear ac-tivation function architecture for optical neural networks,which implements an optical-to-optical nonlinearity byconverting a small portion of the optical input power intoan electrical voltage. The remaining portion of the origi-nal optical signal is phase- and amplitude-modulated bythis voltage as it passes through an interferometer. Foran input signal with amplitude z, the resulting nonlin-ear optical activation function, f(z), is a result of theresponses of the interferometer under modulation as wellas the components in the electrical signal pathway.

A schematic of the architecture is shown in Fig. 1(c),where black and blue lines represent optical waveguidesand electrical signal pathways, respectively. The inputsignal first enters a directional coupler which routes aportion, α, of the input optical power to a photodetec-tor. The photodetector is the first element of an optical-to-electrical conversion circuit, which is a standard com-ponent of high-speed optical receivers for converting an

optical intensity into a voltage. In this work, we assumea normalization of the optical signal such that the totalpower in the input signal is given by |z|2. The optical-to-electrical conversion process consists of the photodetectorproducing an electrical current, Ipd = R ·α|z|2, where Ris the photodetector responsivity, and a transimpedanceamplifying stage, characterized by a gain G, convertingthis current into a voltage VG = G ·R · α|z|2. The out-put voltage of the optical-to-electrical conversion circuitthen passes through a nonlinear signal conditioner witha transfer function, H(·). This component allows for theapplication of additional nonlinear functions to transformthe voltage signal. Finally, the conditioned voltage sig-nal, H(VG) is combined with a static bias voltage, Vb toinduce a phase shift of

∆φ =π

Vπ

[Vb +H

(GRα|z|2

)](2)

for the optical signal routed through the lower port ofthe directional coupler. The parameter Vπ represents thevoltage required to induce a phase shift of π in the phasemodulator. This phase shift, defined by Eq. 2, is a non-

4

linear self-phase modulation because it depends on theinput signal intensity.

An optical delay line between the directional couplerand the Mach-Zehnder interferometer (MZI) is used tomatch the signal propagation delays in the optical andelectrical pathways. This ensures that the nonlinear self-phase modulation defined by Eq. 2 is applied at the sametime that the optical signal which generated it passesthrough the phase modulator. For the circuit shown inFig. 1(c), the optical delay is τopt = τoe + τnl + τrc, ac-counting for the contributions from the group delay of theoptical-to-electrical conversion stage (τoe), the delay as-sociated with the nonlinear signal conditioner (τnl), andthe RC time constant of the phase modulator (τrc).

The nonlinear self-phase modulation achieved by theelectric circuit is converted into a nonlinear amplituderesponse by the MZI, which has a transmission dependingon ∆φ as

tMZI = j exp

(−j∆φ

2

)cos

(∆φ

2

). (3)

Depending on the configuration of the bias, Vb, a largerinput optical signal amplitude causes either more or lesspower to be diverted away from the output port, resultingin a nonlinear self-intensity modulation. Combining theexpression for the nonlinear self-phase modulation, givenby Eq. 2, with the MZI transmission, given by Eq. 3,the mathematical form of the activation function can bewritten explicitly as

f(z) = j√

1− α exp

(−j 1

2

[φb + π

H(GRα|z|2

)

Vπ

])

· cos

(1

2

[φb + π

H(GRα|z|2

)

Vπ

])z, (4)

where the contribution to the phase shift from the biasvoltage is

φb = πVbVπ. (5)

For the remainder of this work, we focus on the casewhere no nonlinear signal conditioning is applied to theelectrical signal pathway, i.e. H(VG) = VG. However,even with this simplification the activation function stillexhibits a highly nonlinear response. We also neglectsaturating effects in the OE conversion stage which canoccur in either the photodetector or the amplifier. How-ever, in practice, the nonlinear optical-to-optical transferfunction could take advantage of these saturating effects.

With the above simplifications, a more compact ex-pression for the activation function response is

f(z) = j√

1− α exp

(−j

[gφ |z|2

2+φb2

])

· cos

(gφ |z|2

2+φb2

)z, (6)

0

1 (a)

Output f (z) ·√

gφ/π

0

1

φb = 1.00π

(b)Transmission

0

1 (c)

0

1

φb = 0.85π

(d)

0

1 (e)

0

1

φb = 0.00π

(f)

0 10

1 (g)

0 10

1

φb = 0.50π

(h)

Input z ·√

gφ/π

Figure 2. Activation function output amplitude (blue lines)and activation function transmission (green lines) as a func-tion of input signal amplitude. The input and output arenormalized to the phase gain parameter, gφ. Panel pairs(a),(b) and (c),(d) correspond to a ReLU-like response, witha suppressed transmission for inputs with small amplitudeand high transmission for inputs with large amplitude. Panelpairs (e),(f) and (g),(h) correspond to a clipped response, withhigh transmission for inputs with small amplitude and re-duced transmission for inputs with larger amplitude.

where the phase gain parameter is defined as

gφ = παGR

Vπ. (7)

Equation 7 indicates that the amount of phase shift perunit input signal power can be increased via the gainand photodiode responsivity, or by converting a largerfraction of the optical power to the electrical domain.However, tapping out a larger fraction optical power alsoresults in a larger linear loss, which is undesirable.

The electrical biasing of the activation phase shifter,represented by Vb, is an important degree of freedom for

5

determining its nonlinear response. We consider a repre-sentative selection, consisting of four different responses,in Fig. 2. The left column of Fig. 2 plots the outputsignal amplitude as a function of the input signal ampli-tude i.e. |f(z)| in Eq. 6, while the right column plotsthe transmission coefficient i.e. |f(z)|2/|z|2, a quantitywhich is more commonly used in optics than machinelearning. The first two rows of Fig. 2, corresponding toφb = 1.0π and 0.85π, exhibit a response which is compa-rable to the ReLU activation function: transmission is lowfor small input values and high for large input values. Forthe bias of φb = 0.85π, transmission at low input valuesis slightly increased with respect to the response whereφb = 1.00π. Unlike the ideal ReLU response, the acti-vation at φb = 0.85π is not entirely monotonic becausetransmission first goes to zero before increasing. On theother hand, the responses shown in the bottom two rowsof Fig. 2, corresponding to φb = 0.0π and 0.50π, are quitedifferent. These configurations demonstrate a saturatingresponse in which the output is suppressed for higher in-put values but enhanced for lower input values. For all ofthe responses shown in Fig. 2, we have assumed α = 0.1which limits the maximum transmission to 1− α = 0.9.

A benefit of having electrical control over the activa-tion response is that, in principle, its electrical bias canbe connected to the same control circuitry which pro-grams the linear interferometer meshes. In doing so, asingle ONN hardware unit can then be reprogrammed tosynthesize many different activation function responses.This opens up the possibility of heuristically selecting anactivation function response, or directly optimizing thethe activation bias using a training algorithm. This re-alization of a flexible optical-to-optical nonlinearity canallow ONNs to be applied to much broader classes ofmachine learning tasks.

We note that Fig. 2 shows only the amplitude responseof the activation function. In fact, all of these responsesalso introduce a nonlinear self-phase modulation to theoutput signal. If desired, this nonlinear self-phase modu-lation can be suppressed using a push-pull interferometerconfiguration in which the generated phase shift, ∆φ, isdivided and applied with opposite sign to the top andbottom arms.

IV. PERFORMANCE AND SCALABILITY

In this section, we discuss the performance of an in-tegrated ONN which uses meshes of integrated opticalinterferometers to perform matrix-vector multiplicationsand the electro-optic activation function, as shown in Fig.1(b),(c). Here, we focus on characterizing how the powerconsumption, computational latency, physical footprint,and computational speed of the ONN scale with respectto the number of network layers, L and the dimension ofthe input vector, N , assuming square matrices. The sys-tem parameters used for this analysis are summarized inTable I and the figures of merit are summarized in Table

Table I. Summary of parameter values

Parameter Value

Modulator and detector rate 10 GHz

Photodetector responsivity (R) 1 A/W

Optical-to-electrical circuit power consumption 100 mW

Optical-to-electrical circuit group delay (τeo) 100 ps

Phase modulator RC delay (τrc) 20 ps

Mesh MZI length (DMZI) 100 µm

Mesh MZI height (HMZI) 60 µm

Waveguide effective index (neff) 3.5

1 10 30Modulator Vπ (volts)

30

35

40

45

50

55

60

65

Opt

ical

-to-e

lect

rical

gain

G(d

BΩ)

P th = 0.1 mW

P th = 1.0 mW

P th = 10.0 mW

Figure 3. Contours of constant activation threshold as a func-tion of the optical-to-electrical gain and the modulator Vπ ofthe activation function shown in Fig. 1(c) with a photodetec-tor responsivity R = 1.0 A/W.

II.

A. Power consumption

The power consumption of the ONN, as shown in Fig.1(b), consists of contributions from (1) the programmablephase shifters inside the interferometer mesh, (2) theoptical source supplying the input vectors, x0, and (3)the active components of the activation function suchas the amplifier and photodetector. In principle, thecontribution from (1) can be made negligible by usingphase change materials or ultra-low power MEMS phaseshifters. Therefore, in this section we focus only on con-tributions (2) and (3) which pertain to the activationfunction.

To quantify the power consumption, we first considerthe minimum input optical power to a single activationthat triggers a nonlinear response. We refer to this as the

6

Table II. Summary of per-layer optical neural network performance using the electro-optic activation function

Scaling Per-layer figures of merit

Mesh Activation N = 4 N = 10 N = 100

Power consumption∗ LN 0.4 W 1 W 10 W

Latency LN L 125 ps 132 ps 237 ps

Footprint LN2 LN 2.5 mm2 6.6 mm2 120.0 mm2

Speed LN2 1.6 × 1011 MAC/s 1 × 1012 MAC/s 1 × 1014 MAC/s

Efficiency∗ N−1 2.5 pJ/MAC 1 pJ/MAC 100 fJ/MAC

∗Assuming no power consumption in the interferometer mesh phase shifters

activation function threshold, which is mathematicallydefined as

Pth =∆φ|δT=0.5

gφ=

VππαGR

·∆φ|δT=0.5, (8)

where ∆φ|δT=0.5 is the is phase shift necessary to gener-ate a 50% change in the power transmission with respectto the transmission with null input for a given φb. Thisthreshold corresponds to z

√gφ/π = 0.73 in Fig. 2(b), to

z√gφ/π = 0.85 in Fig. 2(d), to z

√gφ/π = 0.73 in Fig.

2(f), and to z√gφ/π = 0.70 in Fig. 2(h). In general,

a lower activation threshold will result in a lower opti-cal power required at the ONN input, |x0|2. Accordingto Eq. 8, the activation threshold can be reduced via asmall Vπ and a large optical-to-electrical conversion gain,GR ∼ 1.0 V/mW. The relationship between G and Vπ foractivation thresholds of 0.1 mW, 1.0 mW, and 10.0 mWis shown in Fig. 3 for a fixed R = 1 A/W. Additionally,in Fig. 3 we conservatively assume φb = π which has thehighest threshold of the activation function biases shownin Fig. 2.

If we take the lowest activation threshold of 0.1 mW inFig. 3, the optical source to the ONN would then needto supply N · 0.1 mW of optical power. The power con-sumption of integrated optical receiver amplifiers variesconsiderably, ranging from as low as 10 mW to as highas 150 mW [24–26], depending on a variety of factorswhich are beyond the scope of this article. Therefore, aconservative estimate of the power consumption from theoptical-to-electrical conversion circuits in all activationsis L ·N · 100 mW. For an ONN with N = 100, the powerconsumption per layer from the activation function wouldbe 10 W and would require a total optical input power ofN ·Pth = 100 · 0.1 mW = 10 mW. Thus, the total powerconsumption of the ONN is dominated by the activationfunction electronics.

B. Latency

For the feedforward neural network architecture shownin Fig. 1(a), the latency is defined by the elapsed timebetween supplying an input vector, x0 and detecting itscorresponding prediction vector, xL. In an integrated

ONN, as implemented in Fig. 1(b), this delay is simplythe travel time for an optical pulse through all L-layers.Following Ref. 12, the propagation distance in a squareinterferometer mesh is DW = N · DMZI, where DMZI isthe length of each MZI within the mesh. In the nonlin-ear activation layer, the propagation length will be domi-nated by the delay line required to match the optical andelectrical delays, and is given by

Df = (τoe + τnl + τrc) · vg, (9)

where the group velocity vg = c0/neff is the speed ofoptical pulses in the waveguide. Therefore,

latency = L ·N ·DMZI · vg−1

︸ ︷︷ ︸Interferometer mesh

+L · (τoe + τnl + τrc)︸ ︷︷ ︸Activation function

.

(10)Equation 10 indicates that the latency contribution fromthe interferometer mesh scales with the product LN ,which is the same scaling as predicted in Ref. 12. Onthe other hand, the activation function adds to the la-tency independently of N because each activation circuitis applied in parallel to all N -vector elements.

For concreteness, we assume DMZI = 100 µm andneff = 3.5. Following our assumption in the previoussection of using no nonlinear electrical signal conditionerin the activation function, τnl = 0 ps. Typical groupdelays for integrated transimpedance amplifiers used inoptical receivers can range from τoe ≈ 10 to 100 ps. More-over, assuming an RC-limited phase modulator speed of50 GHz yields τrc ≈ 20 ps. Therefore, if we assume aconservative value of τoe = 100 ps, a network dimensionof N ≈ 100 would have a latency of 237 ps per layer,with equal contributions from the mesh and the activa-tion function. For a ten layer network (L = 10) thetotal latency would be approximately 2.4 ns, still ordersof magnitude lower than the latency typically associatedwith GPUs.

C. Physical footprint

The physical footprint of the ONN consists of the spacetaken up by both the linear interferometer mesh and theoptical and electrical components of the activation func-tion. Neglecting the electrical control lines for tuning

7

each MZI, the total footprint of the ONN is

A = L ·N2 ·AMZI︸ ︷︷ ︸Interferometer mesh

+ L ·N ·Af︸ ︷︷ ︸Activation function

, (11)

where AMZI = DMZI · HMZI is the area of a single MZIelement in the mesh and Af = Df · Hf is the area of asingle activation function.

In the direction of propagation, Df is dominated by thewaveguide optical delay line required to match the delayof the electrical signal pathway. Based on the previousdiscussion of the activation function’s latency, τopt = 120ps corresponds to a total waveguide length of Df ≈ 1 cm.For simplicity, we assume this delay is achieved using astraight waveguide, which results in a large footprint butwith optical losses that can be very low. For example, insilicon waveguides losses below 0.5 dB/cm have been ex-perimentally demonstrated [27]. In principle, incorporat-ing waveguide bends or resonant optical elements couldsignificantly reduce the activation function’s footprint.For example, coupled micro ring arrays have experimen-tally achieved group delays of 135 ps over a bandwidthof 10 GHz in a 0.03 mm × 0.25 mm footprint [28].

Transverse to the direction of propagation, the acti-vation function footprint will be dominated by the elec-tronic components of the optical-to-electrical conversioncircuit. In principle, compact waveguide photodetectorsand modulators can be utilized. However, the compo-nents of the transimpedance amplifier may be challeng-ing to integrate in the area available between neigh-boring output waveguides of the interferometer mesh.One possibility towards achieving a fully integrated opto-electronic ONN would be to use so-called amplifier-freeoptical receivers [26], where ultra-low capacitance detec-tors provide high-speed opto-electronic conversion. Sim-ilarly to the experimental demonstration in Ref. 29,the amplifier-free receiver could be integrated directlywith a high efficiency (e.g. effectively a low Vπ) electro-optic modulator. Compact electro-absorption modula-tors could also be utilized. In addition to achieving acompact footprint, operating without an amplifier wouldalso result in an order of magnitude reduction in bothpower consumption and latency, with the later reducingthe required length of the optical delay line and thus thefootprint.

For the purposes of our analysis, we assume no inte-gration of the electronic transimpedance amplifier and,therefore, that the on-chip components of the activationfunction fit within the height of each interferometer meshrow, Df ≤ DMZI = 60 µm. Under this assumption andfollowing the scaling in Eq. 11, the total footprint of asingle ONN layer of dimension N = 10 would be 11.0 mm× 0.6 mm. Interestingly, following the latency discussionin the previous section, a single ONN layer of dimensionN = 100 would have a footprint of 20.0 mm × 6.0 mm,with equal contribution from the activation function andfrom the mesh.

D. Speed

The speed, or computational capacity, of the ONN, asshown in Fig. 1(a), is determined by the number of inputvectors, x0 that can be processed per unit time. Here, weargue that although our activation function is not fullyoptical, it results in no speed degradation compared to alinear ONN consisting of only interferometer meshes.

The reason for this is that a fully integrated ONNwould also include high-speed modulators and detectorson-chip to perform fast modulation and detection of se-quences of x0 vectors and xL vectors, respectively. Wetherefore argue that the same high-speed detector andmodulator elements could also be integrated between thelinear network layers to provide the optical-electrical andelectrical-optical transduction for the activation function.State of the art integrated transimpedance amplifiers canalready operate at speeds comparable to the optical mod-ulator and detector rates, which are on the order of 50 -100 GHz [24, 30], and thus would not be a limiting factorin the speed of our architecture.

To perform a matrix-vector multiplication on a conven-tional CPU requires N2 multiply-accumulate (MAC) op-erations, each consisting of a single multiplication and asingle addition. Therefore, assuming a photodetector andmodulator rate of 10 GHz means that an ONN can effec-tively performN2·L·1010 MAC/sec. This means that onelayer of an ONN with dimension N = 10 would effectivelyperform 1012 MAC/sec. Increasing the input dimensionto N = 100 would then scale the performance of the ONNto 1014 MAC/sec per layer. This is two orders of magni-tude greater than the peak performance obtainable withmodern GPUs, which typically have performance on theorder of 1012 floating point operations/sec (FLOPS). Be-cause the power consumption of the ONN scales as LN(assuming passive phase shifters in the mesh) and thespeed scales as LN2, the energy per operation is mini-mized for large N (Table II). Thus, for large ONNs thepower consumption associated with the electro-optic con-version in the activation function can be amortized overthe parallelized operation of the linear mesh.

We note that the activation function circuit shown inFig. 1(c) can be modified to remove the matched opticaldelay line by using very long optical pulses. This modi-fication may be advantageous for reducing the footprintof the activation and would result in τopt τele. How-ever, this results in a reduction of the ONN speed, whichwould then be limited by the combined activation delayof all L nonlinear layers in the network, ∼ (L · τele)

−1.

V. COMPARISON WITH THE KERR EFFECT

All-optical nonlinearities such as bistability and sat-urable absorption have been previously considered as po-tential activation functions in ONNs [10, 31]. An al-ternative implementation of the activation function inFig. 1(c) could consist of a nonlinear MZI, with one

8

0 10 20 30 40 50Amplifier gain G (dBΩ)

100

102

104

106(W·m

)−1

(a)α = 0.01α = 0.10α = 0.50Si Kerr

20 40 60 80 100Modulator VπL (V · mm)

101

102

103

104

105

(W·m

)−1

(b)G = 10 dBΩ

G = 20 dBΩ

G = 30 dBΩ

Si Kerr

Figure 4. Nonlinear parameter ΓEO for the electro-optic acti-vation as a function of (a) gain, G, for α = 0.50, 0.10, and 0.01and (b) modulator VπL. The nonlinear parameter associatedwith the optical Kerr effect, ΓKerr in a Silicon waveguide ofcross sectional area A = 0.05 µm2 corresponds to the blackdotted line.

of its arms having a material with Kerr nonlinear op-tical response. The Kerr effect is a third-order opticalnonlinearity which generates a change in the refractiveindex, and thus a nonlinear phase shift, which is pro-portional to the input pulse intensity. In this section wecompare the electro-optic activation function introducedin the previous section [Fig. 1(c)] to such an alterna-tive all-optical activation function using the Kerr effect,highlighting how the electro-optic activation can achievea lower activation threshold.

Unlike the electro-optic activation function, the Kerreffect is lossless and has no latency because it arises froma nonlinear material response, rather than a feedforwardcircuit. A standard figure of merit for quantifying thestrength of the Kerr effect in a waveguide is through theamount of nonlinear phase shift generated per unit inputpower per unit waveguide length. This is given mathe-matically by the expression

ΓKerr =2π

λ0

n2

A, (12)

where n2 is the nonlinear refractive index of the mate-rial and A is the effective mode area. ΓKerr ranges from100 (W·m)−1 in chalcogenide to 350 (W·m)−1 in silicon[32]. An equivalent figure of merit for the electro-optic

feedforward scheme can be mathematically defined as

ΓEO = παRG

VπL, (13)

where VπL is the phase modulator figure of merit. Thefigures of merit described in Eqs. 12-13 can be repre-sented as an activation threshold (Eq. 8) via the rela-

tionship Pth = ∆φ|δT=0.5

ΓL , for a given waveguide length,L where the electro-optic phase shift or nonlinear Kerreffect take place.

A comparison of Eq. 12 and Eq. 13 indicates thatwhile the strength of the Kerr effect is largely fixed bywaveguide design and material choice, the electro-opticscheme has several degrees of freedom which allow it topotentially achieve a stronger nonlinear response. Thefirst design parameter is the amount of power tapped offto the photodetector, which can be increased to gener-ate a larger voltage at the phase modulator. However,increasing α also increases the linear signal loss throughthe activation which does not contribute to the nonlin-ear mapping between the input and output of the ONN.Therefore, α should be minimized as long as the opticalpower routed to the photodetector is large enough to beabove the noise equivalent power level.

On the other hand, the product RG determines theconversion efficiency of the detected optical power intoan electrical voltage. Fig. 4(a) compares the nonlinear-ity strength of the electro-optic activation (blue lines) tothat of an implementation using the Kerr effect in silicon(black dashed line) for several values of α, as a func-tion of G. The responsivity is fixed at R = 1.0 A/W.We observe that tapping out 10% of the optical powerrequires a gain of 20 dBΩ to achieve a nonlinear phaseshift equivalent threshold to that of a silicon waveguidewhere A = 0.05 µm2 for the same amount of input opticalpower. Tapping out only 1% of the optical power requiresan additional 10 dBΩ of gain to maintain this equiva-lence. We note that the gain range considered in Fig.4(a) is well within the regime of what has been demon-strated in integrated transimpedance amplifiers for op-tical receivers [24–26]. In fact, many of these systemshave demonstrated much higher gain. In Fig. 4(a), thephase modulator VπL was fixed at 20 V·mm. However,because a lower VπL translates into an increased phaseshift for a given applied voltage, this parameter can alsobe used to enhance the nonlinearity. Fig. 4(b) demon-strates the effect of changing the VπL for several valuesof of G, again, with a fixed responsivity R = 1.0 A/W.This demonstrates that with a reasonable level of gainand phase modulator performance, the electro-optic ac-tivation function can trade off an increase in latency fora significantly lower optical activation threshold than theKerr effect.

9

VI. MACHINE LEARNING TASKS

In this section, we apply the electro-optic activationfunction introduced above to several machine learningtasks. In Sec. VI A, we simulate training an ONNto implement an exclusive-OR (XOR) logical operation.The network is modeled using neuroptica [33], a cus-tom ONN simulator written in Python, which trains thesimulated networks only from physically measurable fieldquantities using the on-chip backpropagation algorithmintroduced in Ref. 23. In Sec. VI B, we consider themore complex task of using an ONN to classify hand-written digits from the Modified NIST (MNIST) dataset,which we model using the neurophox [34, 35] packageand tensorflow [36], which computes gradients usingautomatic differentiation. In both cases, we model thevalues in the network as complex-valued quantities andrepresent the interferometer meshes as unitary matricesparameterized by phase shifters.

A. Exclusive-OR Logic Function

An exclusive-OR (XOR) is a logic function which takestwo inputs and produces a single output. The output ishigh if only one of the two inputs is high, and low for allother possible input combinations. In this example, weconsider a multi-input XOR which takes N input val-ues, given by x1 . . . xN , and produces a single outputvalue, y. The input-output relationship of the multi-input XOR function is a generalization of the two-inputXOR. For example, defining logical high and low valuesas 1 and 0, respectively, a four-input XOR has an out-put table indicated the desired values in Fig. 5(b). Weselect this task for the ONN to learn because it requiresa non-trivial level of nonlinearity, meaning that it couldnot be implemented in an ONN consisting of only linearinterferometer meshes.

The architecture of the ONN used to learn the XOR isshown schematically in Fig. 5(a). The network consistsof L layers, with each layer constructed from an N ×Nunitary interferometer mesh followed by an array of Nparallel electro-optic activation functions, with each ele-ment corresponding to the circuit in Fig. 1(c). After thefinal layer, the lower N − 1 outputs are dropped to pro-duce a single output value which corresponds to y. Un-like the ideal XOR input-output relationship describedabove, for the XOR task learned by the ONN we nor-malize the input vectors such that they always have anL2 norm of 1. This constraint is equivalent to enforc-ing a constant input power to the network. Additionally,because the activation function causes the optical powerlevel to be attenuated at each layer, we take the high out-put state to be a value of 0.2, as shown in Fig. 1(b). Thelow output remains at a value of 0.0. An alternative tousing a smaller amplitude for the output high state wouldbe to add additional ports with fixed power biases to in-crease the total input power to the network, similarly to

Figure 5. (a) Architecture of an L-layer ONN used to imple-ment an N -input XOR logic function. (b) Red dots indicatethe learned input-output relationship of the XOR for N = 4on an 2-layer ONN. Electro-optic activation functions are con-figured with gain g = 1.75π and biasing phase φb = π. (c)Mean squared error (MSE) versus training epoch. (d) FinalMSE after 5000 epochs averaged over 20 independent trainingruns vs activation function gain. Different lines correspond tothe responses shown in Fig. 2, with φb = 1.00π, 0.85π, 0.00π,and 0.50π. Shaded regions correspond to the range (minimumand maximum) final MSE from the 20 training runs.

the XOR demonstrated in Ref. 23.In Fig. 5(b) we show the four-input XOR input-output

relationship which was learned by a two-layer ONN. Theelectro-optic activation functions were configured to havea gain of g = 1.75π and biasing phase of φb = π. Thisbiasing phase configuration corresponds to the ReLU-likeresponse shown in Fig. 2(a). The black markers indi-cate the desired output values while the red circles indi-cate the output learned by the two-layer ONN. Fig. 5(b)indicates excellent agreement between the learned out-put and the desired output. The evolution of the meansquared error (MSE) between the ONN output and thedesired output during training confirms this agreement,

10

as shown in Fig. 5(c), with a final MSE below 10−5.To train the ONN, a total of 2N = 16 training ex-

amples were used, corresponding to all possible binaryinput combinations along the x-axis of Fig. 5(b). All16 training examples were fed through the network ina batch to calculate the mean squared error (MSE) lossfunction. The gradient of the loss function with respectto each phase shifter was computed by backpropagatingthe error signal through the network to calculate the losssensitivity at each phase shifter [23]. The above stepswere repeated until the MSE converged, as shown in Fig.5(c). Only the phase shifter parameters were optimizedby the training algorithm, while all parameters of theactivation function were unchanged.

To demonstrate that the nonlinearity provided by theelectro-optic activation function is essential for the ONNto successfully learn the XOR, in Fig. 5(d) we plot thefinal MSE after 5000 training epochs, averaged over 20independent training runs, as a function of the activationfunction gain, gφ. The shaded regions indicates the min-imum and maximum range of the final MSE over the 20training runs. The four lines shown in Fig. 5(d) corre-spond to the four activation function bias configurationsshown in Fig. 2.

For the blue curve in Fig. 5(d), which correspondsto the ReLU-like activation, we observe a clear improve-ment in the final MSE with an increase in the nonlinearitystrength. We also observe that for very high nonlinearity,above gφ = 1.5π, the range between the minimum andmaximum final MSE broadens and the mean final MSEincreases. However, the best case (minimum) final MSEcontinues to decrease, as indicated by the lower borderof the shaded blue region. This trend indicates that al-though increasing nonlinearity improves the ONN’s abil-ity to learn the XOR function, very high levels of non-linearity may also prevent the training algorithm fromconverging.

A trend of decreasing MSE with increasing nonlinear-ity is also observed for the activation corresponding tothe green curve in Fig. 5(d). However, the range of MSEvalues begins to broaden at a lower value of gφ = 1.0π.Such broadening may be a result of the changing slopein the activation function output, as shown in Fig. 2(e).For the activation functions corresponding to the red andorange curves in Fig. 5(d), the final MSE decreases some-what with an increase in gφ, but generally remains muchhigher than the other two activation function responses.We conclude that these two responses are not as wellsuited for learning the XOR function. Overall, these re-sults demonstrate that the flexibility of our architectureto achieve specific forms of nonlinear activation functionsis important for the successful operation of an ONN.

B. Handwritten Digit Classification

The second task we consider for demonstrating the ac-tivation function is classifying images of handwritten dig-

its from the MNIST dataset, which has become a stan-dard benchmark problem for ANNs [37]. The datasetconsists of 70,000 grayscale 28×28 pixel images of hand-written digits between 0 and 9. Several representativeimages from the dataset are shown in Fig. 6(a).

To reduce the number of input parameters, and hencethe size of the neural network, we use a preprocessingstep to convert the images into a Fourier-space represen-tation. Specifically, we compute the 2D Fourier trans-form of the images which is defined mathematically asc(kx, ky) =

∑m,n e

jkxm+jkyng(m,n), where g(m,n) is

the gray scale value of the pixel at location (m,n) withinthe image. The amplitudes of the Fourier coefficientsc(kx, ky) are shown below their corresponding imagesin Fig. 6(a). These coefficients are generally complex-valued, but because the real-space map g(m,n) is real-valued, the condition c(kx, ky) = c∗(−kx,−ky) applies.

We observe that the Fourier-space profiles are mostlyconcentrated around small kx and ky, corresponding tothe center region of the profiles in Fig. 6(a). This isdue to the slowly varying spatial features in the images.We can therefore expect that most of the information iscarried by the small-k Fourier components, and with thegoal of decreasing the input size, we can restrict the data

to N coefficients with the smallest k =√k2x + k2

y. An

additional advantage of this preprocessing step is that itreduces the computational resources required to performthe training process because the neural network dimen-sion does not need to accommodate all 282 = 784 pixelvalues as inputs.

Fourier preprocessing is particularly relevant for ONNsfor two reasons. First, the Fourier transform has astraightforward implementation in the optical domainusing techniques from Fourier optics involving standardcomponents such as lens and spatial filters [38]. Second,this approach allows us to take advantage of the factthat ONNs are complex -valued functions. That is to say,the N complex-valued coefficients c(kx, ky) can be han-dled by an N -dimensional ONN, whereas to handle thesame input using a real-valued neural network requires atwice larger dimension. The ONN architecture used inour demonstration is shown schematically in Fig. 6(a).The N Fourier coefficients closest to kx = ky = 0 arefed into an optical neural network consisting of L layers,after which a drop-mask reduces the final output to 10components. The intensity of the 10 outputs are recordedand normalized by their sum, which creates a probabil-ity distribution that may be compared with the one-hotencoding of the digits from 0 to 9. The loss functionis defined as the cross-entropy between the normalizedoutput intensities and the correct one-hot vector.

During each training epoch, a subset of 60,000 imagesfrom the dataset were fed through the network in batchesof 500. The remaining 10,000 image-label pairs were usedto form a test dataset. For a two-layer network withN = 16 Fourier components, Fig. 6(b) compares the clas-sification accuracy over the training dataset (solid lines)

11

Figure 6. (a) Schematic of an optical image recognition setup based on an ONN. Images of handwritten numbers from theMNIST database are preprocessed by converting from real-space to k-space and selecting N Fourier coefficients associated withthe smallest magnitude k-vectors. (b) Test accuracy (solid lines) and training accuracy (dashed lines) during training for atwo layer ONN without activation functions (blue) and with activation functions (orange). N = 16 Fourier components wereused as inputs to the ONN and each vector was normalized such that its L2 norm is unity. The activation function parameterswere gφ = 0.05π and φb = 1.00π. (c) Cross entropy loss during training. (d) Confusion matrix, specified in percentage, for thetrained ONN with the electro-optic activation function.

and testing dataset (dashed lines) while Fig. 6(b) com-pares the cross entropy loss during optimization. Theblue curves correspond to an ONN with no activationfunction (e.g. a linear optical classifier) and the orangecurves correspond to an ONN with the electro-optic acti-vation function configured with gφ = 0.05π, φb = 1.00π,and α = 0.1. The gain setting in particular was selectedheuristically. We observe that the nonlinear activationfunction results in a significant improvement to the ONNperformance during and after training. The final valida-tion accuracy for the ONN with the activation functionis 93%, which amounts to an 8% difference as comparedto the linear ONN which achieved an accuracy of 85%.

The confusion matrix computed over the testingdataset is shown in Fig. 6(d). We note that the pre-diction accuracy of 93% is high considering that onlyN = 16 complex Fourier components were used, and the

network is parameterized by only 2 × N2 × L = 1024free parameters. Moreover, this prediction accuracy iscomparable with the 92.6% accuracy achieved in a fully-connected linear classifier with 4010 free parameters tak-ing all of the 282 = 784 real-space pixel values as inputs[37]. Finally, in Table III we show that the accuracycan be further improved by including a third layer inthe ONN and by making the activation function gain atrainable parameter. This brings the testing accuracyto 94%. Based on the parameters from Table I and thescaling from Table II, the 3 layer handwritten digit clas-sification system would consume 4.8 W while performing7.7× 1012 MAC/sec. Its prediction latency would be 1.5ns.

12

Table III. Accuracy on the MNIST testing dataset after optimization

# Layers Without activation With activation

Untrained Trained∗

1 85.00% 89.80% 89.38%

2 85.83% 92.98% 92.60%

3 85.16% 92.62% 93.89%

∗The phase gain, gφ, of each layer was optimized during training

VII. CONCLUSION

In conclusion, we have introduced an architecture forsynthesizing optical-to-optical nonlinearities and demon-strated its use as a nonlinear activation function in afeed forward ONN. Using numerical simulations, we haveshown that such activation functions enable an ONN tobe successfully applied to two machine learning bench-mark problems: (1) learning a multi-input XOR logicfunction, and (2) classifying handwritten numbers fromthe MNIST dataset. Rather than using all-optical non-linearities, our activation architecture uses intermediatesignal pathways in the electrical domain which are ac-cessed via photodetectors and phase modulators. Specif-ically, a small portion of the optical input power is tappedout which undergoes analog processing before modulat-ing the remaining portion of the same optical signal.Whereas all-optical nonlinearities have largely fixed re-sponses, a benefit of the electro-optic approach demon-strated here is that signal amplification in the electronicdomain can overcome the need for high optical signalpowers to achieve a significantly lower activation thresh-old. For example, we show that a phase modulator Vπof 10 V and an optical-to-electrical conversion gain of 57dBΩ, both of which are experimentally feasible, result inan optical activation threshold of 0.1 mW. We note thatthis nonlinearity is compatible with the in situ trainingprotocol proposed in Ref. 23, which is applicable to ar-bitrary activation functions.

Our activation function architecture can utilize thesame integrated photodetector and modulator technolo-gies as the input and output layers of a fully-integratedONN. This means that an ONN using this activation

suffers no reduction in processing speed, despite usinganalog electrical components. The only trade off madeby our design is an increase in latency due to the electro-optic conversion process. However, we find that an ONNwith dimension N = 100 has a total prediction latencyof 2.4 ns/layer, with approximately equal contributionsfrom the propagation of optical pulses through the in-terferometer mesh and from the electro-optic activationfunction. Conservatively, we estimate the energy con-sumption of an ONN with this activation function to be100 fJ/MAC, but this figure of merit could potentiallybe reduced by orders of magnitude using highly efficientmodulators and amplifier-free optoelectronics [29].

Finally, we emphasize that in our activation function,the majority of the signal power remains in the opticaldomain. There is no need to have a new optical sourceat each nonlinear layer of the network, as is requiredin previously demonstrated electro-optic neuromorphichardware [14, 21, 39] and reservoir computing architec-tures [40, 41]. Additionally, each activation function inour proposed scheme is a standalone analog circuit andtherefore can be applied in parallel. While we have fo-cused here on the application of our architecture as anactivation function in a feedforward ONN, the synthesisof low-threshold optical nonlinearlities using this circuitcould be of broader interest for optical computing as wellas microwave photonic signal processing applications.

ACKNOWLEDGMENTS

This work was supported by a US Air Force Officeof Scientific Research (AFOSR) MURI project (GrantNo FA9550-17-1-0002). I.A.D.W. acknowledges helpfuldiscussions with Avik Dutt.

[1] Vivek K. Pallipuram, Mohammad Bhuiyan, andMelissa C. Smith, “A comparative study of GPU pro-gramming models and architectures using neural net-works,” The Journal of Supercomputing 61, 673–718(2012).

[2] Jeffrey M. Shainline, Sonia M. Buckley, Richard P. Mirin,and Sae Woo Nam, “Superconducting OptoelectronicCircuits for Neuromorphic Computing,” Physical ReviewApplied 7, 034013 (2017).

[3] Bhavin J. Shastri, Alexander N. Tait, Thomas Fer-reira de Lima, Mitchell A. Nahmias, Hsuan-Tung Peng,and Paul R. Prucnal, “Principles of NeuromorphicPhotonics,” arXiv:1801.00016 [physics] , 1–37 (2018),arXiv:1801.00016 [physics].

[4] F. D. Coarer, M. Sciamanna, A. Katumba, M. Freiberger,J. Dambre, P. Bienstman, and D. Rontani, “All-OpticalReservoir Computing on a Photonic Chip Using Silicon-Based Ring Resonators,” IEEE Journal of Selected Top-

13

ics in Quantum Electronics 24, 1–8 (2018).[5] Julie Chang, Vincent Sitzmann, Xiong Dun, Wolfgang

Heidrich, and Gordon Wetzstein, “Hybrid optical-electronic convolutional neural networks with optimizeddiffractive optics for image classification,” Scientific Re-ports 8, 12324 (2018).

[6] Shane Colburn, Yi Chu, Eli Shilzerman, and Arka Ma-jumdar, “Optical frontend for a convolutional neural net-work,” Applied Optics 58, 3179–3186 (2019).

[7] Jose Capmany and Dalma Novak, “Microwave photonicscombines two worlds,” Nature Photonics 1, 319 (2007).

[8] David Marpaung, Chris Roeloffzen, Rene Heideman,Arne Leinse, Salvador Sales, and Jose Capmany, “Inte-grated microwave photonics,” Laser & Photonics Reviews7, 506–538 (2013).

[9] Paolo Ghelfi, Francesco Laghezza, Filippo Scotti,Giovanni Serafino, Amerigo Capria, Sergio Pinna,Daniel Onori, Claudio Porzi, Mirco Scaffardi, Anto-nio Malacarne, Valeria Vercesi, Emma Lazzeri, FabrizioBerizzi, and Antonella Bogoni, “A fully photonics-basedcoherent radar system,” Nature 507, 341 (2014).

[10] Yaser S. Abu-Mostafa and Demetri Psaltis, “OpticalNeural Computers,” Scientific American 256, 88–95(1987).

[11] Demetri Psaltis, David Brady, Xiang-Guang Gu, andSteven Lin, “Holography in artificial neural networks,”Nature 343, 325–330 (1990).

[12] Yichen Shen, Nicholas C. Harris, Scott Skirlo, MihikaPrabhu, Tom Baehr-Jones, Michael Hochberg, Xin Sun,Shijie Zhao, Hugo Larochelle, Dirk Englund, and MarinSoljacic, “Deep learning with coherent nanophotonic cir-cuits,” Nature Photonics 11, 441–447 (2017).

[13] David A. B. Miller, “Self-configuring universal linear op-tical component,” Photonics Research 1, 1 (2013).

[14] Alexander N. Tait, Thomas Ferreira de Lima, Ellen Zhou,Allie X. Wu, Mitchell A. Nahmias, Bhavin J. Shastri,and Paul R. Prucnal, “Neuromorphic photonic networksusing silicon photonic weight banks,” Scientific Reports7, 7430 (2017).

[15] Marina Radulaski, Ranojoy Bose, Tho Tran, ThomasVan Vaerenbergh, David Kielpinski, and Raymond G.Beausoleil, “Thermally Tunable Hybrid Photonic Archi-tecture for Nonlinear Optical Circuits,” ACS Photonics5, 4323–4329 (2018).

[16] Qiaoliang Bao, Han Zhang, Zhenhua Ni, Yu Wang,Lakshminarayana Polavarapu, Zexiang Shen, Qing-HuaXu, Dingyuan Tang, and Kian Ping Loh, “Monolayergraphene as a saturable absorber in a mode-locked laser,”Nano Research 4, 297–307 (2011).

[17] Nam Hun Park, Hwanseong Jeong, Sun Young Choi,Mi Hye Kim, Fabian Rotermund, and Dong-Il Yeom,“Monolayer graphene saturable absorbers with stronglyenhanced evanescent-field interaction for ultrafast fiberlaser mode-locking,” Optics Express 23, 19806 (2015).

[18] Xiantao Jiang, Simon Gross, Michael J. Withford, HanZhang, Dong-Il Yeom, Fabian Rotermund, and Alexan-der Fuerbach, “Low-dimensional nanomaterial saturableabsorbers for ultrashort-pulsed waveguide lasers,” Opti-cal Materials Express 8, 3055 (2018).

[19] A. L. Lentine and D. A. B. Miller, “Evolution of theSEED technology: Bistable logic gates to optoelectronicsmart pixels,” IEEE Journal of Quantum Electronics 29,655–669 (1993).

[20] Arka Majumdar and Armand Rundquist, “Cavity-enabled self-electro-optic bistability in silicon photonics,”Optics Letters 39, 3864 (2014).

[21] Alexander N. Tait, Thomas Ferreira de Lima, Mitchell A.Nahmias, Heidi B. Miller, Hsuan-Tung Peng, Bhavin J.Shastri, and Paul R. Prucnal, “Silicon Photonic Modula-tor Neuron,” Physical Review Applied 11, 064043 (2019).

[22] Edmondo Trentin, “Networks with trainable amplitudeof activation functions,” Neural Networks 14, 471–493(2001).

[23] Tyler W. Hughes, Momchil Minkov, Yu Shi, and Shan-hui Fan, “Training of photonic neural networks throughin situ backpropagation and gradient measurement,” Op-tica 5, 864–871 (2018).

[24] M. N. Ahmed, J. Chong, and D. S. Ha, “A 100 Gb/stransimpedance amplifier in 65 nm CMOS technologyfor optical communications,” in 2014 IEEE InternationalSymposium on Circuits and Systems (ISCAS) (2014) pp.1885–1888.

[25] K. T. Settaluri, C. Lalau-Keraly, E. Yablonovitch, andV. Stojanovic, “First Principles Optimization of Opto-Electronic Communication Links,” IEEE Transactions onCircuits and Systems I: Regular Papers 64, 1270–1283(2017).

[26] K. Nozaki, S. Matsuo, A. Shinya, and M. No-tomi, “Amplifier-Free Bias-Free Receiver Based on Low-Capacitance Nanophotodetector,” IEEE Journal of Se-lected Topics in Quantum Electronics 24, 1–11 (2018).

[27] Shankar Kumar Selvaraja, Peter De Heyn, Gustaf Win-roth, Patrick Ong, Guy Lepage, Celine Cailler, Ar-naud Rigny, Konstantin K. Bourdelle, Wim Bogaerts,Dries Van Thourhout, Joris Van Campenhout, andPhilippe Absil, “Highly uniform and low-loss passive sil-icon photonics devices using a 300mm CMOS platform,”in Optical Fiber Communication Conference (2014), Pa-per Th2A.33 (Optical Society of America, 2014) p.Th2A.33.

[28] Jaime Cardenas, Mark A. Foster, Nicolas Sherwood-Droz, Carl B. Poitras, Hugo L. R. Lira, Beibei Zhang,Alexander L. Gaeta, Jacob B. Khurgin, Paul Morton,and Michal Lipson, “Wide-bandwidth continuously tun-able optical delay line using silicon microring resonators,”Optics Express 18, 26525 (2010).

[29] Kengo Nozaki, Shinji Matsuo, Takuro Fujii, Koji Takeda,Akihiko Shinya, Eiichi Kuramochi, and Masaya Notomi,“Femtofarad optoelectronic integration demonstratingenergy-saving signal conversion and nonlinear functions,”Nature Photonics (2019), 10.1038/s41566-019-0397-3.

[30] G. Yu, X. Zou, L. Zhang, Q. Zou, M. zheng, andJ. Zhong, “A low-noise high-gain transimpedance ampli-fier with high dynamic range in 0.13ım CMOS,” in 2012IEEE International Symposium on Radio-Frequency In-tegration Technology (RFIT) (2012) pp. 37–40.

[31] Yichen Shen, Nicholas C. Harris, Scott Skirlo, MihikaPrabhu, Tom Baehr-Jones, Michael Hochberg, Xin Sun,Shijie Zhao, Hugo Larochelle, Dirk Englund, and MarinSoljacic, “Supplementary information: Deep learningwith coherent nanophotonic circuits,” Nature Photonics11, 441–446 (2017).

[32] C. Koos, L. Jacome, C. Poulton, J. Leuthold, andW. Freude, “Nonlinear silicon-on-insulator waveguidesfor all-optical signal processing,” Optics Express 15,5976–5990 (2007).

14

[33] “Neuroptica: An optical neural network simulator,”https://github.com/fancompute/neuroptica/.

[34] “Neurophox: A simulation framework for unitary neuralnetworks and photonic devices,” https://github.com/

solgaardlab/neurophox/.[35] Sunil Pai, Ben Bartlett, Olav Solgaard, and David

A. B. Miller, “Matrix Optimization on Universal UnitaryPhotonic Devices,” Physical Review Applied 11, 064044(2019).

[36] Martın Abadi, Ashish Agarwal, Paul Barham, EugeneBrevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado,Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghe-mawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving,Michael Isard, Yangqing Jia, Rafal Jozefowicz, LukaszKaiser, Manjunath Kudlur, Josh Levenberg, Dande-lion Mane, Rajat Monga, Sherry Moore, Derek Mur-ray, Chris Olah, Mike Schuster, Jonathon Shlens, BenoitSteiner, Ilya Sutskever, Kunal Talwar, Paul Tucker,Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas,Oriol Vinyals, Pete Warden, Martin Wattenberg, Mar-

tin Wicke, Yuan Yu, and Xiaoqiang Zheng, “Tensor-Flow: Large-Scale Machine Learning on HeterogeneousSystems,” (2015), software available from tensorflow.org.

[37] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner,“Gradient-based learning applied to document recogni-tion,” Proceedings of the IEEE 86, 2278–2324 (1998).

[38] Joseph W. Goodman, Introduction to Fourier Optics(Roberts and Company Publishers, 2005).

[39] H. Peng, M. A. Nahmias, T. F. de Lima, A. N. Tait,and B. J. Shastri, “Neuromorphic Photonic IntegratedCircuits,” IEEE Journal of Selected Topics in QuantumElectronics 24, 1–15 (2018).

[40] L. Larger, M. C. Soriano, D. Brunner, L. Appeltant, J. M.Gutierrez, L. Pesquera, C. R. Mirasso, and I. Fischer,“Photonic information processing beyond Turing: An op-toelectronic implementation of reservoir computing,” Op-tics Express 20, 3241 (2012).

[41] Francois Duport, Anteo Smerieri, Akram Akrout, MarcHaelterman, and Serge Massar, “Fully analogue photonicreservoir computer,” Scientific Reports 6, 22381 (2016).