Experimental quantum speed-up in reinforcement learning agents

V. Saggio1∗, B. E. Asenbeck1, A. Hamann2, T. Stromberg1, P. Schiansky1, V. Dunjko3, N. Friis4,

N. C. Harris5,6, M. Hochberg7, D. Englund6, S. Wolk2,8, H. J. Briegel2,9, and P. Walther1,10

1University of Vienna, Faculty of Physics, Vienna Center for QuantumScience and Technology (VCQ), Boltzmanngasse 5, Vienna A-1090, Austria

2Institut fur Theoretische Physik, Universitat Innsbruck, Technikerstraße 21a, 6020 Innsbruck, Austria3LIACS, Leiden University, Niels Bohrweg 1, 2333 CA Leiden, Netherlands4Institute for Quantum Optics and Quantum Information - IQOQI Vienna,Austrian Academy of Sciences, Boltzmanngasse 3, A-1090 Vienna, Austria

5Lightmatter, 60 State Street, Floor 10, Boston, MA, 02130, USA6Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA

7NOKIA Corporation, New York, NY, USA8Deutsches Zentrum fur Luft- und Raumfahrt e.V. (DLR),

Institut fur Quantentechnologien, Soflingerstr. 100, 89077 Ulm, Germany9Fachbereich Philosophie, Universitat Konstanz, Fach 17, 78457 Konstanz, Germany and

10Christian Doppler Laboratory for Photonic Quantum Computer,Faculty of Physics, University of Vienna, Vienna, Austria.

Increasing demand for algorithms that can learn quickly and efficiently has led to a surge ofdevelopment within the field of artificial intelligence (AI). An important paradigm within AI isreinforcement learning (RL), where agents interact with environments by exchanging signals via acommunication channel. Agents can learn by updating their behaviour based on obtained feedback.The crucial question for practical applications is how fast agents can learn to respond correctly.An essential figure of merit is therefore the learning time. While various works have made use ofquantum mechanics to speed up the agent’s decision-making process, a reduction in learning timehas not been demonstrated yet. Here we present a RL experiment where the learning of an agentis boosted by utilizing a quantum communication channel with the environment. We further showthat the combination with classical communication enables the evaluation of such an improvement,and additionally allows for optimal control of the learning progress. This novel scenario is thereforedemonstrated by considering hybrid agents, that alternate between rounds of quantum and classicalcommunication. We implement this learning protocol on a compact and fully tunable integratednanophotonic processor. The device interfaces with telecom-wavelength photons and features a fastactive feedback mechanism, allowing us to demonstrate the agent’s systematic quantum advantage ina setup that could be readily integrated within future large-scale quantum communication networks.

INTRODUCTION

Rapid advances in the field of machine learning (ML)and in general artificial intelligence (AI) have been pavingthe way towards intelligent algorithms and automation.An important paradigm within AI is reinforcement learn-ing (RL), where agents interact with an environment and‘learning’ is facilitated by feedback exchanges [1]. RL hasapplications in many sectors, from robotics (e.g. robotcontrol [2], text and speech recognition [3]) to the health-care domain (e.g. finding optimal treatment policies [4])to simulation of brain-like computing [5] and neural net-works [6, 7]. RL is also at the heart of the celebratedAlphaGo algorithm [8], able to beat even the most skilledhuman players at the game of Go.

At the same time, thanks to the capability of quantumphysics to outperform classical algorithms, the develop-ment of quantum technologies has been experiencing re-markable progress [9]. Quantum physics offers improvedperformance in a variety of applications, from quantum-enhanced sensing [10] to secure communication [11] andquantum information processing [12]. Quantum mechan-ics also inspires new advantageous algorithms in ML, and

in particular RL [13, 14]. In fact, RL has been suc-cessfully implemented to aid in problems encounteredin quantum information processing itself, e.g. decod-ing of errors [15–17], quantum feedback [18], adaptivecode-design [19], and even the design of new quantum ex-periments [20, 21]. Conversely, quantum technology hasbeen employed to enable a quadratically faster decision-making process for RL agents via the quantization oftheir internal hardware [22–25].

In all of these RL applications, the interaction betweenthe agent and the environment is performed via entirelyclassical communication. Here we consider a novel RLsetting where the agent and the environment can alsocommunicate via a quantum channel [26]. In particular,we introduce a hybrid agent that enables quantum aswell as classical information transfer, and thus combinesquantum amplitude amplification with a classical updatepolicy via a feedback loop. Within such a scenario, it ispossible for the first time to quantify and achieve a quan-tum speed-up in the agent’s learning time with respect toRL schemes based on classical communication only. Thelearning time is defined as the average number of inter-actions until the agent receives a reward (‘accomplishes

arX

iv:2

103.

0629

4v1

[qu

ant-

ph]

10

Mar

202

1

2

Percept si

Reward r

Action ai

ENVIRONMENTAGENT

(a)

(b)

(c)

ENVIRONMENT

ENVIRONMENT

AGENT

AGENT

CLASSICAL CHANNEL

QUANTUM CHANNEL

si r

ai

|si> d |r>

|ai>

FIG. 1: Schematic of a learning agent. (a) An agentinteracts with an environment by receiving perceptual inputsi and outputting consequent actions ai. In the case of thecorrect ai being performed, the environment issues a rewardr that the agent uses to enhance its performance in the nextround of interaction. (b) Agent and environment interactingvia a classical channel, where communication is only possiblevia a fixed preferred basis (e.g. ‘vertical’ or ‘horizontal’). (c)Agent and environment interacting via a quantum channel,where arbitrary superposition states are exchanged.

a specific task’). Such improvement represents a remark-able advantage over previously implemented protocols,and may prove crucial in the development of increasinglycomplex learning devices [5–7].

We demonstrate this protocol using a fully-programmable nanophotonic processor interfacedwith photons at telecom wavelengths. The setup enablesthe implementation of an active feedback mechanism,thus proving suitable for demonstrations of RL algo-rithms. Moreover, such a photonic platform holds thepotential of being fully interfaceable with future quan-tum communication networks thanks to the photons’telecom wavelengths. In fact, a long-standing goal ofthe current development of quantum communicationtechnologies lies in establishing a form of ‘quantuminternet’ [27, 28] — a highly interconnected networkable to distribute and manipulate complex quantumstates via fibres and optical links (ground-based or evenvia satellites). We therefore envisage AI and RL toplay an important role in future large-scale quantumcommunication networks, including a potential quantuminternet, much in the same way that AI forms integralparts of the internet today. Our results thus additionallydemonstrate the feasibility of integrating quantummechanical RL speed-ups in future complex quantumnetworks.

QUANTUM ENHANCEMENT INREINFORCEMENT LEARNING

The conceptual idea of RL is shown in Fig. 1(a). Herean agent (e.g. physical or web-based) interacts with anenvironment by receiving perceptual input, called ‘per-cepts’ si, and outputting specific ‘actions’ ai accordingly.‘Rewards’ r issued by the environment for correct com-binations of percepts and actions incentivize agents toimprove their decision-making, and thus to learn [1].

Although RL has already been shown amenable toquantum enhancements, in practical demonstrations theinteraction between agents and environments has so farbeen restricted exclusively to classical communication,meaning that agents are limited to exchanges in a fixedpreferred basis, e.g. ‘vertical’ or ‘horizontal’ as shown inFig. 1(b). In general, it has been shown that granting anagent access to quantum hardware (while still restrictingit to classical exchanges with the environment) does notreduce the average number of interactions that the agentneeds to accomplish its task, although it allows it to out-put actions quadratically faster [22, 25]. To achieve areduction in learning times, and thus in sample complex-ity, fully quantum-mechanical interactions between agentand environment must be considered.

Our goal is to experimentally demonstrate such aquantum-enhanced RL scenario. We start with consid-ering an agent and environment with access to internalquantum (as well as classical) hardware that can interactby coherently exchanging quantum states |ai〉, |si〉, and|r〉, representing actions ai, percepts si, and rewards r,respectively, via a joint quantum channel, as illustratedin Fig. 1(c). In this case, communication is no longer lim-ited to a fixed preferred basis, but allows for an exchangeof arbitrary superpositions. Agents react to (sequencesof) percepts |si−1〉 with (sequences of) actions |ai〉 ac-cording to a policy π(ai|si−1) that is updated during thelearning process. In particular, this quantum frameworkincludes so-called deterministic strictly epochal (DSE)learning scenarios, which we focus on. Here, ‘epochs’consist of strings of percepts ~s = (s0, · · · , sL−1) and ac-tions ~a = (a1, · · · , aL) of fixed length L, and a final re-ward r, and both ~s = ~s(~a) and r = r(~a) are completelydetermined by ~a (see Methods, Sec. I). Many interestingenvironments are epochal, e.g. in applications of RL toquantum physics [21, 29–31] or popular problems such asplaying Go [8]. A non-trivial feature of the DSE scenariois that the effective behaviour of the environment canbe modelled via a unitary UE on the action and rewardquantum registers (indicated with subscripts A and R,respectively):

UE |~a〉A|0〉R =

{|~a〉A|1〉R if r(~a) > 0|~a〉A|0〉R else

, (1)

which identifies rewarded sequences of actions. Thequantum-enhanced agent can use UE to perform a quan-tum search for rewarded action sequences and teach thefound sequences to a classical agent [26]. We push this

3

θ φ(b)

D0

D1D2 / D3

Grover implementation

H n

HOracle Reflection

Oracle qubit

Target qubits

n

Classical control TTM|0

|1

50:50 splitting ratio

(a) (c)Single-photon

source

Nanophotonic processor

FIG. 2: Experimental setup. (a) Single programmable unit consisting of a Mach-Zehnder interferometer (MZI) equippedwith two fully tunable phase shifters θ and φ. (b) Real-life picture of a single MZI in the processor. The third phase shifter inthe bottom arm of the interferometer is not used. (c) Overview of the setup. A single-photon source generates single-photonpairs at telecom wavelength. One photon is sent straight to a single-photon detector D0, while the other one is coupled into theprocessor and undergoes the desired computation. It is then detected, in coincidence with the photon in D0, either in detectorD1 or D2/D3 after the agent plays the classical/quantum strategy (refer to Fig. 3 for more details). The coincidence events arerecorded with a custom-made Time Tagging Module (TTM). Different areas of the processor are assigned to either the agentor the environment. The former one also comprises a classical control updating its policy. They alternately play a Grover-likesearch to look for the rewarded sequence of actions.

idea further and grant the quantum and the classical partof the agent complete access to each other. In this way,it is possible to create a hybrid agent with a feedbackloop between quantum search and classical policy up-date. This approach allows us to quantify the speed-upin learning time, which is not possible in the general set-ting discussed in [26] (see Methods, Sec. I).

It is known that near-term quantum devices are gen-erally subject to limited coherence times, restricting thenumber of quantum gates that can be performed beforeessential quantum features are lost. However, already afew or even a single coherent query to UE are enough toconstruct a quantum-enhanced RL framework that canoutperform a classical one.

In a classical epoch, the agent prepares the state|~a〉A|0〉R where ~a is determined by directly sampling froma classical probability distribution p(~a). From classicallyinteracting with the environment, the agent determinesthe corresponding reward r(~a) and percepts ~s(~a), anduses the obtained information to update its policy π, thuslearning.

In a quantum epoch, the agent prepares the state|ψ〉A|−〉R, with |ψ〉 =

∑~a

√p(~a) |~a〉 being determined

according to p(~a) dictated by its current classical pol-

icy π, and |−〉 = (|0〉 − |1〉)/√

2. Both states are sentto the environment, which acts on them via UE induc-ing a phase of −1 for all rewarded sequences of actions.The states are sent back to the agent which performs areflection URef = (2|ψ〉〈ψ|A − 1A) over the initial su-perposition state. This leads to an amplitude amplifica-tion of rewarded action sequences similar to the Grover’salgorithm [32]. Parametrising the classical probabilityto find a reward via sin2(ξ) with ξ ∈ [0, 2π], the stepsabove followed by a final measurement on the action reg-

ister result with an increased probability sin2(3ξ) in arewarded action sequence. However, this procedure doesnot reveal the reward or the corresponding sequence ofpercepts. These can be determined only during classicaltest epochs, where the measured sequence of actions isused as input state.

The quantum-enhanced agent plays in a hybrid wayby alternating between these quantum and classical testepochs. Such agents find rewarded sequences of actionsfaster, and hence usually learn faster than entirely classi-cal agents. The learning speed-up manifests in a reducedaverage quantum learning time 〈T 〉Q, that is, the aver-age number of epochs necessary to achieve a predefinedwinning probability QL. In general, a quadratic improve-ment in the learning time given by 〈T 〉Q ≤ α

√J〈T 〉C for

the hybrid agent can be achieved if the maximal numberof coherent interactions between the agent and the envi-ronment scales with the problem size. Here, J quantifiesthe number of rewards the agent needs to find in order tolearn and 〈T 〉C the average classical learning time. How-ever, already a single (coherent) query to UE is enoughto improve the learning time albeit by a constant factor,as demonstrated in the Methods, Sec. I.

EXPERIMENTAL IMPLEMENTATION

Quantized RL protocols can be compactly realized us-ing state-of-the-art photonic technology [33]. In partic-ular the path towards miniaturization of photonic plat-forms holds the advantage of providing scalable architec-tures where many elementary components can be accom-modated on small devices. Here we use a programmablenanophotonic processor with dimensions of 4.9 x 2.4 mm,comprising 26 waveguides fabricated to form 88 Mach-

4

Zehnder interferometers (MZIs) in a trapezoidal config-uration. A quantum gate is implemented by a MZIequipped with two thermo-optic phase shifters, one in-ternal to allow for a scan of the output distribution overθ ∈ [0, 2π] and one external dictating the relative phaseφ ∈ [0, 2π] between the two output modes, as shown inFig. 2(a). This makes each MZI act as a fully-tunablebeam splitter and allows for coherent implementation ofsequences of quantum gates. Information is spatiallyencoded onto two orthogonal modes |0〉 = (1, 0)T and|1〉 = (0, 1)T, which constitute the computational basis.Fig. 2(b) shows how a MZI looks like in our integrateddevice.

Pairs of single photons are generated in the telecomwavelength band from a single-photon source pumpedby laser light at 789.5 nm. One photon is coupled intoone waveguide in the processor to perform a particularcomputation, while the other one is sent straight to asingle-photon detector D0 for heralding. The detectorsare superconducting nanowires that combine extremelyhigh efficiency (up to ∼90%) with a very short deadtime (< 100 ns), thus proving the best candidates forfast feedback at telecom wavelengths. Detection eventsrecorded in D0 and at the output of the processor fallinginto a temporal window of 1.3 ns are registered with atime tagging module (TTM) as coincidence events (seeMethods, Sec. II for more details). Different areas ofthe processor are assigned to the agent and the environ-ment, who alternately perform the aforementioned stepsof the Grover-like amplitude amplification. The agent isfurther equipped with a classical control mechanism thatupdates its learning policy. All of this can be seen inFig. 2(c).

In our experiment, we represent all possible sequencesof actions by a single qubit (|1〉A= rewarded, |0〉A= notrewarded) and use another qubit to encode the reward(|1〉R, |0〉R). This results in a four-level system whereeach level is a waveguide path in our processor, as shownin Fig. 3 (which illustrates only the part of the processorneeded for our computation).

The probability for the agent to choose a correct se-quence of actions is initially set to sin2(ξ) = ε = 0.01,representing, for instance, a single rewarded sequence ofactions out of 100. After a single photon is coupled intothe mode |0A0R〉, the agent puts it into the superpositionstate |ψ〉A = (cos ξ|0〉A + sin ξ|1〉A)|0〉R via the unitaryUp. Next, it can decide to play classically or quantum-mechanically.

Classical strategy. The environment flips the rewardqubit only if the action qubit is in the correct state viathe unitary UE , as shown in Fig. 3(a). Next, the pho-ton is coupled out and detected in either D1 or D2 withprobability cos2(ξ) and sin2(ξ), respectively. Only if D2is triggered (i.e. the agent has been rewarded), a classi-cal feedback mechanism updates the policy π, and thusp(~a) and ε, via rules given by the projective simulationframework (see Methods, Sec. I).

Quantum strategy. Here, the theoretical framework de-

Classical circuit

Quantum circuit

AGENT

UpUH0

UH1

UH0

UH1-1

-1

URef

UpUE

UE

D1

D2

D1D3

|0A1R

|0A0R

|1A0R

|1A1R

|0A1R

|0A0R

|1A0R

|1A1R

(a)

(b)

AGENTENVIRONMENT

AGENT AGENTENVIRONMENT

FIG. 3: Circuit implementation. One photon is coupledinto the |0A0R〉 spatial mode and undergoes different unitariesdepending on whether a classical (a) or a quantum (b) epochis implemented. The waveguides highlighted in yellow showthe photon’s possible paths. Identity gates are representedwith straight waveguides.

scribed above is exploited to speed up the learning pro-cess. As illustrated in Fig. 3(b), after the reward qubitis rotated to |−〉R via the unitaries UH0 and UH1, theenvironment acts as an oracle via UE . Consecutively, theagent reverses the effect of UH0 and UH1, and then it per-forms the reflection URef . Next, a measurement in thecomputational basis of the action register is performed,leading to detection of a rewarded sequence of actions inD3 with increased probability sin2(3ξ). The classical testepoch needed to determine the reward is not experimen-tally implemented for practical reasons, since the circuitvisibility is bigger than 0.99.

In general, any Grover-like algorithm faces a drop inthe amplitude amplification after the optimal point isreached. Keeping in mind that each agent will reachthis optimal point at different epochs, one can identifythe probability ε = 0.396 (see Methods, Sec. I) untilwhich it is beneficial for all agents to use a quantumstrategy, as on average they will observe more rewardsthan with the classical strategy. As soon as this prob-ability is surpassed, it is advantageous for the agentsto switch to an entirely classical strategy. This com-bined quantum-classical strategy thus avoids the winningprobability drop without introducing any additional over-heads in terms of experimental resources.

RESULTS

Here, we show the experimental comparison betweenquantum and classical strategies.

At the end of each classical epoch, we record outcomes1 and 0 for the rewarded and not rewarded behaviour, ob-taining a binary sequence whose length equals the num-ber of played epochs for the classical learning strategy,and half of the number of epochs for the quantum strat-egy. To enable a fair comparison between the quantumand classical scenarios, the reward in the quantum caseis distributed — i.e. averaged — over the two epochsneeded to determine a test sequence of actions ~a and its

5

0.0

0.2

0.4

0.6

0.8

1.0

η(QL=0.37)=0.39

Non-distributed case

Distributed caseη

0

10

1

0 20 40 60

0 100 200 300 400 500Even epochs

Even epochs0 200 400

All epochs

(a)

(d)

(b)

(c)

0.0

0.2

0.4

0.6

0.8

1.0

TheoryExperiment

Quantum strategy

Classical strategy

Combined strategy

Classical strategy

η

TheoryExperiment

η

η

0 60

T aQ2 =50

FIG. 4: Behaviour of the average reward η for different learning strategies. The solid line represents the theoreticaldata simulated with n = 10, 000 agents, while the dots represent the experimental data measured with n = 165 agents. Theshaded regions indicate the errors associated to each single data point. (a) η of agents playing a quantum (blue) or classical(orange) strategy. (b) η accounting, in the quantum case, for rewards obtained only every second epoch, compared to (c) thecase where the reward is distributed over the two epochs an agent needs to acquire it. (d) Comparison between the classicaland combined quantum-classical case, where an advantage over the classical case is visible. Here, the agents stop the quantumstrategy at their best performance (at ε = 0.396) and continue playing classically. The inset shows the point at which the agentplaying the quantum strategy reaches the predefined winning probability QL = 0.37, after 〈T 〉Q = 100 epochs.

reward. The reward is then averaged over many differ-ent agents playing independently of one another. Fig. 4shows the average reward η for the different learningstrategies.

The theoretical data is simulated for n = 10, 000agents and the experimental data obtained from n = 165.Fig. 4(a) visualizes the quantum improvement originat-ing from the use of amplitude amplification in compar-ison with a purely classical strategy. For completeness,we also show in Fig. 4(b) and (c) the comparison be-tween not distributing and distributing the reward overtwo epochs in the quantum case.

As soon as ε = 0.396 is reached, η starts decreas-ing. Our setup allows the agents to always choose thefavorable strategy by switching from quantum to classi-cal as soon as the second becomes more advantageous.In this way, such combined strategy always outperformsthe purely classical scenario, as shown in Fig. 4(d).

We define QL = 0.37 as the winning probability. Notehowever that any probability below ε = 0.396 can bedefined as the winning probability QL. The learning time〈T 〉 for QL decreases from 〈T 〉C = 270 in the classicalcase to 〈T 〉Q = 100 in the combined quantum-classicalcase. This implies a reduction of 63%, which fits well to

the theoretical values T theoryC = 293 and T theoryQ = 97,taking into account small experimental imperfections.

As the combined configuration prevents the averagereward from dropping, it is particularly relevant for real-life applications when a Grover search (with its intrin-sic overshooting drawback) is implemented. An averagereward that saturates at a high niveau without a sub-sequent drop can also be achieved by employing otheralgorithms like the fixed-point algorithm [34]. However,they show a less favorable speed-up than Grover-like am-plitude amplification, especially considering the limitedsize of our integrated processor.

6

In general, an agent can experience a quadratic speed-up in its learning time if it can perform an arbitrary num-ber of coherent Grover iterations [35] even if the numberof actual rewarded sequences is unknown [36].

CONCLUSIONS

We have demonstrated a novel RL protocol where in-formation is alternately communicated via a quantumand a classical channel. This makes it possible to eval-uate the agent’s performance resulting in a speed-up inthe learning time, and gain optimal control of the learn-ing process. Learning agents relying on purely classicalcommunication are therefore outperformed. At the sametime, emerging photonic circuit technology provides theadvantages of compactness, full tunability and low-losscommunication, thus proving suitable for RL algorithms

where active feedback mechanisms, even over long dis-tances, need to be implemented.

We highlight that although photonic architecturesprove particularly suitable for these types of learningalgorithms, the presented demonstration is based on anovel theoretical background that is general and appli-cable to any quantum platform. Moreover, as the fieldof integrated optics is moving towards the fabricationof increasingly large devices, this demonstration couldbe extended to more complex quantum circuits allowingfor processing of high-dimensional states, and thuspaving the way for achieving superior performance inincreasingly complex learning devices.

∗Corresponding author: valeria.saggio@univie.ac.at.

[1] Sutton, R. S. & Barto, A. G., Reinforcement Learning:An Introduction (MIT press, Cambridge, 1998).

[2] Johannink, T. et al. Residual Reinforcement Learn-ing for Robot Control. In 2019 International Con-ference on Robotics and Automation (ICRA), Mon-treal, QC, Canada, 6023-6029 (IEEE, 2019). URLhttps://doi.org/10.1109/ICRA.2019.8794127.

[3] Tjandra, A., Sakti, S. & Nakamura, S. Sequence-to-Sequence ASR Optimization via Reinforcement Learn-ing. In 2018 IEEE International Conference on Acous-tics, Speech and Signal Processing (ICASSP), Cal-gary, AB, Canada, 5829-5833 (IEEE, 2018). URLhttps://doi.org/10.1109/ICASSP.2018.8461705.

[4] Komorowski, M., Celi, L. A., Badawi, O., Gordon,A. C. & Faisal A. A. The artificial intelligence clin-ician learns optimal treatment strategies for sepsis inintensive care. Nat. Med. 24, 1716-1720 (2018). URLhttp://doi.org/10.1038/s41591-018-0213-5.

[5] Thakur, C. S. et al. Large-scale neuromorphic spik-ing array processors: A quest to mimic the brain.Frontiers in neuroscience 12, 891 (2018). URLhttps://doi.org/10.3389/fnins.2018.00891.

[6] Shen, Y. et al. Deep learning with coherent nanopho-tonic circuits. Nat. Photon. 11, 441 (2017). URLhttps://doi.org/10.1038/nphoton.2017.93.

[7] Steinbrecher, G. R., Olson, J. P., Englund, D.& Carolan, J. Quantum optical neural networks.npj Quantum Information 5, 1-9 (2019). URLhttps://doi.org/10.1038/s41534-019-0174-7.

[8] Silver, D. et al. Mastering the game of Go withouthuman knowledge. Nature 550, 354-359 (2017). URLhttp://dx.doi.org/10.1038/nature24270.

[9] Arute, F. et al. Quantum supremacy using a pro-grammable superconducting processor. Nature 574, 505-510 (2019). URL https://doi.org/10.1038/s41586-019-1666-5.

[10] Giovannetti, V., Lloyd, S. & Maccone, L. Advances inquantum metrology. Nat. Photon. 5, 222 (2011). URLhttps://doi.org/10.1038/nphoton.2011.35.

[11] Sergienko, A. V. (ed.) Quantum Communications andCryptography (CRC press, Boca Raton, 2018). URLhttps://doi.org/10.1201/9781315221120.

[12] Monroe, C. Quantum information processing withatoms and photons. Nature 416, 238-246 (2002). URLhttps://doi.org/10.1038/416238a.

[13] Dong. D., Chen, C., Li, H & Tarn, T.-J.Quantum Reinforcement Learning. IEEE T.Syst, Man Cy. B 38, 1207-1220 (2008). URLhttps://doi.org/10.1109/TSMCB.2008.925743.

[14] Dunjko, V. & Briegel, H. J. Machine learning & arti-ficial intelligence in the quantum domain: a review ofrecent progress. Rep. Progr. Phys. 81, 074001 (2018).URL https://doi.org/10.1088/1361-6633/aab406.

[15] Baireuther, P., O’Brien, T. E., Tarasinski, B. &Beenakker, C. W. J. Machine-learning-assisted correc-tion of correlated qubit errors in a topological code.Quantum 2, 48 (2018). URL https://doi.org/10.22331/q-2018-01-29-48.

[16] Breuckmann, N. P. & Ni, X. Scalable Neural Network De-coders for Higher Dimensional Quantum Codes. Quan-tum 2, 68-92 (2018). URL https://doi.org/10.22331/q-2018-05-24-68.

[17] Chamberland, C. & Ronagh, P. Deep neural decoders fornear term fault-tolerant experiments. Quant. Sci. Techn.3, 044002 (2018). URL https://doi.org/10.1088/2058-9565/aad1f7.

[18] Fosel, T., Tighineanu, P., Weiss, T. & Marquardt, F.Reinforcement Learning with Neural Networks for Quan-tum Feedback. Phys. Rev. X 8, 031084 (2018). URLhttps://doi.org/10.1103.PhysRevX.8.031084.

[19] Poulsen Nautrup, H., Delfosse, N., Dunjko, V., Briegel,H. J. & Friis, N. Optimizing quantum error correc-tion codes with reinforcement learning. Quantum 3,215 (2019). URL https://doi.org/10.22331/q-2019-12-16-215.

[20] Krenn, M., Malik, M., Fickler, R., Lapkiewicz, R. &Zeilinger, A. Automated Search for new Quantum Ex-periments. Phys. Rev. Lett. 116, 090405 (2016). URLhttp://doi.org/10.1103/PhysRevLett.116.090405.

[21] Melnikov, A. A. et al. Active learning machinelearns to create new quantum experiments. Proc.Natl. Acad. Sci. U.S.A. 115, 1221-1226 (2018). URLhttp://dx.doi.org/10.1073/pnas.1714936115.

[22] Paparo, G. D., Dunjiko, V., Makmal, A., Martin-

7

Delgrado, M. A. & Briegel, H. J. Quantum Speedup forActive Learning Agents. Phys. Rev. X 4, 031002 (2014).URL http://dx.doi.org/10.1103/PhysRevX.4.031002.

[23] Dunjko, V., Friis, N. & Briegel, H. J. Quantum-enhanced deliberation of learning agents usingtrapped ions. New J. Phys. 17, 023006 (2015). URLhttps://doi.org/10.1088/1367-2630/17/2/023006.

[24] Jerbi, S., Poulsen Nautrup, H., Trenkwalder, L. M.,Briegel, H. J. & Dunjko, V. A framework for deep energy-based reinforcement learning with quantum speed-up(2019). Preprint at https://arxiv.org/abs/1910.12760.

[25] Sriarunothai, T. et al. Speeding-up the decision mak-ing of a learning agent using an ion trap quantum pro-cessor. Quantum Sci. Technol. 4, 015014 (2019). URLhttp://dx.doi.org/10.1088/2058-9565/aaef5e.

[26] Dunjko, V., Taylor, J. M., & Briegel, H.J. Quantum-Enhanced Machine Learning.Phys. Rev. Lett. 117, 130501 (2016). URLhttp://dx.doi.org/10.1103/PhysRevLett.117.130501.

[27] Kimble, H. J. The quantum internet. Nature 453, 1023-1030 (2008). URL https://doi.org/10.1038/nature07127.

[28] Cacciapuoti, A. S. et al. Quantum Internet: Net-working Challenges in Distributed Quantum Com-puting. IEEE Network 34, 137-143 (2020). URLhttps://doi.org/10.1109/MNET.001.1900092.

[29] Denil, M. et al. Learning to Perform Physics Experi-ments via Deep Reinforcement Learning (2016). Preprintat https://arxiv.org/abs/1611.01843.

[30] Bukov, M. et al. Reinforcement Learn-ing in Different Phases of Quantum Con-trol. Phys. Rev. X 8, 031086 (2018). URLhttp://dx.doi.org/10.1103/PhysRevX.8.031086.

[31] Poulsen Nautrup, H. et al. Operationally meaningfulrepresentations of physical systems in neural networks(2020). Preprint at https://arxiv.org/abs/2001.00593.

[32] Grover, L. K. Quantum mechanics helps in searching fora needle in a haystack. Phys. Rev. Lett. 79, 325 (1997).URL https://doi.org/10.1103/PhysRevLett.79.325.

[33] Flamini, F. et al. Photonic architecture for reinforce-ment learning. New. J. Phys. 22, 045002 (2020). URLhttps://doi.org/10.1088/1367-2630/ab783c.

[34] Yoder, T. J., Low, G. H. & Chuang, I. L. Fixed-Point Quantum Search with an Optimal Number ofQueries. Phys. Rev. Lett. 113, 210501 (2014). URLhttps://doi.org/10.1103/PhysRevLett.113.210501.

[35] Hamann, A. et al. A hybrid agent for quantum-accessiblereinforcement learning (2020). In preparation.

[36] Boyer, M., Brassard, G., Hoyer, P. & Tappa, A. Tightbounds on quantum searching. Fortschr. Phys. 46, 493(1998). URL https://doi.org/10.1002/3527603093.ch10.

[37] Briegel, H. J. & De las Cuevas, G. Projective simulationfor artificial intelligence. Sci. Rep. 2, 1-16 (2012). URLhttp://doi.org/10.1038/srep00400.

[38] Kim, T., Fiorentino, M. & Wong, F. N. C.Phase-stable source of polarization-entangledphotons using a polarization Sagnac interfer-ometer. Phys. Rev. A 73, 012316 (2006). URLhttp://doi.org/10.1103/PhysRevA.73.012316.

[39] Saggio, V. et al. Experimental few-copy multipartite en-tanglement detection. Nat. Phys. 15, 935-940 (2019).URL http://doi.org/10.1038/s41567-019-0550-4.

[40] Harris, N. C. et al. Quantum transport sim-ulations in a programmable nanophotonic pro-cessor. Nat. Photon. 11, 447-452 (2017). URL

http://doi.org/10.1038/nphoton.2017.95.[41] Marsili, F. et al. Detecting single infrared photons with

93% system efficiency. Nat. photon. 7, 210-214 (2013).URL https://doi.org/10.1038/nphotonc.2013.13.

Acknowledgments: The authors thank Lee A.Rozema, Irati Alonso Calafell and Philipp Jenke for helpwith the detectors. A.H. acknowledges support from theAustrian Science Fund (FWF) through the project P30937-N27. V.D. acknowledges support from the DutchResearch Council (NWO/ OCW), as part of the Quan-tum Software Consortium programme (project number024.003.037). N.F. acknowledges support from the Aus-trian Science Fund (FWF) through the project P 31339-N27. H.J.B. acknowledges support from the AustrianScience Fund (FWF) through SFB BeyondC F7102, theMinisterium fur Wissenschaft, Forschung, und KunstBaden-Wurttemberg (Az. 33-7533-30-10/41/1) and theVolkswagen Foundation (Az. 97721). P.W. acknowl-edges support from the research platform TURIS, theEuropean Commission through ErBeStA (no. 800942),HiPhoP (no. 731473), UNIQORN (no. 820474),EPIQUS (no. 899368), and AppQInfo (no. 956071),from the Austrian Science Fund (FWF) through CoQuS(W1210-N25), BeyondC (F 7113) and Research Group(FG 5), and Red Bull GmbH. The MIT portion of thework was supported in part by AFOSR award FA9550-16-1-0391 and NTT Research. Author contributions:V.S. and B.E.A. implemented the experiment and per-formed data analysis. A.H., V.D., N.F., S.W. and H.J.B.developed the theoretical idea. T.S. and P.S. providedhelp with the experimental implementation. N.C.H.,M.H., and D.E. designed the nanophotonic processor.V.S., S.W., and P.W. supervised the project. All theauthors contributed to writing the paper. Competinginterests: The authors declare they have no compet-ing interests. Data and material availability: Datarelated to this work may be available upon request.

8

METHODS

I. Quantum-enhanced learning agents

In this section, we develop an explicit method for com-bining a classical reinforcement learning agent with quan-tum amplitude amplification. Our approach for such hy-brid agents goes beyond the ideas of Ref. [26] by intro-ducing a feedback loop between classical policy updateand quantum amplitude amplification. The developedmodel allows us to determine achievable improvementsin sample complexity, and thus learning time. In addi-tion, the final policy of our agent has similar propertiesas the policy of the underlying classical agent, leading toa comparable behavior as discussed in more detail in [35](a paper dedicated to discuss the theoretical backgroundof the hybrid agent presented here more specifically).

In the following, we concentrate on simple determinis-tic, strictly epochal (DSE) environments. That is, the in-teraction between the agent and the environment is struc-tured into epochs where each epoch starts with the samepercept s0, and at each time step i an action-perceptpair (ai, si) is exchanged. At the end of each epoch, af-ter L action-percept pairs are communicated, a rewardr ∈ {0, 1} is given to the agent. The rules of the gameare deterministic and time independent, such that per-forming a specific action ai after receiving a percept si−1always leads to the same following percept si.

The behaviour of an agent is determined by its pol-icy described by the probability π(ai|si−1) to performthe action ai given the percept si−1. In determinis-tic settings, the percept si is completely determinedby all previous performed actions a1, · · · , ai such thatπ(ai|si−1) = π(ai|a1, · · · , ai−1). Thus, the behaviour ofthe agent within one epoch is described by sequences ofactions ~a = (a1, · · · , aL) and their corresponding proba-bilities

p(~a) =

L∏i=1

π(ai|a1, · · · , ai−1). (A.1)

The learning agent implemented here uses a policy basedon projective simulation [37], where each sequence of ac-tions ~a is associated with a weight factor h(~a) initializedto h = 1. Its policy is defined via the probability distri-bution

p(~a) =h(~a)∑

~a′h(~a′)

. (A.2)

If the agent has played the sequence ~a, it updates thecorresponding weight factor via

h(~a) → h(~a) + λr(~a), (A.3)

where λ = 2 in our experiment. In general, the updatemethod for quantum-enhanced agents is not limited toprojective simulation and can be used to enhance any

classical learning scenario, provided that p(~a) exists andthat the update rule is solely based on the observed re-wards.

We generalize the given learning problem to the quan-tum domain by encoding different sequences of actions ~ainto orthogonal quantum states |~a〉 defining our computa-tional basis. In addition, we create a fair unitary oracularvariant UE of the environment [26], whose behaviour isdescribed by

UE |~a〉 =

{|~a〉 if r(~a) = 0−|~a〉 else

. (A.4)

The unitary oracle UE can be used to perform, for in-stance, a Grover search or amplitude amplification forrewarded sequences of actions by performing Grover it-erations

UG =(

2|ψ〉〈ψ| − 1)UE (A.5)

on an initial state |ψ〉. A quantum-enhanced agent with

access to UE can thus find rewarded sequences of actionsfaster than a corresponding classical agent, defined by thesame initial policy π(ai|si−1) and update rules, without

access to UE .In general, the optimal number k of Grover iterations

UkG|ψ〉 depends on the winning probability

q =∑

{~a|r(~a)6=0}

p(~a) (A.6)

via k ∼ 1/√q [32]. In the following, we assume that q

is known at least to a good approximation. This is forinstance possible if the number of winning sequences |~a〉is known. However, a similar quantum-enhanced learningagent can be also developed if q is unknown by adaptingmethods from [36] as described in [35].

I.1. Description of the agent

A hybrid agent that learns faster than a classical onecan be constructed by alternating between quantum am-plitude amplification and classical policy update by re-peatedly performing the following steps:

1. Given the classical probability distribution p(~a),determine the success probability q based on thecurrent policy and prepare the quantum state

|ψ〉 =∑{~a}

√p(~a)|~a〉. (A.7)

2. Apply the optimal number of Grover iterationk(√q) leading to

|ψ′〉 = UkG|ψ〉 (A.8)

and perform a measurement on |ψ′〉 in the com-putational basis to determine a test sequence ofactions ~a.

9

3. Play one classical epoch by using the test sequence~a determined in step 2 and record the correspond-ing sequence of percepts ~s(~a) and the reward r(~a).

4. Update the classical policy π(ak|sk−1) and the re-sulting probability distribution p(~a) according toA.3.

There exists a limit Q on q determining whether it ismore advantageous for the agent to perform a Groveriteration with k(

√q) ≥ 1 or sample directly from p(~a)

(therefore k(√q) = 0) to determine ~a. In the latter case

the agent would only interact classically (as in step 3)with the environment.

After each epoch, a classical agent receives a rewardwith probability q = ε = sin2(ξ). We assume that theagent can use one epoch to either perform one Groveriteration (step 2) or to determine the reward of a giventest sequence ~a (step 3). Thus, for k = 1 it receives afterevery second epoch a reward with probability sin2(3ξ).As a consequence, we define the expected average rewardof an agent playing a classical strategy as ηC = 2 sin2(ξ)and of an agent playing a quantum strategy with k = 1as ηQ = sin2(3ξ). For q < Q, ηQ > ηC , meaning that thequantum strategy proves advantageous over the classicalcase. However, as soon as ηC = ηQ (at Q = 0.396),a classical agent starts outperforming a correspondingquantum agent which still performs Grover iterations.

Determining the winning probability q exactly suchthat q = ε as in the example presented here is not al-ways possible. In general, additional information like thenumber of possible solutions and model building helpsto perform this task. Note that a Q smaller than 0.396should be chosen if q can only be estimated up to somerange. To circumvent this problem, methods like Groversearch with unknown reward probability [36], or fixed-point search [34], can be used to determine if and howmany steps of amplitude amplification should be per-formed [35].

I.2. Learning time

We define the learning time T as the number of epochsan agent needs on average to reach a predefined winningprobability QL. The above described quantum-enhancedagent can reach the probability Q on average with lessepochs than its classical counterpart. However, once bothreach Q, they need on average the same number of epochsto reach Q + ∆Q with 0 ≤ ∆Q < 1 − Q. Therefore,we choose QL ≤ Q in order to quantify the achievableimprovement of a hybrid agent compared to its classicalcounterpart. In our experiment, we choose QL = 0.37 todefine the learning time.

Let `J = {~a1, · · · ,~aJ} be a time-ordered list of all therewarded sequences of actions an agent has found untilit reaches QL. Note that the actual policy πj , and thuspj , of our agents depend only on the list `j of observedrewarded sequences of actions, and this is independent

of whether it has found them via classical sampling orquantum amplitude amplification. As a result, a classicalagent and its quantum-enhanced version are describedby the same policy π(`j) and behave similar if they havefound the same rewarded sequences of actions. However,the quantum-enhanced agent finds them faster.

In general, the actual policy and overall success prob-ability might depend on the found rewarded action se-quence. Thus the number J of observed rewarded actionsequences necessary to learn might vary. However, thisis not the case for the here reported experiment. Thelearning time in this case can be determined via

T (J) =

J∑j=1

tj (A.9)

where tj determines the number of epochs necessary forthe agent to find the next rewarded sequence ~aj after ithas observed j − 1 rewards. For a purely classical agent,the average time is given by

〈tj〉C =1

qj(A.10)

where qj is the actual success probability. This time isquadratically reduced to

〈tj〉Q =α√qj

(A.11)

for the quantum-enhanced agent. Here, α is a parameterdepending only on the number of epochs needed to createone oracle query UE [26] and on whether qj is known.In the case considered here, we find α = π/4. As aconsequence, the average learning time for the quantum-enhanced learning agent is given by

〈T (J)〉Q =

J∑j=1

α√qj

(A.12)

≤ α√J

√√√√ J∑j=1

1

qj(A.13)

≤ α√J√〈T (J)〉C (A.14)

where we used the Cauchy-Schwarz inequality in the sec-ond step. The classical learning time typically scales with〈T 〉C ∼ AK for a learning problem with episode length Kand the choice between A different actions in each step.The number J of observed rewarded sequences in orderto learn depends on the specific policy update and some-times also on the list `J of observed rewarded sequencesof actions. For an agent sticking with the first rewardedaction sequence, we would find J = 1. However, typ-ical learning agents are more explorative, and commonscalings are J ∼ K such that we find for these cases

〈T (J)〉Q ∼√

log(〈T (J)〉C)√〈T (J)〉C . (A.15)

10

This is equivalent to a quasi-quadratic speed-up in thelearning time if it is possible to perform arbitrary num-bers of Grover iterations.

In more general settings, there exist several possible`J with different length J such that the learning time〈T (J)〉 needs to be averaged over all possible `J , whichleads again to a quadratic speed-up in learning [35].

I.3. Limited coherence times

In general, all near-term quantum devices allow forcoherent evolution only for a limited time and thus amaximal number n of Grover iterations. For winningprobabilities q = sin2 ξ with (2n+ 1)ξ ≤ π/2, performingn Grover iterations leads to the highest probability offinding a rewarded action.

Again, we assume that the actual policy of an agentonly depends on the number of observed rewards anagent has found. As a consequence, the average timea quantum-enhanced agent limited to n Grover iter-ations needs to achieve the success probability Q <sin2(π/(4n+ 2)) is given by

〈T (J, n)〉Q =

J∑j=1

α0n+ 1

sin[(2n+ 1)ξj ]2(A.16)

with sin2 ξj = qj and α0 determining the number of

epochs necessary to create one oracle query UE . For α0 =1, n >> 1 and (2n+1)ξJ � π/2 we can approximate thelearning time for the quantum-enhanced agent via

〈T (J, n)〉Q ≈J∑j=1

1

4nqj=〈T (J)〉C

4n(A.17)

where we used sinx ≈ x for x � 1. In general, itcan be shown [35] that the success probability Qn =sin2(π/(4n+ 2)) can be reached by a quantum-enhancedagent limited to n Grover iterations in a time

〈T (n)〉Q ≤ γ〈T 〉Cn

(A.18)

where γ is a factor depending on the specific setting.In our case Eq. A.16 can be used to compute the lowerbound for the average quantum learning time, with α0 =n = 1. For the classical strategy Eq. A.10 is used.

II. Experimental details

A continuous wave laser (Coherent Mira HP) is used topump a single-photon source producing photon pairs inthe telecom wavelength band. The laser light has a cen-tral wavelength of 789.5 nm and pumps the single pho-ton source at a power of approximately 100 mW. The

source is a periodically poled KTiOPO4 non-linear crys-tal placed in a Sagnac interferometer [38, 39], where theemission of single photons occurs via a type II Spon-taneous Parametric Down-Conversion (SPDC) process.The crystal (produced by Raicol) is 30 mm long, set toa temperature of 25 ◦C, has a poling period of 46.15 µmand is quasi-phase matched for degenerate emission ofphotons at 1570 nm when pumping with coherent laserlight at 785 nm. As the processor is calibrated for awavelength of 1580 nm, we shift the wavelength of thelaser light to 789.75 nm in order to produce one photonat 1580 nm (that is then coupled into the processor) andanother one at 1579 nm (the heralding photon).

The processor used for the experiment is a silicon-on-insulator (SOI) type, designed by the Quantum Photon-ics Laboratory at MIT (Massachusetts Institute of Tech-nology) [40]. Each programmable unit on the device actsas a tunable beam splitter implementing the unitary

Uθ,φ =

(eiφ sin θ

2 eiφ cos θ2cos θ2 − sin θ

2

)(A.19)

where θ and φ are the internal and external phases (asin Fig. 2(b)) set via thermo-optical phase shifters con-trolled by a voltage supply. The achievable precision forphase settings is higher than 250 µrad. The bandwidthof the phase shifters is around 130 kHz. The waveguides,spatially separated from one another by 25.4 µm, are de-signed to admit one linear polarization only. The highcontrast in refractive index between the silicon and sil-ica (the insulator) allows for waveguides with very smallbend radius (less than 15 µm), thus enabling a high com-ponent density (in our case 88 MZIs) on small areas (inour case 4.9 x 2.4 mm). Given the small dimensions,the in- (and out-)coupling is realized with the help ofSi3N4 − SiO2 waveguide arrays (produced by Lionix In-ternational), that shrink (and enlarge) the 10 µm opticalfibers’ mode to match the 2 µm mode size of the waveg-uides in the processor. The total input-output loss isaround 7 dB. The processor is stabilized to a temper-ature of 28 ◦C and calibrated at 1580 nm for optimalperformance. To reduce the black-body radiation emis-sion due to the heating of the phase shifters when voltageis applied, wavelength division multiplexers with a trans-mission peak centered at 1571 nm and bandwidth of 13nm are used before the photons are sent to the detectionapparatus. In our processor, two external phase shiftersin the implemented circuits were not responding to thesupplied voltage. This defects were accounted for by de-ploying an optimization procedure.

The single-photon detectors are multi-element super-conducting nanowires (produced by photonSpot) withefficiencies up to 90% in the telecom wavelength band.They have a dark count rate of v 100 c.p.s, low timingjitter (hundreds of ps) and a reset time < 100 ns [41].