software and gateware development for sirius bpm ... › icalepcs2017 › posters ›...

Legend:

L. M. Russo, Beam Diagnostics GroupBrazilian Synchrotron Light Laboratory, LNLS, Brazil

THPHA149

The Brazilian Synchrotron Light Laboratory (LNLS) is in the final stages of developing an open-source BPM system for Sirius, a 4th-generation synchrotron light source under construction in Brazil. The system is based on the MicroTCA.4 standard comprising AMC FPGA boards carrying FMC digitizers and a CPU module. The software is built with the HALCS framework and employs a service-oriented architecture (SOA) to export a flexible interface between the gateware modules and its clients, providing a set of loosely-coupled components favoring reusability, extensibility and maintainability. In the paper, the BPM system will be discussed in detail focusing on how specific functionalities of the system are integrated and developed in the framework to provide SOA services. In particular, two domains will be covered: (i) gateware modules, such as the ADC interface, acquisition engine and digital signal processing; (ii) software services counterparts, showing how these modules can interact with each other in a uniform way, easing integration with control systems.

Software and Gatew are Development for Sirius BPM Ele ctronics Using a

Service-Orient ed Architecture

BPM Project Gateware

BPM Project Software using HALCS framework

Protocol Conversion

Peripheral #1

Peripheral #2

Bridge #1

Peripheral #3.1

Peripheral #3.2

Peripheral #4

Communication Interface

Figure 1. Generic Hardware Architecture.

Communication Interface

Peripheral Controller #4

Peripheral Controller #3.2

Peripheral Controller #3.1

External protocol

Intra-Controller protocol

Peripheral Controller #2

Peripheral Controller #1

Figure 2. SOA-based Software Architecture.

Figure 6. BPM Software Architecture.

Client Layer

Broker Layer

Service Layer

Device Access Layer

HALCS DataflowIntroduction

Pipe Protocol

MLM Protocol

Board Support

HAL

Dispatch Engine

Broker

MLM Protocol

Kernel

PCIe

LLIO

DEVIO

SMIO #2 (e.g., Data Acquisition)

SMCH

SMPRSMPR

SMCH

SMIO #1 (e.g., FMC130 Control)

MALAMUTE Broker

Client App #2 (e.g., CLI interface)

Client App #1 (e.g., EPICS IOC)

Eth (TCP)

Figure 3. HALCS Framework Architecture.

LinksHALCS Framework: https://github.com/lnls-dig/halcsBPM EPICS IOC: https://github.com/lnls-dig/bpm-epics-iocBPM Gateware: https://github.com/lnls-dig/bpm-gw DSP Cores: https://github.com/lnls-dig/dsp-cores Infra Cores: https://github.com/lnls-dig/infra-coresTiming Receiver: https://github.com/lnls-dig/timing-receiver-gw

Summary● SOA principles applied to low-level software maximize reuse of commonly used

functionalities● Best used in conjunction with isolated hardware components with minimal interaction

among each other● Succesfully deployed in the BPM and the upcoming MicroTCA.4 Timing Receiver

projects

Generic Hardware Architecture

SOA-based Software Architecture

● Standard communication interfaces: PCIe, UART, etc.

● Hierarchical Design, e.g. based on open-source Wishbone Bus Protocol

● Peripherals with minimal interaction, acting as isolated components

● Desirable to have knowledge about internal components such as: unique ID, name, address range, version, capabilities

● Software abstracts hardware components as services

● Uses a common protocol to communicate with hardware device

● Services can coordinate themselves by using an Intra-Controller protocol

● Protocols acts as a flexible/extensible API

● HALCS Framework implements SOA principles, using an Inversion of Control design paradigm

● Uses a common RPC protocol to expose services functionalities

● Broker provides discoverability and reliability to services using Mailbox messaging pattern

● Services (SMIO) register functions

● Services can use additional abstractions:

● SMCH for external chips: AD9510 clock distributor and PLL, Si57x clock oscillator, etc.

● SMPR for external protocols: SPI, I2C, etc.

● DEVIO: Event-driven reactor engine

● LLIO: Hardware Abstaction Layer

● Reusable gateware components, based on Wishbone B4

● Self-Describing Bus (SDB) aware components:

● Easier for software to be gateware-agnostic: dynamic offsets, version, capabilities

● Software can act as a userspace driver

● Application logic pushed to Client Layer

Figure 5. BPM Hardware Architecture.

Switch. Clock

DSPData,

Controls & Clock

cos

-sin

I

Q

NCO

CORDIC (Rect-Pol)

tbt amp.

CIC

CIC

CIC

CIC

tbt

tbt

CORDIC (Rect-Pol)

fofb amp.

fofb

fofb

CICmon. amp.

x4

DSP Data

Signal Deswap

x2

x2

FMC ADC Board

RFFE Board

LEDsTrigger

Backplane

Addr. SpaceWB Slave

DAQ

Addr. SpaceWB Slave

Trigger Mux

WB SlaveAddr. Space

WB SlaveAddr. Space

Switching Clk. Gen.

WB Master

WB Crossbar

SDB

WB Slave

To all WB Slaves

Addr. SpaceWB Slave

Heartbeat & LEDs

Addr. SpaceWB Slave

Trigger Interface

delay

diff. sum

adc

tbt

fofb

mon.

adc

tbt

fofb

mon.

xyq

sum

fofb amp. 1 to 4

Data

MMC

Addr. SpaceWB Slave

Diagnos- tics

Serial

Addr. Space

PCIe Interface

WB MasterAddr. Space

UART (optional)

WB Master

FPGA

DDR3 SDRAM

2GB

MGT Transceiver

WB SlaveAddr. Space

FMC ADC Interface

WB SlaveAddr. Space

A. Sp.WB Slv.

ADC Clock

A. Sp.WB Slv.

FMC Misc.

1. Caller instantiates LLIO

2. Caller instantiates DEVIO

3. Caller registers SMIOs manually or via SDB

4. SMIOs registers into Malamute Broker

5. Clients may send/receive messages to any SMIO

MSGPIPE

DEVIO PIPE DEVIO PIPE: REGISTER_SMIOREGISTER_SMIO_ALLUNREGISTER_SMIOUNREGISTER_SMIO_ALL

MSG PIPE: External Protocol

Event loop

CMD PIPE: $TERM

MALAMUTE:RPC protocol

Event loop

RPC over Malamute

CMDPIPE

RPC over Malamute

RPC over Malamute

Figure 4. HALCS Dataflow.

HALCS Architecture

Device Interface

RPC Client Interface

Trigger Control & Status

Trigger Interface

Device Interface

RPC Client Interface

Heartbeat&LEDs

LEDs Control & Status

Device Interface

RPC Client Interface

IPMI InfoBoard Status

Diagnostics

x2Device Interface

RPC Client Interface

Testing Functions

LEDsTrigger Control

FMC Status

FMC Misc.

x2Device Interface

RPC Client Interface

DSP Configuration

Swithching Control

DSP

Device Interface

RPC Client Interface

Trigger Mux

Mux Control

x2

Device Interface

Support Functions

Status

Data Transfer

Trigger Control

RPC Client Interface

DAQ

x2

I2C

Oscillator Control Control &

StatusSPI

PLL Control

Device Interface

RPC Client Interface

FMC ADC Clock

x2x2Device Interface

RPC Client Interface

ADC Control

SPI

Control & Status

FMC ADC

Dispatch Engine (DEVIO)

SDB lib

Device Interface

RPC Client Interface

Deswap Control

Deswap

x2

External protocol

Intra-Controller protocol

RPC client protocol

Application-Specific Module (SMIO)SMIO

HAL (LLIO)

PCIe

To all RPC Client Interfaces

To all RPC Client Interfaces

To all Clients

MALAMUTE BrokerTo all Clients

EPICS DriverCommand-Line

Interface

Client Layer

Broker Layer

Service Layer

Device Access Layer