Senior DesignSenior Design May 07-06 May 07-06

High Performance High Performance Optical InterconnectOptical Interconnect

April 24, 2007April 24, 2007

IntroductionIntroductionTeam MembersTeam Members

Layth Al-JalilLayth Al-Jalil Jay BeckerJay Becker Adam FritzAdam Fritz David SheetsDavid Sheets

AdvisorsAdvisors Dr. Mani MinaDr. Mani Mina Dr. Arun SomaniDr. Arun Somani Dr. Robert WeberDr. Robert Weber

Client: Lockheed Martin

Presentation OutlinePresentation Outline

Project OverviewProject OverviewIndustry Application (High speed fiber optic Industry Application (High speed fiber optic

networking)networking)Operating environment, users and uses, assumptions Operating environment, users and uses, assumptions

and limitations, expected end productand limitations, expected end product Approach Used Approach Used

Research, design, implementation, testingResearch, design, implementation, testing Organizational DetailsOrganizational Details

Project status, resources and schedulesProject status, resources and schedules Closure MaterialsClosure Materials

Project evaluation, lessons learned, closing summaryProject evaluation, lessons learned, closing summary

TerminologyTerminology

Bus:Bus: Shared medium network topology Shared medium network topology COTSCOTS: : CConsumer onsumer ooff ff tthe he sshelf, e.g. “COTS equipment”helf, e.g. “COTS equipment” LASERLASER: : LLight ight aamplification by mplification by sstimulated timulated eemission of mission of rradiation adiation NICNIC: : NNetwork etwork iinterface nterface ccard, in this case, a card that accepts data ard, in this case, a card that accepts data

over a PCI-e bus and transfers it into a serial fiber optic signalover a PCI-e bus and transfers it into a serial fiber optic signal NodeNode: A node is the combination of a network interface card and the : A node is the combination of a network interface card and the

host computerhost computer Primary ControllerPrimary Controller: This is the node that acts as the scheduler, i.e. : This is the node that acts as the scheduler, i.e.

“the master”“the master” SecondarySecondary: This is a node that is controlled by the primary : This is a node that is controlled by the primary

controller, i.e. “the slave”controller, i.e. “the slave” SATA: SATA: Serial ATA; A computer bus technology primarily designed Serial ATA; A computer bus technology primarily designed

to transfer data to and from optical drivesto transfer data to and from optical drives SMA:SMA: Coaxial RF connectors to interface with coaxial cable Coaxial RF connectors to interface with coaxial cable TDMTDM: Time division multiplexing : Time division multiplexing

Problem StatementProblem Statement

Use COTS equipmentUse COTS equipment Understand how to interconnect two to Understand how to interconnect two to

four processors using 10 Gbps technologyfour processors using 10 Gbps technology Use a fiber optic mediaUse a fiber optic media $5,000 budget$5,000 budget Develop most capable system with this Develop most capable system with this

fundingfunding

Project OverviewProject OverviewOriginal Task (design project):Original Task (design project): Design a fiber optic network that will Design a fiber optic network that will

allow communication up to 10 Gbpsallow communication up to 10 GbpsRevised Task (research project):Revised Task (research project): Design a fiber optic network with 2.5 Design a fiber optic network with 2.5

Gbps communication which takes into consideration common 10 Gbps communication which takes into consideration common 10 Gbps issues and provides direction for scaling up to 10 GbpsGbps issues and provides direction for scaling up to 10 Gbps

(Topology Overview)

Uni-directional data flow

Node 1

Coupler

Tx Rx

1

3

2

4

Node 2

Coupler

Tx Rx

1

3

2

4

Node 3

Coupler

Tx Rx

1

3

2

4

Nod

e 0

Tx

Rx

Industry Relevance and ApplicationIndustry Relevance and Application

Avionics industry trends:Orion Space Vehicle (Lockheed Martin)Network centric warfare

Applications in the private sector:HDTVIn-flight entertainment systems

Operating EnvironmentOperating Environment

InitialInitial High Speed Systems Engineering laboratoryHigh Speed Systems Engineering laboratoryDependable Computing laboratory Dependable Computing laboratory

ClientClient Military aircraft and vehiclesMilitary aircraft and vehiclesSpacecraft (Orion)Spacecraft (Orion)

ProjectedProjectedCommercial airlinersCommercial airlinersWide area networksWide area networks Courtesy: GlobalSecurity.org

End Users and UsesEnd Users and Uses UsersUsers

Pilots, passengers, and crew (aerospace vehicles)Pilots, passengers, and crew (aerospace vehicles)Engineers (Lockheed Martin)Engineers (Lockheed Martin)Scientists and students (ISU)Scientists and students (ISU)

UsesUsesProvide research on the capabilities and limitations of Provide research on the capabilities and limitations of

accelerated communication between processing accelerated communication between processing nodesnodes

Provide medium for streaming high resolution video Provide medium for streaming high resolution video throughout a platform with minimal latencythroughout a platform with minimal latency

Provide insight into different protocols, processor Provide insight into different protocols, processor configurations, and future fiber optic technology configurations, and future fiber optic technology solutionssolutions

Assumptions and LimitationsAssumptions and Limitations Lockheed Martin provided a budget of $5000 for Lockheed Martin provided a budget of $5000 for

purchase of COTS equipmentpurchase of COTS equipment System developed at 2.5 Gbps can be scaled to 10 System developed at 2.5 Gbps can be scaled to 10

Gbps with COTS equipmentGbps with COTS equipment Processor boards will be provided by either Iowa State Processor boards will be provided by either Iowa State

University or Lockheed MartinUniversity or Lockheed Martin Development of this system must be largely drawn from Development of this system must be largely drawn from

existing research existing research Availability of compatible network componentsAvailability of compatible network components Distance between nodes using multi-mode fiber optic Distance between nodes using multi-mode fiber optic

cabling cannot exceed 300 meterscabling cannot exceed 300 meters Lack of familiarity with development platform softwareLack of familiarity with development platform software

Expected End ProductExpected End Product

A fiber optic network utilizing a bus topology A fiber optic network utilizing a bus topology which operates at approximately 2.5 Gbps which operates at approximately 2.5 Gbps and a plan for scaling up to 10 Gbps with and a plan for scaling up to 10 Gbps with adequate financial resources adequate financial resources

Custom software that ensures reliable data Custom software that ensures reliable data transfer between processing nodes across transfer between processing nodes across the networkthe network

Documentation summarizing the processes Documentation summarizing the processes and conclusions of experimentation, design, and conclusions of experimentation, design, and implementationand implementation

Project StatusProject StatusAccomplishments:Accomplishments: Explored available fiber optic technology solutions Explored available fiber optic technology solutions Completed a detailed hardware design Completed a detailed hardware design Ordered and received fiber optic cabling, couplers, and Ordered and received fiber optic cabling, couplers, and

splitterssplitters Experimented and tested coupler/splitter configuration Experimented and tested coupler/splitter configuration (to (to

interface boards with the fiber optic link) interface boards with the fiber optic link) Selected and tested prototype development platform Selected and tested prototype development platform

(Xilinx Virtex II Pro boards) (Xilinx Virtex II Pro boards) Developed software algorithms to enable reliable Developed software algorithms to enable reliable

communication between boards communication between boards Initial testing into connection of fiber optic cabling and Initial testing into connection of fiber optic cabling and

passive elements between nodes to complete the optical passive elements between nodes to complete the optical networknetwork

Approach UsedApproach Used(Research)(Research)

Available fiber optic technology solutionsAvailable fiber optic technology solutionsXilinx Virtex BoardsXilinx Virtex BoardsMyrinetMyrinet

Networking protocols compatible with our Networking protocols compatible with our chosen topology (Loop)chosen topology (Loop)TDMTDMDWDMDWDMCDMACDMAEthernetEthernet

Programming languages involved in hardware Programming languages involved in hardware controlcontrolVHDL,VHDL, C, C++, assembly level languages, etc.C, C++, assembly level languages, etc.

Approach UsedApproach Used(Preliminary System Designs)(Preliminary System Designs)

NIC

Switch

NIC

NIC

NIC

Switch

Switch

Switch

PC

PC

PC

PC

Fiber Optic Interconnect (RING Topology)

PCI Bus

PCI Bus

PCI Bus

PCI Bus

Fiber Optic Interconnect

Fiber Optic Interconnect

Fiber Optic Interconnect

Fiber Optic Interconnect

Initial design provided signal termination and bi-directional signal traffic.

Signal termination was provided by fiber optic switch which was opened when fiber optic transceiver fed a signal into the switch. When no signal was fed, the switch would be closed (allowing traffic to pass through the node).

Such a topology cost $14000, thus exceeding our budget.

Approach UsedApproach Used(Double Ring Design)(Double Ring Design)

Approach UsedApproach Used(Current System Design)(Current System Design)

The current design does not realize a complete ring The current design does not realize a complete ring because without signal termination fiber optic line because without signal termination fiber optic line becomes saturated, and f/o transceiver does not output a becomes saturated, and f/o transceiver does not output a value therein.value therein.

Node 1

Coupler

Tx Rx

1

3

2

4

Node 2

Coupler

Tx Rx

1

3

2

4

Node 3

Coupler

Tx Rx

1

3

2

4

Nod

e 0

Tx

Rx

Differential signal from transceiver in absence of a complete loop.

Differential signal from transceiver in presence of a complete loop. The optic signal is degraded by line noise that is not terminate, thus acculumating.

Approach UsedApproach Used(Technologies Involved)(Technologies Involved)

Myrinet/Ethernet board: could not modify Myrinet/Ethernet board: could not modify software running on boards.software running on boards.

Xilinx Virtex II Pro X: Provided 10 Gbps Xilinx Virtex II Pro X: Provided 10 Gbps solution over multiple channels; became solution over multiple channels; became Virtex 4.Virtex 4.

Xilinx Virtex II Pro: 3 Gbps solution that is Xilinx Virtex II Pro: 3 Gbps solution that is available in the lab.available in the lab.

Approach UsedApproach Used (Software Design)(Software Design)

PL

B

Bus transceiver

PPC

OP

B

Aurora

TX FIFO

PLB to OPB Bridge

RX FIFOTX BRAM

RX BRAM

Network

Application Memory

Approach UsedApproach Used (Software Design)(Software Design)

PowerPC processorPowerPC processor

Program is stored in application memoryProgram is stored in application memory

Communication through OPBCommunication through OPB

Data transfer through PLBData transfer through PLB

Bus Transceiver interfaces PLB and AuroraBus Transceiver interfaces PLB and Aurora

Approach UsedApproach Used(Software Design)(Software Design)

The controller calculates each node’s transmission needs and allocates a The controller calculates each node’s transmission needs and allocates a proportional “transmission window” for the next time slot.proportional “transmission window” for the next time slot.

Approach UsedApproach Used(Implementation)(Implementation)

Strategy:Strategy: Implement components piecewise, then integrateImplement components piecewise, then integrate Ensure proper performance before integratingEnsure proper performance before integrating

Development components: Development components: Established reliable communication between boards Established reliable communication between boards Developed an organized communication scheme that Developed an organized communication scheme that

would manage signal transmissions across the would manage signal transmissions across the networknetwork

Designed and tested a coupler/splitter configuration to Designed and tested a coupler/splitter configuration to interface processing boards with the fiber optic linkinterface processing boards with the fiber optic link

Integrated fiber optic cabling and coupler/splitter Integrated fiber optic cabling and coupler/splitter configuration between boards to complete the fiber configuration between boards to complete the fiber optic connectionoptic connection

Approach UsedApproach Used(Testing)(Testing)

Communication between two processors located on the Communication between two processors located on the same board same board

Communication between two processors located on Communication between two processors located on separate boardsseparate boards

Terminal characteristics of coupler/splitter configuration Terminal characteristics of coupler/splitter configuration to verify proper functionalityto verify proper functionality

Fiber optics and passive elements were integrated Fiber optics and passive elements were integrated between two processing nodesbetween two processing nodes

Tested complete network functionalityTested complete network functionality

Project EvaluationProject EvaluationMilestonesMilestones Relative Relative

ImportanceImportanceEvaluation ScoreEvaluation Score Resultant ScoreResultant Score

Problem definitionProblem definition 5%5% 100%100% 5%5%

ResearchResearch 20%20% 100%100% 20%20%

Technology selectionTechnology selection 20%20% 100%100% 20%20%

End-product designEnd-product design 15%15% 66.66%66.66% 10%10%

Prototype implementationPrototype implementation 15%15% 66.66%66.66% 10%10%

End-product testingEnd-product testing 5%5% 66.66%66.66% 3.33%3.33%

End-product documentationEnd-product documentation 5%5% 90%90% 4.5%4.5%

Project reviewsProject reviews 5%5% 90%90% 4.5%4.5%

Project reportingProject reporting 5%5% 90%90% 4.5%4.5%

End-product demonstrationEnd-product demonstration 5%5% 100%100% 5%5%

TotalTotal 100%100% 86.83%86.83%

(Previously defined passing score: 80% )

Resources and SchedulesResources and Schedules(This Semester)(This Semester)

Hours (current)Hours (current) Cost Cost ($10.50)($10.50)

Adam FritzAdam Fritz 161161 $1690.50$1690.50

David SheetsDavid Sheets 251251 $2635.50$2635.50

Layth Al-JalilLayth Al-Jalil 148148 $1554$1554

Jay BeckerJay Becker 167.5167.5 $1758.75$1758.75

Xilinx Virtex II Pro Xilinx Virtex II Pro development boardsdevelopment boards 00 DonatedDonated

Multi-mode fiber cablingMulti-mode fiber cabling 00 $250$250

2 PC’s2 PC’s 00 DonatedDonated

Misc. Hardware (splitters, Misc. Hardware (splitters, couplers, etc.)couplers, etc.) 00 $1,900$1,900

Poster BoardPoster Board 1010 $25$25

TotalsTotals 727.5727.5 $9813.75$9813.75

(Personal Effort and Resource Requirements)

Resources and SchedulesResources and Schedules

Future RecommendationsFuture Recommendations

Fiber optic elements and network complexity can Fiber optic elements and network complexity can be expanded uponbe expanded upon

A TDM scheme could be implemented to govern A TDM scheme could be implemented to govern the signal transmissions across the networkthe signal transmissions across the network

Obtain necessary equipment for increasing Obtain necessary equipment for increasing communication rates to 10 Gbpscommunication rates to 10 Gbps

Continue development of hardware capable of Continue development of hardware capable of handling the necessary digital signal processinghandling the necessary digital signal processing

Closure MaterialsClosure Materials Project Evaluation:Project Evaluation:

Valuable learning experience for group membersValuable learning experience for group members Team has gained an understanding of how to interconnect Team has gained an understanding of how to interconnect

multiple processors and a fiber optic mediamultiple processors and a fiber optic media Commercialization:Commercialization:

This product is intended to serve as a research tool for students This product is intended to serve as a research tool for students at Iowa State University and engineers at Lockheed Martinat Iowa State University and engineers at Lockheed Martin

Risk and ManagementRisk and Management Lack of familiarity with subject matterLack of familiarity with subject matter

Lessons learned:Lessons learned: Signal processing issues related to fiber optic communicationsSignal processing issues related to fiber optic communications Vendors are reluctant to allow public access to product prices, Vendors are reluctant to allow public access to product prices,

specs, and quality customer servicespecs, and quality customer service Troubleshooting issues related to controlling hardware through Troubleshooting issues related to controlling hardware through

softwaresoftware

Closing SummaryClosing Summary

Questions?Questions?

We believe our design for this fiber optic We believe our design for this fiber optic connection of processor nodes will prove to be a connection of processor nodes will prove to be a

cost effective and beneficial resource for Lockheed cost effective and beneficial resource for Lockheed Martin and Iowa State University in studying the Martin and Iowa State University in studying the

capabilities and limitations of gigabit capabilities and limitations of gigabit communication technology.communication technology.