Embedded System:Embedded System:VV22IP PhoneIP PhoneDesign & Design &

DevelopmentDevelopmentD96921014D96921014郭人豪郭人豪

OutlineOutline

IntroductionIntroduction Function ListFunction List DesignDesign Obstacles, Workaround and Bug FixObstacles, Workaround and Bug Fix Rule of ThumbRule of Thumb ConclusionConclusion

IntroductionIntroduction

What is VWhat is V22IP Phone?IP Phone? Voice and Video over IP PhoneVoice and Video over IP Phone Extend from H.323Extend from H.323

Video use H.263, MPEG4 SP, H.264 BPVideo use H.263, MPEG4 SP, H.264 BP Audio use G.711, G.726, G.729, AMR-NB, AMR-Audio use G.711, G.726, G.729, AMR-NB, AMR-

WB, iLBC, etc.WB, iLBC, etc. Large LCD screen with Web Surf APLarge LCD screen with Web Surf AP

3.5” ~ 7”3.5” ~ 7” Browser, Mail Client, Widget, Media player, etBrowser, Mail Client, Widget, Media player, et

c.c.

Hardware Design of VHardware Design of V22IP IP PhonePhone

3com, Agere3com, Agere Separated Codec & AP ProcessorSeparated Codec & AP Processor Fast but Complex and High closeFast but Complex and High close

TI, Broadcom, QualcommTI, Broadcom, Qualcomm Integrated DSP in AP ProcessorIntegrated DSP in AP Processor Complex Software Designed and Hard to poComplex Software Designed and Hard to po

rtrt Freescale, ADI, RenesasFreescale, ADI, Renesas

AP Processor with Multimedia AcceleratorAP Processor with Multimedia Accelerator Relatively Low Cost but limited capabilitiesRelatively Low Cost but limited capabilities

Separated Codec & Separated Codec & ProcessorProcessor

AP Processor Modem / Codec

Ethernet Switch

LCDNetwork I/O

Audio I/O

CAM

Keypad I/OTouchPanelHook StateLED Control

Integrated DSP in AP Integrated DSP in AP ProcessorProcessor

Processor Core

Internal Bus

DSP

LCD

CAM

Human I/ONetwork I/OAudio I/O

AP Processor w/ Multimedia AP Processor w/ Multimedia AcceleratorAccelerator

AP Processor

ASIC MMAccelerator

LCD

CAM

Human I/ONetwork I/OAudio I/O

VV22IP Phone CapabilitiesIP Phone Capabilities

VideoVideo Encoder / DecoderEncoder / Decoder Multi format supportMulti format support Adapted Buffer ControlAdapted Buffer Control Broken Frame RecoveryBroken Frame Recovery PiP, PoPPiP, PoP Lips syncLips sync Bitrate, Frame interval, Framerate, etc.Bitrate, Frame interval, Framerate, etc.

VV22IP Phone CapabilitiesIP Phone Capabilities

AudioAudio Encoder / DecoderEncoder / Decoder Acoustic Echo CancellationAcoustic Echo Cancellation Acoustic Echo SuppressionAcoustic Echo Suppression Jitter ControlJitter Control Voice Activity DetectionVoice Activity Detection White Noise GeneratorWhite Noise Generator 3-way talking, Muxer, etc.3-way talking, Muxer, etc.

VV22IP Phone CapabilitiesIP Phone Capabilities

Dialing ToneDialing Tone EU, US, JP formatEU, US, JP format Tone GeneratorTone Generator Human interactive tone generatorHuman interactive tone generator

VV22IP Phone CapabilitiesIP Phone Capabilities

ProtocolsProtocols SIP protocol (w/ T.120)SIP protocol (w/ T.120) RTP, RTCP streaming protocolRTP, RTCP streaming protocol RTSP streaming controlRTSP streaming control SRTP, SRTCP securitySRTP, SRTCP security HTTP for firmware update & Web-base UI CoHTTP for firmware update & Web-base UI Co

ntrolntrol DHCP client, 802.1q VLAN 802.1p, 802.1p QoSDHCP client, 802.1q VLAN 802.1p, 802.1p QoS ICMP ping, ARP dup-host detection, etc.ICMP ping, ARP dup-host detection, etc.

VV22IP Phone CapabilitiesIP Phone Capabilities

APPsAPPs Phone Book, Contact ListPhone Book, Contact List CalendarCalendar Web BrowserWeb Browser Electric White Board, Screen CapElectric White Board, Screen Cap XML support for SIP extensionXML support for SIP extension

SoftDSPSoftDSP Idea from FreescaleIdea from Freescale

Using Hardware ASIC for Video CodecUsing Hardware ASIC for Video Codec Too complex for embedded systemToo complex for embedded system Core2Duo Process only 3-5 fps for 1080p AVCCore2Duo Process only 3-5 fps for 1080p AVC

Audio Codec is relatively easyAudio Codec is relatively easy Using AP Processor to encode / decode audio fUsing AP Processor to encode / decode audio f

ramerame Low cost and can upgrade for upcoming audio Low cost and can upgrade for upcoming audio

codeccodec Performance is affected by OS and CPU powerPerformance is affected by OS and CPU power

Design RequirementDesign Requirement Audio Heartbeat must be called every 10 or 20 msAudio Heartbeat must be called every 10 or 20 ms Video Heartbeat must be called every 10, 20, or 40 msVideo Heartbeat must be called every 10, 20, or 40 ms Hook state must be process at onceHook state must be process at once TouchPanel must be interactive at once but can be execuTouchPanel must be interactive at once but can be execu

ted laterted later i.e. play click sound at once and insert event into listi.e. play click sound at once and insert event into list

While streaming, Audio must be always clear and no glitWhile streaming, Audio must be always clear and no glitchch Video is reasonable smoothVideo is reasonable smooth Hook state is always first priorityHook state is always first priority Web surfing or other AP may have performance penalty while strWeb surfing or other AP may have performance penalty while str

eaming is oneaming is on LED display pattern is always real-timeLED display pattern is always real-time And on and on and on…..And on and on and on…..

System DesignSystem DesignMain Phone thread

Hook threadMultiMedia

thread

SIP threadLED control

thread

ToneGenthread

KeypadRead thread

UI Interactivethread

……

Generate and Monitor

Execute thread

Event Queue

Create Events and Insert

Main Phone Process

System DesignSystem Design

Hook stateMultiMedia

state

SIP state

LED controlstate

ToneGenstate

KeypadRead state

UI Interactivestate

Dispatch

Loop Every 10 ms

Idle state

Phone Middleware

System DesignSystem Design

UI Code

I/O Routine

LED ControlHook stateKeypad(Human Interact)

Network I/OAudio / Video I/O

Callback functions

Virtual Functionsfor implementation

Phone Middleware

System DesignSystem Design

UI Code

I/O Routine Network I/OAudio / Video I/OHook StateLED ControlEtc.

IPC call

Callback functions

KeypadTouch Panel

Comparison of Comparison of Interrupt / PollInterrupt / Poll

High LoadingHigh Loading Error resilience or high performanceError resilience or high performance

Low CPU powerLow CPU power How to design queue length?How to design queue length?

Mutex, thread overhead, program styleMutex, thread overhead, program style Resource starvation, implementation faultResource starvation, implementation fault

TimingTiming How to make sure it runs on-time?How to make sure it runs on-time?

Comparison of Combine / Comparison of Combine / Separated UISeparated UI

OverheadOverhead Need IPC to communicateNeed IPC to communicate Need CPU to process, waste of already Need CPU to process, waste of already

critical CPU loadcritical CPU load ComplexityComplexity

Need to define internal protocol for Need to define internal protocol for communicatecommunicate

OS support, performanceOS support, performance Why choose two-tier design?Why choose two-tier design?

ResultsResults

Code sizeCode size 2.5MB vs. 8MB (plus C++ lib)2.5MB vs. 8MB (plus C++ lib)

Footprint sizeFootprint size 5.5MB vs. 20MB5.5MB vs. 20MB

Client testClient test 500K bulk dial test passed500K bulk dial test passed

Audio ProblemAudio Problem

DELAY!!!!DELAY!!!! Linux 2.4 use OSS audio driverLinux 2.4 use OSS audio driver

2.6 starting support ALSA audio driver2.6 starting support ALSA audio driver ALSA provide low/zero latency delayALSA provide low/zero latency delay

TIA-810A require phone to have 100ms delTIA-810A require phone to have 100ms delay from end to enday from end to end Mic → audio driver → audio encoder → packetMic → audio driver → audio encoder → packet

ized → network → muxer → audio decoder → sized → network → muxer → audio decoder → speakerpeaker

How to minimize the delay?How to minimize the delay?

Audio ProblemAudio Problem WHERE IS THE CPU POWER???WHERE IS THE CPU POWER???

VAD use little CPU, so does white noise, most audVAD use little CPU, so does white noise, most audio codec except G.726.io codec except G.726.

AES use moderate CPU but still surviveAES use moderate CPU but still survive Not in 4-way talking or aboveNot in 4-way talking or above

PEOPLE DON’T WANT AES!!!PEOPLE DON’T WANT AES!!! AEC is much much more complicatedAEC is much much more complicated According to a pre-Bell Lab engineer, there are two dAccording to a pre-Bell Lab engineer, there are two d

esign in embedded system, 16ms of 1024ms or 64ms esign in embedded system, 16ms of 1024ms or 64ms of 128msof 128ms

The complexity grows exponentially with detect rangThe complexity grows exponentially with detect rangee

How about a phone without Speaker?How about a phone without Speaker?

Video ProblemVideo Problem

Video Input chip not compatible with AP Video Input chip not compatible with AP ProcessorProcessor PAL in okay to transform to CIFPAL in okay to transform to CIF Big problem in NTSC to CIFBig problem in NTSC to CIF

640x240 to 352x288 but no upscale (240 → 288 ups640x240 to 352x288 but no upscale (240 → 288 upscale)cale)

The ghostly Preview WindowsThe ghostly Preview Windows Sometimes, the hardware design is just canSometimes, the hardware design is just can

’t match the software requirement’t match the software requirement Open the preview window 1x1, configure the PrP, Open the preview window 1x1, configure the PrP,

close the window, call UI to flush the bitmapclose the window, call UI to flush the bitmap

LED ProblemLED Problem

Use AP Processor’s GPIO port and CPU Use AP Processor’s GPIO port and CPU clock to emulate LED on/off periodclock to emulate LED on/off period Not accurateNot accurate Actually, totally waste of CPU and out of Actually, totally waste of CPU and out of

sync in such a short periodsync in such a short period You can’t throw the CPU in the LED You can’t throw the CPU in the LED

routing, there are Audio and Video are routing, there are Audio and Video are waiting for you.waiting for you.

So, how about throw away the LED So, how about throw away the LED indicator and use LCD only?indicator and use LCD only?

Extension Board ProblemExtension Board Problem

Custom Chip for ext. board signaling and Custom Chip for ext. board signaling and LED flashingLED flashing II22C interface, which is buggy and affect the C interface, which is buggy and affect the

audio chip controlaudio chip control Custom protocol, which is also buggy and Custom protocol, which is also buggy and

need to bug on bugneed to bug on bug No-one-can-understand spec, which is told by No-one-can-understand spec, which is told by

their senior through phone and personal guidetheir senior through phone and personal guide The packet with no payload after each commandThe packet with no payload after each command Time interval between commandsTime interval between commands Write-read and read-write-read periodWrite-read and read-write-read period

LCD ProblemLCD Problem

Are you kidding? LCD problem?Are you kidding? LCD problem? Yes, and it’s really a painful experienceYes, and it’s really a painful experience

CPU spec is for reference only (the CPU spec is for reference only (the pirate code?)pirate code?) Demo board comes with QVGA panelDemo board comes with QVGA panel CPU spec support up to VGA panelCPU spec support up to VGA panel But not VGA video with PIP!But not VGA video with PIP!

Amplifier ProblemAmplifier Problem

Not embedded system related but is Not embedded system related but is funnyfunny The gain value write in Audio chip manual The gain value write in Audio chip manual

is not the actually measured value on is not the actually measured value on boardboard

The auto-gain or pre-gain function on chip The auto-gain or pre-gain function on chip has the same name but different has the same name but different implementation with the one in customer implementation with the one in customer requirementrequirement

Hardware design and power managementHardware design and power management

Protocol ProblemProtocol Problem

The SDP in SIP is a RFC standardThe SDP in SIP is a RFC standard But not in the engineer’s mind of the But not in the engineer’s mind of the

ATA box designerATA box designer Yes, the compatibility problemYes, the compatibility problem

But this time, there are no reference, no But this time, there are no reference, no guide, no document, no source code.guide, no document, no source code.

System ProblemSystem Problem

In Linux 2.4 you have no clock_gettime() but geIn Linux 2.4 you have no clock_gettime() but gettimeofday()ttimeofday() Which is extremely vulnerable while NTP settimeofdWhich is extremely vulnerable while NTP settimeofd

ay()ay() Embedded system often omit the external RTC clock Embedded system often omit the external RTC clock

for cost issuefor cost issue How can I do?How can I do?

The invincible watchdog in embedded systemThe invincible watchdog in embedded system Phone crash, UI crash, power lost, in-experience usePhone crash, UI crash, power lost, in-experience use

r manipulation, maintenance, Server crash, etc.r manipulation, maintenance, Server crash, etc.

Resource ProblemResource Problem

Flash is used widely in embedded systeFlash is used widely in embedded systemm Read lightningly fast, write damnly slowRead lightningly fast, write damnly slow Block all other routine while writing to flaBlock all other routine while writing to fla

shsh Buggy interface in KernelBuggy interface in Kernel

Multiple write will crash, use fwrite insteadMultiple write will crash, use fwrite instead Fwrite can’t measure timeFwrite can’t measure time

Configuration lost, Bad bits in flash, etc.Configuration lost, Bad bits in flash, etc.

Rule of ThumbRule of Thumb

Customer don’t know what they want, just Customer don’t know what they want, just push the product and ads to them!push the product and ads to them! Likewise in sales. Make sure you do not admit every Likewise in sales. Make sure you do not admit every

function what they just think of while stay in toilet.function what they just think of while stay in toilet. Embedded system is customized and Embedded system is customized and

extendableextendable But they are NOT everything. Think before agree But they are NOT everything. Think before agree

the requirement from customerthe requirement from customer Don’t argue with engineersDon’t argue with engineers

Try to help them workaround the bug or negotiate Try to help them workaround the bug or negotiate with customer to have another waywith customer to have another way

ConclusionConclusion Choosing RIGHT BSP will shorter the development quiChoosing RIGHT BSP will shorter the development qui

te a lotte a lot Think about what you’re designing before you designThink about what you’re designing before you design There are always a workaround there, just you may noThere are always a workaround there, just you may no

t find itt find it We are developing embedded system! Throw away the compatWe are developing embedded system! Throw away the compat

ibility!ibility! Choosing the RIGHT HARDWARE is the same importaChoosing the RIGHT HARDWARE is the same importa

nt as good code-syntax writing nt as good code-syntax writing Boys, this is not the well-known x86 systemBoys, this is not the well-known x86 system

Please always keep in mind that the CPU performance Please always keep in mind that the CPU performance is limited while coding on embedded systemis limited while coding on embedded system The RAM is limited too, don’t memcpy() on every stuff!The RAM is limited too, don’t memcpy() on every stuff!

Thank youThank you

Please have some feedback.Please have some feedback.