Download - Arduino based GUI for Touchscreen Displays
![Page 1: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/1.jpg)
Arduino based GUI for Touchscreen Displays
Helen Schloh, B.Sc. Informatics
Supervisor: Andreas MäderSecond Supervisor: Bernd SchützSubmission Date: 31.3.2021
16.02.2021
![Page 2: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/2.jpg)
Outline
• Research Question• Definitions• Hardware
• Setup• Touchscreen• Arduino• Other Devices
• Software• Touch Detection• Display• Event handling• Simulation
• Evaluation• Outlook
![Page 3: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/3.jpg)
Research QuestionWhat am I researching and why?
• Research setup• Develop GUI library for given Touchscreen Display and Arduino Due
• Purpose• Using the Display during ES exercises• Investigate performance of setup• Explore use cases
• Goal• Detect Gestures reliably• Draw GUI Elements on Display• Maximize Quality of speed and memory usage• Derive recommendations for using the GUI library
![Page 4: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/4.jpg)
Definitions
• I²C• Communication protocol• Synchronous• Serial
• UART (Universal asynchronous receiver and transmitter)• Hardware Device for Communication• Asynchronous• Serial
• (Hardware) Interrupt: • Processor stops all tasks to execute interrupt routine
![Page 5: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/5.jpg)
Hardware
![Page 6: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/6.jpg)
Setup
![Page 7: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/7.jpg)
HardwareWhat does the System look like?
FPGA
Pins/Custom protocol
UART/Custom protocol
I2C/Custom procotol
Display
Touchpanel
[1][4]
![Page 8: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/8.jpg)
HardwareArduino Due
• Microcontroller Board• 84 MHz clock
• Serial Communication• I²C : 100 KHz• UART
• Serial: (max 115200 bps)• SerialUSB: (max 480 Mbps)
• Memory• 512 kB ROM• 64,32 kB RAM
[1]
![Page 9: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/9.jpg)
HardwareTouchscreen
• EA TFT070‐84ATS Touchscreen• Display
• 800x480 RGB • 33,3 MHz parallel Interface• Cannot be sent by Arduino
• Touch Panel• 1792x1024 dots• I²C Schnittstelle [4]
![Page 10: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/10.jpg)
Other DevicesWindows PC and FPGA
• Windows PC: Acer Swift 1
• FPGA• DE0-Nano
Development Board• 50 MHz clock
[2]
[3]
![Page 11: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/11.jpg)
Software
![Page 12: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/12.jpg)
SoftwareArchitecture
• Object oriented C++• OO‐metaphor fits GUI Element nesting• ES students know Java Swing• Arduino runs with C++
Detecting GesturesModeling finite Automaton
as Class
Display GUIDevelop GUI Elements as
Classes
Handle Touch EventsImplement Listener
Pattern
![Page 13: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/13.jpg)
SoftwareTouch Detection
![Page 14: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/14.jpg)
Interface ‐ Touch
FPGA
pins/Custom protocol
UART/Custom protocol
I2CCustom procotol
Display
Touchpanel
[1][4]
![Page 15: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/15.jpg)
Touch RecognitionHow is Touch Information retrieved?
• Attach interrupt on touch occurring
• Retrieve touch information
![Page 16: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/16.jpg)
Touch RecognitionHow are Touch Gestures recognized?
• Modeling Finite Automaton• Class as Automaton• Functions as States• Function calls for transition
• Change of state depending on• Number of Touchpoints (Fingers on Touchpanel): TP• Time passed since entering state: Timer• Position of Touches: x[n],y[n], n‐th Touch
![Page 17: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/17.jpg)
Touch Detection ‐ Gestures
>0 Finger downagain
>1 Finger pressed
Finger up
Drag Single Tap
Double Tap
Click and hold
1 Finger pressed
Timer > 0.5s
NoGesture
TP == 1
TP > 1
Timer > 0.5s
TP > 1
![Page 18: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/18.jpg)
Touch Detection ‐ Gestures
>0 Finger downagain
>1 Finger down
Finger up
Drag Single Tap
Double Tap
Click and hold
1 Finger pressed
NoGesture
TP > 0
TP == 0 &&Timer < 0.5s
Timer > 0.3s
TP > 0 &&Timer < 0.3s
TP == 0
![Page 19: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/19.jpg)
Touch Detection ‐ Gestures
>0 Finger downagain
>1 Finger down
Finger up
Drag Single Tap
Double Tap
Click and hold
1 Finger down
NoGesture
TP == 1
TP > 1
TP > 1
Δx[0] > 10 ||Δy[0] > 10
Δx[0] > 10 ||Δy[0] > 10
![Page 20: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/20.jpg)
Touch Detection – Multitouch Gestures
>0 Finger downagain
>1 Finger down
Finger up
Drag Single Tap
Double Tap
Click and hold
Pinch
1 Finger down
NoGesture
TP > 1
,Direction
TP < 1
![Page 21: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/21.jpg)
SoftwareDisplay
![Page 22: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/22.jpg)
Interface – DisplayHow are GUI Elements displayed?
FPGA
pins/Custom protocol
UART/Custom protocol
I2CCustom procotol
Display
Touchpanel
[1][4]
![Page 23: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/23.jpg)
Interface – DisplayWhat does the Arduino communicate to the FPGA?
Description Opcode [8 bit] Body
Clear Display 0x0 ‐
Set drawing Position 0x1 x [10 bit]y [10 bit]
Set drawing Color 0x4 Red [8 bit]Green [8 bit]Blue [8 bit]
Set pixel,Increment drawing position
0x2 ‐
Draw bitmap 0x3 Width [10 bit]Height [10 bit]Bitmap [(width*height) bit]
![Page 24: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/24.jpg)
Interface – DisplayDrawing Bitmaps
0 1 1 0 0 0 0 0
0 0 1 1 0 0 0 0
0 0 0 1 1 0 0 0
0 0 0 0 1 1 0 0
0 0 0 0 1 1 1 0
0 0 0 0 1 1 0 0
0 0 0 1 1 0 0 0
0 0 1 1 0 0 0 0
0 1 1 0 0 0 0 0
![Page 25: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/25.jpg)
Overview of GUI ElementsWhich Elements were implemented?
![Page 26: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/26.jpg)
Software Event Handling
![Page 27: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/27.jpg)
Event handlingHow does the Arduino react to Touch Events?
![Page 28: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/28.jpg)
Event handlingExample
![Page 29: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/29.jpg)
Event handlingExample
![Page 30: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/30.jpg)
Event handlingHow does the rectangle change its color?
![Page 31: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/31.jpg)
SoftwareSimulation
![Page 32: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/32.jpg)
Software – SimulationWhy do we need a simulation?
FPGA
pins/Custom protocol
UART/Custom protocol
I2CCustom procotol
Display
Touchpanel
[1][4]
![Page 33: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/33.jpg)
Software – SimulationWhy do we need a simulation?
FPGA
pins/Custom protocol
UART/Custom protocol
I2CCustom procotol
Display
Touchpanel
[1][4]
![Page 34: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/34.jpg)
Software ‐ SimulationUsing a Windows PC to simulate Touchscreen and Arduino
TouchpanelDisplay
Mouse EventsComputer Window [3]
![Page 35: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/35.jpg)
Software – SimulationUsing a Windows PC as Display
UART/Custom protocol
I2CCustom procotol
Touchpanel
Display
Computer Window
[1]
[3]
[4]
![Page 36: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/36.jpg)
Evaluation
![Page 37: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/37.jpg)
Research QuestionHow am I answering these questions now?
• Research setup• Develop GUI library for given Touchscreen Display and Arduino Due
• Purpose• Using the Display during ES exercises• Investigate performance of setup• Explore use cases
• Goal• Detect Gestures reliably• Draw GUI Elements on Display• Maximize Quality of speed and memory usage• Derive recommendations for using the GUI library
![Page 38: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/38.jpg)
Research QuestionHow am I answering these questions now?
• Research setup• Implemented Gesture Detection• Created drawable GUI Elements• Handled Touchevents• Simulated FPGA functions on Windows PC
![Page 39: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/39.jpg)
Research QuestionHow am I answering these questions now?
• Goal• The System can detect Gestures reliably
• Single tap, double tap, click and hold, dragging, …• GUI Elements on Display were implemented
• Basic shapes, Buttons, Option groups, radio Buttons, …• Maximize Quality of speed and memory usage
• Use bitmaps for faster drawing (8‐times faster)• Sparsely use of Component attributes• Library: 4% ROM storage• Use fastest possible serial interface of Arduino (SerialUSB)
![Page 40: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/40.jpg)
Research QuestionHow am I answering these questions now?
• Goal• Derive recommendations for using the GUI library
• Touchscreens are very versatile and can adapt to many use cases• Quality of performance and memory usage are sufficient for applications without real time requirements
• Should not be used for images/ streaming/ animations• Certain GUI Elements are more useful than others for Arduino (Textfields need Keyboard,…)
• Use multiple pages instead of scrolling• Avoid large GUI elements• Avoid dynamic creation of GUI elements
![Page 41: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/41.jpg)
Research QuestionHow am I answering these questions now?
• Purpose• Using the Display during ES exercises
• Intuitive (OO) library• Easy to use for unexperienced students (Java inspired)
• Investigate performance of setup• I identified the fastest way to send Data to display• Benchmarking by using Timers for functions
• Explore use cases• Implemented Basic GUI Elements to be used for developing more complex GUIs• I could give recommendations when to use the library
![Page 42: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/42.jpg)
Outlook
• Next steps• Documentation, testing• Multi touch recognition• Further Evaluation: Benchmarking, RAM usage
• Possible Extensions to the Project• GUI
• Layout Manager, multiple pages,…• Anti aliasing
• FPGA• Connect to FPGA• More complex commands to FPGA
![Page 43: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/43.jpg)
Questions?
![Page 44: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/44.jpg)
Sources ‐ Images
• [1] Arduino: https://cdn‐reichelt.de/bilder/web/artikel_ws/A300/ARDUINO_DUE_01_NEU.jpg
• [2] FPGA: https://www.intel.com/content/www/us/en/programmable/b/de0‐nano‐dev‐board.html
• [3] Acer Swift 1: https://www.preis.de/produkte/Acer‐Swift‐1‐SF114‐32/5014185.html
• [4] Touchscreen: https://www.mouser.de/ProductDetail/ELECTRONIC‐ASSEMBLY/EA‐TFT070‐84ATS?qs=f9yNj16SXrLEoCqnHqqifg%3D%3D
![Page 45: Arduino based GUI for Touchscreen Displays](https://reader030.vdocument.in/reader030/viewer/2022020622/61ec75c22cad6f7f5463e35f/html5/thumbnails/45.jpg)
Sources
• Touchscreen: https://www.reichelt.de/tft‐display‐7‐0‐154x86mm‐800x480‐dot‐24bit‐rgb‐pcap‐touch‐ea‐tft070‐84ats‐p260435.html?CCOUNTRY=445&LANGUAGE=de
• Arduino: https://store.arduino.cc/arduino‐due• FPGA: https://www.digikey.de/de/product‐highlight/t/terasic‐tech/de0‐nano‐soc