speed up your gis server - run gis software on solid-state drives (ssd)
DESCRIPTION
Solid-State Drives (SSDs) are recently becoming popular storage mediums in laptops, and are praised for their speed and shock resistance. However, there has been some debate over whether or not SSDs are ready for the data center. GIS software is known for its high demands on server hardware. Especially hard-drives became a bottleneck when serving map tiles, rendering map images or running spatial computations. SSDs seem to be one possibility to increase IO performance, but optimized server hardware is rare and still very expensive. We contacted a Korean maker specialising in high-performance SSD servers, mainly used to provide video-on-demand services, and we received two of their enterprise servers for testing and benchmarking with FOSS4G software. In this presentation we present our benchmarking results and explain the strong and weak points by comparing traditional hard-drives and solid-state disks.TRANSCRIPT
GIS software on solid-state drives (SSD)FOSS4G2011 - #1
Speed up your GIS serverGIS software on solid-state drives (SSD)
GIS software on solid-state drives (SSD)FOSS4G2011 - #2
Speed up your GIS serverGIS software on solid-state drives (SSD)
Seongbong Kim
Daniel Kastl
GIS software on solid-state drives (SSD)FOSS4G2011 - #3
Motivation
● Involved in pgRouting project● Lots of questions about
performance● pgRouting's bottleneck is
loading data from hard disks into memory
● How to be as fast as Google?
There is no simple answer ...
GIS software on solid-state drives (SSD)FOSS4G2011 - #4
… what about running
PostgreSQL/pgRouting on SSD?
GIS software on solid-state drives (SSD)FOSS4G2011 - #5
Who is using SSD with a laptop?
GIS software on solid-state drives (SSD)FOSS4G2011 - #6
Who is using SSD on a server?
GIS software on solid-state drives (SSD)FOSS4G2011 - #7
What is a solid-state drive?
A solid-state drive (SSD) is a data storage device that uses solid-state memory to store persistent data with the intention of providing access in the same manner of a traditional block i/o hard disk drive.
SSDs are distinguished from traditional hard disk drives (HDDs), which are electromechanical devices containing spinning disks and movable read/write heads.
In contrast, SSDs use microchips which retain data in non-volatile memory chips and contain no moving parts.
http://en.wikipedia.org/wiki/Solid-state_drive
GIS software on solid-state drives (SSD)FOSS4G2011 - #8
SSD vs. HDD (1)
Characteristic Solid-state drive Hard disk drive
Spin-up time short long
Random access time ~ 0.1 ms 5-10 ms
Read latency time short long
Read performance consistent varying
Moving parts no yes
Weight/size smaller larger
Write longevity type dependent unlimited
GIS software on solid-state drives (SSD)FOSS4G2011 - #9
SSD vs. HDD (2)
Characteristic Solid-state drive Hard disk drive
Parallel operation possible no
Shock, altitude, vibration
resistant not resistant
Magnetic susceptibility no possible
Software encryption limited yes
Costs (energy) lower high
Costs (capacity) high “low”
GIS software on solid-state drives (SSD)FOSS4G2011 - #10
MLC vs. SLC
Multi-Level-Cell
● shorter lifespan
(10.000 write cycles)
● higher data density
● slower throughput
● less efficient writing
● less expensive
Single-Level-Cell
● longer lifespan
(100.000 write cycles)
● lower data density
● higher throughput
● more efficient writing
● more expensive
GIS software on solid-state drives (SSD)FOSS4G2011 - #11
Test Server Specification
Model Name Intel(R) Xeon(R) CPU - E5620 @ 2.40GHz
Processors 6
CPU cores 4
Cache Size 12.288 KB
Total Memory 24.675.980 KB (24 GB)
RAID 0
Total SSD 750 GB (SLC)
OS CentOS 5.6
GIS software on solid-state drives (SSD)FOSS4G2011 - #12
Benchmark
GIS software on solid-state drives (SSD)FOSS4G2011 - #13
Data & Tools
● Geodata● Raster: Landsat imagery North America● Vector: OpenStreetMap Planet
● Processing● PostgreSQL import with Imposm● Image processing with GDAL● Routing topology with osm2po● Image rendering with Mapserver (Fast CGI)
GIS software on solid-state drives (SSD)FOSS4G2011 - #14
Ambitious Plans
● File system benchmarks
● WMS map rendering from raster data
● WMS map rendering from vector data (DB)
● Tile seeding from raster data
● Tile seeding from vector data (DB)
● Serving from tile cache
GIS software on solid-state drives (SSD)FOSS4G2011 - #15
Reality
● File system benchmarks
● WMS map rendering from raster data
● WMS map rendering from vector data (DB)
● Tile seeding from raster data
● Tile seeding from vector data (DB)
● Serving from tile cache
GIS software on solid-state drives (SSD)FOSS4G2011 - #16
Benchmarking Tools
● Apache JMeter for WMS
● pgBench for PostgreSQL
● FIO for file system
＝＞ WMS Shootout
GIS software on solid-state drives (SSD)FOSS4G2011 - #17
File I/O Benchmark
Sequential Read Sequential Write Random Read Random Write0
500
1000
1500
2000
2500
3000
3500
4000
16k32k64k128k256k512k1024k
GIS software on solid-state drives (SSD)FOSS4G2011 - #18
GIS software on solid-state drives (SSD)FOSS4G2011 - #19
GIS software on solid-state drives (SSD)FOSS4G2011 - #20
GIS software on solid-state drives (SSD)FOSS4G2011 - #21
GIS software on solid-state drives (SSD)FOSS4G2011 - #22
Use Case - VOD or FTP
Requirements● Mainly reading access● Fast read rate● Serving data
Example● Tile server● Data server
http://www.maptiler.org/
GIS software on solid-state drives (SSD)FOSS4G2011 - #23
Use Case - DB / FS only
Requirements● Combined writing
and reading access● Fast read rate
Examples● Geo-database● Map rendering● Data processing
GIS software on solid-state drives (SSD)FOSS4G2011 - #24
Lessons Learned
● Different field of expertize● Relatively small amount of community
resources● Standards & best practices● Comparability● PostgreSQL/PostGIS – how to tune it right?
… well, everyone asks for benchmarks
GIS software on solid-state drives (SSD)FOSS4G2011 - #25
GIS software on solid-state drives (SSD)FOSS4G2011 - #26
Want to benchmark your Open Source software?
GIS software on solid-state drives (SSD)FOSS4G2011 - #27
Thank you!
LSD Tech Korea