WelcomeITU Innovators

IT KartelletTalk

Coffee & Networking

ITU Innovators

● Student Entrepreneurial OrganizationDriven by volunteers

● VisionConnecting students with startup companies, advisors, incubators, events, networking, ...

● EventsArranging and attending talks, workshops, competitions, meetings, ...

IT Kartellet

● IT-consultancy● Software development● Hosting● Moodle/Drupal experience● ASP.NET (MVC)

Copenhagen Tech Talks

Scaling Moodle

● Independent student project fall 2013 under the supervision of Kasper Østerbye

● Collaboration with the IT department of ITU (LearnIT)

● Collaboration with ITK (Moodle consulting)

Scalability, Performance Testing & TuningIT Kartellet and friendsPresented by ITU Innovators

What is Moodle?

● Modular Object-Oriented Dynamic Learning Environment

● Open source● Open University - 700.000 users, 6.000

courses● So it does scale - but how?

Scalability basics

● Vertical / scaling up● Horizontal / scaling out

○ X-axis○ Y-axis○ Z-axis

● Combining○ Performance○ Fault tolerance○ Fault isolation

Moodle architecture

Minimum Example setup

● 20.000 students● High availability requirements -

no downtime is allowed● All components should be

fault tolerant

CBS - Learn

● 2.500 students● Permits for small

windows of downtime

● Should handle large file uploads from games and digital media students

ITU - LearnIT

Moodle as a black box

● ...or a number of black boxes● Large community● Developed over many years● Limited documentation● How can we make the best out of it?

Testing approach

● Repeatable● Automated● Well documented ● Organized● Comparable● Targeted

User request simulation

User request simulation

● Simply HTTP Requests● Realistic input and randomization● Client-side measurements● Why jMeter?

Server-side measurements

● Monitor/record during test● Compare to everyday usage● Invaluable monitoring tools

○ iostat○ inotify○ htop

Server-side measurements - iostat

● Monitoring of cpu/storage activity● Repeated measure with given interval● Most relevant columns

○ iowait, avgqu-sz, await● More details: http://linuxcommand.org/man_pages/iostat1.html

Server-side measurements - inotify

● Monitor all files access, open, write etc.● More details:

○ inotifywatch http://linux.die.net/man/1/inotifywatch

○ inotify-win https://github.com/thekid/inotify-win / JNotify http://jnotify.sourceforge.net/

Server-side measurements - htop

● When does the server give up - when is it trashing itself in terms of CPU / memory?

● Adjust the server settings accordingly - max number of threads, concurrent users etc.

Immediate visualisation

Immediate visualisation

Test and visualisation process

Shell script JMeter R script

Number of usersList of serversRamp-up

Status codeUserResponse time Diagram

Server-side

Variousmeasurements

R scriptDiagram

Case exampleFile storage and cache

● Moodle requires shared cache

File storage and cache

Data properties

● Shared vs. Local● Volatile vs. Non-Volatile

Local

Shared

Shared

Local

Strings, plugin data, config, database meta

Update Logs

File-store, Temporary files

VolatileHtmlpurifier, question data, course contacts

Non-Volatile

Baseline (single shared NFS)

MoodleHDD - File store - Language strings - Plugins data - Config - Htmlpurifier - Question data - Course contacts - Database meta - Temporary files - Update logs

Web server 1

Moodle

Web server 2

NFS

Local volatile cache

MoodleHDD - File store - Htmlpurifier - Question data - Course contacts - Temporary files - Update logs

Web server 1NFS

APC - Language strings - Plugins data - Config - Database meta

Moodle

Web server 2

APC - Language strings - Plugins data - Config - Database meta

Shared volatile cache

MoodleHDD - File store - Temporary files - Update logs

Web server 1

NFS

Memcached

RAM - Language strings - Plugins data - Config - Database meta - Htmlpurifier - Question data - Course contacts

Moodle

Web server 2

Case exampleDatabase

Database load

● Sessions● User access rights● Courses● File meta data● ...

Single instance

● Easy to set up● ACID● Single point of failure● All load on a single server● Vertical scaling

Cluster setup

● Redundancy (X-axis)● Specialisation (sharding)

○ Y-axis○ Z-axis

Scaling Open Source

● Open source code - not build with scaling in mind - will only scale “prettily” if redesigned, but we can do it!○ A lot can be achieved by configuration

● A different approach to contributing to open source!○ We were imagining to be able to walk right in and

point out where moodle developers were wrong○ However, we ended up not changing a single line of

moodle code

Questions?

Coffee & Networking