IBM Lotus Domino 8.5.3 server performance: IBM Lotus Notes 8.5.3 performance

Andy Nolet, Software Engineer, IBM

Angelo Lynn, Software Engineer, IBM

Wu W Huang, Software Engineer, IBM

November 2011

© Copyright International Business Machines Corporation 2011. All rights reserved.

Summary: IBM Lotus Domino 8.5.3 and IBM Lotus Notes 8.5.3 have been optimized to reduce the transactions from the client to the server. These optimizations yield a reduction in server resources, thus lowering the Total Cost of Ownership. This white paper shows the reductions in processor and disk utilization achieved with large numbers of Notes 8.5.3 clients running on a single Domino 8.5.3 server.

Table of Contents1 Introduction ............................................................................................................................... 1 2 N85mail and N853mail workloads defined ................................................................................ 2 3 Test configurations .................................................................................................................... 3 4 AIX test results .......................................................................................................................... 5 4 Linux on System z results ......................................................................................................... 8 6 Appendix: Explanation of Lotus Notes.ini settings ................................................................... 14 7 Conclusion .............................................................................................................................. 15 8 Resources ............................................................................................................................... 15 9 About the authors .................................................................................................................... 15

1 IntroductionThis white paper presents information on the latest workload developed by the IBM® Lotus® Domino® Performance team. The new workload, named “N853mail,” continues the line of workloads that are designed to mimic the transactions performed by the IBM Lotus Notes® client previously discussed in the developerWorks® article series beginning with “IBM Lotus Domino 8.5 server performance, part 1: IBM Lotus Notes performance.”

Here we report on the benefits of running the IBM Lotus Notes 8.5.3 client together with the Domino 8.5.3 server; specifically, we demonstrate the reduction in server resources such as processor and disk usage that are realized compared with Notes/Domino 8.5.x (see figure 1).

1

Figure 1. Server resource reduction with Lotus Domino 8.5.3

AIX Window s zLinux0

2

4

6

8

10

12

14

16

12.5

14

99

34

11

23

Server Resource Reduction with Domino 8.5.3With 4000 Simulated Notes Users

%CPU Busy Total Disk Ops/S Total Disk Bytes

Per

cent

Impr

ovem

ent

The measurements in this report use performance workloads that mimic the behavior of thousands of Notes clients performing common messaging and calendar operations against a single Domino server.

2 N85mail and N853mail workloads definedTwo workloads were used, N85mail, which simulates the API calls generated by the Notes 8.5.x client; and N853mail, which performs the same high-level operations but with the API calls generated by a Lotus Notes 8.5.3 client.

Similarities between the two workloadsN85mail and N853mail workloads use the N85mailinit workload to create the seed database and perform the same Notes client actions:

1. Refresh the Inbox2. Read 5 messages, reply to one every other cycle3. Send a message to a user4. Send a message to three users every other cycle5. Schedule an appointment once every 24 cycles6. Schedule an invitation once every 24 cycles7. Move one document to the Lotus NotesBench folder8. Delete two documents9. Accept an invitation once every 24 cycles10. Poll for new mail11. Repeat

Both workloads run with the same Notes.ini settings, and each workload cycle is 780 seconds, with all client transactions spread out over the entire 780 seconds.

Differences between the two workloads• N85mail makes 47.5 calls to the server per cycle, mimicking the Notes 8.5.x client, whereas

N853mail makes 32 calls to the server per cycle, mimicking the Notes 8.5.3 client, in which:

2

◦ Reply to a message does 3.5 fewer client transactions per cycle.◦ Move a message does 5 fewer client transactions per cycle.◦ Delete a message does 7 fewer client transactions per cycle.

• N85mail workload does an average of 14600 client transactions per minute for 4000 users; whereas N853mail workload does an average of 9900 client transactions per minute for 4000 users.

• N85mail makes the call of read entries on the Inbox with a value of 50, while N853mail uses a value of 100.

NOTE: The call of read entries in the N853Mail workload was changed to match the Notes 8.5.3 client, which was optimized to make fewer calls to the server for read entries but attempts to read more data with each call and buffers them for subsequent view refreshes.

3 Test configurationsIn general, the tests were set up with 4000 users registered in the Domino Directory on the server. At the beginning of the test, each user has a mail file that is roughly 170 MB of compressed documents, 3000 messages in the Inbox, and roughly 380 messages in the Trash, which are set to expire and be deleted at the rate of two messages every 15 minutes throughout the measurement.

These tests have transaction logging enabled with the “Favor runtime” setting, and mail journaling is set to journal all messages locally. Lotus Domino Domain Monitoring (DDM) probes are enabled for messaging and the OS, and all users have mail rules that block mail from 10 users external to the test.

With Lotus Domino 8.5.x we enabled document compression on the mail databases, reducing their size from approximately 250MB down to approximately 170MB. Additionally, we enabled Domino Attachment Object Store (DAOS) once the mail databases were created and also enabled DAOS on the mail boxes and journal files.

The tests use the default settings of the Domino server and mail template; however, some settings were changed to enhance performance. The changes used on all platforms are shown in table 1, with any additional platform-specific items discussed in later in the paper in the section for the respective platform.

Refer to the Appendix section in this paper for further details on what the Lotus Notes.ini settings below accomplish.

Table 1. Domino server and mail templates settings

Configuration Lotus Domino 8.5.2 Server

Lotus Domino 8.5.3 Server

Lotus Domino 8.5.3 Server

Mail Template Mail85.ntf Mail85.ntf Mail85.ntf

Workload n85mail n85mail n853mail

Transaction Log Enabled / Favor runtime Enabled / Favor runtime Enabled / Favor runtime

DAOS Enabled Enabled Enabled

3

Mail Journaling All messages local All messages local All messages local

Additional, non-default mail database options

• Don’t overwrite free space

• Disable automatic update of views

• Compress document data

• Use DAOS

• Don’t overwrite free space

• Disable automatic update of views

• Compress document data

• Use DAOS

• Don’t overwrite free space

• Disable automatic update of views

• Compress document data

• Use DAOS

ServerTasks Replica, Router, Update, AMgr, AdminP, Sched, CalConn, RnRMgr, LDAP

Replica, Router, Update, AMgr, AdminP, Sched, CalConn, RnRMgr, LDAP

Replica, Router, Update, AMgr, AdminP, Sched, CalConn, RnRMgr, LDAP

Lotus Notes.ini additions

NLCache_Size=67108864,Server_Show_Performance=1,RouterDbCacheSize=6100,Schedule_No_Validate=1,NSF_DBcache_Maxentries=5100,Debug_NSF_Show_Allstats=1,Create_R85_Databases=1,Debug_NSF_Compress_All_Lotus Notes=1,Create_R85_Log=1,Enable_LZ1_Encrypted_Lotus Notes=1

NLCache_Size=67108864,Server_Show_Performance=1,RouterDbCacheSize=6100,Schedule_No_Validate=1,NSF_DBcache_Maxentries=5100,Debug_NSF_Show_Allstats=1,Create_R85_Databases=1,Debug_NSF_Compress_All_Lotus Notes=1,Create_R85_Log=1,Enable_LZ1_Encrypted_Lotus Notes=1

NLCache_Size=67108864,Server_Show_Performance=1,RouterDbCacheSize=6100,Schedule_No_Validate=1,NSF_DBcache_Maxentries=5100,Debug_NSF_Show_Allstats=1,Create_R85_Databases=1,Debug_NSF_Compress_All_Lotus Notes=1,Create_R85_Log=1,Enable_LZ1_Encrypted_Lotus Notes=1

NOTE: The results in this paper are from benchmarks executed in a controlled environment. Although effort was made during the creation of the benchmark to include typical user operations, it is likely that real users will make different use of Lotus Domino than the narrow range of function that is tested by the benchmark.

The numbers herein should therefore be used primarily to understand the relative performance of the Domino releases; they do not represent recommendations for real-world deployment. For assistance with capacity planning, we recommend you consult your hardware vendor and IBM TechLine.

Also, though we show results on a variety of hardware platforms, these configurations are not of uniform capacity. Our intent here is to focus on the performance of Lotus Domino itself, and this data should not be used to compare platforms against each other.

Test runsThe tests run for this study were:

• Test run #1: Domino 8.5.2 Enterprise Server with the N85mail workload simulating Lotus Notes 8.5 Clients, to be used as a baseline.

• Test run #2: Domino 8.5.3 Enterprise Server with the N85mail workload simulating Lotus Notes 8.5.2 Clients, to show the benefit of migrating the Domino mail server to version 8.5.3 and the Notes clients to 8.5.2.

4

• Test run #3: Domino 8.5.3 Enterprise Server with the N853 mail workload simulating Lotus Notes 8.5.3, to show the benefit of migrating both the Domino mail server and the Notes clients to 8.5.3.

4 AIX test resultsIn this section we discuss the results from our tests on the AIX® platform. Table 2 summarizes the configuration we used.

Table 2. AIX test configurationHardware model IBM Power 570 (9117-MMA)CPUs for test / speed 1 physical and 1 logical 3.5-GHz CPUMemory 16 GBActive physical drives IBM FastT Fiber Channel storage, 6 trays

each with 14 15000-RPM disk drives

6 trays are configured into 1 Logical RAID5 volume

Active logical volumes 1 RAID5 logical volume for Domino binaries, Domino data, and DAOS;

1 RAID0 local drive for transaction loggingOperating system AIX 7.1 64 bit

Test hardware was a IBM Power 570 (9117-MMA) system, which is based on POWER6™ processor-based technology, configured into two Logical partitions (LPARs). The test LPAR was configured with a single physical CPU and a single logical CPU, which was capped and did not use entitlement for additional CPU resources.

Capping the available CPU processor power let us accurately measure the load at a higher level than would be possible with the variability in CPU allocated to the LPAR using virtualized CPUs and entitlement.

We configured storage by using an IBM DS4000 Series (FastT) into a single Logical file system that supported the Domino binaries, Domino data files, mail journal database, and the DAOS NLO files. We used the AIX Advanced Journal File system (JFS2) configured with RAID 5.

All mail database files, log files, Mail*.box, and DAOS files are recreated before starting a test, to ensure that every test is started with the databases in the exact same state. The Transaction log files are recreated for every test and located on dedicated storage configured using RAID 0.

In all tests the NSF Buffer Pool Size was explicitly set to be the same (512MB). Note that the default size for 32-bit Lotus Domino is 512 MB, and for 64-bit Lotus Domino 8.5.x the default NSF Buffer Pool Size is 1 GB when there is >4 GB RAM in the system.

Network access was through a single 1-GB Ethernet adapter running in full duplex mode.

Figure 2 shows the Percent CPU Busy (System and Application) for each simulated user load at steady state.

5

Figure 2. Percent CPU busy for each simulated user load at steady state

Percent CPU Busy

15

2738

48

1323

3342

0

10

20

30

40

50

60

1000 2000 3000 4000

Simulated Users

N85Mail D852 Srvr N85Mail D852 TXNS N853Mail

Figure 3 shows the total disk operations (Reads and Writes) for each simulated user load at steady state.

Figure 3. Total disk operations for each simulated user load at steady state

Total Disk Operations/sec

193

377

558

740

182

355

520670

0100200300400500600700800

1000 2000 3000 4000

Simulated Users

N85Mail D852 Srvr N85Mail D852 TXNS N853Mail D853 Srver

6

Figure 4 shows the total disk megabytes transferred per second for each simulated user load at steady state.

Figure 4. Total disk MBs transferred per second for each simulated user load at steady state

Total Disk Megabytes Transferred/sec

1.7

3.4

5

7

1.6

3.24.7

6.2

012345678

1000 2000 3000 4000

Simulated Users

N85Mail D852 Server N85Mail D852 TXNS N853Mail

Table 3 summarizes the resource usage at 4000 users.

Table 3. Resource usage at 4000 usersDomino Server 64-bit Lotus Domino

8.5.2Lotus Domino 8.5.3

Notes Client simulated Lotus Notes 8.5.2 Lotus Notes 8.5.3

Percent change when migrating both client and server to v8.5.3

CPU % Busy 48 45 42 -12%Total disk operations/sec 740 753 670 -9% Total disk reads/sec 142 155 153 8% Total disk writes/sec 598 599 518 -13%Total disk MBs transferred/sec 7 6.8 6.2 -11% Total disk MBs read/sec 1 1 1 0% Total Disk MBs written/sec 6 5.8 5.1 -15%Client transactions/minute 14609 14607 9873 -32%Shared memory used (MB) 1696 1689 1701 0%Process memory used (MB) 138 141 142 3%Network (MB/sec) 1.83 1.8 1.78 -2%

7

So, it's easy to see that lowering CPU and disk writes is a Total Cost of Ownership (TCO) win.

4 Linux on System z resultsLet's now discuss the results for Linux® on System z. Table 4 summarizes the configuration we used.

Table 4. Linux on System z test configuration

Model z9 2094–S54CPUs for test / Speed 5 Dedicated Processors from one LPAR Memory 12 GBActive Physical Drives DS8300Active Logical Volumes 64x4 LVM mail dbs, and each volume for

Lotus Notes data, Domino address book, mailbox, and transaction log.

OS SUSE SLES 10 with some updates for Lotus Domino 8.5.2 and Lotus Domino8.5.3

The Notes.ini settings were as follows:

Platform_statistics_enabled=1Server_pool_tasks=100NSF_Buffer_Pool_Size_MB=386NLCACHE_SIZE=104857600

All performance test results reported in this section come from one LPAR on a series z9 model 2094-S54. This LPAR used five dedicated processors for the performance measurements and was configured as a single DPAR (Domino partition running an instance of the Domino server). This machine booted SUSE Linux Enterprise Server (SLES) 10 with some updates for Lotus Domino 8.5.2 and Lotus Domino 8.5.3.

The performance test LPAR was configured with 12 GB of memory. We used a single-gigabit Ethernet Open System Architecture (OSA) card, and our LAN was isolated to avoid other network traffic interference from unrelated activities.

All disks were allocated from an Enterprise Storage Server (DS8300) array with each disk configured as 3390 model 9. Client mail databases were distributed evenly over 64 Logical Volume Management (LVM) file systems, each allocated across four volumes in a single LVM, providing 28 GB of useable space per file system.

There separate volumes were allocated for the Domino execution, Notes data, Domino address book, mailboxes 1 to 4, and transaction logging. An EXT3 file system was used on Linux for System z.

Figure 5 shows Percent CPU Busy usage, where Lotus Domino 8.5.2 with 8.5.2 client transactions is in red, Lotus Domino 8.5.3 with 8.5.2 client transactions is in green, and Lotus Domino 8.5.3 with 8.5.3 client transactions is in blue.

8

Figure 5. Percent CPU busy for Linux on System z

Percent CPU Busy

6

12

18

26

3

11

17

23

0

5

10

15

20

25

30

1000 2000 3000 4000

Simulated Users

N85Mail N85Mail D852 TXNS N853Mail

Figure 6 shows the total disk I/O operations per second at different sets of simulated user loads.

Figure 6. Total disk operations/sec for Linux on System z

Total Disk Operations/sec

236404

595

808

234389

570

778

0100200300400500600700800900

1000 2000 3000 4000

Simulated Users

N85Mail N85Mail D852 TXNS N853Mail

9

Figure 7 shows the total disk megabytes transferred per seconds at different sets of simulated user loads.

Figure 7. Total disk megabytes transferred/sec for Linux on System z

Total Disk Megabytes Transferred/sec

1.94

3.55

5.25

7.08

1.933.42

5.05

6.84

012345678

1000 2000 3000 4000

Simulated Users

N85Mail N85Mail D852 TXNS N853Mail

Table 5 shows the resource usage at 4000 users at steady state for Lotus Domino 8.5.2 versus Lotus Domino 8.5.3.

Table 5. Resource usage at 4000 users, Domino 8.5.2 vs. 8.5.3 with Notes 8.5.2 client transactionsLotus Domino server Lotus Domino

8.5.2Lotus Domino 8.5.3

Lotus Notes client simulated

Lotus Notes 8.5.2 Lotus Notes 8.5.3

Percent change when migrating both client and server to 8.5.3

CPU % busy 25.6 25.2 23.3 -9%Total disk operations/sec 808 850 778 -4%

Total disk reads/sec 151 165 183 21%Total disk writes/sec 656 684 594 -9%Total disk MBs transferred/sec

7.08 7.26 6.84 -3%

Total disk MBs read/sec 0.59 0.65 0.72 22%Total disk MBs written/sec

6.49 6.61 6.12 -6%

Shared memory used (MB)

1138 1148 1153 1%

Process memory used (MB)

237 243 237 0%

Network (MB/sec) 1.98 2.01 1.94 -2%

10

When we upgraded both client and server from 8.5.2 to 8.5.3 on Linux on System z, our results showed 9% CPU improvement, 4% improvement on total disk operations per second, and 3% on total disk megabytes transferred per second.

Lotus Domino 8.5.3 and Lotus Notes 8.5.3 optimized the transactions sent from the client to the server so, as is the case for the AIX platform, upgrading both server and client from 8.5.2 to 8.5.3 can improve TCO on Linux on System z.

5 Microsoft Windows 2008 test resultsTable 6 summarizes the configuration for our testing on Microsoft® Windows® 2008.

Table 6. Windows test configurationModel X3850CPUs for test / speed 4 / 3669-MHz Memory 8 GBActive physical drives 6 trays each with 14 15000-RPM disk

drivesActive logical volumes 6OS Windows 2008 R2, hotfix 979149

All performance results reported here were tested on an IBM X3850 server configured with four processors running 3669 MHz, along with 8 GB of memory running Windows 2008 R2. The server had a single Domino partition installed, and we used a single-gigabit Ethernet OSA card.

Our LAN was isolated to avoid other network traffic interference from unrelated activities. All disks were allocated from a DS4000 storage array with 48 disk drives configured as six RAID 0 logical units. There are separate volumes allocated for the Domino execution, Notes data, Domino address book, mailboxes 1 to 4, and transaction logging.

Clients were configured to run n85mail and n853mail workloads.

Figure 8 shows CPU % busy usage, where Lotus Domino 8.5.2 with 8.5.2 client transactions is in red, Lotus Domino 8.5.3 with 8.5.2 client transactions is in green, and Lotus Domino 8.5.3 with 8.5.3 client transactions is in blue.

11

Figure 8. Percent CPU busy for Windows

Percent CPU Busy

4.610.7

20.5

29

3.99

15.9

25.5

05

101520253035

1000 2000 3000 4000

Simulated Users

N85Mail N85Mail D852 TXNS N853Mail

Figure 9 shows the total disk I/O operations per second at different sets of simulated user loads.

Figure 9. Total disk I/O operations/sec

Total Disk Operations/sec

2.13.7

5.8

8.1

1.93.6

5.4

7.5

0123456789

1000 2000 3000 4000

Simulated Users

N85Mail N85Mail D852 TXNS N853Mail

12

Figure 10 shows the total disk megabytes transferred per second at different sets of simulated user loads.

Figure 10. Total disk megabytes transferred/sec

Total Disk Megabytes Transferred/sec

2.24.1

6.4

9.1

2

4

6

8.5

0

2

4

6

8

10

1000 2000 3000 4000

Simulated Users

N85Mail N85Mail D852 TXNS N853Mail

Table 7 shows the resource usage at 4000 users at steady state for Lotus Domino 8.5.2 versus Lotus Domino 8.5.3.

Table 7. Resource usage at 4000 users, Domino 8.5.2 vs. 8.5.3 with Notes 8.5.2 client transactionsLotus Domino server 32 bit

Lotus Domino 8.5.2

Lotus Domino 8.5.3

Lotus Notes client simulated

Lotus Notes 8.5.2 Lotus Notes 8.5.3

Percent change when migrating both client and server to 8.5.3

CPU % busy 29 29 25 -14%Total disk operations/sec

802 842 779 -3%

Total disk reads/sec 162 177 180 11%Total disk writes/sec 640 684 599 -6%Total disk MBs transferred/sec

9.02 9.4 8.87 -2%

Total disk MBs read/sec 1.83 1.96 1.98 8%Total disk MBs written/sec

7.18 7.43 6.89 -4%

Shared memory used (MB)

1636 1629 1586 -3%

Process memory used (MB)

257 257 257 0%

Network (MB/sec) 2.35 2.4 2.29 -2%

13

When we upgraded both client and server from 8.5.2 to 8.5.3 on Windows 2008 R2 server, our results showed 14% CPU improvement, 3% improvement on total disk operations per second, and 2% on total disk megabytes transferred per second.

Running Lotus Domino 8.5.3 with Lotus Notes 8.5.3 has optimized the transactions sent from the client to the server so, once again, upgrading from both the server and clients from 8.5.2 to 8.5.3 can improve TCO on Windows 2008 R2.

6 Appendix: Explanation of Lotus Notes.ini settingsTable 8 lists some Notes.ini settings and their definitions.

Table 8. Notes.ini definitionsNotes.ini parameter ExplanationCreate_R85_Databases Applies to Domino 8.5 servers. Set it to 1 to create

new ODS53 databases on Domino 8.5 servers.

Create_R85_Log Applies to Domino 8.5 servers. Set it to 1 to create Domino Transaction Logging in Domino 8.5 format.

Debug_NSF_Compress_All_Lotus Notes Set it to 1 to enable document compression for all databases on the Domino server.

NSF_COMPRESS_TXN_LOGS Set it to 1 to enable compression of log records (used by zLinux / zOS).

DEBUG_ENABLE_SYS_V_SHM Set it to 1 to let Lotus Domino use System V for shared memory instead of mmap files on UNIX platforms.

DEBUG_NSF_SHOW_ALLSTATS New in Lotus Domino 8. Set it to 1 to enable Domino I/O stat collection for database metadata, views, and objects

EVENT_CORRELATION_POOL_SIZE. Specifies the size of the event correlation pool used by the Event task and DDM.

Enable_LZ1_Encrypted_Lotus Notes Set it to 1 to enable LZ1 compression of encrypted documents in mail journal database.

EVENT_POOL_SIZE Set this variable to specify the amount of memory used for unprocessed events, unprocessed notifications, and event suppression. The default setting is 5242880 (5MB).

Log_mailrouting Specifies the level of logging of router events performed by the router process.

MEM_AddressableMemSizeMB Set to override the default setting for the amount of memory Lotus Domino will see as "addressable" memory (where VALUE is in megabytes, and the default varies by platform).

MEM_EnablePreAlloc Set it to 1 to enable pre-allocation of shared memory.

NLCACHE_SIZE Specifies the size of Namelookup Cache, in bytes. The default value is 16MB.

14

NSF_Buffer_Pool_Size_MB Specifies the maximum size (in bytes) of the NSF buffer pool, a section of memory dedicated to buffering I/O transfers between Lotus Domino and disk storage.

NSF_Dbcache_Maxentries Specifies the number of databases that a server can hold in its database cache at one time.

RouterDbCacheSize Specifies the size of Router Database cache used to cache mail databases in the router process.

SCHEDULE_NO_VALIDATE Enables or disables whether SchedMgr validates its busytime database entry on a daily basis, as follows: 0 (default) - Enables validation 1 - Disables validation

Server_Show_Performance Applies to servers. Specifies whether or not server performance events are displayed on the console. Set this variable to 1 to display server performance events on console.

ServerTasks Specifies the tasks that begin automatically at server startup and continue until the server is shut down.

Server_Pool_Tasks Total number of physical threads in the Lotus Domino server (DbServer) threadpool.

7 ConclusionLotus Domino 8.5.3 and Lotus Notes 8.5.3 are optimized to reduce the transactions from the client to the server. These optimizations result in a reduction in server resources, lowering the TCO on the AIX, Linux on System z, and Microsoft Windows 2008 platforms.

8 Resources• Participate in the IBM Lotus Notes/Domino 8.5 discussion forum.

• Contribute to the IBM Lotus Domino wiki.

• Refer to the developerWorks® IBM Lotus Notes and Domino product page.

• Read the developerWorks article, “IBM Lotus Domino 8.5 server performance, part 1: IBM Lotus Notes performance.”

9 About the authorsAndrew Nolet is a member of the Lotus Domino Performance team. You can reach Andy at anolet@us.ibm.com.

Angelo Lynn is a member of the Lotus Domino Performance team, with primary focus on the performance of the Lotus Domino Server on Windows-based platforms. You can reach Angelo at anglynn@us.ibm.com.

15

Wu W Huang is a member of the Lotus Domino Performance team, with primary focus on Lotus iNotes and XPages on Windows and Lotus Domino performance on System z. You can reach Wu Huang at wuhuang@us.ibm.com.

Trademarks

• AIX, developerWorks, Domino, IBM, iNotes, Lotus, and Notes, are trademarks or registered trademarks of IBM Corporation in the United States, other countries, or both.

• Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States, other countries, or both.

• Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

• Other company, product, and service names may be trademarks or service marks of others.

16