CERN IT Department CH-1211 Genve 23 Switzerlandt COOL Performance Tests ATLAS Conditions Database example Romain Basset, IT-DM October 2008 CERN IT Department CH-1211 Genve 23 Switzerlandt Outline Description ATLAS conditions database Different performance tests Database population Results Conclusion CERN IT Department CH-1211 Genve 23 Switzerlandt Description ATLAS COOL reference workloads Source : https://twiki.cern.ch/twiki/bin/view/Atlas/CoolRefWork, R. Hawkings.https://twiki.cern.ch/twiki/bin/view/Atlas/CoolRefWork Folder(s) N-Folders N-channels Data #per run kB/run ID/POOL/POOLDIR_n232string(160 char)110 CALO/POOL/POOLDIR_n1732string(160 char)187 MUON/MDT/RT11174CLOB 4.5k MUON/MDT/T011174CLOB 3k MUON/POOL/POOLDIR132string(160 char)15 GLOBAL/DETSTATUS/TISUMM1503 floats63.6 TDAQ/DCS/TESTDCS_n floats TDAQ/DCS/TESTDCS_n floats126000 CERN IT Department CH-1211 Genve 23 Switzerlandt Performance tests Two main cases : Insertion from online or from calibration jobs Data readback from reconstruction jobs Two types of data in the DB schema : Simple version folders (DCS) Multi-version folders (tags, user tags) The tests should cover both cases with a mix of both types of data. CERN IT Department CH-1211 Genve 23 Switzerlandt Performance tests 2 other questions with performance implications: Is the Oracle partitioning a possible solution to ease data management in few years ? Is index compression a good solution to decrease the size of the DB ? To answer these questions, we will also compare the performance of partitioned tables and compressed index to the normal schema. CERN IT Department CH-1211 Genve 23 Switzerlandt Database population A test DB had to be generated from scratch. The AthenaDBTestRec package provided in the ATLAS software was used to insert and retrieve the data. Several clients were running in parallel, but it was not possible to split the work for the DCS data. The duplicates for partitioned tables and index compression were created using Data Pump to accelerate the process. CERN IT Department CH-1211 Genve 23 Switzerlandt Test datasets List of the different datasets : Data typeAmount (year)Size (GB)Rows ATLAS COOL DB1154- DCS12460,000,000 DCS ,000,000 DCS ,000,000 DCS partitioned w/ global index ,000,000 DCS partitioned w/ local index ,000,000 DCS w/ comp. index ,000,000 MV w/ CLOB10.51,100,000 MV w/ CLOB52.55,500,000 MV w/ CLOB10511,000,000 MV CLOB w/ comp. index ,000,000 CERN IT Department CH-1211 Genve 23 Switzerlandt 10-year dataset : SV folder The equivalent of a 10 years old SV folder was populated in about 5 days. CERN IT Department CH-1211 Genve 23 Switzerlandt Inserting in a SV folder As seen in previous slide, insertion in a SV folder remains stable over time. Only 1% increase after 10 years. CERN IT Department CH-1211 Genve 23 Switzerlandt 10-year dataset : MV folder The equivalent of a 10 years old MV folder was populated in about 2 days. CERN IT Department CH-1211 Genve 23 Switzerlandt Inserting in a MV folder As seen in previous slide, insertion in a MV folder remains stable over time. The increase is less than 2% after 10 years. CERN IT Department CH-1211 Genve 23 Switzerlandt Data retrieval in SV folder CERN IT Department CH-1211 Genve 23 Switzerlandt Data retrieval in MV folder CERN IT Department CH-1211 Genve 23 Switzerlandt Table partitioning To ease the data management on the long term, we proposed to use Oracle partitioning (eg: to put old data offline). The partitioning should be transparent to the software to reduce the implementation cost. Several problems were found with this idea (IOV overlap) as well as performance issues. CERN IT Department CH-1211 Genve 23 Switzerlandt Partitioning problem Readback test on a partitioned table CERN IT Department CH-1211 Genve 23 Switzerlandt Conclusion The tests show no performance problem when scaling up to a 10-year dataset. The parallel readback tests are the next in line. The preliminary tests by R. Hawkings are good so we are confident. Partitioning and index compression are still being tested. CERN IT Department CH-1211 Genve 23 Switzerlandt Questions ? / Discussion ?