xrootd monitoring and control harsh arora cern. setting up service monalisa service monalisa...
TRANSCRIPT
![Page 1: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/1.jpg)
Xrootd Monitoring andControl
Harsh AroraCERN
![Page 2: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/2.jpg)
Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module
![Page 3: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/3.jpg)
Collecting Xrootd Parameters
MonalisaService
ExternalStorage
XrootdSystem/
Network Info
Sends monitoringdata
Control XrootdServices
![Page 4: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/4.jpg)
Proposed Architecture to Monitoring & Control the Xrootd in ALICE
Long HistoryDB
MonALISA
Network Monitoring
ApMon
XROOTDSERVER
SystemMonitoring
MonaLisaXrootd Repository
Aggregated Data
Alerts
Actions
Iosif Legrand June 2012
ControlModule
Control /
Update
MonALISA
Network Monitoring
ApMon
XROOTDSERVER
SystemMonitoring
ControlModule
MonALISA
Network Monitoring
ApMon
XROOTDSERVER
SystemMonitoring
ControlModule
![Page 5: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/5.jpg)
Run a MonALISA service on each host running an XROOTD server ( a new dedicated group )
Control the xrootd server Start / Stop / Update Dynamically Change conf parameters
Collect Monitoring information from the xrootd servers using the new & improved monitoring functionality from xrootd
Perform full network measurements and tests (RTT , available bandwidth, topology )
Monitor the storage system used by xrootd servers Create a dedicated MonALISA repository
5Iosif Legrand June 2012
Proposed Architecture to Monitoring & Control the Xrootd in ALICE
![Page 6: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/6.jpg)
Active Available Bandwidth measurements between all the ALICE grid sites (2)
Iosif Legrand October 20106
6 Iosif Legrand June 2012
ALICE measurements between all VO BoxesImportant to have similar Data for Xrootd servers
![Page 7: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/7.jpg)
7
Advantages of Using a dedicated MonALISA group to monitor and control all the XROOTD servers
Easy to maintain and update a critical service for Offline computing
Significantly improve the monitoring information and will
help to better understand the way storage system are used in the ALICE Computing model (Distributed Storage for Data)
Control the xrootd servers and can dynamically configure systems based on how they are used.
Monitor the true network connectivity Monitor the real storage used by xrootd servers Monitor the connection and transfer per client / job Can really help in developing optimization procedures for
complex data flows
Iosif Legrand June 2012
![Page 8: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/8.jpg)
% of CPU Used by Monalisa
![Page 9: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/9.jpg)
ParametersCollection and Verification
![Page 10: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/10.jpg)
% of time spent by CPU in User Mode
Monalisa Host Monitoring Reported by Xrootd
![Page 11: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/11.jpg)
ParametersOpen_files Srv_rd_mbytes
Srv_wr_files
![Page 12: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/12.jpg)
Parameters(Cont.)Srv_rd_files Srv_wr_mbytes
Space_free
![Page 13: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/13.jpg)
Average Load over last 15 min.Load15
![Page 14: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/14.jpg)
Ambiguous Parameters
Connected Clients Number of Jobs Requiring a Thread
![Page 15: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/15.jpg)
Dedicated MLRepository
![Page 16: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/16.jpg)
![Page 17: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/17.jpg)
Detailed Monitoring per Client
Monitoring per Client Patch to xrootd-3.1.0
![Page 18: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/18.jpg)
Client Monitoring Example
Num
ber
of
Byte
s
![Page 19: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/19.jpg)
Additional Information provided by monitoring each client from the xrootd
server Global Map of Data Transfers Debug individual clients Understand better the effect of long RTT
![Page 20: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/20.jpg)
Preparing Correlations Plots
Bad xrootd Server Detection High CPU Usr with Low Disk Throughput -ve correlation b/w Traffic and Disk
IOUtil(%) High Disk IOUtil(%) with Less Read Write
Rate – Probably Damaged Disk High Load Value with Less number of
clients
![Page 21: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/21.jpg)
Monalisa on Each Server
Local CPU Monitoring Info Understanding Network Topology Traffic Information Monitoring directly the storage system Control
![Page 22: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/22.jpg)
Future Plan
![Page 23: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/23.jpg)
Control via Monalisa
Start/Stop xrootd server
Upgrade
![Page 24: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/24.jpg)
Problems
Latest version of xrootd-3.2.1 Incompatible with xrootd-ftsofs-1.1.0
![Page 25: Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service Monalisa Service Monalisa Repository Test Xrootd Server ApMon Module](https://reader036.vdocument.in/reader036/viewer/2022062409/5697bff11a28abf838cbb144/html5/thumbnails/25.jpg)
Thank You