dost 2016 cloud without failures

OpenStack: Eine Cloud ohne Fehler? Höhere Zuverlässigkeit durch gezieltes Provozieren von Fehlern!

Using Fault Injection to Increase Cloud Reliability!

Götz Brasche / Jorge Cardoso CTO IT PL RnD and Director CSI / Lead Architect Cloud Operations and Analytics

21.06. - 22.06.2016

Cologne, Germany

1

• 47% aller Probleme in Datenzentren resultieren aus Ausfällen von Cloudiensten

• Dauer der Ausfälle reicht von 40 Minuten bis zu fünf Tagen

• Kosten pro Ausfall durchschnittlich 690.204 USD

• Kosten pro Minute knapp 7.000 USD

2

Unplanned downtimeis caused by*

software bugs … 27% hardware … 23%human error … 18%network failures … 17%natural disasters … 8%

* Marcus, E., and Stern, H. Blueprints for High Availability: Designing Resilient Distributed Systems. John Wiley & Sons, Inc., 2003.

3

Google's 2007 found annualized failure rates (AFRs) for drives

1 year old 1.7%3 year old >8.6%

Eduardo Pinheiro, Wolf-Dietrich Weber, and Luiz André Barroso. 2007. Failure trends in a large disk drive population. In Proc. of the 5th USENIX conference on File and Storage Technologies (FAST '07). USENIX Association, Berkeley, CA, USA, 2-2.

4

OpenStack is Growing

Apr 6, 2016

March 2016

5

OpenStack User Survey: A snapshot of OpenStack users’ attitudes and deployments. April 2016. (https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf). Fig. 2.1, Page 9.

Key User Interests

6

What Companies Think About…

Cloud Reliability

7

Gartner• Member of the Magic

Quadrant for x86 Server

Virtualization

Infrastructure

• Member of the Magic

Quadrant for Integrated

Systems

Cloud

Infrastructure

Product

Innovation AwardFrost & Sullivan 2013

10,000Huawei employees working

in cloud computing and

dedicated to meeting every

IT requirement

No. 1Industry-leading performance

according to the SPECvirt

server virtualization

performance benchmark

Best of Show

Award

NominationInterop 2013

DCD Blueprint

AwardChina's first in the data

center industry

100,000Desktops in the world's

largest-scale deployment

Huawei Cloud Computing Investment & Rewards

8

OpenStack and FusionSphere

9

Fusion Sphere Architecture

VMware vSphere

Third-party Virtualization Architecture

FusionCompute FusionStorage FusionNetwork

Huawei Virtualization Architecture

Server Storage Network & Security

Physical Infrastructure

Cloud Storage APIHuawei Open API

eBackup

UltraVR

Backup & DR

SNMP/REST NBI

FusionSphere SOI

FusionManager

Portal

RBAC

Alarm

Log

Open APIResource

ManagementConfiguration API Adapter

Cloud Storage

ManagementVDC/VPC

10

Market Recognitions Grows…

2 times higher scalability

Industry-leading performance in the

SPECvirt test

For the first time in 3 years, Gartner has

introduced a new company, Huawei

(FusionSphere), into the Magic Quadrant

for x86 Server Virtualization

Infrastructure.

HypervisorSPECvirt

ScoreRanking

FusionSphere 5.0 632 1

Linux 6.4 (KVM) 625 2

ESXi 5.1 472 3

http://www.spec.org/virt_sc2013/results/specvirt_sc2013_perf.html

vSphere

5.1

vSphere

6.0

FusionSphere

5.0

3264

128

Nodes per cluster

• High performance

− < 5% CPU performance overheads

− Support for database, email, and

ERP and CRM services

• High reliability

− Proactive event detection

− Active/Standby management nodes

− Upgrade without service interruption

− Multi-level disaster recovery plans

Support for Critical Applications

VMware

MicrosoftOracle

Parallel

CitrixHUAWEI

LEADERSCHALLENGERS

NICHE PLAYERS VISIONARIES

As of July 2014COMPLETENESS OF VISION

Source: Gartner (July 2014)

AB

ILIT

Y T

O E

XE

CU

TE

Red Hat

Member of the Gartner Magic Quadrant

for x86 Server Virtualization

Infrastructure

11

FusionSphere 5.1 Key Performance Indicators

Physical Server/VM Performance Indicator Value

Max. number of vCPUs (virtual SMP) per VM 128

Max. memory size per VM 4 TB

Max. virtual disk capacity per VM 64 TB

Max. number of virtual disks per VM 60

Max. number of virtual NICs per VM 12

Max. number of logical CPU cores per physical server 480

Max. memory size per physical server 12 TB

Max. number of powered-on VMs per physical server 1024

12

FusionSphere 5.1 Key Performance Indicators

Management Indicator Value

Max. number of physical servers per logical cluster 128

Max. number of VMs per logical cluster 3000

Max. number of logical clusters supported by a VRM node 32

Max. number of hosts supported by a VRM node 1024

Max. number of VMs supported by a VRM node 10,000

Max. number of VRM nodes that can be cascaded 16

Max. number of physical servers supported by cascaded VRM nodes 4096

Max. number of VMs supported by cascaded VRM nodes 80,000 (best practice in the industry)

15

FAILURES ARE INEVITABLE!

THE BEST WE CAN DO IS BE PREPARED FOR THEM AND LEARN FROM THEM

TEST, REPAIR, LEARN & PREDICT !

Kripa Krishnan, Technical Program Director from Google

16

One reason [Netflix]: It’s the lack of control over the underlying hardware, the inability to configure it to ensure 100% uptime

Why does using a cloud infrastructure requires advanced approaches for resiliency?

17

A program designed to increase resilience by purposely injecting

major failuresDiscover flaws and subtle dependencies

Amazon AWS: GameDay

“That seems totally bizarre on the face of it, but as you dig down, you end up finding

some dependency no one knew about previously […] We’ve had situations where we

brought down a network in, say, São Paulo, only to find that in doing so we broke our

links in Mexico.”

18

Google DIRT (Disaster Recovery Test) Annual disaster recovery & testing exercise

8 years since inception

Multi-day exercise triggering (controlled) failures in systems and process

Premise 30-day incapacitation of headquarters following a disaster

Other offices and facilities may be affected

When “Big disaster”: Annually for 3-5 days

Continuous testing: Year-round

Who 100s of engineers (Site Reliability, Network, Hardware, Software, Security, Facilities)

Business units (Human Resources, Finance, Safety, Crisis response etc.)

Google: DiRT

Source http://flowcon.org/dl/flowcon-sanfran-2014/slides/KripaKrishnan_LearningContinuouslyFromFailures.pdf

19

Netflix: Chaos MonkeyFewer alerts for

ops team

Amazon EC2 and Amazon RDS Service Disruption in the US East RegionApril 29, 2011

September 20th, 2015: Amazon’s DynamoDB service experienced an availability issue in their US-EAST-1

Transfer traffic to east region

20

Huawei: Butterfly Effect

-- Butterfly Effect System --Enables to Automatically Test and Repair OpenStack and Cloud

Applications

CLOUD APPLICATION

HUAWEI FusionSphere

The system works by intentionally injecting different failures, test the ability to

survive them, and learn how to predict and repair failures preemptively

Failure

Repair

Test

21

Approach to Cloud Reliability

1

2 3

4

7

5

6

Design & Deploy

Test Infrastructure

Monitoring Facilities

Design & Execute Fault-Injection Plan

Identify Damages

Predict Future Errors

Automatic Repair

Repair & Learn

OpenStack

Inject Fault F51

Fault F51(mysql unavailable)

OpenStack

22

Design & Execute Fault-Injection PlanBest way to avoid failure: Fail constantly

Kill cinder database(Simulate update failure)

Introduce delay in messages(Full-scale traffic shows where the real bottlenecks are)

Operation ErrorOPENSTACK_KEYSTONE_URL = "http://%s:5000/v2.0" % OPENSTACK_HOST

Operation Error/etc/nova/nova.confDelete: auth_strategy=keystone

Remove driver to HDRemove access to NFS(Simulate hardware failure)

The main testing framework of OpenStack is called Tempest, an opensource project with more than 2000 tests: only black-box testing (test only access the public interfaces)

2

OpenStack

OpenStackimage list

server create

security group create

Learning

OpenStack

Inject Fault F51

Learning

OpenStackimage list

server create

security group create

Learning

OpenStack

Monday

OpenStackimage list

Monday

OpenStack

Tuesday

OpenStackimage list

Tuesday

OpenStackimage list

Tuesday

?

OpenStackimage list

Tuesday


OpenStack

Inject Fault F51

Faults Behavior


Root Cause

$ openstack image list

2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUGdogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.553 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004acd50> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2808:05:30.555 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.565 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2808:05:30.566 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.567 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.573 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0049b090> acquired _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.576 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.581 23842 DEBUG oslo.middleware.request_id [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] <BE_ID> (backward<<<) new x-openstack-request-id: <BE_ID><iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181>__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:95 <<keystone-main.service>>2016-04-28 08:05:30.585 11693 DEBUG eventlet.wsgi.server [-] (11693) accepted ('192.168.4.231', 51774) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 <<glance-api.service>>2016-04-28 08:05:30.588 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1>__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<glance-api.service>>2016-04-28 08:05:30.589 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<glance-api.service>>2016-04-28 08:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Determining version of request: GET /v1/images/detail Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:47 <<glance-api.service>>2016-04-28 08:05:30.590 11693DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Matched version: v1 process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:72 <<glance-api.service>>2016-04-2808:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Using url versioning process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:60 <<glance-api.service>>2016-04-28 08:05:30.591 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] new path /v1/images/detail process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:73 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.auth.identity.v2 [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Making authentication request tohttp://192.168.4.231:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:86 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.session [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - -REQ_DOMA] <BE_ID> (bridge>>>) Call from keystoneclient/session.py to Keystone <BE_ID> from request /usr/lib/python2.7/site-packages/keystoneclient/session.py:373 <<glance-api.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<keystone-admin.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call__/usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<keystone-admin.service>>2016-04-28 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:310 <<keystone-admin.service>>2016-04-28 08:05:30.598 23841 INFOkeystone.common.wsgi [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] POST http://192.168.4.231:35357/v2.0/tokens <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException_enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0740f610> acquired _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.600 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.603 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.618 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148<<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f002c5ed0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.628 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-

Root Cause Localization

$ egrep ‘ERROR | TRACE’ /var/log/nova/nova-api.log | wc

$ grep ‘req-’ /var/log/glance/glance-api.log

Typically, the first thing to look in the logs when you hit a

problem is for the error message or the stack trace that has

the details on which part of the python module the problem

is generated, for that particular component.

This should explain the problem in detail, as well as its origin.

You should also look for the exception that gets logged for

any failure, which will also help identify the root cause of the

failure. If there are no exception/stack trace messages getting

dumped to the logs, it means that the problem might not be of

severe impact and the user can continue with their operation.

However, in such cases, at least a warning message will get

logged to make sure that we don't miss out on anything that's

happening in the FusionSphere environment.

The root cause of most of the issues seen in FusionSphere can

be triaged with the default logging levels. If the support team

requires more detailed logging, you can enable the debug

logs and perform the action again to capture all the details.

To have a stable environment, you want to detect failure

promptly and determine causes efficiently. With a distributed

system, it's even more important to track the right items to

meet a service-level target. By knowing where the logs are and

how to manage them, you can analyze most issues you

encounter, allowing you to keep your environment running

smoothly.

Limitations of Troubleshooting Approaches

Although today’s programs are orders of magnitude more complex than those of 30 years ago, many people still use printf to log to console or local disk, and use some combination of manual inspection and regular expressions to locate specific messages or patterns.FEBRUARY 2012, VOL. 55, NO. 2, COMMUNICATIONS OF THE ACM

Manual, complex, error-prone, and expensive

http://www.slideshare.net/tomoya/openstack-at-ntt-resonant-lessons-learned-in-web-infrastructure

One (very) simple command: glance.images.list()Size: 316K # messages:1499 # DEBUG: 1068 # INFO: 23 # Others: 408

DOCOMO has shown

(100GB and 80M lines)/day for

100 nodes

42

Activity Monitoring2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUGdogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.553 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004acd50> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2808:05:30.555 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.565 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2808:05:30.566 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.567 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.573 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0049b090> acquired _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.576 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.581 23842 DEBUG oslo.middleware.request_id [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] <BE_ID> (backward<<<) new x-openstack-request-id: <BE_ID><iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181>__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:95 <<keystone-main.service>>2016-04-28 08:05:30.585 11693 DEBUG eventlet.wsgi.server [-] (11693) accepted ('192.168.4.231', 51774) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 <<glance-api.service>>2016-04-28 08:05:30.588 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1>__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<glance-api.service>>2016-04-28 08:05:30.589 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<glance-api.service>>2016-04-28 08:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Determining version of request: GET /v1/images/detail Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:47 <<glance-api.service>>2016-04-28 08:05:30.590 11693DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Matched version: v1 process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:72 <<glance-api.service>>2016-04-2808:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Using url versioning process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:60 <<glance-api.service>>2016-04-28 08:05:30.591 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] new path /v1/images/detail process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:73 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.auth.identity.v2 [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Making authentication request tohttp://192.168.4.231:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:86 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.session [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - -REQ_DOMA] <BE_ID> (bridge>>>) Call from keystoneclient/session.py to Keystone <BE_ID> from request /usr/lib/python2.7/site-packages/keystoneclient/session.py:373 <<glance-api.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<keystone-admin.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call__/usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<keystone-admin.service>>2016-04-28 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:310 <<keystone-admin.service>>2016-04-28 08:05:30.598 23841 INFOkeystone.common.wsgi [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] POST http://192.168.4.231:35357/v2.0/tokens <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException_enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0740f610> acquired _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.600 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.603 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.618 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148<<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f002c5ed0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.628 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-

43


44


Activity Monitoring2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Releasedcreation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile[iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value,waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile[iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004acd50> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.555 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Releasedcreation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value,waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile[iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile[iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.566 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.567 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.573 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94<<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.57423842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0049b090> acquired_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.576 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.58123842 DEBUG oslo.middleware.request_id [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] <BE_ID> (backward<<<) new x-openstack-request-id: <BE_ID><iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:95 <<keystone-main.service>>2016-04-28 08:05:30.585 11693 DEBUG eventlet.wsgi.server [-](11693) accepted ('192.168.4.231', 51774) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 <<glance-api.service>>2016-04-28 08:05:30.588 11693 DEBUGoslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation<BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<glance-api.service>>2016-04-2808:05:30.589 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<glance-api.service>>2016-04-28 08:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Determining version ofrequest: GET /v1/images/detail Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:47 <<glance-api.service>>2016-04-2808:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Matched version: v1process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:72 <<glance-api.service>>2016-04-28 08:05:30.590 11693 DEBUGglance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Using url versioning process_request/usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:60 <<glance-api.service>>2016-04-28 08:05:30.591 11693 DEBUGglance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] new path /v1/images/detail process_request/usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:73 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.auth.identity.v2 [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Making authentication request to http://192.168.4.231:35357/v2.0/tokens get_auth_ref/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:86 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.session [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (bridge>>>) Call from keystoneclient/session.py to Keystone <BE_ID> from request /usr/lib/python2.7/site-packages/keystoneclient/session.py:373 <<glance-api.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>

Logi

cal t

ime

0

10

20

30

40

50

60

30%70%

Variability

Warning Y

Warning A

Warning B

Warning C

Error Z

Distributed Execution Graph

Google's Tracing System (see also X-Trace and Magpie) Originally created to understand the system behavior from a search request

Today Google's production clusters generate >1 TB/day of sampled trace data

Dapper: a Large-Scale Distributed Systems Tracing Infrastructure

Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, Chandan Shanbhag, Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Google, Inc. (2010).

// Java:Tracer t = Tracer.getCurrentTracer();String request = ...;if (hitCache())

t.record("cache hit for " + request);else

t.record("cache miss for " + request);

Annotation 41 Java and 68 C++ applications have

custom annotations to better understand

intra-span activity

Overall approach When a thread handles a traced control path, Dapper attaches a trace

context to thread-local storage.

When the control flow library (threading, control flow, RPC) is used to

schedule callbacks, Dapper attached a trace context.

Performance Basic instrumentation as small as possible and record only a fraction

of all traces using sampling (1/1000).

Daemon uses < 0.3% CPU during collection. Small memory footprint

47

Dapper

Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, Chandan Shanbhag, Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Google, Inc. (2010).

FingerprintLo

gica

l tim

e

0

10

20

30

40

50

60

30%70%

Variability

Warning Y

Warning A

Warning B

Warning C

Error Z

Distributed Execution Graph

http://acumendatasystems.com/how-fingerprint-biometric-time-clocks-work/

failureOK

Fingerprint Analytics

Can you identify your failures?

Key Challenges Analysis New matching algorithms Real-time fingerprint detection Fingerprint prediction Alignments and nested FP

M8

M1

Service a Service kService b

$ openstack image list Fault 15aOK

51

Application Fields

Failure Mode and Effect Analysis (FMEA)

Security and Intrusion detection Performance Analysis

Bottleneck

Cloud Accountability

52

OpenStack Engineers positions

Rapid prototyping of cool ideas: propose it today, code it, and show it running in

3 months…

Internship positions for MSc students

Fault injection, fault models, fault libraries, fault plans, brake and rebuild systems

all day long, …

Innovative PoCs

Solving difficult challenges of real problems using quick and dirty prototyping

Join the Cause!

53

Industry-Academia Workshop on

Cloud Reliability and Resilience

This workshop intends to bring

together industry and academia

to identify the most relevant

requirements in the field of cloud

reliability and resilience, on one

hand, and existing state-of-the-art

solutions, on the other.

We invite engineers, scientists,

regulators, and experts to discuss

and contribute to the creation of a

new generation of highly reliable

cloud platforms.

November Event

BERLIN 7-8 NOVEMBER 2016 6, Ernst-Reuter-Platz 710587 BerlinGermany

Copyright©2016 Huawei Technologies Co., Ltd. All Rights Reserved.

The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product

portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive

statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time

without notice.

HUAWEI ENTERPRISE ICT SOLUTIONS A BETTER WAY

Thank you!

dost 2016 cloud without failures

Internet