advanced trouble ticket systems steve peck operations manager global research noc i2 joint techs –...

Advanced Trouble Advanced Trouble Ticket SystemsTicket Systems

Steve PeckSteve Peck

Operations ManagerOperations Manager

Global Research NOCGlobal Research NOC

I2 Joint Techs – July 18, I2 Joint Techs – July 18, 20062006

Advanced Trouble Ticket Advanced Trouble Ticket SystemsSystems

If you look at R&E networks as a If you look at R&E networks as a system, there are quite a few system, there are quite a few components that need to components that need to interoperate well to achieve the interoperate well to achieve the desired result, the advanced desired result, the advanced networking that we know and love networking that we know and love so well.so well.


While the primary focus of advanced While the primary focus of advanced networking is the incredible networking is the incredible bandwidth, the dazzling speeds, the bandwidth, the dazzling speeds, the amazing advanced services, and the amazing advanced services, and the remarkable high-end applications that remarkable high-end applications that run over the networks, there are some run over the networks, there are some basic operational elements that help basic operational elements that help provide the foundation of support for provide the foundation of support for all of the great gee-whiz bang stuff that all of the great gee-whiz bang stuff that gets the front page coverage.gets the front page coverage.


One of the “basic” operational One of the “basic” operational elements that help facilitate the R&E elements that help facilitate the R&E system is the system is the Trouble Ticket System Trouble Ticket System (TTS)(TTS). .

With the tremendous changes and With the tremendous changes and advancement in networking, the advancement in networking, the trouble ticket system can no longer be trouble ticket system can no longer be just “basic,” a glorified word processor, just “basic,” a glorified word processor, or a simplistic database like thingy that or a simplistic database like thingy that doubles as a work order system, or doubles as a work order system, or project management tool.project management tool.


Advanced networks Advanced networks require advanced require advanced

trouble ticket trouble ticket systems.systems.


FlexibilityFlexibility- While a TTS needs to be robust, it While a TTS needs to be robust, it does need to be lightweight and does need to be lightweight and flexible enough for customization and flexible enough for customization and integration with other NOC systems integration with other NOC systems and tools. and tools.

- Large enterprise wide behemoths like Large enterprise wide behemoths like Peregrine or Remedy can be an Peregrine or Remedy can be an obstacle to the flexibility required of a obstacle to the flexibility required of a NOC.NOC.


FlexibilityFlexibility- The TTSThe TTS should scale to the needs of should scale to the needs of both the NOC and the user community.both the NOC and the user community.

- Members of the user community like Members of the user community like to have a place to look up the status of to have a place to look up the status of a ticket on the web. NOC’s should a ticket on the web. NOC’s should offer this service.offer this service.

- Sometimes it is helpful for users to Sometimes it is helpful for users to actually access the system itself, with actually access the system itself, with limited user permissions, to look at limited user permissions, to look at their tickets, and even to provide their tickets, and even to provide updates.updates.


Intuitive User InterfaceIntuitive User Interface- A helpful way to organize the user A helpful way to organize the user interface is to nest certain fields within interface is to nest certain fields within the schema so as to not inundate the the schema so as to not inundate the user with 17 trillion fields laid out on a user with 17 trillion fields laid out on a flat screen.flat screen.

- Should oShould organize certain fields into rganize certain fields into ticket types, based on the action ticket types, based on the action needed. These fields reside in nested needed. These fields reside in nested dependencies.dependencies.

- This way related fields can be This way related fields can be organized sensibly, and only seen when organized sensibly, and only seen when needed.needed.


Ticket TypesTicket Types


Scheduled Maintenance Ticket-Type Scheduled Maintenance Ticket-Type WindowWindow


Provision/Modify/Decom Ticket Type Provision/Modify/Decom Ticket Type Window Window


Another fundamental question asked Another fundamental question asked with dependent nested fields is “Is this with dependent nested fields is “Is this

network impacting?”network impacting?”


Intuitive User InterfaceIntuitive User Interface- Define fields with different user Define fields with different user permissions - write or read only, some permissions - write or read only, some for Engineering, some for the Service for Engineering, some for the Service Desk (front-line staff), etc.Desk (front-line staff), etc.

- Make certain fields mandatory before Make certain fields mandatory before changing the ticket’s status, handing it changing the ticket’s status, handing it back to the other group, and especially back to the other group, and especially before closing the ticket. before closing the ticket.

- Helps to capture the information that Helps to capture the information that really is needed and necessary.really is needed and necessary.


Advanced Reporting Advanced Reporting CapabilitiesCapabilities- - Network availability based on node, Network availability based on node, circuit, etc. circuit, etc.

- Trend reporting: Trend reporting: - Specific device failures (cards, etc.) Specific device failures (cards, etc.) - Problem source Problem source - How long a ticket is in the hands of How long a ticket is in the hands of the Servicethe Service Desk, Engineering, Vendor, Desk, Engineering, Vendor, Customer.Customer.- Other options, if system is robust - Other options, if system is robust enough enough


Advanced Reporting Advanced Reporting CapabilitiesCapabilities- Time to resolution Time to resolution - Business activity during a specific Business activity during a specific time frametime frame

- How many tickets createdHow many tickets created- What kind of ticketsWhat kind of tickets- When createdWhen created- Created by whoCreated by who- Break stats out by network or Break stats out by network or supportsupport operation, etc.operation, etc.


Advanced Reporting Advanced Reporting CapabilitiesCapabilities- Make sure you have the granularity - Make sure you have the granularity needed to perform the reports and needed to perform the reports and searches you need. More fields may searches you need. More fields may mean a bit more work, but it saves you mean a bit more work, but it saves you time because…time because…

- - NOTHINGNOTHING can be can be WORSEWORSE than than staying up all night staying up all night MANUALLYMANUALLY going going through 215 outage tickets, looking for through 215 outage tickets, looking for RMA numbers for a report!!!RMA numbers for a report!!!


CARDINAL RULE OF CARDINAL RULE OF TICKETING!TICKETING!

Your system is only as accurate Your system is only as accurate as the data you put in it!!!as the data you put in it!!!


Managing Work FlowManaging Work Flow- Master Ticket and Sub-Task Master Ticket and Sub-Task (parent/child(parent/childrelationships)relationships)

- Helps with issues that require Helps with issues that require multiple separate actions, often by multiple separate actions, often by different support groups (Field Service different support groups (Field Service Requests, RMA’s, customer Requests, RMA’s, customer notification, etc.) notification, etc.)

- Helps keep tickets tidier, rather than - Helps keep tickets tidier, rather than the infamous massively long ticket with the infamous massively long ticket with everything in it including the kitchen everything in it including the kitchen sink and the bit bucket.sink and the bit bucket.


Managing Work FlowManaging Work Flow- How do you handle the issue of a How do you handle the issue of a problem with one of your networks, problem with one of your networks, causing a related problem with another causing a related problem with another network you support? network you support?

- Each deserves a separate ticket, for Each deserves a separate ticket, for record keeping purposes, etc. record keeping purposes, etc.

- Must have a way to automatically Must have a way to automatically copy the original ticket to the other copy the original ticket to the other network project (or queue), then edit network project (or queue), then edit the new ticket as appropriate.the new ticket as appropriate.


Managing Work FlowManaging Work Flow- Need ways to segregate tickets by the Need ways to segregate tickets by the specific networks you are supporting.specific networks you are supporting.

- But also need ways to see an But also need ways to see an aggregation of tickets, to help manage aggregation of tickets, to help manage workflow, for reporting purposes, etc. workflow, for reporting purposes, etc.


Managing Work FlowManaging Work Flow- An excellent method to manage NOC An excellent method to manage NOC work flow is through a series of “Next work flow is through a series of “Next Action” fields: Item, Day, Time.Action” fields: Item, Day, Time.

- These can be viewed through a tool like These can be viewed through a tool like “TickMon.”“TickMon.”

Sort by date and time (expired, Sort by date and time (expired, upcoming)upcoming)

Lists Next Action ItemLists Next Action Item

Sorts by network, support group, or Sorts by network, support group, or user user


Managing Work FlowManaging Work Flow


Managing Work FlowManaging Work Flow- Automated escalation – to Automated escalation – to management, network administration, management, network administration, etc.etc.

- Switch time to UTC.Switch time to UTC.

- Links to internal documentation: field Links to internal documentation: field definitions, time zone converters, definitions, time zone converters, databases, tools, linkage to vendor databases, tools, linkage to vendor sites for case and RMA info, shipping sites for case and RMA info, shipping updates (Fed Ex, etc.)updates (Fed Ex, etc.)


Customer NotificationCustomer Notification- TTS should provide the raw data for TTS should provide the raw data for customer notification.customer notification.

- Make sure editing is done in the Make sure editing is done in the ticketing system, and not through the ticketing system, and not through the notification tool interface.notification tool interface.

- This way the ticket information is This way the ticket information is always the most up-to-date. always the most up-to-date.


Customer Notification ToolCustomer Notification Tool


Calendar FunctionsCalendar Functions- Have the capability to automatically - Have the capability to automatically publish calendars from the TTS for:publish calendars from the TTS for:

- Scheduled MaintenancesScheduled Maintenances

- Field Service Requests- Field Service Requests

- RMAs- RMAs- Maintenance FreezesMaintenance Freezes

- Outages- Outages


Database IntegrationDatabase Integration- Find Circuit(s) from Route Find Circuit(s) from Route - Find Element(s) from Element Find Element(s) from Element - Find Circuit(s) from Customer Find Circuit(s) from Customer - Find Circuit(s) from Element Find Circuit(s) from Element

- Need solid database back-end in Need solid database back-end in your TSS!!!your TSS!!!


Trouble Ticket ExchangeTrouble Ticket Exchange- Designed to allow quick and clear Designed to allow quick and clear sharing of information between trouble sharing of information between trouble ticket systems in a secure and reliable ticket systems in a secure and reliable environment.environment.

- The exchange of this data is handled The exchange of this data is handled via automated email between the ticket via automated email between the ticket systems.systems.

- Scripts and parsers in the respective Scripts and parsers in the respective ticket systems receive such email and ticket systems receive such email and insert the data into the appropriate insert the data into the appropriate tickets and fields. tickets and fields.


Trouble Ticket ExchangeTrouble Ticket Exchange- The end user (home ticket system), The end user (home ticket system), will see their own ticket properly will see their own ticket properly updated with data from a foreign user updated with data from a foreign user (external ticket system), and vice (external ticket system), and vice versa.versa.

- Clearly developed workflow - Clearly developed workflow guidelines and rules are needed to guidelines and rules are needed to ensure a trouble-free ticket exchange. ensure a trouble-free ticket exchange.

- Loops between the ticketing systems Loops between the ticketing systems are another possible danger in an are another possible danger in an automated ticket exchange. automated ticket exchange.


Trouble Ticket ExchangeTrouble Ticket Exchange- Potentially a very powerful tool for Potentially a very powerful tool for NOC-to-NOC communication.NOC-to-NOC communication.

- Global NOC doing it within OSG - Global NOC doing it within OSG between Support Centers (IU -> Fermi between Support Centers (IU -> Fermi Lab, EGEE)Lab, EGEE)

- Currently not done very often. Currently not done very often. Difficult and time consuming.Difficult and time consuming.

- Easier between like systems (Remedy Easier between like systems (Remedy to Remedy)to Remedy)


Trouble Ticket ExchangeTrouble Ticket Exchange- There are many challenges, as each There are many challenges, as each ticketing system has its own unique ticketing system has its own unique features, characteristics, and features, characteristics, and restrictions.restrictions.

- Trouble ticket exchange is an ever Trouble ticket exchange is an ever changing process as ticket systems will changing process as ticket systems will evolve, schema changes, etc. Serious evolve, schema changes, etc. Serious challenge to keep updated. challenge to keep updated.

- - But how does this scale when your But how does this scale when your NOC needs to exchange information NOC needs to exchange information and tickets with a number of NOC’s, and tickets with a number of NOC’s, each with their own TTS?each with their own TTS?


ConclusionConclusion- With the challenges being presented - With the challenges being presented to NOCs today - optical networking, to NOCs today - optical networking, dynamic provisioning, monitoring dynamic provisioning, monitoring issues, security, etc, the need for issues, security, etc, the need for seemless and meaningful seemless and meaningful communication between NOC’s communication between NOC’s continues to grow.continues to grow.

- NOC-to-NOC communication is - NOC-to-NOC communication is essential in the advanced R&E world.essential in the advanced R&E world.


ConclusionConclusion-The sharing of information between The sharing of information between NOC’s must increase. We must view NOC’s must increase. We must view the NOC’s, and the important roles the NOC’s, and the important roles they play, as a key part of the R&E they play, as a key part of the R&E system. TTS needs to have a central system. TTS needs to have a central role in this.role in this.

- I invite the community to think about - I invite the community to think about these communication and operational these communication and operational issues.issues.


QUESTIONS?QUESTIONS?


Steve PeckSteve Peck

Global Research NOCGlobal Research NOC

[email protected]@iu.edu

http://globalnoc.iu.eduhttp://globalnoc.iu.edu

advanced trouble ticket systems steve peck operations manager global research noc i2 joint techs –...

Documents

separate ticket

long ticket

trouble ticket system

noc systems

work flow master ticket

amazing advanced services

user community

work order system