san antonio findings
TRANSCRIPT
-
8/9/2019 San Antonio findings
1/68
-
8/9/2019 San Antonio findings
2/68
-
8/9/2019 San Antonio findings
3/68
-
8/9/2019 San Antonio findings
4/68
-
8/9/2019 San Antonio findings
5/68
-
8/9/2019 San Antonio findings
6/68
-
8/9/2019 San Antonio findings
7/68
-
8/9/2019 San Antonio findings
8/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
2
(31) Summary of Interview
(32) NR 1B MPDE Photos
(33) Work Item 233-11-001, Reservation Task Request 33
(34) SAN ANTONIO Engineering Log of Nov 09
(35) NR 2A MPDE Summary of Inspection(36) NR 2A MPDE Photos
(37) MARMC Lab Rpt #2009NN03572 of 4 Dec 09
(38) MARMC Lab Rpt #2010NN00038 of 7 Jan 10
(39) MARMC Lab Rpt #2009NN03466 of 23 Nov 09
(40) MPDE Lube Oil Wetted Parts List
(41) LO Service System Description and Diagram
(42) E-mail of 10 Nov 09
(43) LO Piping Material Specification
(44) LO Pump Parts List
(45) LO System Valve Assembly
(46) LO Service System 6 Inch Valve Diagram(47) LO Service System H Valve Diagram
(48) S9233-DL-010
(49) MRC 2331 W-1R
(50) Investigator Notes (CoS)(3M Program Review)
(51) Maintenance Bucket Pictures
(52) SAN ANTONIO LO Log Aug 09
(53) SAN ANTONIO Engineering Log of Sep 09
(54) SAN ANTONIO LO Log Sep 09
(55) SAN ANTONIO LO Log Oct 09
(56) SAN ANTONIO LO Log Nov 09
(57) memo 5830 of 14 Jan 10 (Muslin Bag)
(58) Muslin Bag LO Flush Photos
(59) LO Flush Manual
(60) NSTM 233
(61) CLARUS Reports
(62) Summary of Interview
(63) Fairbanks Morse Condition Reports
(64) memo 5830 of 14 Jan 10 (Bearing
Inspection)
(65) Bearing Damage Photos
(66) E-mail of 15 Jan 10
(67) JFMM Volume IV Chapter 4
(68) memo 5830 of 17 Dec 09 (MPDE MaterialHistory Logs)
(69) EM01 Quarterly Schedule
(70) MRC Card 2640 M-2
(71) MRC 2331 18M-12R
(72) EM02 Quarterly PMS Schedule
(73) MRC 2331 R-12W
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
9/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
3
(74) memo 5830 of 17 Dec 09 (3M Program)
(75) MRC 2331 R-13W
(76) MRC 2331 R-14M
(77) MRC 2331 U-35
(78) MRC 2620 U-1(79) MRC 2620 U-3
(80) memo 5830 of 15 Jan 10 (LOQM)
(81) SAN ANTONIO SAMM Oil Sample DB 6 Dec 09
(82) EOCC Procedures
(83) memo 5830 of 15 Jan 10 (Eng Dept Trng)
(84) Investigator Notes (CoS) (Training Program Review)
(85) E-mail of 22 Dec 09
(86) E-mail of 29 Dec 09
(87) Summary of Interview
(88) Summary of Interview
(89) Summary of Interview(90) ECS PP Brief
(91) E-mail 13 Oct 06
(92) PRESINSURV NORFOLK VA 132000Z Apr 07
(93) PRESINSURV NORFOLK VA 061500Z Dec 07
(94) CASREP 07075 08099 Summary
(95) e-mail of 16 Nov 07
(96) CRO Predeployment PowerPoint of Aug 08
(97) LPD 17 Deployment Status Report 27 Nov 08 to 12 Dec
08
(98) ail of 15 Dec 08
(99) E-mail of 12 Oct 06
(100) E-mail of 22 Oct 06(101) Post FOA Support PowerPoint
(102) e-mail 5 Sep 07
(103) SAN ANTONIO TSO List
(104) E-mail of 5 Jan 10
(105) E-mail 8 Jan 10
(106) NAVSEA Philadelphia Code 911 LPD 17 Class ICAS
Status Update
(107) ICAS PP Brief
(108) memo 5830 of 15 Jan 10
(109) LPD 17 Class NTSP 1 Jul 09 (Final-Approved)
(110) Letter of Promulgation for the ManagementResponsibilities
(111) LPD Training POM 12 PP Brief
(112) PSMD LPD 17 6 Jan 04
(113) PSMD LPD 17 Change One 25 Feb 05
(114) PSMD LPD 17 Change Two 9 Apr 07
(115) E-mail of 11 Jan 10
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
10/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
4
(116) PEO Washington DC 301804Z Mar 09 LPD 17 Class FY09
Interim Training Plan for Main Propulsion
(117) PEO Washington DC 171810Z Oct 09 LPD 17 Class FY10
Interim Training Plan for Main Propulsion
(118) LPD 17 Class MPDE-SSDG-ECS Interim Training PP(119) E-mail of 18 Dec 09
(120) EOOW PQS NAVEDTRA 43104
(121) Engineering Collaterals and Qualifications PQS
NAVEDTRA 43704
(122) CO, USS SAN ANTONIO (LPD 17) ltr 5100 Ser 00 of 26
Mar 08
(123) CO, USS SAN ANTONIO (LPD 17) ltr 5100 Ser 00 of 3
Jun 09
(124) CO, USS SAN ANTONIO (LPD 17) ltr 5100 Ser 00 of 1
Sep 09
PRELIMINARY STATEMENT
1. Pursuant to enclosure (1), and in accordance with reference(a), I conducted a command investigation into the facts and
circumstances surrounding the discovery of metal shavings on the
oil side of three of four of USS SAN ANTONIO (LPD 17) Main
Propulsion Diesel Engines (MPDE) and a review of the maintenance
performance and quality control efforts by shipyard facilities,
intermediate maintenance activities, and SAN ANTONIO. Enclosure
(2), a list of relevant acronyms and abbreviations, is included
to assist in a review of the investigation.
2. Executive Summarya. Metal Shavings
(1) In October 2008, MARMC/NNSY voyage repair team
conducted significant repairs to the MPDE lube oil service
system piping onboard SAN ANTONIO due to failure of several
welded joints. The repair included a partial use of carbon
steel piping. Starting June 2009 during CNO PMA, NNSY conducted
permanent repairs of the piping that included the replacement of
approximately 80% of the existing external lube oil servicepiping with new 300 series stainless steel piping. After the
completion of the piping repairs, CLARUS using the NAVSEA
procedure conducted a flush of the piping. However, as designed
the flushing procedure was inadequate to completely remove
contaminants from the lube oil service system.
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
11/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
5
(2) In November and December 2009, NSSA Diesel
Inspectors and SAN ANTONIO personnel discovered foreign matter
in the lube oil service system for NR 1A, 1B, and 2A MPDE. The
foreign matter included 300 series stainless steel. The
probable source of the 300 series stainless steel found in NR1A, 1B, and 2A MPDE lube oil service systems was the byproducts
of the production and installation of stainless steel lube oil
service system piping. The cause of the stainless steel was
potentially the result of poor weld joint fit up during new
construction that created contaminate traps that could allow for
an intermittent release of contaminates.
(3) In November 2009, NSSA determined that depot level
maintenance was required to address excessive crankshaft main
bearing and thrust bearing wear in 1A MPDE, 1B MPDE, and 2A
MPDE. This bearing damage is the primary component of MPDEcorrective maintenance since completion of the PMA in October
2009. This damage is not related to the metal shavings found in
the MPDE lube oil service systems. Bearing failure root cause
is lube oil contamination that occurred during the Shipbuilding
process to include all repairs and maintenance prior to the PMA.
Total estimated cost of this additional MPDE corrective
maintenance is $7.5M. The operational impact is such that
repairs may preclude SAN ANTONIO executing her next scheduled
deployment.
(4) An alternate means of contamination causing bearing
damage based on technical evaluation of bearing failures on LPD
17 and LPD 21 has been proposed by NAVSEA during a recent video
teleconference with Fleet Forces Command, Second Fleet,
Commander Naval Surfaces Forces Atlantic and Expeditionary
Strike Group Two on 8 January. This hypothesis suggests that
the lube oil contaminates that are circulated back to the MPDE
sump via the self-cleaning pre-filter are continuously reduced
in physical size by the macerating action of the lube oil
service pumps until the contaminates are small enough to pass
through the filter and duplex strainers. Contaminates that pass
through the filter and strainer complex could still be large
enough to cause bearing damage. NAVSEA is considering anemergent alteration to eliminate this mechanism of contaminate
introduction into the diesel complex by filtering out
contamination discharged by the filter back to the sump.
(5) Ships Force was slow to recognize lube oil
contamination for a variety of long term issues. Command
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
12/68
-
8/9/2019 San Antonio findings
13/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
7
(6) An external review to validate if Sailor/Officertraining and NAVSEA conditioned based maintenance reach backsystems were in place to support the lead ship of a new classthat had a crew that was "right sized" with these externalassumptions in place,
(7) A review of the automated systems designed to supportshipboard operations including the Engineering Control System(ECS), Integrated Condition Assessment System (ICAS) and TotalShip Training System (TSTS) was conducted to evaluate theeffectiveness of these systems to support ship's force's abilityto operate the propulsion plant per the current ship's manningdocuments,
(8) An audit by using the appropriate Technical WorkDocuments (TWD) and associated Objective Quality Evidence (OQE)
against Joint Fleet Maintenance Manual (JFMM), NAVSEA StandardItems, Specification Items, and other appropriate technicalguidance to assess Depot and MSR corrective maintenanceperformance,
(9) A review of ship's force corrective and preventivemaintenance by examining preventive maintenance scheduling andexecution,
(10) Oral interviews with the chain of command,
(11) A review of NAVSEA technical documentationdescribing how SAN ANTONIO class ships are to manned, trained and
provided external support was reviewed to determine completenessand effectiveness of execution,
(12) A briefing from NAVSEA representatives on theEngineering Control System (ECS) and Integrated Condition
Assessment System (ICAS), and
(13) A review of the syllabus for training provided bySurface Warfare Officer School (SWOS) to prospective SAN ANTONIOclass Commanding Officers, Executive Officers and ChiefEngineers.
4. The Investigating Officer has the following priorengineering/maintenance experience: USS HUNLEY (AS 31) Planning
and Estimating Officer (12 months), Repair Duty Officer (12
months), and Auxiliaries Officer (11 months); USS LEXINGTON (AVT
16) Main Engines Officer (19 months); Steam Engineering Officer
of the Watch (EOOW) Instructor and Course Coordinator (30
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
14/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
8
months); and USS MOUNT HOOD (AE 32), Chief Engineer (CHENG) (20
months).
5. The following personnel assisted in the conduct of the
investigation (prior engineering/maintenance experience listed):
a. Chief of Staff, Expeditionary
Strike Group TWO: BS Mechanical Engineering, University of
Wisconsin 1983; MA Systems Engineering, Naval Postgraduate
School 1991; Nuclear Power School 1984; Nuclear Power Training
Unit, Idaho Falls (A1W Prototype) 1985; USS LONG BEACH (CGN 9),
Number Two Engine Room Division Officer (36 months); USS
ARKANSAS (CGN 41), Main Propulsion Assistant (MPA) (16 months);
USS TRUXTUN (CGN 35), CHENG (22 months); Atlantic Fleet Nuclear
Propulsion Examination Team (NPEB), Junior Team Member (28
months); and USS EISENHOWER (CVN 69) Reactor Officer (28months).
b. N4, Expeditionary Strike Group
TWO: USS SCOTT (DDG 995), MPA (30 months); USS CHINOOK (PC 9),
CHENG (36 months); Mk 5 Special Operations Craft, Officer in
Charge/Assistant Officer in Charge (38 months); USS KEARSARGE
(LHD 3), MPA (24 months); USS WASP (LHD 1), CHENG (18 months);
and Auxiliary Floating Dry Dock Dynamic, Commanding Officer (23
months).c. N43, Naval Surface Force
Atlantic: USS NASSAU (LHA 4), Assault Officer (24 months),
Auxiliaries Officer (18 months), and Repair Officer (six
months); Afloat Training Group, Atlantic, Training Liaison
Officer (24 months); and USS OAK HILL (LSD 51), CHENG (13
months).
d. Engineering Support,
Expeditionary Strike Group TWO: OAK HILL, Electrical Officer (28
months).
e. Engineering Support,
Expeditionary Strike Group TWO: NASSAU, Electrical Officer (17months).
f. Diesel Engine Inspector
(Code 216.5) Diesel Engine Inspector, Norfolk Naval Shipyard:
USS MESA VERDE (LPD 19), Top Snipe (24 months); USS SAN ANTONIO,
Pre-Commissioning Unit (PCU) for MESA VERDE (six months); USS
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
15/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
9
NEW ORLEANS, PCU for MESA VERDE (six months); USS CARTER HALL
(LSD 50), PCU for MESA VERDE; OAK HILL, M-Division Leading Chief
Petty Officer (eight months); and HSV-X-1, Lead Engineman (12
months).
g. Maintenance and Material
Management System Chief Petty Officer, Amphibious Squadron SIX:
USS REID (FFG-30) (36 months); USS MCKEE (AS-41) (24 months);
and USS CORMORANT (MHC 57), EA01 Leading Petty Officer (48
months).
h. , Norfolk Ship Support Activity (NSSA):
Merchant Marine Officer (9yrs) U.S Coast Guard licensed
Engineering Officer on medium speed diesel powered ships; U.S.
Navy Port Engineer (12 years) DDG Class/CG Class Class
maintenance management; NSSA LSD/LPD Class Team Leader (threeyears) Managed RMC Waterfront Operations Department LSD/LPD
4/LPD 17 Class Maintenance Teams; NSSA CVN/Amphibious Ship
Division Manager (one year) - Principal Waterfront Operations
Department advisor in matters relating to CVN and amphibious
ship maintenance.
i. , Code 212.5, Nondestructive
Testing Level III, Quality Assurance Supervisor, NSSA: Retired
HTC; engineering plant operator/watch stander (MMOW) (four
years); ship/submarine maintenance, repair and quality assurance
(22 years); and contractor oversight (seven years).
j. Quality Assurance
Specialist with Code 212.5, Norfolk Naval Shipyard Engineering
and Technical Fleet Support Department: Retired HTC; NAVSEA
certified NDT Level II Inspector (22 years); and NAVSEA
certified NDT Level III Examiner (eight years).
k. Project Team Leader, Norfolk Naval
Shipyard (NNSY), USS BATAAN (LHD 5): prior experience as
Supervisory Assistant Project Superintendent: SAN ANTONIO, WASP,
USS NASSAU, and USS GEORGE WASHINGTON (CVN 73); NNSY Production
Department (30 years).
6. Expeditionary Strike
Group TWO, and Expeditionary Strike
Group TWO provided administrative support in the course of this
investigation.
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
16/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
10
7. provided legal
guidance in the course of this investigation.
FINDINGS OF FACT
Metal Shavings Discovery and Causation
Introduction
1. During a deployment to Fifth Fleet Area of Operationsbetween 9 October and 25 October 2008, SAN ANTONIO suffered
repetitive weld joint failures in the MPDE lube oil service
system piping. [Encls. 3, 4, 5)]
2. The weld joint failures were a result of poor weldtechniques. [Encls. (5, 6)]
3.The poor weld techniques during new construction causedexcessive oxidation and stainless steel globules inside the lube
oil service system piping. [Encl. (6)]
4. The poor weld joint fit up during new construction createdtraps. [Encls. (5, 6)]
5. From 2 to 23 November 2008, MARMC/NNSY voyage repair teamconducted repairs on the lube oil piping inport Bahrain. [Encls.
(3, 4)]
6. The repair included piping replacement using 304 stainlesssteel and carbon steel piping. [Encls. (3, 4)]
7. The repairs did not include a flush of the lube oil system.[Encls. (3, 4)]
8. Upon completion of repairs, the lube oil service systemserving each engine was operated in normal configuration. [Encl.
(7)]
9. Ships force continued to monitor lube oil duplex strainercontent until the baskets were clear of foreign material and
particulate. [Encl. (7)]
10. On 27 March 2009, SAN ANTONIO returned from the deployment.[Encl. (8)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
17/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
11
11. On 17 June 2009, SAN ANTONIO began a Chief of NavalOperations Planned Maintenance Availability (CNO PMA). [Encl.
(8)]
12. During the CNO PMA, NNSY replaced approximately 80% of theexisting external lube oil service system piping serving allfour MPDEs with new 300 series stainless steel piping. [Encls.
(9-15)]
13. After completion of the lube oil service system pipingrepairs, a flush of the lube oil system occurred. [Encl. (9)]
14. During the CNO PMA, contract personnel conducted thefollowing MPDE repairs: NR 2A1 lube oil service pump, NR 1A and
NR 1B MPDEs valve cover software replacement, NR 1B MPDE
crankcase fan oil seal, and NR 2A MPDE low pressure air lineline. [Encl. (9)]
15. On 2 October 2009, SAN ANTONIO completed a light offassessment (LOA). [Encl. (8)]
16. On 7 October 2009, SAN ANTONIO conducted sea trials.[Encl. (8)]
NR 1A MPDE Foreign Matter Discovery Timeline
17. On 29 July 2009, work started to replace NR 1A MPDEcylinder head (16 total) valve cover gaskets and software
crankcase in accordance with NSTM S9233-DL-MMM-020 (Chapter 7)
and NSTM S9233-DL-MMM-010. [Encl. (16)]
18. On 19 August 2009, CHENG, closed out NR 1A MPDElube oil sump. [Encl. (17)]
19. On 11 September 2009, work completed to replace NR 1A MPDEcylinder head (16 total) valve cover gaskets and software
crankcase in accordance with NSTM S9233-DL-MMM-020 (Chapter 7)
and NSTM S9233-DL-MMM-010. [Encl. (16)]
20. If normal cleanliness standards are adhered to during theexecution of the work, a comprehensive lube oil service system
flush is not required. [Encl. (18, 19)]
21. Normal cleanliness standards were met prior to commencementof production work. [Encl. (20)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
18/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
12
22. On 7 October 2009 at 1903 while operating, NR 1A MPDEsuffered an engineering casualty: unusual noise and vibration
on NR 9 cylinder. [Encl. (21)]
23. On 7 October 2009 at 1955, a lube oil casualty sample wastaken from NR 1A MPDE. [Encl. (21)]24. On 14 October 2009, CASREP submitted on NR 1A MPDEregarding loud metallic noise. [Encl. (22)]
25. The CASREP indicated no unusual conditions were noted on NR9 cylinder. [Encl. (22)]
26. On 21 October 2009, CASCOR submitted on NR 1A MPDEregarding loud metallic noise. [Encl. (22)]
27. On 22 November 2009, Naval Surface Warfare Center CarderockDivision-Ship Systems Engineering Station Philadelphia (NSWCCD-
SSES Philly) reported abnormal wear levels in NR 1A MPDE based
on a lube oil sample submitted on 7 October 2009. [Encl. (23)]
28. Due to the abnormal wear levels, NSWCCD-SSES Phillyrecommended an inspection of NR 1A MPDE. [Encl. (23)]
29. On 25 November 2009, the Commanding Officer, SAN ANTONIO,requested Port Engineer, and , CHENG,
consult with NSWCCD-SSES Philly, for MPDE
inspection recommendations. [Encl. (23)]
30. On 25 November 2009, requestedand Norfolk Naval
Shipyard Diesel Inspectors, provide inspection recommendations.
[Encl. (23)]
31. On 25 November 2009, conducted the inspectionof the lube oil service system and an inspection of NR 1A MPDE.
[Encls. (23-25)]
32. During the inspection of NR 1A MPDE, the engine remainedstatic and was not barred over. [Encl. (25)]
33. The inspection of NR 1A MPDE included a visual inspectionof cam bearing and journals, and rod bearing and journals, and
main bearing and journals. [Encl. (25)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
19/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
13
34. The online strainer exhibited no particulates. [Encls.(25, 26)]
35. The off-line strainer exhibited ferrous particulate (fine)on the magnet that if accumulated would be less than 1% of themagnet cylinder surface. [Encls. (25, 26)]
36. The back-flushing filter exhibited particulate: metalshavings, paint chips, gasket material, fine debris located in
remnants of fibrous material in 20 candles of 66. [Encls. (25,
26)]
37. The cam covers accesses exhibited no debris orparticulates. [Encls. (25, 26)]
38.The dry sump exhibited no solid material. [Encl. (25)]
39. In the dry sump, there was free water standing in the oil.[Encl. (25)]
40. recommended the wet sump be pumped out andcertified gas free for further inspection. [Encl. (25)]
41. On 1 December 2009, wet sump cleaned and gas freed. [Encls.(25)]
42. During the initial visual inspection of the wet sump, thewet sump exhibited a higher than expected amount of babbitt
material. [Encl. (25)]
43. In the wet sump, there was free water standing in theresidual oil. [Encl. (25)]
44. On 15 December 2009, on-line and off-line Rocker DuplexStrainers removed by Ships Force and inspected by DEI
. [Encl. (24)]
45. On-line and Off-line Rocker Strainers exhibited free water,dirt, and paint chips (white, terracotta, and yellow). [Encl.(24)]
46. Rocker Sump opened by Ships Force and inspected by DEI[Encl. (24)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
20/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
14
47. Rocker Sump exhibited free water, dirt, and paint chips.[Encl. (24)]
NR 1B MPDE Foreign Matter Discovery Timeline
48. On 21 July 2009, work tasked to repair NR 1B MPDE crankcasefan oil seal. This repair had an estimated start date of 27July 2009 and was signed complete on 25 August 2009. [Encl.
(27)]
49. On 27 July 2009, work started to repair NR 1B MPDEcrankcase fan oil seal. [Encl. (27)]
50. On 29 July 2009, work started to replace NR 1B MPDE cylinderhead (16 total) valve cover gaskets and software crankcase in
accordance with NSTM S9233-DL-MMM-020 (Chapter 7) and NSTM
S9233-DL-MMM-010. [Encl. (28)]
51. On 25 August 2009, work completed on NR 1B MPDE crankcasefan oil seal. [Encl. (27)]
52. On 26 August 2009 at 1140, Chief Engineer,closed out NR 1B MPDE lube oil sump. [Encl. (17)]
53. On 11 September 2009, work completed to replace NR 1B MPDEcylinder head (16 total) valve cover gaskets and software
crankcase in accordance with NSTM S9233-DL-MMM-020 (Chapter 7)
and NSTM S9233-DL-MMM-010. [Encl. (28)]
54. If normal cleanliness standards are adhered to during theexecution of the work, a comprehensive lube oil service system
flush is not required. [Encl. (18, 19)]
55. Work would not have commenced prior to cleanlinessstandards being met. [Encls. (29-31)]
56. On 7 October 2009 at 1010 while operating NR 1B MPDE, NR 1BMPDE suffered an engineering casualty: hot bearing on NR5 main
bearing (172 degrees F). [Encl. (21)]
57. The hot bearing was a 21 degree spread from the nexthighest bearing temperature. [Encl. (21)]
58. On 7 October 2009 at 1010, NR 5 main bearing (172 degreesF) was 21 degrees F higher than the next highest bearing
temperature. [Encl. (21)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
21/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
15
59. On 1 December 2009, filter removed for NR 1B MPDE due toissues with NR 1A and 2A MPDE. [Encl. (24)]
60. Lube oil results for NR 1B had been satisfactory (did notindicate need to conduct inspection). [Encl. (24)]61. Filter element removed. Set on work bench. Amount ofbabbitt material on filter and filter candles. Had fibrous
material as well. No further inspection. [Encl. (24)]
62. On 15 December 2009, on-line and off-line strainers wereremoved by ships force and inspected by DEI [Encl.
(24)]
63.On-line strainer exhibited fine ferrous particulate on themagnet, accumulated debris did not exceed the circumference of a
dime. [Encls. (24, 32)]
64. Off-line strainer exhibited fine ferrous particulate on themagnet, accumulated debris did not exceed the circumference of a
dime. [Encls. (24, 32)]
65. On 15 December 2009, on-line and off-line Rocker DuplexStrainers removed by Ships Force and inspected by DEI
[Encl. (24)]
66. On-line and Off-line Rocker Strainers exhibited dirt,gasket material, paint chips and free water. [Encls. (24, 32)]
67. Rocker Sump opened by Ships Force and inspected by DEI[Encl. (24)]
68. Rocker Sump exhibited free water, dirt, and paint chips.Sump bottom was coated with a large amount of fine bronze
material and non-ferrous matter similar to babbitt. [Encls. (24,
32)]
69. On 16 December 2009, 1B MPDE lube oil sump pumped, 600gallons of lube oil remained at the pump end of the sump. [Encl.
(24)]
70. On 17 December 2009, 1B MPDE lube oil sump opened forinspection. [Encl. (24)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
22/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
16
71. At the sump cover opening ferrous particulate, paint chips,dirt and carbon were immediately visualized before entering the
sump. [Encls. (24, 32)]
72. The DEI hand skimmed the oil layer at the sump coveropening; this revealed fine particulate (bronze in color), nonferrous particulate resembling babbit. [Encls. (24, 32)]
NR 2A MPDE Foreign Matter Discovery Timeline
73. On 22 July 2009, work tasked to replace NR 2A MPDE lowpressure air line serving the engine barring gear (turning
gear). [Encl. (33)]
74. On 25 July 2009, work started and completed to replace NR2A MPDE low pressure air lube oil line serving the engine
barring gear (turning gear). [Encl. (33)]
75. The low pressure air line consisted of 3/8 inch stainlesssteel tube. [Encl. (33)]
76. On 13 August 2009, , Chief Engineer, closed out NR2A MPDE lube oil sump. [Encl. (17)]
77. On 14 August 2009 at 1815, NR 2A MPDE lube oil strainercheck point complete by Engineering Duty Officer in preparation
for engine flush. [Encl. (17)]
78. On 6 October 2009 at 0750, NR 2A MPDE emergency stopped dueto NR 8 hot bearing. [Encl. (21)]
79. On 6 October 2009 at 0822, casualty sample taken for NR 2AMPDE. [Encl. (21)]
80. On 6 October 2009 at 1210 NR 2A MPDE lube oil straineropened and inspected SAT. [Encl. (21)]
81. On 6 October 2009 at 1250, NR 2A MPDE lube oil servicesystem duplex strainer tested satisfactory. [Encl. (21)]
82. On 6 October 2009 at 1315, DEI, andverified positive lube oil flow on NR 2A MPDE NR 8 bearing.
[Encl. (21)]
83. On 6 October 2009 at 1734, bearing inspection on NR 2A MPDENR8 bearing revealed no apparent damage. [Encl. (21)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
23/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
17
84. DEI, and the CHENG closed out NR 8 bearing.[Encl. (21)]
85. On 2 November 2009 at 1043, NR 2A MPDE stopped due to hotbearing casualty. [Encl. (34)]
86. On 2 November 2009 at 1102, casualty sample drawn on NR 2AMPDE NR 2 bearing. [Encl. (34)]
87. On 2 November 2009 at 1154, Diesel Engine Inspector, CHENG, and inspected NR 2A MPDE NR 1 and
NR 2 main bearing. [Encl. (35)]
88. SAN ANTONIO personnel and found a quarter ofan inch ferrous metal sliver and several non-ferrous particles.
[Encls. (35, 36)]
89. On 2 November 2009, NR 2A MPDE lube oil service systemtagged out for inspection of strainer and back-flushing filter.
[Encls. (35, 36)]
90. The strainer candles were dislodged. [Encl. (35)]91. The strainer exhibited ferrous particulate (fine) on themagnet that if accumulated would be not exceed the area of a
dime. [Encls. (35, 36)]
92. The back-flushing filter exhibited ferrous and non-ferrousover 75% of the candles. [Encl. (35)]
93. The ferrous particles exhibited on the back-flushing filtermeasured under one half inch when stretched. [Encls. (35, 36)]
Identification of Foreign Matter
NR 1A MPDE
94. The foreign debris removed from NR 1A MPDE sump includedorganic and metallic materials. [Encl. (37)]
95. The organic materials consisted of items similar to paint,textile floor coverings, synthetic polymer Nylon, rubber,
hydraulic oil, Kevlar, glass fibers, hair, polypropylene, and
cellulose fibers. [Encl. (37)]
96. The metallic materials consisted of material similar to a300 Series Stainless Steel alloy. [Encl. (37)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
24/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
18
97. Three of the metallic samples provided appear to be from amachining operation. [Encl. (37)]
98. For comparison MARMC personnel analyzed known standards of304 Series Stainless Steel, 316 Series Stainless Steel, 321Series Stainless Steel, and 347 Series Stainless Steel alloys.
[Encl. (37)]
99. MARMC personnel cannot identify the precise composition ofthe metallic materials due to possible contamination. [Encl.
(37)]
NR 1B MPDE
100. The debris removed from NR 1B MPDE are consistent withcarbon steel of undetermined origin, and 300 Series Stainless
Steel alloys from a machining operation. [Encl. (38)]
NR 2A MPDE
101. The metallic debris removed from NR 2A MPDE lube oilservice system back-flushing filter consisted of a material
similar to a 300 Series Stainless Steel alloy from a machining
peration. [Encl. (39)]o
Internal Source of the Foreign Matter
MPDE
102. There are no parts of the MPDE wetted by lube oil thatcontain stainless steel. [Encl. (40)]
LO Service System
103. Each MPDE has its own lube oil sump, lube oil servicepumps, self-cleaning filter, heat exchanger and duplex strainer.
Lube oil is drawn from the sump via the installed positive
displacement lube oil pumps. [Encl. (41)]
104. Lube oil discharged from the lube oil service pumps entersa self-cleaning filter. [Encl. (41)]
105. The self-cleaning filter filtration is 30 micron absoluteand is equipped with an external (15PSID) and internal (29PSID)bypass that ensures the MPDE is not starved for oil if a high
differential pressure condition occurs in the filter. [Encl.
(421)]
106. The external bypass valve has been recently directed byappropriate technical authority to be disabled via a blank
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
25/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
19
flange to prevent contaminates bypassing the filter. [Encl.
(41)]
107. The filter has a rotating set of internal elements thatalternately filter the lube oil when in service. [Encl. (41)]
108. When the internal elements are not in service, they areflushed with lube oil discharged from the lube oil service
pumps. [Encl. (41)]
109. The filter effluent is discharged directly back to thesump, which results in the total mass of contaminates remaining
constant in the MPDE sump. [Encl. (6, 41)]
110. The filtered lube oil discharged from the self-cleaningfilter flows through a heat exchanger, header pressureregulating valve, a duplex strainer (55 micron absolute), and
then into the diesel complex where the lube oil is ultimately
discharged into the MPDE lube oil sump. [Encl. (41)]
111. The attached transfer pump and purifier bowl assembly ofthe lube oil purifier come into contact with the lube oil.
[Encl. (42)]
112. The transfer pump has carbon steel rotors and a bronzehousing. [Encl. (42)]
113. The purifier bowl is made of a special stainless steelalloy. [Encl. (42)]
114. The paring disc is made of 316 Series Stainless Steel.[Encl. (42)]
115. If the paring disc and purifier bowl come into contact,this would cause noticeable sound and vibration. [Encl. (42)]
116. Any contact will be evident on the paring disc top andbottom surfaces. [Encl. (42)]
117. The lube oil piping system contains 304 Series StainlessSteel, 316 Series Stainless Steel alloy and Carbon Steel (from
voyage repair). [Encl. (43)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
26/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
20
118. MPDE lube oil service pump special washers used to fastenthe pump casing to its suction inlet bell and mechanical seal
cover assembly are made from stainless steel. [Encl. (44)]
119. MPDE lube oil service pump special washers do not come incontact with lube oil during the normal operation of the lubeoil service system. [Encl. (44)]
120. The valves in the lube oil service system do not containstainless steel. [Encls. (45-47)]
MPDE Bearings
121. The MPDE bearings consist of a steel backing, top layerin overlay, babbit, and copper. [Encl. (48)]t
Internal Cause of Foreign Matter Introduction Into the LO SystemLO Service System Filling/Strikedown Procedures
122. MRC 2331 W-1R requires the drain and refill of the MPDElube oil loop seals. [Encl. (49)]
123. During the performance of MRC 2331 W-1R, SAN ANTONIOpersonnel indicated the use of a blue plastic bucket to perform
the maintenance check. [Encls. (50, 51)]
124. The blue plastic bucket is used to perform other tasks inthe propulsion spaces. [Encls. (50, 51)]
NR 1A MPDE
125. On 18 August 2009, the lube oil quality management (LOQM)log annotated zero gallons in the NR 1A MPDE lube oil sump.
[Encl. (52)]
126. On 9 September 2009 at 1738, the prior to transfer (PTT)sample on lube oil tank 6-65-2-F was satisfactory. [Encl. (53)]
127. On 9 September 2009 at 1855, a lube oil transfer from lubeoil tank 6-65-2-F to NR 1A MPDE lube oil sump commenced. [Encl.
(53)]
128. On 9 September 2009 at 2113, a lube oil transfer from lubeoil tank 6-65-2-F to NR 1A MPDE lube oil sump of 1647 gallons
completed. [Encl. (53)]
129. On 10 September 2009, the LOQM log annotated 2000 gallonsin the NR 1A MPDE lube oil sump. [Encl. (54)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
27/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
21
130. On 15 September 2009, the LOQM annotated 1750 gallons inthe NR 1A MPDE lube oil sump. [Encl. (54)]
131. On 18 September 2009 at 0915, the PTT sample on lube oiltank 6-68-2-F was satisfactory. [Encl. (53)]
132. On 18 September 2009 at 1350, there was 538 gallons of oiltransferred to NR 1A MDPE lube oil sump from lube oil tank 6-68-
2-F. [Encl. (53)]
133. On 8 October 2009 at 0329, the PTT sample from lube oiltank 6-122-1-F satisfactory. [Encl. (21)]
134. On 21 October 2009 at 1340, the PTT sample on lube oiltank 6-68-2-F was satisfactory. [Encl. (21)]
135. On 22 October 2009, the LOQM log annotated 1640 gallons inthe NR 1A MPDE lube oil sump. [Encl. (55)]
136. A lube oil transfer is not documented in the engineeringlog. [Encl. (21)]
137. On 23 October 2009 at 0639, the PTT sample on lube oiltank 6-121-1-F was satisfactory. [Encl. (21)]
138. On 4 November 2009 at 1548, NR 2 40WT lube oil purifierstarted purifying NR 1A MPDE sump to 6-117-1-F IAW PTLO. [Encl.
(34)]
139. On 19 November 2009, the LOQM log annotated 1657 gallonsin the NR 1A MPDE lube oil sump. [Encl. (56)]
140. A lube oil transfer is not documented in the engineeringlog. [Encl. (34)]
141. On 30 November 2009 at 0914, NR 1 40WT lube oil purifiertransfer pump started taking suction from NR 1A MPDE lube oil
sump discharging to lube oil tank 6-63-2-F IAW PTLO. [Encl.
(34)]
142. On 30 November 2009 at 1453, NR 1 40WT lube oil purifiertransfer pump secured IAW PTLO. [Encl. (34)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
28/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
22
143. On 30 November 2009 at 1453, a 1355 gallon transfer fromNR 1A MPDE lube oil sump to lube oil tank 6-63-2-F completed.
[Encl. (34)]
NR 1B MPDE144. On 18 August 2009, there was a 10 gallon increase in 1BMPDE sump (280 gallons to 290 gallons). No Prior to Transfer
sample taken. [Encl. (52)]
145. A lube oil transfer is not documented in the engineeringlog. [Encl. (17)]
146. On 9 September 2009 at 1247, the PTT sample on lube oiltank 6-68-2-F was satisfactory. [Encl. (53)]
147.On 9 September 2009 at 1309, NR 1 40 WT lube oil purifiertaking suction from lube oil tank 6-68-2-F to NR 1B MPDE. [Encl.
(53)]
148. On 9 September 2009 at 1328, the PTT sample on lube oiltank 6-46-2-F was satisfactory. [Encl. (53)]
149. On 9 September 2009 at 1329, a lube oil transfer commencedfrom lube oil tank 6-68-2-F to NR 1B MPDE sump. [Encl. (53)]
150. On 9 September 2009 at 1738, the PTT sample on lube oiltank 6-65-2-F was satisfactory. [Encl. (53)]
151. On 9 September 2009 at 1729, a transfer of 2118 gallons oflube oil to NR 1B MPDE was completed. [Encl. (53)]
152. On 10 September 2009, the LOQM log annotated 2025 gallonsin NR 1B MPDE sump. [Encl. (54)]
153. On 14 September 2009, the LOQM log annotated 2225 gallonsin NR 1B MPDE sump. [Encl. (54)]
154. A lube oil transfer is not documented in the engineeringlog. [Encl. (53)]
155. On 15 September 2009, the LOQM log annotated 1769 gallonsin NR 1B MPDE sump. [Encl. (54)]
156. A lube oil transfer is not documented in the engineeringlog. [Encl. (53)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
29/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
23
157. On 22 October 2009, the LOQM log annotated 1940 gallons inNR 1B MPDE lube oil sump. [Encl. (55)]
158. A lube oil transfer is not documented in the engineeringlog. [Encl. (21)]
159. On 23 October 2009 at 0639, the sample on lube oil tank6-121-1-F was satisfactory. [Encl. (21)]
160. On 23 October 2009, the LOQM log annotated 1769 gallons inNR 1B MPDE lube oil sump. [Encl. (55)]
161. A lube oil transfer is not documented in the engineeringlog. [Encl. (21)]
NR 2A MPDE162. On 4 August 2009, the LOQM log annotated 380 gallons in NR2A MPDE lube oil sump. [Encl. (52)]
163. A lube oil transfer is not documented in the engineeringlog. [Encl. (17)]
164. On 8 August 2009, the LOQM log annotated 280 gallons in NR2A MPDE lube oil sump. [Encl. (52)]
165. A lube oil transfer is not documented in the engineeringlog. [Encl. (17)]
166. On 11 August 2009, the LOQM log annotated zero gallons inNR 2A MPDE lube oil sump. [Encl. (52)]
167. A lube oil transfer is not documented in the engineeringlog. [Encl. (17)]
168. On 15 September 2009 at 0850, the PTT sample on lube oiltank 6-122-1-F was satisfactory. [Encl. (53)]
169. On 15 September 2009 at 0923, NR 2 40WT lube oil purifieraligned and started filling NR 2A MPDE. [Encl. (53)]
170. On 15 September 2009, the LOQM log annotated 1416 gallonsin NR 2A MPDE lube oil sump. [Encl. (54)]
171. A lube oil transfer is not documented in the engineeringlog. [Encl. (53)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
30/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
24
172. On 26 September 2009, the LOQM log annotated 1657 gallonsin NR 2A MPDE lube oil sump. [Encl. (54)]
173. A lube oil transfer is not documented in the engineeringlog. [Encl. (53)]
174. On 7 October 2009, the LOQM annotated 1421 gallons in NR2A MPDE lube oil sump. [Encl. (55)]
175. A lube oil transfer is not documented in the engineeringlog. [Encl. (21)]
176. On 8 October 2009 at 0329, the PTT sample on lube oil tank6-122-1-F is satisfactory. [Encl. (21)]
177.On 8 October 2009 at 0340, NR 2 40WT lube oil transferpump aligned and started taking suction from 6-122-1-F and
discharging to NR 2A MPDE sump IAW PTLO. [Encl. (21)]
178. On 8 October 2009 at 0407, secured from transfer of lubeoil to NR 2A MPDE Sump IAW PTLO. [Encl. (21)]
179. On 9 October 2009, 1657 gal annotated in LOQM Logs, +237gal from previous day. [Encl. (55)]
180. The lube oil transfer is documented in the engineering logas occurring two days earlier and accounts for the 179 gallon
increase. [Encl. (21)]
181. On 16 October 2009 at 1236, a PTT sample on lube oil tank6-122-1-F was satisfactory. [Encl. (21)]
182. On 23 October 2009 at 0639, a PTT sample on lube oil tankon 6-121-1F was satisfactory. [Encl. (21)]
183. On 23 October 2009 at 0702, NR 2 40WT lube oil purifieraligned and started suction from 6-122-1-F discharging to NR 2A
MPDE sump. [Encl. (21)]
184. On 23 October 2009 at 0742, secured from transferring 40WTlube oil to NR 2A MPDE Sump. Transferred 445 gal. [Encl. (21)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
31/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
25
FOR OFFICIAL USE ONLY
Possible External Source of the Foreign Matter
LO System Flushing Components
185. The flushing procedure involves the use of muslin bagsthat contain kevlar to catch debris. [Encls. (57, 58)]
External Cause of the Foreign Matter Introduction Into the LO
System
Lube Oil Service System Flushing Process
186. The flushing process for the LPD 17 Class is NAVSEA S9233-DM-HBK-010-007 Diesel Engine Maintenance Standard Flush LPD 17,
MPDE Lube Oil System and Engine. [Encl. (59)]
187. The flushing procedure consists of the following steps:a. The MPDE sump is drained and refilled with a high
detergent, low viscosity special diesel lube oil.
b. The installed lube oil service pumps, self-cleaning
filter element and duplex strainer elements are removed to
prevent a low flow area that could allow the accumulation of
contaminants.
c. A flush rig with pumps, heaters, filters and associated
manifolds are installed as a temporary system.
d. The pumps provide a pressure source for the lube oil,
the heaters maintain lube oil temperature/viscosity at the
required value and the manifolds aligned flushing lube oil to
the appropriate components.
e. Muslin bags are installed in the temporary filter and
are used to capture any contaminants.
f. The individual flushes are timed flushes that are
complete once the contamination found in the muslin bags meets
acceptance criteria.
g. The temporary pumps take suction from the main sump and
in series with the filter discharge clean oil into the externaland/or internal piping where the contaminants in the piping are
discharged into the sump.
h. The general sequence is that the external lube oil
piping is flushed, then the MPDE internal lube oil
-
8/9/2019 San Antonio findings
32/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
26
piping/components are flushed followed by a combined (external
and internal) flush.
i. Upon completion of the flushes the sump is pumped down
and cleaned.
j. The LO Cooler flush can be accomplished at anytime
during the flushing evolution.
k. In addition to the temporary manifold installed to align
flushing lube oil to the external and/or internal piping, two
temporary valves were installed to bypass both pressure
regulating valves serving the lube oil system.
l. The government approved procedure fails to indicate the
required position of these valves at the start of the externalpiping flush.
[Encl. (60)]
188. NAVSEA S9233-DM-HBK-010-007 Note 2 (page 6) states themanual exceeds the requirements of NAVSEA S9086-HB-STM-010 NSTM
Chapter 233 Diesel Engines. [Encl. (59)]
189. NAVSEA S9233-DM-HBK-010-007 does not contain a requirementfor the lube oil sump to be cleaned prior to commencing the
flush. [Encl. (59)]
190. NSTM Chapter 233 paragraph 233-8.21.12.f states that priorto flushing the engine sump shall be drained and wiped clean
with lint free rags. [Encl. (60)]
191. NAVSEA S9233-DM-HBK-010-007 does not indicate location oramount of quadraplex filters required or installed during
flushing. [Encl. (59)]
192. NSTM Chapter 233, Figures 233-8-1 and 233-8-2 depictfilters installed immediately downstream of the flushing pump
and immediately upstream of the sump. [Encl. (60)]
193. NAVSEA S9233-DM-HBK-010-007 Note 18 states that allcheckpoints shall be verified, logged on enclosure (4) of the
procedure and signed by performing activity representative and
DEI before the next step can be performed. Verify with ships
force representative only if required. [Encl. (59)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
33/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
27
194. NAVSEA S9233-DM-HBK-010-007 Checkpoint 2 states that DEIor Ship Representative and the contractor shall inspect all
accessible engine areas to verify all foreign material and
debris has been removed. [Encl. (59)]
195. NAVSEA S9233-DM-HBK-010-007 Temporary valves (TV 1 and 2)are installed in the external piping, however the alignment of
these valves are not indicated in Table 1 (Valve Alignment).
Paragraph 3-1.13.1 requires that TVs be opened for a minimum of
5 minutes; however this occurs during the flushing process
allowing the potential for the valves to be improperly aligned
at the start of the flush. [Encl. (59)]
196. NAVSEA S9233-DM-HBK-010-007, paragraphs 1-5.1 and 1-5.4require removal of number 1 and 2 LO service pumps respectively,
prior to commencing flush. [Encl. (59)]
197. The lube oil pumps were not removed prior to the flush.[Encls. (61, 62)]
198. NAVSEA S9233-DM-HBK-010-007 does not require thecontractor to document the foreign material found during the
flush of the system. [Encl. (60)]
Damage Caused to the MPDE Bearings (to include cost)
Bearing Inspection Results
199. An inspection of the following bearings has beenconducted: NR 1A MPDE-Nr 2, 4, 6, and 8 main bearings and thrust
bearing; NR 1B MPDE-all main bearings and thrust bearing; and NR
2A MPDE-all main bearings and thrust bearing. [Encl. (63, 64)]
200. The bearing inspections revealed that all bearing surfacesshowed debris passed through. [Encl. (63-65)]
201. The bearing surfaces reflected debris similar to thefollowing items: industrial, bead blasting material, material
consistent with pipe welding process, dirt and sand, and water.
[Encl. (63-65)]
202. All main bearing backings reflected fretting at the shellback ends with oil was found on the lower bearing steel-backs
during the removal due to inadequate bearing crush. [Encl. (63-
65)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
34/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
28
203. NR 1A MPDE thrust bearing lower shell exhibited high heatmarks on the steel backing, which reflect a previous hot bearing
during operation. [Encl. (63-65)]
204. The date and time of the damage to NR 1A thrust bearingcannot be determined due to no ICAS records, i.e., systeminoperable or switched off. [Encl. (63, 64)]
205. NR 1B MPDE thrust bearing surfaces exhibited the passingof large and small debris. [Encl. (63-65)]
206. A 3 millimeter wide and 2 millimeter deep groove wasvisualized in the babbitt surface and embedded debris remained
in the bearing surface and journal. [Encl. (63-65)]
207.The thrust journal has grooves and cuts matching thedebris tracks in the bearing surface. [Encl. (63-65)]
208. The thrust bearing face showed wear consistent withGeisling mis-alignment/improper spacing. [Encl. (63-65)]
209. NR 2A MPDE thrust bearing surfaces exhibited the passingof small debris. [Encl. (63-65)]
210. The wear patterns were not consistent with the lowoperating hours. [Encl. (63, 64)]
211. All bearings removed and inspected meet the discardcriteria in Chapter 4 of the LPD17 Class PC2.5 STC Volume 1
Technical Manual. [Encl. (63, 64)]
212. The current cost estimate to repair NR 1A, 1B, and 2A MPDEis $7.5 million dollars. [Encl. (66)]
Maintenance Performance and Quality Control Efforts
Shipboard Engineering Programs
MPDE Material History
213. The Joint Fleet Maintenance Manual, Volume IV, Chapter 4Diesel Engines, paragraph 4.4b requires that the CommandingOfficer shall ensure that each inspection/assessment is entered
in the applicable machinery history with a brief description of
any repair before operating conditions found at the conclusion
of the assessment/inspection. [Encl. (67)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
35/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
29
214. The Joint Fleet Maintenance Manual, Volume IV, Chapter 4Diesel Engines, paragraph 4.5d requires that Engine Operation
Hours since Overhaul and Engine Operating Hours since
Commissioning, at a minimum, shall be kept for all MPDEs, SSDGs,
and EDGs. For Colt Pielstick Engines on LSD-41/49, LPD-17, andLHD-8 Class ships, operating hours since cylinder head
refurbishment and individual power pack change outs shall also
be kept. [Encl. (67)]
215. The Joint Fleet Maintenance Manual, Volume IV, Chapter 4Diesel Engines, Paragraph 4.6b(4) Commanding Officers shall
ensure all diesel records are readily available to include
Diesel Maintenance/history records to include all maintenance
and significant items accomplished on the engine since last
engine overhaul. [Encl. (67)]
216. SAN ANTONIO maintained material history logs usingstandard green hardback log books. [Encl. (68)]
217. The material history documented hours for each MPDE.[Encl. (68)]
218. The material history did not document inspection orssessment for each MPDE. [Encl. (68)]a
3M Program
219. On 19 September 2009, EM01 conducted PMS Check: 2331-017-49 U-11 Change Main Engine Lube Oil System Lubricating Oil.
[Encl. (69)]
220. The 13 Week Accountability Log does not clearly record theindividual that conducted the check. [Encl. (69)]
221. On 20 September 2009, EM01 conducted PMS Check: 2331-017-49 U-35 Clean and Inspect Rocker Arm Lube Oil Strainer. [Encl.
(69)]
222. The 13 Week Accountability Log does not clearly record theindividual that conducted the check. [Encl. (69)]
223. On 20 September 2009, EM01 completed PMS Check: 2331-017-49 U-11 Change Main Engine Lube Oil System Lubricating Oil.
[Encl. (69)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
36/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
30
224. The 13 Week Accountability Log does not clearly record theindividual that conducted the check. [Encl. (69)]
225. On 20 September 2009, EM01 completed PMS Check: 2331-017-49 U-35 Clean and Inspect Rocker Arm Lube Oil Strainer. [Encl.(69)]
226. The 13 Week Accountability Log does not clearly record theindividual that conducted the check. [Encl. (69)]
227. On 22 September 2009, EM01 conducted PMS Check: 2640/006-98 M-2 Clean and Inspect Diesel Engine Purifier Bowl. [Encl.
(69)]
228. The 13 Week Accountability Log does not clearly record theindividual that conducted the check. [Encl. (69)]
229. The check requires a tag out. [Encl. (70)]230. On 1 October 2009, EM01 completed PMS Check: 2331-017-4918M-8R Measure and Record Crankshaft Deflection Readings. [Encl.
(69)]
231. The 13 Week Accountability Log does not clearly record theindividual that conducted the check. [Encl. (69)]
232. On 29 October 2009, EM01 completed PMS Check: 2640/006-98M-2 Clean and Inspect Diesel Engine Purifier Bowl (Tag Out # E-
1281). [Encl. (69)]
233. The 13 Week Accountability Log does not clearly record theindividual that conducted the check. [Encl. (69)]
234. On 29 October 2009, EM01 conducted PMS Check: 2640/006-98M-2 Clean and Inspect Diesel Engine Purifier Bowl. [Encl. (69)]
235. The 13 Week Accountability Log does not clearly record theindividual that conducted the check. [Encl. (69)]
236. On 12 November 2009, EM01 conducted PMS Check: 2331/017-4918M-12R Test Injector Nozzle. [Encl. (69)]
237. The 13 Week Accountability Log does not clearly record theindividual that conducted the check. [Encl. (69)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
37/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
31
238. The check requires a tag out. [Encl. (71)]239. The 13 Week Accountability Log has N/A annotated for thetag out number. [Encl. (69)]
240. On 25 November 2009, EM01 conducted PMS Check: 2640/006-98M-2 Clean and Inspect Diesel Engine Purifier Bowl. [Encl. (69)]
241. The 13 Week Accountability Log does not clearly record theindividual that conducted the check. [Encl. (69)]
242. On 27 October 2009, EM02 conducted PMS Check: 2640/006-98M-2 Clean and Inspect Diesel Engine Purifier Bowl. [Encl. (72)]
243. The 13 Week Accountability Log does not clearly record theindividual that conducted the check. [Encl. (72)]
244. MRC 2331 R-12W MPDE Sump Samples is required: weekly inuse. [Encl. (73)]
245. SAN ANTONIO personnel last performed the check in March2009. [Encl. (50, 74)]
246. SAN ANTONIO personnel lined out MRC 2331 R-12W on MIP.[Encl. (50, 74)]
247. MRC 2331 R-13W MPDE Rocker Arm LO Viscosity Sample usingOTC is required: weekly in use. [Encl. (75)]
248. SAN ANTONIO personnel last performed the check in March2009. [Encl. (50, 74)]
249. SAN ANTONIO personnel lined out MRC 2331 R-13W on MIP.[Encl. (50, 74)]
250. MRC 2331 R-14M MPDE Sump Shore Lab Samples is required:Monthly in use. [Encl. (76)]
251. SAN ANTONIO personnel typically performed the check butdid not take credit for performance of the check for PMS
purposes. [Encl. (50)]
252. MRC 2331 U-35 C/I Rocker Arm LO Strainer is required: (a)>5 PSID; and (b) Change Rocker Arm LO. [Encl. (77)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
38/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
32
253. SAN ANTONIO personnel did not document the check for NR 2AMPDE, following change out of lube oil in August 2009, in 3M
Program. [Encl. (50)]
254. MRC 2620 U-1 C/I Duplex LO Strainer is required: (a) >15PSID; (b) LO Change; and (c) 1/watch 1st 48 hours aftermaintenance. [Encl. (78)]
255. SAN ANTONIO personnel did not perform check after lube oilchange out or after maintenance. [Encl. (50)]
256. MRC 2620 U-3 Renew LO Filter Elements is required: (a) >15PSID; (b) As directed by results of LO analysis; and (c) Change
LO. [Encl. (79)]
257.SAN ANTONIO personnel did not document the check inconjunction with a lube oil change out.[Encl. (50)]
258. Ships Force provided Memorandum of Record reportingcompletion of MRC. [Encl. (50)]
Lube Oil Quality Management Program
259. SAN ANTONIO personnel made no meaningful attempt tomonitor JOAP/NOAP monthly preventive maintenance lube oil
samples. [Encl. (81)]
260. Chain of command review of the LOQM logs were cursory andnot performed on a daily basis. [Encl. (81)]
261. Quarterly calibration of the onboard KITTEWAKE lube oilanalysis equipment was not performed. [Encl. (81)]
262. Over the period of February to May 2009, the SAMM Databaseconsistently revealed high levels of engine wear products to
include Iron (Fe) in NR 1A, 1B, and 2A MPDE lube oil. [Encl.
(80, 81)]
263. EOCC Procedure Hot Bearing in a Main Engine (MHMEB) doesnot require a visual inspection of the strainers or filter mediafor the presence of babbit which would provide an immediate
indication of bearing failure. [Encl. (82)]
264. EOCC Procedure Unusual Noise and Vibration in a MainPropulsion Diesel Engine (MNVME) does not require a visual
inspection of the strainers or filter media for the presence of
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
39/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
33
babbit which would provide an immediate indication of bearing
failure. [Encl. (82)]
Training Program
265. An audit of FLTMPS TYCOM required courses associated withlube oil systems was conducted and the following deficiencieswere noted:
a. One of fifty required Sailors have completed the LPD 17
Class Lube Oil System Familiarization Course (LPD-005)
b. Zero of two required Sailors have completed the
Propulsion Alarm and Indication Course (A-651-0047)
c. One of fifty required Sailors have completed the LPD 17
Class Fuel Oil System Familiarization Course (LPD-004)
d. Five of twenty-three required Sailors have completed the
LPD 17 Class Electrical Systems Familiarization Course (LPD-006)
e. Two of sixty-five required Sailors have completed the
LPD 17 Class Electrical Generation and Distribution Course (LPD-
007)
f. One of twenty-three required Sailors have completed the
LPD 17 Class AC and Refrigeration Course (LPD-014)
g. Two of fifty-eight required Sailors have completed the
LPD 17 Class Reverse Osmosis Familiarization Course (LPD-017)
h. Six of eighty-nine required Sailors have completed the
LPD 17 Class Compressed Air System Familiarization Course (LPD-
019)
i. Five of eighty-eight required Sailors have completed the
LPD 17 Class Engineering Control System (ECS) Course (LPD-032)
j. Two of forty-five required Sailors have completed the
LPD 17 Class Shipboard Status Monitor System Course (LPD-036)
k. Three of forty-two required Sailors have completed the
LPD 17 Class Main Propulsion Diesel Engine Operation Course
(LPD-103A)
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
40/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
34
l. Two of fifty-eight required Sailors have completed the
LPD 17 Class Ships Service Diesel Generator Operation Course
(LPD-103B)
m. One of seventy-four required Sailors have completed theLPD 17 Class Steering Gear Machinery System Operation Course
(LPD-103D)
n. Two of twenty-five required Sailors have completed the
LPD 17 Class Controllable Pitch Propeller Operation Course (LPD-
103E)
o. Two of twenty-five required Sailors have completed the
LPD 17 Class Fuel Oil Purifier Operation Course (LPD-104)
p. Two of twenty-five required Sailors have completed theLPD 17 Class MPDE LO Purifier Operation Course (LPD-105A)
q. Two of twenty-five required Sailors have completed the
LPD 17 Class Reduction Gear LO Purifier Operation Course (LPD-
105B)
r. Two of seventeen required Sailors have completed the LPD
17 Class Electrical System Operation Course (LPD-106)
s. Three of fifteen required Sailors have completed the LPD
17 Class Refrigeration Plant Operation Course (LPD-114B)
t. One of twenty-five required Sailors have completed the
LPD 17 Class Reverse Osmosis Operation Course (LPD-117)
u. Three of forty-five required Sailors have completed the
LPD 17 Class Shipboard Status Monitor Subsystem Operation Course
(LPD-136)
v. One of thirty-eight required Sailors have completed the
LPD 17 Class Steering Gear Machinery Maintenance Course (LPD-
203D)
w. Three of forty-eight required Sailors have completed the
LPD 17 Class Controllable Pitch Propeller Maintenance Course
(LPD-203E)
x. Zero of thirty-five required Sailors have completed the
LPD 17 Class Fuel Oil Purifier Maintenance Course (LPD-204)
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
41/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
35
y. One of thirty-five required Sailors have completed the
LPD 17 Class Main Reduction Gear LO Purifier Maintenance Course
(LPD-205A)
z. Two of twenty-five required Sailors have completed theLPD 17 Class Fuel Oil Purifier Operation Course (LPD-104)
[Encl. (83)]
266. Engineering Department has not completed many required NKOcourses including courses associated with ship control systems
(ECS/MCS), main propulsion and lubricating oils. [Encl. (84)]
267. Three months of training records for the period ofFebruary to December 2009 were unavailable. [Encl. (84)]
268.Despite a post deployment leave period in March/April2009, MP Division reported 100% attendance at all documented
training (typical reports stated that 25 of 25 Sailors attended
training). [Encl. (84)]
269. All MP Division training conducted since February 2009 wasdocumented as being conducted by the same second class petty
officer. [Encl. (84)]
270. The SAN ANTONIO Training Instruction requires that aSubject Matter Expert conduct the training. [Encl. (84)]
271. Since February 2009, MP Division conducted training threetimes on a system that did not exist on SAN ANTONIO: Diesel
Engine Brake System. [Encl. (84)]
272. From February to December 2009, training was not conductedon MPDE bearing high temperature casualties, loss of lube oil
pressure or loss of lube oil cooling casualties. [Encl. (84)]
273. Two lesson plans (Lubricating Oil Storage, Transfer andPurification and Diesel Engine Clutch and Brake System) were
dated December 2000. [Encl. (84)]
274. Five lesson plans (Lubricating Oil Quality Management,Diesel Main Reduction Gear, Bearing Records, Air System
Casualties and Diesel Engine Lube Oil System were dated 2005
(various months). [Encl. (84)]
275. The Diesel Lube Oil System lesson plan:
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
42/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
36
a. Did not address the self-cleaning filter or strainer
inspection criteria and potential sources/types of
contamination, or typical lube oil casualties (low level, low
pressure, loss of cooling (high temperature)
b. Had incorrect or incomplete criteria for the shift clean
and inspection of the lube oil strainers
[Encl. (84)]
276. The Lube Oil Storage, Transfer and Purification Systemlesson plan did not address required samples or appropriate lube
oil log entries in support of transfers. [Encl. (84)]
277. The Lube Oil Quality Management lesson plan did notaddress NSTM requirement to periodically test lube oil analysis
equipment, or weekly lube oil sample analysis when in operation,or the means to obtain the lube oil test results via the SAMMs
database and the requirement to maintain lube oil in
specification. [Encl. (84)]
SAN ANTONIO Class Engineering Systems
Shipboard Automated Maintenance Management System
278. Shipboard Automated Maintenance Management (SAMM) Systemprovides electronic means for ships force to track lube oil
analysis results. Ships force samples are sent to the lab.
Results are posted on the internet in the SAMMS folder. [Encl.
(85, 86)]
279. Based on oil sample results, recommendations to ship'sforce are developed by NSWC Philadelphia and forwarded to the
ship via email. [Encl. (85, 86)]
280. PREDICT is contracted by the Navy to conduct lube oilsample analysis. The lab is required to provide analysis of
samples two working days after the sample is received at the
lab. [Encl. (85, 86)]
281. Currently NSWC Philadelphia has tasked PREDICT to conductferrography analysis on samples for LPD 17 class ships. [Encl.(85, 86)]
282. Lube oil sample shipping costs are paid for by the programunder NSWC Philadelphia. [Encl. (85, 86)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
43/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
37
283. Based on location of the ship, and the methodology used toship samples (transfer to support ship, inport, etc) samples may
be received in the PREDICT lab out of sequence. [Encl. (85, 86)]
284. SAN ANTONIO has a stand alone version of SAMMS. Insteadof automatically sending data to SAMMS, ships force mustdownload information to a disc, and send information via the
internet. [Encl. (85, 86)]
285. With a stand alone system, ships force does not havedirect access to the SAMMS folder or lube oil analysis trend
data. [Encl. (85, 86)]
286. NSWC Philadelphia calibrates Kittiwake equipment onboardships every two years. [Encl. (85, 86)]
287. Training on SAMMS is scheduled for ships, and additionaltraining is available upon request. [Encl. (85, 86)]
288. Crew members from SAN ANTONIO received SAMMS training inApril 2008, July 2008, and August 2008 prior to the ships
deployment. [Encl. (85, 86)]
289. Ships force had the capability via their Port Engineer togain access to the monthly lube oil samples but did not attempt
to do so because of their level of knowledge of the system.
[Encl. (87-89)]
290. The current work around is that based on oil sampleresults recommendations to ship's force are developed by NSWC
Philadelphia and forwarded to the ship via email. [Encl. (87-
89)]
Engineering Control System
291. LPD 17 class Engineering Control System (ECS) is amicroprocessor-based real-time distributed data acquisition,
control, processing, and display system that:
a. provides remote monitoring and control of propulsion,electrical, auxiliary, fuel, damage control, and ballast systems
b. consists of 20 operator Multi-Function Workstations
(MFWS) and 35 Data Acquisition Units (DAUs)
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
44/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
38
c. uses the Shipboard Wide Area Network (SWAN) to
distribute data to and from ECS consoles
[Encl. (90)]
292. SWAN and ECS are commercial off the shelf systemsintegrated and maintained by Raytheon Incorporated during thenew construction process. [Encl. (90)]
293. DAU circuit boards, also called VME Remote Terminal Units(VRTU), are the interface between engineering plant sensors and
ECS and are designed to accept inputs from multiple sensors.
[Encl. (90)]
294. The DAU configuration allows interruption of multiplesensor signals if one sensor electrically fails. [Encl. (90)]
295. Troubleshooting of the DUA is complex and tedious becausethe technician must search each signal wired to the VRTU for the
failed sensor. [Encl. (90)]
296. After ship delivery in 2006, SAN ANTONIO and an outsidemaintenance community documented problematic ECS operation,
troubleshooting, and repair. [Encl. (91)]
297. PREINSURV findings during Final Contract Trials held from26-30 March 2007 included:
a. Significant deficiencies associated with the SWAN that
negatively impacted ECS operation.
b. LPD 17 Class integrated support was marginal at best.
[Encl. (92)]
298. PREINSURV findings during Material Inspection (MI)completed 26-30 November 2007 included:
a. Ships force lacked the fundamental troubleshooting tools
to attain self-sufficiency in effectively maintaining the SWAN
and ECS because of system software upgrades and installation ofalterations without supporting technical documentation
b. Technical documentation was not available because of
proprietary rights (i.e., MPDE engine controls technical
documentation)
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
45/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
39
c. Integrated logistic support necessary to troubleshoot
analog and digital signals between installed equipment and ECS
DAUs did not exist
d. At the start of the MI, ships force and NSSA technicianswere actively troubleshooting 55 out of service faults and 71
unknown faults displayed at ECS consoles
e. Optimal manning, existing level of knowledge, and lack
of technical documentation had combined to exceed available
ships force resources to operate the engineering plant
f. During underway portion the MI, an average of 20 alarms
per minute displayed by ECS during underway operations.
g. Watchstanders had become conditioned to ignore alarms.[Encl. (93)]
299. ECS maintenance after INSURV focused on correcting shorepower interface problems and restoring DAU circuit boards to
reduce the number of ECS faults in support of the February 2008
OPEVAL. [Encl. (94, 95)]
300. Significant post OPEVAL maintenance included theintermittent loss of speed and heading at the Bridge and
Steering Helm console. [Encl. (94)]
301. Technical community approach was to complete a system wideassessment of SWAN to determine root cause of signal failure.
[Encl. (96)]
302. NAVSEA PMS 317 funded an onboard SWAN subject matterexpert for the entire deployment to assist ships force with
troubleshooting/repair and defining SWAN/ECS interface problems.
[Encl. (97, 98)]
303. LPD 17 Class precommissioning crews received SWAN and ECStraining directly from Raytheon during the commissioning
process. [Encl. (91)]
304. Upon ship arrival in Norfolk, no RMC technicians weretrained in the ECS system. [Encl. (91)]
305. NAVSEA-Philadelphia Code 955 has been designated as InService Engineering Agent for ECS. [Encl. (91)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
46/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
40
306. Access to Raytheon for shipboard ECS technical support wasconvoluted due lack of a contract vehicle that could be used by
the Regional Maintenance Centers (RMCs) for emergent
troubleshooting and repair. Raytheon support is funded by
NAVSEA PMS 317. [Encl. (100)]
307. A post delivery support assist process was used by the RMCMaintenance Team to interface with PMS 317 and Raytheon on a
case by case basis. This process resulted in delays in system
restoration. [Encls. (101, 102)]
308. ECS reliability issues caused SAN ANTONIO to set onaverage three additional engineering watch standers during
normal operation. [Encl. (87, 88)]
309.In mid-December 2009, the Commanding Officer had approvedapproximately fifty Temporary Standing Orders (TSOs) that
authorized ships force to modify EOSS or add additional duties
to watch standers to locally monitor, operate or verify the
proper operation of equipment vice remote operation and
monitoring via the Engineering Control Systems (ECS). [Encl.
(87, 89, 90, 103)]
ICAS
310. ICAS provides a distributed set of six personal computers(PC) workstations and associated Portable Data Acquisition
(PDA), Portable Data Terminal (PDT) and Data Acquisition Set-64
(DAS-64) capable of obtaining and displaying engineering plant
equipment status data. [Encl. (104)]
311. ICAS maintains records of individual plant equipment in aship unique Configuration Data Set (CDS), for monitoring and
trend analysis functions. [Encl. (104)]
312. ICAS is a non-mission critical subsystem that only runs asa stand alone application on the unclassified side of the SWAN
network. [Encl. (104)]
313. The Channel B connection (dual hone) to SWAN modificationsfor each ICAS work station were completed during the Post
Shakedown Availability (2 April 2007 25 June 2007). [Encl.
(105)]
314. During the PSA sea trials, the Main Machinery Room NR 1ECS console and ICAS console desynched a total of three times:
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
47/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
41
ECS console desynched one time; ICAS console desynched two
times. [Encl. (105)]
315. The desynchronization of one console did not affect theother console. [Encl. (105)]
316. NAVSEA ICAS and ECS representatives were onboard duringsea trials. [Encl. (105)]
317. On 18 July 2007, NAVSEA ICAS representatives left SANANTONIO with all five ICAS workstations and one ICAS viewport
operational. [Encl. (105)]
318. In October 2007, NAVSEA ICAS representative participated in an ICMP assessment aboard SAN ANTONIO. [Encl.
(105)]
319. In October 2007, NAVSEA ICAS representativevisited SAN ANTONIO to update logsheets and install new ECS
software. [Encl. (105)]
320. On 1 March 2008, NAVSEA ICAS representativereceived email from stating ICAS was up and running,
but needed to install a new logsheet. [Encl. (105)]
321. On 7 March 2008, NAVSEA ICAS representative received email stating that during an OPEVAL: "ICAS fault
caused ECS to desync during ballast operations with a LCU in the
well delaying the evolution." [Encl. (105)]
322. In July 2008, NAVSEA ICAS representativeattempted to further assist ship via distance support (email)
but lost communication. [Encl. (105)]
323. The LPD 17 Class de-synchronization issue is caused by theShip Wide Area Networks (SWAN) handling of network
communications amongst the various ECS consoles. [Encl. (106)]
324. The de-synchronization problem was initially encounteredduring SAN ANTONIOs builder trials (prior to ICAS
installation). [Encl. (106)]
325. The recommended solution was the five ICAS workstationsbecome dual honed with a Channel B connection and network card.
[Encl. (106)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
48/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
42
326. The installed solution differed in that a shared ChannelB connection with the neighboring ECS console was implemented.
[Encl. (106)]
327. The five ICAS consoles thus shared a pre-existing channelB connection with a neighboring ECS console. [Encl. (106)]328. The installation of the GIG E network onboard USS NEWORLEANS (LPD 18) in Spring 2008 has further mitigated the
potential of de-synchronization since it is more robust than the
original ATM solution. [Encl. (106)]
329. NEW ORLEANS has been using ICAS since Spring 2008. [Encl.(106)]
330.Although SAN ANTONIO received the GIG E upgrade in (fall2009) she is reluctant to utilize the ICAS consoles for fear of
desynchronizing the entire control system. [Encl. (106)]
331. NAVSEA ICAS representatives believe that SAN ANTONIO haslost confidence in ICAS and will provide her training and a ship
rider during her upcoming trial in January 2010. [Encl. (106)]
332. Ships force has mitigated the risk associated with ICASbeing secured on SAN ANTONIO by setting additional watches and
manually logging and trending propulsion plant parameters.
[Encl. (88)]
333. Operating parameters and other equipment informationmonitored by ICAS is designed to be automatically sent off ship.
[Encl. (107)]
334. This interface has not been enabled on LPD 17, so shipsforce would conduct a manual push of the data off ship on a
periodic basis if ICAS was operational. [Encl. (107)]
335. Off ship subject matter experts would review the data andmake recommendations with regard to operations and condition
based maintenance. [Encl. (107)]
336. Until recently, only automated reports associated for airconditioning systems has been funded and available for ships
force although ICAS is designed to monitor the entire propulsion
plant. [Encl. (108)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
49/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
43
337. ICAS support would be in place to monitor MPDE performancefor LPD 19 deployment. [Encl. (5)]
Navy Training System Plan
338. The NTSP for USS San Antonio (LPD 17) Class AmphibiousTransport Dock Ship is NSTP N85-NTSP-S-30-9603B (01 Jul 09).[Encl. (109)]
339. NSTP N85-NTSP-S-30-9603B page xii states that the 01 July2009 version was prepared to document changes since the
previously approved NSTP in May 2003. [Encl. (109)]
340. NSTP N85-NTSP-S-30-9603B pages I-3 to I-4 states: LPD 17Class ships were introduced into the Fleet beginning with the
commissioning of USS SAN ANTONIO in January 2006. Despite the
relative newness of the Class, modest improvements,modernization and changes have already been made that
necessitate updating this NTSP. These ships were built with the
primary goal of reducing the total ownership cost (TOC) while
maintaining and improving readiness and reducing manning through
greater emphasis on technological enhancements, distance
learning, continuous learning systems, job/training aids,
distributed/joint learning, virtual reality, and multi-media
training. Introduction of these technologies provides the crew
and embarked forces with training tools that provide the
flexibility to adjust to meet mission objectives. The use of
electronic media has greatly improved the overall capacity for
the ship to train itself. This advance training technology
insertion, combined with implementation of advance Condition
Based Maintenance technologies and the use of Integrated
Condition Assessment System (ICAS) to eliminate work and
increase system readiness, provides the LPD 17 class with a
comprehensive program that reduces TOC and meets reduced manning
requirements. [Encl. (109)]
341. NSTP N85-NTSP-S-30-9603B page I-5 states: TSTS. TheTSTS provides the LPD 17 Class ships with centralized
coordination and management of training. It maintains the
individual and team training records including courses/eventsrequired and completed, qualifications attained, and evaluations
of training effectiveness. ICW courses can be taken from any
work station computer or tactical console with the results
interfacing directly with TSTS. TSTS also supports and records
the results of stand alone training exercises using the embedded
training features of specific systems. The Navy Integrated
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
50/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
44
Application Product Suite (NIAPS) and Navy Knowledge Online
(NKO) Afloat will replace TSTS functionality once the LPD 17
Class ICWs and IETMs are compatible with and hosted on NKO.
[Encl. (109)]
342. The Ships Electronic Classroom was never finished and iscurrently used as the ships gym. [Encl. (83, 88)]
343. The TSTS Server is located in the Learning Resource Center(LRC) but is not connected to the ships LAN or desk top
computers in the LRC. [Encl. (83, 88)]
344. The work around for TSTS is that classes that should beresident on the TSTS are available via Navy Knowledge Online
(KNO). Course completion is automatically posted to FLTMPS
database. Qualifications are manually tracked using onboarddata bases. [Encl. (83, 87, 88)]
345. Engineering Department has a low completion rate of NKOcourses. [Encl. (83)]
346. In the opinion of the Training Officer, the training department is too small (approximately four
Officers/Sailors) to track and record training. [Encl. (83)]
347. The ships training instruction was signed by the currentCommanding Officer in February 2009. [Encl. (83)]
348. The Training Officer was unfamiliar with several aspectsof the ships training instruction including lesson plan
approval, record keeping requirements and monthly training
reports. [Encl. (83)]
349. NSTP N85-NTSP-S-30-9603B page I-7 under OrganizationalLevel states: The LPD 17 maintenance strategy calls for the
maximum use of onboard O-Level maintenance capabilities through
state of the art maintenance aids such as the Module Test and
Repair capability, ICAS, ETMs, and IETMs. PM is accomplished in
accordance with the ships Planned Maintenance System (PMS)Schedule. The performance of PM maximizes the service life and
minimizes equipment down time. PM tasks include condition
directed and time directed tasks. Every effort should be made
to reduce the number of time driven tasks and make optimum use
of condition monitoring equipment instead [Encl. (109)]
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
51/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
45
350. NSTP N85-NTSP-S-30-9603B page I-7 under CorrectiveMaintenance states: CM incorporates the use of ETMs/IETMs and
interactive diagnosis to reduce MTTR. The installation of ICAS
affords shore based knowledge aboard ship, thereby reducing
trouble shooting time and maintenance-induced failures andimproving the operational efficiency of the ship. [Encl. (109)]
351. NSTP N85-NTSP-S-30-9603B Appendix A discusses a LPD 17Class Stakeholder meeting and training summit held 22 January
2008. [Encl. (109)]
352. The Stakeholders meeting identified five major systemsaboard the LPD 17 Class ships that presented critical operation,
maintenance and training concerns. These systems included the
Main Propulsion Diesel Engines (MPDE), the Ship Service Diesel
Generators (SSDG), the Engineering Control System (ECS), theShip Wide Area Network (SWAN), and the MK 46 MOD 30mm Gun
Weapon System(GWS). All of these systems except the MK 46 GWS
are commercial off the shelf systems that are unique to LPD 17
Class and for which no Equipment/System/Subsystem (E/S/S) Navy
Training System Plan has yet been developed and no formal Navy
schoolhouse/pipeline training currently exists. A major outcome
of this meeting was that TEAM SHIPS and all the Stakeholders
established, in accordance with OPNAVINST 1500.76A, a Training
Planning Process Methodology (TRPPM) Advisory Board to plan the
way ahead to determine the long term training solution and
update this LPD 17 Class NTSP. [Encl. (109)]
353. The LPD 17 Class training summit was called to addressthe action items identified during the Stakeholders Meeting.
The four main tasks assigned were to:
a. Describe the immediate training requirements for USS SAN
ANTONIO (LPD 17)
b. Describe initial training for LPD 17 Class Pre-
Commissioning Crews
c. Determine what the interim training plan for LPD 17
Class follow-on crews should be
FOR OFFICIAL USE ONLY
-
8/9/2019 San Antonio findings
52/68
Subj: COMMAND INVESTIGATION OF DIESEL ENGINE AND RELATED
MAINTENANCE AND QUALITY ASSURANCE ISSUES ABOARD USS SAN
ANTONIO (LPD 17)
46
d. Develop the process for determining the formal long term
training solution for the LPD 17 Class and brief the Way
Ahead.
[Encl. (109)