Optimizing Calibration Intervals Using Weibull Analysis at Eli Lilly
Eli Lilly implements Weibull analysis to determine optimal calibration intervals, highlighting the importance of avoiding excessive preventive maintenance, addressing infant mortality issues, and accurately identifying out-of-tolerance issues to enhance equipment reliability. The company's robust measurement program emphasizes measurement uncertainty calculations, guard banding, statistical calibration tolerance levels, and ISO 17025 accreditation for its corporate calibration laboratory. Insights reveal that most calibration PMs are set at one-year intervals, with a focus on ensuring instruments remain within assigned tolerances.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Using Weibull Analysis to Determine Calibration Intervals Ronald F. Reimer, PE, CRE, CMRP Eli Lilly and Company
A Key Reliability Principle If it ain t broke; don t fix it! Infant Mortality definition: failures right after invasive interventions. Too much preventive maintenance (PM) usually hurts the equipment and reduces reliability. Not: If it fails, do the PM twice as often. We started having evidence that we were performing our calibration PMs too often 2
Calibration Background Eli Lilly does over 50,000 calibrations a year. Our data indicated our instruments had issues about 2% of the time. It was thought the majority of these issues were out of tolerance (OOT) issues. But, further investigation revealed that the majority of the issues were infant mortality issues or were otherwise non- OOT related. 3
Other Issues vs Calibration OOT Site C: 621 Work Orders 12 Issues (1.9%) 0 Calibration OOT 12 Other issues Site A (for a particular instrument set): 6890 Work Orders 155 Issues (2.2%) 22 Calibration OOT (0.3%) 112 Other errors 21 Bad components, wires, etc. Note: 1. Site B: 7629 Work Orders 129 Issues (1.7%) 16 Calibration OOT (0.2%) 47 Other issues 34 Bad components 32 Broken wires, connectors, etc. The actual OOT rate is 1/10 of the apparent rate. More than half of the issues were caused by the calibration process itself. 2. 4
Purpose of a Calibration PM 1. A calibration PM does not normally calibrate the instrument: it checks to see if the instrument is still within its assigned tolerances. 2. Only if the instrument drifted out of tolerance (OOT), does the PM adjust the instrument back into tolerance. 3. The main purpose of a calibration PM is to check to see if the instrument is still within tolerance. A calibration PM does NOT protect against: Failed instruments Broken/loose wires Events that damage an instrument. End of life failures Human errors 5
Lillys Measurement Program Eli Lilly has a very strong measurement program: Measurement uncertainty calculations for critical instruments Guard banding requirements Statistical based calibration tolerance levels Proven acceptable ranges Product control strategies Top tier instruments Redundant instruments for highly critical areas ISO 17025 accredited corporate calibration laboratory 6
Setting PM Intervals 93% of our calibration PMs intervals are one year or less. There were two main drivers for these short calibration intervals: 1) That s the way we have always done it 2) A fear of lot bracketing Lot bracketing pertains to the length of time between calibration checking intervals. If the As-Found condition of the instrument is OOT, then all the lots manufactured since the last passing PM are now suspect. This concern kept most calibration intervals to 1 year or less. 7
Reasons to Re-examine Intervals 1. Calibration intervals were based on old history and concerns over lot bracketing (not based on science or observed data). 2. The calibration process itself creates errors and deviations. One Site s Temperature Data, 2008 to 2015: 7852 Work orders for 875 instruments 13 Calibration OOT (Annualized rate=0.2%) 66 Broken wires, connectors, errors, other (Annualized rate=1%) 3. Instrument technology has improved and most instruments are very stable. One OEM guarantees 5-10 years instrument stability However, instrument manufacturers do NOT give a recommended PM interval. They say it depends on how we use the instrument 8
Additional Reasons 4. Pfizer increased their intervals in 2008 and saw no negative change in pass rates; a Lilly site in 2010 had similar positive results. 5. The perceived risk of an OOT instrument is high, but the demonstrated risk has actually been very low. After searching through our CAPA database (2009-2015) we found: No quality impact on product from an OOT condition One quality impact due to the calibration process itself This outstanding result is due to our robust measurement system described earlier. Our Project Goal: Set calibration intervals for field instruments with a Weibull statistical distribution using actual Lilly data. 9
Scope of This Project Out of Scope QC/QA Labs Corporate Calibration Laboratory Analytical instruments HPLC Total Organic Carbon pH Conductivity Optical Density In Scope Temperatures Levels Flow Pressure Time Speed Humidity 10
Weibull Analysis Waloddi Weibull developed the Weibull distribution in 1937. The Weibull distribution has become the leading statistical model in the world for analyzing equipment life data. Information from a Weibull plot: Can determine survivability after a period of time. Beta Slope of line o <1 Infant mortality o B=1 Independent of age o B>1 Wear out 11
Example Weibull Chart-2 Example: 3 instruments OOT; 7 still in tolerance after one year Sample data 218, 395 days
Example Weibull Chart-3 Example: 3 instruments OOT; 97 still in tolerance after one year 218, 723 days
Using Lilly Data and Weibull Analysis Site X Temp Transmitter Example 6890 WO s since 1/1/2006 21 OOT due to drift 99.7% pass rate Weibull predicts 95% of instruments are still in tolerance at 9.8 years. So why check yearly? 15
Risk Mitigation: 3 Levels, 2 Caps Three passing levels 98% for Critical Operating Data (COD) instruments 95% for other critical instruments (Q, HSE, Business) 90% for non-critical instruments Capped maximum intervals 5 years for COD instruments 10 years for other critical instruments 16
Weibull Data The Weibull Analysis is performed with data representing a single failure mode, which, for this project, was defined as an OOT condition due to calibration drift. Some instrument failures may have nothing to do with calibration drift. It may be due to broken or loose wires, a bad I/O card, or a total failure. These issues were NOT considered OOT for this analysis but were treated as a suspension. All the maintenance data for each instrument were reviewed and failures due to OOT drift were separated from failures due to other reasons. 17
The Process Obtain 7+ years of calibration and maintenance history for each instrument. Review and clean up the data. Sometimes need to look at other documents to determine if OOT condition due to drift or other issues. Group like instruments together (more on this later). Perform Weibull Analysis to determine recommended intervals to enable pass rates of 98%, 95%, and 90%. Send recommendations to sites for their review, modification, and approval. Then update CMMS. 18
Sample of Weibull Analysis Results The following table shows some results for one site: Some instruments showed their intervals could be significantly extended; some showed their intervals could be moderately extended and some instruments already had the correct interval or maybe should be shortened. 19
Concerns with this Concept 1. Previous History (resistance to change) 2. Lot Bracketing 3. Instrument Grouping 20
Not the Way We Always Done This Cavemen : Citizens Against Virtually Everything 21
Watch out for Icemen International Coalition of Experts 22
Manage Resistance to Change We ve always calibrated every 6 months We re following OEM recommendations We have already perfectly adjusted our intervals Frequent communication needs to happen to help people with these concerns. Usually need to talk about it at least three different times. Be prepared to accept that some people will never get over it. 23
Concerns with Lot Bracketing Extending the calibration checking intervals will cause more manufacturing lots to be bracketed when an OOT condition is found. But we have never had a case with product impact from a single OOT instrument. This is because our product control strategies are not dependent upon single instruments. This is a common concern and it takes several discussions, with data, to help people get comfortable with extending intervals. 24
Protecting Lillys Measurement Program Eli Lilly has a robust measurement system currently in place. Increasing calibration checking intervals will not affect the following: 1. Current instrument tolerances and ranges 2. Guard banding 3. Maintaining a 95% or greater pass rate for critical instruments 4. Alarm settings 25
Instrument Grouping Scheme This technical concern was brought up by several instrument engineers. They felt the analysis was only valid if all the following were taken into account: Instruments Function (level, flow, temperature ) Technology (mass flow, vortex ) Manufacturer Model Number Required Tolerance Environment 26
Difficulties There was not enough data available to do statistical analysis at the lower levels. There simply was not enough OOT readings as the population size got smaller and smaller. Recall that <2% of issues were apparent OOT. Therefore, to get 3 OOT readings, you need 150+ readings. Since, the actual OOT due to instrument drift may be as low as 1/10 of the apparent OOT, even more readings would be needed (1500+). 27
Group Data Example 432 Pressure Transmitters at Site A 23 Manufacturers (57% tied to one OEM) 20 OOT over eight years 6 OOT with the 57% OEM (note: 57% of 20 OOT is 11. So this OEM is doing better than the other OEMs) 6 other OEMs had the remaining 14 OOT results. 16 OEMs had no OOTs 50 Unique model numbers 57 Unique tolerance limits 28
Group Data Example (cont) Able to plot the one OEM and the entire group Pass 98% 95% 6 yrs 90% 12 yrs All 2 yrs 5 yrs 11 yrs OEM X 3 yrs OEM X OEM X performs better than the other instruments. 29
Group Data Example (cont) Assume we use the All results for all 432 instruments: As a group, 95% of all the instruments will still be within tolerance after 5 years. However, OEM X will have a greater percentage within tolerance while the red OEMs should have a little more out of tolerance. Pass OEM X 98% 3 yrs 95% 6 yrs 90% 12 yrs All 2 5 11 But as a whole, 95% are still within tolerance after 5 years. 30
Group Data Results Able to group all instrument by function (pressure, speed, level, etc.) Subdivided two groups by technology (when data was available) Flow Air Liquid D/P, Magnetic, Mass, Vortex Shedding Level Pressure, Probe, Wave There just wasn t enough OOTs to allow grouping of other functions into smaller divisions. 31
Extending Calibration Intervals Pros + Less chance of making a mistake + Less damage to the equipment + Less overall deviations (due to less mistakes and damage) + Improved craft safety due to less maintenance activities + Less downtime required from Operations, more uptime + Less churn: fewer work orders: fewer schedules, less coordination activities, less paperwork, less LOTO, less instruments off line, less return to service activities. Cons - An OOT finding will implicate more lots, but the chance of product impact is very, very low if you have a strong product control strategy. 32
Projected Impact Cut number of calibration PMs in half Improved manufacturing performance Less churn Improved instrument reliability Improved safety by not putting operators and mechanics in harms way (ladders, LOTO) We are currently in the middle of implementation Five sites have implemented the new intervals. They are seeing a combined reduction of over 10,000 calibrations/year. Will be verifying instrument performance later to ensure results 33
Risk of Intrusive Checking How often do you check your tire pressure? Every day? Every fill up? As needed? Every time your check the air, a little air escapes. Eventually, air will be needed. Why? Was the tire leaking or did the checking cause the lower air pressure? What if checking the tire pressure cost $100 each time and you had a 5% chance of wearing out the valve stem? What if you couldn t operate the car for two hours while you were checking the pressure? 34
Summary Instrument technology has improved considerably in the last twenty years. Many digital instruments are showing very little drift. The data shows too many PMs hurts the equipment. Most intervals were set based on that s the way we ve always done it or based on a fear of lot bracketing. Lilly has years of calibration results to help make statistical based interval decisions Lilly s robust measurement program allows an increase in calibration intervals with low risk. Increasing calibration intervals will help the instruments perform better, have fewer deviations, improve safety, and reduce costs. 35
Thank You! Any questions? 36
