Questions and Answers Related to ISPE Quality Metrics Pilot Program, Wave 2
At the plenary session of the ISPE/FDA/PQRI Quality Manufacturing Conference held in Washington, DC from 1–3 June 2015, ISPE announced that its important Quality Metrics Initiative will continue with a Pilot Program, Wave 2.
A face-to-face meeting was held on 2 June 2015 at the ISPE/FDA/PQRI Quality Manufacturing Conference when there was a presentation of the objectives, benefits, a tentative list of potential Wave 2 metrics and a summary time plan. There was also an opportunity for interested companies to ask questions related to ISPE Pilot Program Wave 2. Similar sessions were also delivered as teleconferences on 16 and 18 June at which there were at least 50 companies represented. Questions and Answers were:
- Are there criteria or prerequisites required for companies to join?
Not really.
If a company only works with CMOs there may be challenges, however, it does depend on how a company is working with CMOs and there could be options. It is recommended companies contact and discuss with McKinsey at ispe_pilot@mckinsey.com
In Wave 2 of the Pilot Program we would like to broaden the types of companies and technologies of participants. For example we would like more small companies, generics, laboratories, sterile products and biological drug substances. - Are there templates to complete and what are these?
Yes, there are Excel-based templates with detailed definitions of each data point collected. There is one template for each site and one for product-application level for each company. Examples of similar style templates used in Wave 1 are given in A Report from the Pilot Project – Wave 1, Appendix 3, which is now available to all ISPE members. The Report has been added to the ISPE bookstore as free to members, and moderate cost to non-members.
- What assistance is given to sites when their first language is not English, for example when completing quality culture surveys?
McKinsey has local translators who provide a translated version of the culture survey. The site leads are encouraged to review it and offer suggestions if a particular translation point needs to be modified. It has also been found helpful for a company site leader to provide some internal guidance to help site employees understand questions better.
- Can a site repeat the quality culture survey in Wave 2 even though it completed a survey in Wave 1?
This question should be discussed with McKinsey since it is not the intention to repeat surveys to achieve ‘better’ answers. The time period between Wave 1 and Wave 2 is too short for a significant culture change at a site. If there was an understanding issue then McKinsey will make a judgment.
- Is there a fixed start date?
No, within a certain timeframe. Wave 2 recruitment has started, however, it is recognized that companies will take different amounts of time to complete contracts and confidentiality agreements. Wave 2 is largely a retrospective data collection exercise although companies have the option to collect some current data – it will be the company’s choice. Hence a fixed start date is not set. It is planned that ‘data lock’ will be at end November 2015 and it is, therefore, unlikely that a company could be accepted into the program after about middle of September.
- What amounts of data are to be collected in Wave 2?
12 or 24 months for new participants, and 9 or 12 months for Wave 1 participants (which have already supplied 15 months of data). It will depend on a company’s ability to supply. The longer period for which we have data, the better our ability to detect trends.
- How will Wave 2 fit alongside publication of the FDA FRN/Guidance?
Wave 2 is starting now and depending on when the FDA FRN/Guidance is issued may produce findings to inform FDA during the commenting period, or may provide findings to inform FDA after the end of the formal commenting period but before issue of final FRN/Guidance. Findings should help companies and FDA with implementation of a standardized quality metrics program. If the FRN/Guidance comes out early in the Pilot we may even be able to test during Wave 2 any variation of a metric that is not in the Pilot currently.
- If a site sources countries in addition to the US, should metric data be only for those products supplied to US or can data for all countries by provided?
It is a company/site’s choice.
- From Wave 1 findings, is it possible to say what ‘good’ looks like?
Wave 1 findings did not specify what ‘good’ looked like. There are not target ranges to aim for e.g. different technologies are likely to produce different ranges of values. Companies will, however, be able to see how they compare to similar groups via the confidential benchmarking exercise. Typical ranges of values are given in the Wave 1 report, which is now available to all ISPE members. It is considered that trending is more important than absolute values and we hope to detect more trending in Wave 2 as one of our objectives.
- Are all data points to be collected? What happens if a site cannot provide all data points?
Ideally we would like to collect all data points, however, there is some flexibility regarding a site that cannot provide all data points. If a data point is not provided, that site will not be able to compare their performance on that metric with other peer sites.
- Are we targeting recruitment of specific technologies?
Yes we would like to expand on Wave 1. Wave 1 had diversity, however, if we have more sites for steriles, biological, drug substances, generics, OTCs, CMOs and from emerging markets we may be able to establish more findings using multivariate analysis if sample sizes are large enough.
- Some metrics depend on technology – will there be correction factors developed?
We try to examine data for different technologies separately and make comparisons within technologies if we have sufficient sample size. Other factors are considered e.g. number of products manufactured at a site, estimates of a site’s level of product complexity. As mentioned above, we are seeking to expand our diversity of technology to try to make comparisons using multivariate analysis and to try to observe trends.
- What is the fee for participating?
There is not a simple answer, however, fees are based on those used in Wave 1. The fee structure depends on size of company, number of sites per company as well as other factors. The following slide was presented at the teleconferences:
Given the complexity it is appropriate that interested companies talk directly to McKinsey using the following link to arrange contact: ispe_pilot@mckinsey.com
If you are interested in joining other companies in this exciting exercise, please contact McKinsey at ispe_pilot@mckinsey.com
FAQs from the 2014 ISPE – FDA cGMP Conference
The following FAQs are based on output from ISPE – FDA cGMP Conference June 3rd 2014, Baltimore, MD
- What is the goal of the ISPE Quality Metrics Pilot program?
ISPE has launched this operational Pilot to demonstrate the feasibility and value of standard Quality Metrics to assess quality performance.
The primary objectives for the Pilot’s initial phase are to:- Test a set metrics based on industry and FDA input to date and which represents both leading and lagging indicators, as well as measures of site, product and quality system performance
- Test harmonization of metric definitions
- Test feasibility of data collection across different companies
- Inform continued industry input to the FDA
- What are the benefits of joining the Pilot?
In addition to objectives of the Pilot, participants will gain added benefits of:
- Influencing the output from ISPE Quality Metrics program in terms of choice of metrics, definitions, ease of data collection based on actual experience
- Having a blinded comparison to the overall industry average and among technology platform peers (provided sufficient sample size is achieved)
- Gaining a head-start on enhancing internal procedures for metric collection along with a set of metric definitions
- Developing insight into implications for metric implementation
- What is the scope of the ISPE Pilot – is it restricted to US supplied products only?
The site must be registered with the US FDA. However, sites have the choice of supplying data for products supplied to the US Market only or for all products at that site. It is recommended that sites do what is easiest for them.
- What is the role of McKinsey in this Pilot program? Will FDA and/or ISPE have access to individual company data?
ISPE will partner with McKinsey to conduct the Pilot program. McKinsey will collect and analyze data confidentially and will share individual results with participating companies. McKinsey's role will be to serve as an independent third party to collect and validate reported data to ensure that confidentiality around company-specific data is preserved, as well as to ensure consistency in interpretation and reporting by individual companies.
McKinsey will conduct confidential plant-level data gathering and analysis. McKinsey will establish a confidential database to be used only for this pilot. The data will be stored in a separate database, used only for the purposes of the ISPE Quality Metrics Pilot program. After the pilot is completed McKinsey will run the analyses and prepare 2 deliverables:- Blinded industry-level analyses that will be shared with ISPE and all participants
- Company-specific benchmarking reports showing where your site(s) stand compared to peers
McKinsey will not use the data in its client service or share individual company data with anyone outside the participating company. Blinded data and analytics will be shared with participating companies and with the ISPE pilot team only. The FDA will not be provided with access to the dataset or to individual company data, only high level analysis based on findings from the data as published in the ISPE report. - What is the cost to participating companies for the Pilot?
McKinsey will charge fees to cover its analytic work only, such as creation of a database, data collection and analysis: $30,000 for first site of companies with >$1 billion annual revenue, $15,000 each additional site. For small companies (<$1 billion annual revenue), first site cost is subsidized at $5,000.
- Are definitions of metrics agreed and what definitions will be used?
One of the main drivers for the Pilot Project is to assist industry and FDA in creating industry standard definitions for the metrics under study. These standard definitions have been developed following public consultation and review. These definitions are now set for the Pilot and will be used as a starting point for Wave 1 of the Pilot. ISPE may amend them if appropriate in the light of experience and knowledge gained from the Pilot.
The McKinsey Support Team will work with sites to collect data based on the agreed definitions. - Will ISPE examine what metrics could be normalized? What will be denominators and numerators?
Each of the metrics in the study includes a detailed definition of what each of the denominators and numerators in the calculation are. Data are ‘normalized’ to some extent in that calculations include values for number of lots dispositioned in order to determine a rate value. Companies will have access at the end of the Pilot to blinded comparison with their technology peers.
- Is the ISPE Pilot going to harmonize with other groups e.g. GPhA and OTC, with respect to Quality Metrics?
FDA has stated that they will support one or many pilots. ISPE is trying to keep abreast of other potential pilots and embraces a co-operative approach. Ideally ISPE does not wish for duplicate pilots. With this in mind ISPE is actively communicating with other associations and will maintain an open dialogue with all other interested parties. ISPE is proceeding with the Pilot and hopes for broad industry engagement.
- What types of companies and technologies should participate?
We currently have interest from branded, generic and over-the-counter (OTC) companies, of different sizes and across various platform technologies. We are also talking to contract manufacturing organizations (CMOs). The range of companies we are talking to is wide.
- What will be involved for a company participating in Wave 1 of the Study?
Wave 1 will involve the collection of 12 months of retrospective data plus 3 months of prospective data for all of the metrics included by each participating company.
- What will Wave 2 of the ISPE Pilot involve?
At this stage, the scope of Wave 2 has not been finalized.
We are targeting to have results from Wave 1 by the end of 2014, however we acknowledge that collection of 12 months of retrospective data plus 3 months of prospective data could prove difficult for all companies to achieve in the time period.
Once Wave 1 is complete, there will be a discussion regarding scope and objectives of the next phase of the study and it is hoped that Wave 2 could start with additional companies in early 2015. The goal is to utilize Wave 1 to develop lessons learned for further refinement or enhancement before entering Wave 2. - Could Wave 2 of this Pilot involve FDA?
FDA is not likely to be involved as they are expecting industry to take the lead with pilot studies. However, outputs from Wave 1 will be discussed with FDA and that may influence the direction for Wave 2.
- What resources will be required for the collection of data to complete Wave 1?
Each participating company will need to provide their own resources to collect and report data into the ISPE Pilot program. The McKinsey Support Team will be available to provide telephone/ email support to assist with clarifications as required.
- Does the team recognize the difficulty in collecting historical data?
ISPE would like to encourage companies to be part of the Pilot and lack of historical data should not be a barrier. The McKinsey Support Team will talk to companies, on a site-by-site basis, about the availability of different types of data.
- How do metrics group into product? Are we going to report, for example OOS by product?
The list of quantitative metrics assesses site, product and quality system performance. Several metrics will be measured by site and product, including Lot Acceptance Rate Confirmed OOS Rate and Product Complaints Rate. All metrics measure site performance.
- Will algorithms be developed and who will have access to the data?
There is no ‘hidden’ algorithm to judge overall quality performance. Data will be collected, blinded, and analyzed prior to preparation of the final report. There will be no access by ISPE to any individual site data.
- How will data be interpreted, particularly if standard definition-based metrics are hard to achieve?
It is acknowledged, that despite the tightness of the metric definitions, there may be collection or interpretation issues. The McKinsey Support Team will work site-by-site to make sure companies are able to talk through their concerns and to make sure they understand the definitions. If issues remain, they will be captured – this will still provide good learning from the Pilot.
- Will a participating company be able to review the report as there is concern regarding comparability based on site size and complexity?
Each participating company will get a chance to review a pre-publication version of the report, which will be circulated for information. In this report, your site will have access to averages and ranges for comparable sites.
- Will individual site or company data be identified?
Individual site data will not be provided to ISPE or FDA. Data representing particular Platform Technologies, for example, solid dosage form, aseptic sterile dosage form and large molecule monoclonal antibody manufacture, will not be grouped and reported on if there are insufficient data points collected to ensure ‘blinding’ of contributions from individual sites.
A company can choose whether to be identified or not as a Pilot participant in the ISPE reports. - Do we have the right balance of leading and lagging indicators for the future?
Wave 1 includes a blend of leading and lagging indicators designed to measure quality performance. The definition of leading and lagging indicators is not consistently applied and the determination depends on the risk criteria considered. For example, all ‘consensus’ metrics identified by FDA in a expert workshop, “Measuring Pharmaceutical Quality through Manufacturing Metrics and Risked-Based Assessment” held on 1 and 2 May 2014 hosted by the Engelberg Center for Health Care Reform at Brookings [1] are included in the Pilot. These ‘consensus’ metrics are all considered lagging whilst two, Lot Acceptance Rate and Confirmed OOS Rate are considered possibly leading.
Wave 1 also includes two surveys to assess leading metrics of quality culture and process capability and these could be used to guide if more quantitative metrics could be considered for Wave 2. ISPE has debated the balance extensively, however, the Pilot cannot cover everything. There is balance in what can be achieved initially.
Wave 2 of the study will benefit from the Wave 1 outcomes and it may be appropriate to revise/refine and include different, additional or fewer metrics in the next phase.
This program will be a journey for both industry and the supporting ISPE/McKinsey teams. - Should data from biologics sites be combined or not?
It is intended to differentiate in the report by platform technology. However, sufficient data within a platform will be required to make comparisons.
- Are these metrics indicative of compliance? Will they be useful?
We are not collecting compliance records so we will not be able to make a correlation. We are collecting data related to complaints and recalls, which may give us some limited correlations. The scope of the Quality Metrics Pilot is much larger than mere compliance and seeks to establish measures of broader ‘quality’ of the establishment.
- Will there be a recommendation on a Risk-Based Inspection Planning algorithm arising from this Pilot?
ISPE will issue a report that may include analysis with regard to correlations; however, the Pilot currently has insufficient data and information regarding the FDA requirements to develop an algorithm. It is understood that any prospective algorithm will include more than just quality metrics.
- Why are we proposing to measure Quality Culture by site vs by company? Would you expect differences in Quality Culture reported from different sites?
Experience shows that quality culture is related to site. Many companies have different cultures at different sites. It is therefore important to do a site-by-site comparison.
It is anticipated that that there will be site-to-site differences. McKinsey noted that they see significant differences in quality culture between sites within a company in their POBOS benchmarking.
The Quality Culture assessment comprises of a series of 15 agree/disagree questions around 5 dimensions of mindset, integrity, leadership, governance and capabilities. The ISPE team agreed these questions. - Who will be involved in the Quality Culture Survey?
We will work with each company to decide. The survey is designed for shop floor personnel. The process of collection could be via either on-line submission or paper-based.
- What level of personnel will be involved in the Quality Culture survey?
Ideally, as many people as possible at all levels. If this is not feasible we will review for the next phase of the study.
- Is there any overlap between the ISPE Pilot and the McKinsey POBOS definitions?
The ISPE Pilot is a completely separate database. Some definitions are the same as POBOS, some slightly different and some very different.
- Would it be possible to facilitate carryover from sites that have already submitted data as part of POBOS benchmarking into the ISPE Pilot?
McKinsey is prepared to work with sites to make data submission as easy as possible for companies.
[1]http://www.brookings.edu/events/2014/05/01-measuring-pharmaceutical-quality
The Brookings Institution