Title 39
SECTION 3010.323
3010.323 Evidence - introduction and reliance upon studies and analyses.
§ 3010.323 Evidence - introduction and reliance upon studies and analyses.(a) Statistical studies. All statistical studies offered in evidence in hearing proceedings or relied upon as support for other evidence shall include a comprehensive description of the assumptions made, the study plan utilized, the procedures undertaken, and references from the academic literature supporting the procedures undertaken. Machine-readable data files, program files, workbooks, and all other necessary materials to enable independent replication of the results or program output if requested by the Commission or parties shall be provided in the form of a compact disk or other media or method approved in advance by the Secretary. Where a computer analysis is employed to obtain the result of a statistical study, all of the submissions required by paragraph (b) of this section shall be furnished, upon request. In addition, for each of the following types of statistical studies, the following information should be provided:
(1) Market research. The following information shall be provided:
(i) A clear and detailed description of the sample, observational, and data preparation designs, including definitions of the target population, sampling frame, units of analysis, questionnaires or data collection instruments, survey variables, and the possible values;
(ii) An explanation of methodology for the production and analysis of the major survey estimates and associated sampling errors;
(iii) A presentation of response, coverage and editing rates, and any other potential sources of error associated with the survey's quality assurance procedures;
(iv) A discussion of data comparability over time and with other data sources;
(v) A complete description and assessment of the effects of all editing and imputation employed;
(vi) Identification of all applicable statistical models considered and the reasons the model based procedures and/or models were selected over other models or procedures, when model-based procedures are employed; and
(vii) An explanation of all statistical tests performed and an appropriate set of summary statistics summarizing the results of each test.
(2) Other sample surveys. The following information shall be provided:
(i) A clear description of the survey design, including the definition of the universe under study, the sampling frame and units, and the validity and confidence limits that can be placed on major estimates; and
(ii) An explanation of the method of selecting the sample and the characteristics measured or counted.
(3) Experimental analyses. The following information shall be provided:
(i) A complete description of the experimental design, including a specification of the controlled conditions and how the controls were realized; and
(ii) A complete description of the methods of making observations and the adjustments, if any, to observed data.
(4) Econometric studies. The following information shall be provided:
(i) A presentation of the economic theory and assumptions underlying the study;
(ii) A complete description of the econometric model(s) and the reasons for each major assumption and specification;
(iii) The definition of the variables selected and the justification for their selection;
(iv) For any alternative model whose computed econometric results influenced the choice of the preferred model, a statement of the reasons for rejecting that alternative, an identification of any differences between that alternative and the preferred model with respect to variable definitions, equation forms, data, or estimation methods, and, upon request, the computed econometric results for that alternative;
(v) A reference to a detailed description in a text, manual, or technical journal for every econometric technique used in the estimation process and the reasons for selecting the technique, or, in the alternative, a description and analysis of the technique that is sufficient for a technical evaluation;
(vi) Summary descriptions and source citations for all input data and, upon request, a complete listing of the data. Complete descriptions of any alterations, adjustments, or transformations made to the data as received from the original sources, and the reasons for making the alterations, adjustments, or transformations;
(vii) A complete report of the econometric results including, where applicable coefficient estimates, standard errors and t-values, goodness-of-fit statistics, other appropriate test statistics, the variance/covariance matrix of the estimates, and computed residuals for results computed from samples composed of fewer than 250 observations, and, upon request, other computed residuals; and
(viii) Descriptions of all statistical tests of hypotheses and the results of such tests.
(5) All other studies involving statistical methodology. The following information shall be provided:
(i) The formula used for statistical estimates;
(ii) The standard errors of each component estimated;
(iii) Test statistics and the description of statistical tests and all related computations, and final results; and
(iv) Summary descriptions of input data, and upon request the actual input data shall be made available at the offices of the Commission.
(b) Computer analyses. (1) In the case of computer studies or analyses which are being offered in evidence, or relied upon as support for other evidence, a foundation for the reception of such materials must be laid by furnishing a general description of the program that includes the objectives of the program, the processing tasks performed, the methods and procedures employed, and a listing of the input and output data and source codes (or a showing pursuant to paragraph (b)(3) of this section as to why such codes cannot be so furnished) and such description shall be furnished in all cases. For the purpose of completing such foundation, the following additional items shall be deemed presumptively necessary and shall be furnished upon request of a party, the Commission, or the presiding officer, unless the presumption is overcome by an affirmative showing. The following information shall be provided:
(i) For all input data, designations of all sources of such data, and explanations of any modifications to such data made for use in the program;
(ii) Definitions of all input and output variables or sets of variables;
(iii) A description of input and output data file organization;
(iv) For all source codes, documentation sufficiently comprehensive and detailed to satisfy generally accepted software documentation standards appropriate to the type of program and its intended use in the proceeding;
(v) All pertinent operating system and programming language manuals;
(vi) If the requested program is user interactive, a representative sample run, together with any explanation necessary to illustrate the response sequence;
(vii) An expert on the design and operation of the program shall be provided at a technical conference to respond to any oral or written questions concerning information that is reasonably necessary to enable independent replication of the program output; and
(viii) Computer simulation models offered in evidence or relied upon as support for other evidence, shall be bound by all applicable provisions of this paragraph (b) and the separate requirements of paragraph (a) of this section, to the extent that portions of the simulation model utilize or rely upon such studies. Information that compares the simulation model output results to the actual phenomena being modelled, using data other than those from which the model was developed, shall be separately identified and submitted as evidence supporting the test and validation of the simulation model. Separate statements concerning the model limitations, including limiting model design assumptions and range of data input utilized in model design, shall be provided. Where test and validation of the entire simulation model are not possible, test and validation information shall be provided for disaggregate portions of the model. If disaggregate testing and validation are not possible, separate statements to that effect and statements regarding operational experts' review of model validity shall be provided.
(2) Upon timely and otherwise proper request of a party, or sua sponte, the Commission or the presiding officer may rule that matters other than those listed in paragraphs (b)(1)(i) through (viii) of this section are necessary to establish the foundation for reception of the evidence concerned and must be furnished.
(3) When the requestor is other than the Commission or the presiding officer, the cost of producing the material required in paragraphs (b)(1)(iv), (vi), and (vii) of this section, shall be borne by the requesting party unless otherwise ordered, for good cause shown by the requestor. When the Commission or the presiding officer is the requestor, it may assume or equitably allocate such costs for good cause shown by the requester.
(4) If the recipient of a request for materials pursuant to this paragraph (b) asserts that compliance with the request would conflict with patent, copyright, trade secret or contract rights applicable to the requested material, the recipient shall immediately notify the requestor and the presiding officer. If valid, the presiding officer shall devise means of accommodating such rights. Such means may include protective orders, including access under protective conditions to the computer facilities of the recipient of a request, making material available for inspection, compensation, or other procedures, according to the nature of the right affected by compliance with this paragraph (b). If the presiding officer determines that compensation is necessary to accommodate the affected right, the cost of compensation shall be borne in the same manner that paragraph (b)(3) of this section prescribes for bearing the costs referenced there. If such right cannot be accommodated by reasonable compensation, or by protective orders or other procedures, and, as a result, materials required by this paragraph (b) cannot be provided, the presiding officer shall determine, in the presiding officer's discretion, whether evidence that relies upon the materials not provided shall be admissible or afforded limited weight.
(c) Other studies and analyses. In the case of all studies and analyses offered in evidence in hearing proceedings or relied upon as support for other evidence, other than the kinds described in paragraphs (a) and (b) of this section, there shall be a clear statement of the study plan, all relevant assumptions and a description of the techniques of data collection, estimation and/or testing. In addition, there shall be a clear statement of the facts and judgments upon which conclusions are based, together with an indication of the alternative courses of action considered and the steps taken to ensure the validity, accuracy, and reliability of the evidence. Tabulations of input data, workbooks, and all other materials necessary to replicate results shall be made available upon request at the offices of the Commission.
(d) Expedition. The party who offers studies or analyses in evidence shall expedite responses to requests made pursuant to this section for data or other information. Responses shall be served on the requesting party, and notice thereof filed with the Secretary in accordance with the provisions of § 3010.127 no later than 3 days after a request is made under § 3010.322(e)(2) or no later than 14 days after a request is made under § 3010.322(e)(3).