Outcomes Research Primer

Jay F. Piccirillo, M.D.
Washington University School of Medicine

Michael G. Stewart, M.D., M.P.H.
Baylor College of Medicine

Richard E. Gliklich, M.D.
Massachusetts Eye & Ear Infirmary

Bevan Yueh, M.D.
Section of Otolaryngology, Department of Surgery and the Robert Wood Johnson Clinical Scholars Program, Department of Medicine, Yale University School of Medicine. Dr. Yueh is a Robert Wood Johnson Clinical Scholars Fellow.

Correspondence and reprint requests:

Jay F. Piccirillo, M.D.
Box 8115, 517 S. Euclid Avenue
Washington University School of Medicine
St. Louis, MO 63110
Phone 314-362-7394 -- Fax 314-362-7522

This article was reprinted from: Piccirillo JF, Stewart MG, Gliklich RE, Yue b: Outcomes research primer. Otolaryngology-Head and Neck Surgery 1997; 117 (4):380-87. ©1997 American Academy of Otolaryngology-Head and Neck Surgery Foundation, Inc. All rights reserved.

Outcomes Research is the scientific study of the outcomes of diverse therapies used for a particular disease, condition, or illness.1,2 The goals of this type of research are to document treatment effectiveness, to create treatment guidelines, and to study the impact of insurance status or reimbursement polices on outcomes of care.3,4 Increasingly, otolaryngology providers are being asked by third-party payers, managed care organizations, and other organizations to document treatment outcomes.

The American Academy of Otolaryngology-Head and Neck Surgery Foundation (AAO-HNSF) has recently begun a research grant program especially targeted to support outcomes research projects in otolaryngology. Each year more instruction courses are offered at the Annual AAO-HNSF meeting on Outcomes Research. Clearly, the AAO-HNS membership are more interested in Outcomes Research than ever before. For that reason, we decided to prepare this Primer on Outcomes Research. It is our hope that this Primer will provide the interested reader with a "starting" point to explore the published literature on Outcomes Research. We do not claim that this Primer is exhaustive and admit that many excellent articles on Outcomes Research may have been omitted. Furthermore, we recognize that the field of Outcomes Research continues to grow and we plan periodic updates to this Primer to include new articles describing these new developments.

The Primer is divided into three main parts: Key Articles, General Articles, and Otolaryngology-Specific Articles. As the name implies, the Key Articles section includes those articles we believe are fundamental to the field of Outcomes Research. These articles are, or will be, "classics" in the field. Because of their importance, we have provided a short annotated bibliography to help the reader decide whether the article is of particular importance to him or her. The General Articles section contains articles of a general medical nature and the Otolaryngology-Specific section contains articles pertaining to otolaryngology.

References:

1. Piccirillo JF. Outcomes research and otolaryngology. Otolaryngol Head Neck Surg. 1994;111:764­769.

2. Roper WL, Winkenwerder W, Hackbarth GM, Krakauer H. Effectiveness in health care. An initiative to evaluate and improve medical practice. N Engl J Med. 1988;319:1197­1202.

3. Epstein AM. The outcomes movement­­will it get us where we want to go? N Engl J Med. 1990;323:266­269.

Relman AS. Assessment and accountability. The third revolution in medical care. N Engl J Med. 1988;319(18):1220­1222.

Key Articles

1. Bergner M. Quality of life, health status, and clinical research. Med Care. 1988;27:S148­S156.

The traditional endpoints of clinical research are morbidity and mortality, and not quality of life. Clinical information on the relative usefulness and sensitivity of quality of life instruments is often lacking. The author discusses these issues and makes suggestions about techniques and investigations to assist with these problems.

2. Cella DF, Bonomi AE. Measuring quality of life: 1995 update. Oncology. 1995;9:47­60.

This is an excellent review article which discusses the definition of quality of life, and statistical and methodological issues with quality of life assessment. The authors provide a summary of a large number of quality of life instruments.

3. Charlson ME, Pompei P, Ales HL, et al. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chron Dis. 1987;40:373­383.

The authors describe the validation of a weighted comorbidity scaling system, and its usefulness in predicting one-year mortality in patients with breast cancer. Basically, comorbid conditions (diseases other than the disease of interest) are weighted according to severity, and an index score is calculated which assists in prospective prognostic stratification.

4. Chassin MR, Kosecoff J, Park RE, et al. Does inappropriate use explain geographic variations in the use of health care services? JAMA. 1987;258:2533­2537.

The authors studied the degree of appropriateness of coronary angiography, upper gastrointestinal endoscopy, and carotid endarterectomy using established protocols in three different geographic areas. Across all three geographic regions, the authors noted relatively high levels of inappropriate utilization. In addition, they find no relationship between levels of appropriateness and different levels of utilization. Therefore, inappropriate use does not explain geographic variations in health care utilization.

5. D'Agostino RB, Kwan H. Measuring effectiveness: What to expect without a randomized control group. Med Care. 1995;33:AS95­AS105.

Randomized controlled trials are considered the ideal way to evaluate treatment efficacy. However, such trials are not always possible, and even when possible, they are often performed with such restrictions that they do not provide the true measures of the effectiveness of the treatment in the "real world" or under clinical conditions of usual practice. This article reviews the use of non-randomized studies to measure effectiveness when a randomized controlled trial is not possible.

6. Donabedian A. The quality of care: How can it be assessed? JAMA. 1988;260:1743­1748.

This is an overview of issues in determining quality of health care. Two components of quality (technical performance and interpersonal skills) are discussed with respect to cost, techniques in sampling and measurement, and sources of data. Distinctions between structure, process and outcomes data are drawn.

7. Eddy DM. Clinical decision making: from theory to practice. Cost-effectiveness analysis. A conversation with my father. JAMA. 1992;267:1669-75.

The first of a series of articles that introduces principles of cost-effectiveness, including its rationale, methods and applications, through a series of imaginary conversations. Questions are asked and answered with examples from clinical practice. Avoids use of technical economic language.

8. Eisenberg JM. Economics. JAMA. 1995;273:1670­1671.

This article highlights major findings about the managed care industry from 1994 to 1995. Studies suggest that favorable patient selection and decreases in in-patient utilization were partly responsible for HMO savings. The differences in health outcomes between fee-for-service or managed care plans are small; satisfaction outcomes are mixed.

9. Ellwood PM. Shattuck lecture­-outcomes management. A technology of patient experience. N Engl J Med. 1988;318:1549­1556.

This special report discusses the problems faced by the American medical system and suggests that a new technology of patient experience will help patients, payers, and providers make rational medical choices. Outcomes management consists of a common patient-understood language of health outcomes. The author describes several different aspects of outcomes management.

10. Epstein AM. The outcomes movement­­will it get us where we want to go? N Engl J Med. 1990;323:266­269.

The author considers the forces that have brought about the outcomes movement and the direction it has taken. The goals of the movement and likely impediments to its progress are discussed.

11. Feinstein AR. Clinical biostatistics. XLI. Hard science, soft data, and the challenges of choosing clinical variables in research. Clin Pharmacol Ther. 1977;22:485­498.

The choice of variables for clinical research should be selected based on their importance to the condition under study. The essence of science is not the use of "hard" variables over "soft" variables, but rather, the selection of variables which are reliable and accurate. Soft variables which describe many of the important aspects of clinical care can be made more scientific through techniques of clinimetrics and then be included in clinical studies.

12. Foundation for Health Services Research. Health Outcomes Research: A Primer. Washington, D.C. 1993.

This primer provides an overview of health services and outcomes research.

13. Gill TM, Feinstein AR. A critical appraisal of the quality of quality­of­life measurements. JAMA. 1994;272:619­626.

The authors review the published literature on quality of life and identify a lack of a well-accepted and generally used definition of quality of life. Authors state that because quality of life is a uniquely personal perspective, patient-specific measures should be used.

14. Greenfield S. The state of outcomes research: are we on target? [editorial]. N Engl J Med. 1989;320(17):1142­1143.

The author argues that large national data bases should be constructed to collect information on the outcomes for patients with different conditions.

15. Guyatt GH, Bombardier C, Tugwell PX. Measuring disease­specific quality of life in clinical trials. CMAJ. 1986;134:889­895.

The authors discuss the development, validation, and usefulness of disease-specific quality of life measures in clinical research. A detailed description is provided for techniques of item selection, item reduction, item format, and validation.

16. Iezzoni LI. Severity of illness measures. Comments and caveats. Med Care. 1990;28(9):757­761.

This editorial presents information on analyzing severity of illness measures and understanding problems and pitfalls of their use.

17. Kassirer JP. The quality of care and the quality of measuring it. New Engl J Medicine. 1993;329:1263­1265.

This essay about the motivations and intentions behind the movements to establish practice guidelines discusses implications of the rigid applications of these standards.

18. Katz JN, Larson MG, Phillips CB, Fossel AH, Liang MH. Comparative measurement sensitivity of short and longer health status instruments. Med Care. 1992;30:917­925.

In the study, patients were presented five different health status measures to assess whether short measures of health status are as sensitive as longer measures. Overall, short health status measures are more responsive then longer, more established measures.

19. Leape LL, Park RE, Solomon DH, Chassin MR, Kosecoff J, Brook RH. Does inappropriate use explain small­area variations in the use of health care services? JAMA. 1990;263:669­672.

Using information from Medicare beneficiaries in several adjacent counties in the same state, the investigators assess whether inappropriate utilization accounts for the large differences in utilization in adjacent counties. Although inappropriate utilization is noted, there are no large differences between counties. Therefore, inappropriate use does not explain variation in health care utilization between different counties in the same state.

20. Lohr KN. Outcome measurement: concepts and questions. Inquiry. 1988;25:37­50.

The need for the accurate measurement of patient outcomes is self-evident. The author describes outcomes research and its use in the measurement of quality of care. The author suggests four areas of improvement to link outcomes to the measurement of process of care.

21. Patrick DL, Deyo RA. Generic and disease­specific measures in assessing health status and quality of life. Med Care. 1989;27:S217­S232.

This article discusses the differences between generic and disease-specific outcomes instruments. Generic measures may be more generalizable and therefore useful for policy studies. Disease-specific instruments may be more sensitive to clinical changes. Reviews considerations selecting an appropriate instrument.

22. Phelps CE. The methodologic foundations of studies of the appropriateness of medical care. N Engl J Med. 1993;329:1241­1245.

Exploration of potential flaws in studies reporting on appropriateness of care. Considers how such studies may incorrectly "diagnose" appropriate practice patterns as inappropriate (and vice versa). Draws a mathematical analogy to the imperfect sensitivities and specificities of all diagnostic tests.

23. Piccirillo JF. Outcomes research and otolaryngology. Otolaryngol Head Neck Surg. 1994;111:764­769.

An overview of the historical development of the outcomes research movement and the ways outcomes research is different from traditional clinical research. The article also provides four methodological requirements for outcomes research. Examples of otolaryngology conditions that seem particularly suited to study with outcomes methodology are presented.

24. Piccirillo JF, Feinstein AR. Black­box mathematics and medical practice. Arch Otolaryngol Head Neck Surg. 1993;119:147­155.

Two articles presented in the otolaryngology literature using outcomes methodology are presented. These articles present results that are at odds with usual clinical practice. The reasons for this discrepancy are discussed and an argument made for improvements in the quality of clinical research provided.

25. Pliskin N, Taylor AK. General principles: cost benefit and decision analysis. In: Bunker J, Barnes B, Mosteller F, eds. Costs, Risks and Benefits of Surgery. New York: Oxford University Press; 1977:5­27.
Examples of cost benefit and decision analysis are provided in this textbook chapter. Concepts such as the time value of money and the use of utility functions to assess non-traditional outcomes are introduced. Several concepts are illustrated with clinical scenarios.

26. Relman AS. Assessment and accountability. The third revolution in medical care. N Engl J Med. 1988;319(18):1220­1222.

The author describes the historical changes in medicine since 1940 and argues that we have entered a third revolution as a result of uncontrolled expansion in health services and unsustainable growth in expenses. This third era of medicine is based on the consensus for the need for assessment accountability in health care.

27. Roper WL, Winkenwerder W, Hackbarth GM, Krakauer H. Effectiveness in health care. An initiative to evaluate and improve medical practice. N Engl J Med. 1988;319:1197­1202.

A detailed discussion of the effectiveness initiative. Several examples from different clinical conditions are provided to illustrate how outcomes research methodology can improve patient care.

28. Rubin HR, Gandek B, Rogers WH, Kosinsji M, McHorney CA, Ware JE, Jr. Patients' ratings of outpatient visits in different practice settings. Results from the Medical Outcomes Study. JAMA. 1993;270:835­840.

A standard and valid questionnaire of patient satisfaction with their outpatient visit is used to assess satisfaction across general medical practices, such as fee for service, group practice and prepaid HMO. Implications of patient satisfaction information are provided.

29. Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312:71­72.

This essay that argues that evidence-based medicine is the conscientious application of the best available evidence to making decisions about patient care. It includes a description of what evidence-based medicine, discusses its merits, and counters the most common criticisms.

30. Tarlov AR, Ware JE, Jr., Greenfield S, Nelson EC, Perrin E, Zubkoff M. The Medical Outcomes Study: an application of methods for monitoring the results of medical care. JAMA. 1989;262:925­930.

This important paper describes the methods and results of the Medical Outcomes Study. This was a two-year observational study designed to help understand how specific components of the health care system affect outcomes.

31. Wennberg J, Gittelsohn A. Variations in medical care among small areas. Sci Am. 1982;246:120

The authors discuss the differences in utilization rates of many surgical procedures in adjacent counties in Vermont. They conclude that differences are largely the result of the different mix of specialists and the procedures they prefer, rather than the health status of patients.

32. Wennberg JE. Outcomes research, cost containment, and the fear of health care rationing. N Engl J Med. 1990;323:1202­1204.

The authors discuss implications of different rates of health care utilization on health care costs, and the potential effects of rationing of health care. The use of outcomes research to identify health outcomes of procedures and medical treatments is less important than society addressing important issues of supply, demand and availability of health care, given fixed budgets and limited resources.

---

General Articles

1. Assessing Medical Technologies. Washington, DC: National Academy Press; 1985.

2. Committee on Methods for Setting Priorities for Guidelines Development, Institute of Medicine. In: Field MJ, ed. Setting Priorities for Clinical Practice Guidelines. Washington,D.C. National Academy Press; 1995:

3. Aaronson NK. Quality of life assessment in clinical trials. Controlled Clinical Trials. 1989;10:195S­208S.

4. Aaronson NK. Quality of life research in clinical trials. A need for common rules and language. Oncology. 1990;4:59­66.

5. Anderson JP, Kaplan RM, Berry CC, Bush JW, Rumbaut RG. Interday reliability of function assessment for a health status measure: The Quality of Well­Being Scale. Med Care. 1989;27:1076­1084.

6. Apgar V. A proposal for a new method of evaluation of the newborn infant. Anesth Analg. 1953;32:260­267.

7. Baldwin LM, Inui TS, Stenkamp S. The effect of coordinated, multidisciplinary ambulatory care on service use, charges, quality of care and patient satisfaction in the elderly. J Community Health. 1993;18:95­108.

8. Bergner M. Measurement of health status. Med Care. 1985;23(5):696­704.

9. Bergner M, Bobbitt RA, Carter WB, Gilson BS. The Sickness Impact Profile: development and final revision of a health status measure. Med Care. 1981;19:787­805.

10. Berwick DM. Continuous improvement as an ideal in health care. New England Journal of Medicine. 1989;320:53­56.

11. Berwick DM, Godfrey AB, Roessner J. Curing Health Care. San Francisco, CA: Jossey­Bass; 1990.

12. Bjelle A. Functional status assessment. Curr Opin Rheumatol. 1991;3:280­285.

13. Borchardt WW. Patient perception of quality in the clinic setting and relevant strategies for improvement. Coll Rev. 1994;Spring:72­97.

14. Brazier JE, Harper R, Jones NMB, et al. Validating the SF­36 health survey questionnaire: new outcome measure for primary care. BMJ. 1992;305:160­164.

15. Brook RH, Lohr KN. Monitoring quality of care in the medicare program. Two proposed systems. JAMA. 1987;258:3138­3141.

16. Brook RH, McGlynn EA. Maintaining quality of care. In: Ginzberg E, ed. Health Services Research. Key to Health Policy. Cambridge, MA: Harvard University Press; 1991:284­314.

17. Brook RH, Ware JE, Davies­Avery A, et al. Overview of adult health status measures fielded in Rand's Health Insurance Study. Med Care. 1979;17:1­131.

18. Brook RH, Ware JE, Rogers WH, et al. Does free care improve adults' health? Results from a randomized trial. N Engl J Med. 1983;309:1426­1434.

19. Busschbach JJV, Horikx PE, van den Bosch JMM, De la Riviere AB, de Charro FT. Measuring the quality of life before and after bilateral lung transplantation in patients with cystic fibrosis. Chest. 1994;105:911­917.

20. Caine N, Harrison SC, Sharples LD, Wallwork J. Prospective study of quality of life before and after coronary bypass grafting. BMJ. 1991;302:511­516.

21. Carey RG, Lloyd RC. Measuring Quality Improvement in Health Care. A Guide to Statistical Process Control Applications. New York, NY: Quality Resources; 1995.

22. Cella DF, Cherin EA. Quality of life during and after cancer treatment. Comp Ther. 1988;14:69­75.

23. Cella DF, Tulsky DS. Measuring quality of life today: Methodologic aspects. Oncology. 1990;4:29­38.

24. Cella DF, Tulsky DS, Gray G, et al. The Functional Assessment of Cancer Therapy Scale: Development and validation of the general measure. J Clin Oncol. 1993;11:570­579.

25. Chassin MR, Kosecoff J, Park RE, et al. Indications for selected medical and surgical procedures: a literature review and ratigns of appropriateness: coronary angiography, publication R­3204/1­CWF­HF­HCFA­PMT­RWJ. Santa Monica,CA: The Rand Corp. 1986.

26. Chassin MR, Kosecoff J, Solomon DH, Brook RH. How coronary angiography is used. Clincal determinants of appropriateness. JAMA. 1987;258(18):2543­2547.

27. Cleary PD, McNeil BJ. Patient satisfaction as an indicator of quality of care. Inquiry. 1988;25:25­36.

28. Davies AR, Ware JE. Involving consumers in quality of care assessment. Health Aff (Millwood). 1988;7:33­48.

29. Daugherty C, Ratain MJ, Grochowski E, et al. Perceptions of cancer patients and their physicians involved in phase I trials. J Clin Oncol. 1995;13:1062­1072.

30. de Haes JCJM, van Knippenberg FCE. Quality of life of cancer patients: review of the literature. In: Aaronson NK, Beckmann J, eds. The Quality of Life of Cancer Patients. New York, NY: Raven Press; 1987:167­182.

31. Deyo RA, Carter WB. Strategies for improving and expanding the application of health status measures in clinical settings. A researcher­developer viewpoint. Med Care. 1992;30:MS176­86; discus.

32. Deyo RA, Patrick DL. The significance of treatment effects: The clinical perspective. Med Care. 1995;33:AS286­AS291.

33. Donovan JL, Frankel SJ, Eyles JD. Assessing the need for health status measures. J Epi & Comm Health. 1993;47:158­162.

34. Eisenberg JM. Clinical economics. A guide to the economic analysis of clinical pratice. JAMA. 1989;262(20):2879­2886.

35. Elinson J, In discussion: Donabedian A, Spitzer W, Tarlov A. Advances in health assessment conference discussion panel. J Chron Dis. 1987;40 Suppl. 1:183S­191S.

36. Elmore JG, Feinstein AR. A bibliography of publications on observer variability (final installment). J Clin Epidemiol. 1992;45:567­580.

37. Epstein AM, Hall JA, Tognetti J, et al. Using proxies to evaluate quality of life. Med Care. 1989;27:S91­S98.

38. Feinstein AR. The pre­therapeutic classification of co­morbidity in chronic disease. J Chron Dis. 1970;23:455­469.

39. Feinstein AR. Clinical biostatistics. XVI. The process of prognostic stratification. Part 2. Clin Pharmacol Ther. 1972;13:609­624.

40. Feinstein AR. Clinical biostatistics. XIV. The purposes of prognostic stratification. Clin Pharmacol Ther. 1972;13:285­297.

41. Feinstein AR. The 'chagrin factor' and qualitative decision analysis. Arch Intern Med. 1985;145:1257­1259.

42. Feinstein AR. Para­analysis, faute de mieux, and the perils of riding on a data barge. J Clin Epidemiol. 1989;42:929­935.

43. Feinstein AR, Horwitz RI. Double standards, scientific methods, and epidemiologic research. N Engl J Med. 1982;307:1611­1617.

44. Ferrans CE, Powers MJ. Quality of Life Index: Development and psychometric properties. Adv Nurs. 1985;8:15­24.

45. Fowler FJ, Cleary PD, Magaziner J, Patrick DL, Benjamin KL. Methodological issues in measuring patient­reported outcomes: The agenda of the work group on outcomes assessment. Med Care. 1994;32, Supplement:JS65­JS76.

46. Fries JF, Spitz PW. The hierarchy of patient outcome. In: Spilker B, ed. Quality of Life Assessments in Clincal Trials. New York, NY: Raven Press, Ltd. 1990:25­35.

47. Garnick DW, Hendricks AM, Comstock CB. Measuring quality of care: fundamental information from administrative datasets. Int J Qual Health Care. 1994;6:163­177.

48. Geigle R, Jones SB. Outcomes measurement: a report from the front. Inquiry. 1990;27:7­13.

49. Glaser EM. Using behavioral science strategies for defining the state­of­the­art. J Applied Behav Sci. 1980;16:79­92.

50. Goldberg HI, Martin DP, Christensen DB, Neighbor WE, Inui TS, LoGerfo JP. The Health of the Public Program at the University of Washington: a new role for academic medical centers. Academic Medicine. 1991;66:499­505.

51. Goldmann DA. Contemporary challenges for hospital epidemiology. Am J Med. 1991;91:8S­15S.

52. Gonnella JS, Hornbrook MC, Louis DZ. Staging of disease. A case­mix measurement. JAMA. 1984;251(5):637­644.

53. Gotay CC, Korn EL, McCabe MS, Moore TD, Cheson BD. Quality­of­life assessment in cancer treatment protocols: Research issues in protocol development. J Natl Cancer Inst. 1992;84:575­579.

54. Gouveia WA, Bungay KM, Massaro FJ, Ware JE, Jr. Paradigm for the management of patient outcomes [see comments]. Am J Hosp Pharm. 1991;48:1912­1916.

55. Greco PJ, Eisenberg JM. Changing physicians' practices. New England Journal of Medicine. 1993;329:1271­1274.

56. Greene MG, Adelman RD, Friedmann E, Charon R. Older patient satisfaction with communication during an initial medical encounter. Soc Sci Med. 1994;38:1279­1288.

57. Greenfield S, Aronow HU, Elashoff RM, Watanbe D. Flaws in mortality data. The hazards of ignoring comorbid disease. JAMA. 1988;260:2253­2255.

58. Greenfield S, Kaplan S, Ware JE, Jr. Expanding patient involvement in care. Effects on patient outcomes. Ann Intern Med. 1985;102:520­528.

59. Greenfield S, Nelson EC. Recent developments and future issues in the use of health status assessment measures in clinical settings. Med Care. 1992;30:MS23­MS41.

60. Griner PF, Mayewski RJ, Mushlin AI, Greenland P. Selection and interpretation of diagnostic tests and procedures. Ann Intern Med. 1981;94:553­600.

61. Guyatt G, Feeny D, Patrick D. Issues in quality of life measurement in clinical trials. Controlled Clinical Trials. 1991;12:81S­90S.

62. Guyatt GH, Cook DJ. Health status, quality of life, and the individual. Commentary. JAMA. 1994;272:630­631.

63. Harrell FE, Lee KL, Califf RM, Pryor DB, Rosati RA. Regression modelling strategies for improved prognostic prediction. Stat Med. 1984;3:143­152.

64. Hays RD, Anderson R, Revicki D. Psychometric considerations in evaluating health­related quality of life measures. Quality of Life Research. 1993;2:441­449.

65. Hays RD, Kravitz RL, Mazel RB, et al. The impact of patient adherence on health outcomes for chronic disease patients in the Medical Outcomes Study. Journal of Behavioral Medicine. 1994;17:347­360.

66. Hays RD, Stewart AL. Construct validity of MOS health measures. In: Stewart AL, Ware JE, eds. Measuring functioning and well­being: The Medical Outcomes Study approach. Durham,N.C. Duke University Press; 1992:325­342.

67. Hays RD, Stewart AL. Sleep measures. In: Stewart AL, Ware JE, eds. Measuring functioning and well­being: The Medical Outcomes Study approach. Durham,N.C. Duke University Press; 1992:235­259.

68. Health Services Research Group. A guide to direct measures of patient satisfaction in clinical practice. Can Med Assoc J. 1992;146:1727­1731.

69. Horwitz RI, Feinstein AR. The application of therapeutic­trial principles to improve the design of epidemiologic research: a case­control study suggesting that anticoagulants reduce mortality in patients with myocardial infarction. J Chronic Dis. 1981;34:575­583.

70. Iezzoni LI. Using risk­adjusted outcomes to assess clinical practice: an overview of issues pertaining to risk adjustment. Ann Thorac Surg. 1994;58:1822­1826.

71. Iezzoni LI, Daley J, Heeren T, et al. Using administrative data to screen hospitals for high complication rates. Inquiry. 1994;31:40­55.

72. Iezzoni LI, Foley SM, Heeren T, et al. A method for screening the quality of hospital care using administrative data: preliminary validation results. QRB Qual Rev Bull. 1992;18:361­371.

73. Iezzoni LI, Greenberg LG. Widespread assessment of risk­adjusted outcomes: lessons from local initiatives. Jt Comm J Qual Improv. 1994;20:305­316.

74. Iezzoni LI, Restuccia JD, Shwartz M, et al. The utility of severity of illness information in assessing the quality of hospital care. The role of the clinical trajectory. Med Care. 1992;30:428­444.

75. Jaeschke R, Guyatt GH. How to develop and validate a new quality of life instrument. In: Spilker B, ed. Quality of Life Assessments in Clinical Trials. New York, N.Y. Raven Press, LTD; 1990:47­57.

76. Jenkins CD. Assessment of outcomes of health intervention. Soc Sci Med. 1992;35:367­375.

77. Jenkins CD, Jono RT, Stanton BA, Stroup­Benham CA. The measurement of health­related quality of life: major dimensions identified by factor analysis. Soc Sci Med. 1990;31:925­931.

78. Juniper EF, Guyatt GH, King DR. Comparison of methods for selecting items for a disease­specific Quality of Life questionnaire: Importance versus factor analysis. Quality of Life Research. 1994;3:52­53.

79. Kahn KL, Rogers WH, Rubenstein LV. Measuring quality of care with explicit process criteria before and after implementation of the DRG­based prospective payment system. JAMA. 1990;264:1969­1973.

80. Kahn KL, Roth CP, Kosecoff J, et al. Indications for selected medical and surgical procedures: a literature review and ratings of appropriateness: diagnostic upper gastrointestinal endoscopy, publication R­3204/4­CWJ­HF­HCFA­PMT­RWJ. Santa Monica, CA: The Rand Corp. 1986.

81. Kaplan MH, Feinstein AR. The importance of classifying initial co­morbidity in evaluating the outcome of diabetes mellitus. J Chronic Dis. 1974;27:387­404.

82. Kaplan RM, Anderson JP, Wu AW, et al. The Quality of Well­Being Scale: Applications in AIDS, cystic fibrosis, and arthritis. Med Care. 1989;27:S27­S43.

83. Katz S, ed. The Portugal Conference: measuring quality of life and functional status in clinical and epidemiological research. J Chron Dis. 1987;40:459­650.

84. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of disease classification system. Crit Care Med. 1985;13:818­829.

85. Kosecoff J, Kanouse DE, Rogers WH, et al. Effects of the National Institutes of Health Consensus Development Program on physcian practice. JAMA. 1987;258:2708­2713.

86. Krahn M, Gafni A. Discounting in the economic evaluation of health care interventions. Med Care. 1993;31:403­418.

87. Krause NM, Jay GM. What do gobal self­rated health items measure? Med Care. 1994;32:930­942.

88. L'Abbe KA, Detsky AS, O'Rourke K. Meta­analysis in clinical research. Ann Intern Med. 1987;107:224­233.

89. Laffel G, Blumenthal D. The Case for Using Industrial Quality Management Science in Health Care Organizations. JAMA. 1989;262:2869­2873.

90. Langner SR, Hutelmyer C. Patient satisfaction with outpatient human immunodeficiency virus care as delivered by nurse practitioners and physicians. Holist Nurs Pract. 1995;10:54­60.

91. Leebov W, Vergare M, Scott G. Patient satisfaction: a guide to practice enhancement. Oradell,N.J. Medical Economics Company, Inc. 1990.

92. Lewis CC, Pantell RH, Kieckheffer GM. Assessment of children's health status: Field test of new approaches. Med Care. 1989;27:s54­s65.

93. Lewis JR. Patient views on quality care in general practice: literature review. Soc Sci Med. 1994;39:655­670.

94. Light RJ, Pillemer DB. Summing up. The Science of Reviewing Research. Cambridge, MA: Harvard University Press; 1984.

95. Maklan CW, Greene R, Cummings MA. Methodological challenges and innovations in patient outcomes research. Med Care. 1994;32, Supplement:JS13­JS21.

96. McCormick B. Can research change the way MDs practice medicine? Hospitals. 1990;64:32­6, 30.

97. Merrick NJ, Fink A, Brook RH, et al. Indications for selected medical and surgical procedures: a literature review and ratings of appropriateness: carotid endarterectomy, publication R­3204/6­CWJ­HF­HCFA­PMT­RWJ. Santa Monica, CA: The Rand Corp. 1986.

98. Mills JL. Sounding Board: Data torturing. New England Journal of Medicine. 1993;329:1196­1199.

99. Mitchell JB, Bubolz T, Paul JE, et al. Using medicare claims for outcomes research. Med Care. 1994;32, Supplement:JS38­JS51.

100. Moinpour CM, Feigl P, Metch B, Hayden, Meyskens FL, Crowley J. Quality of life endpoints in cancer clinical trials: Review and Recommendations. J Natl Cancer Inst. 1989;81:485­495.

101. Moloney TW, Paul B. The consumer movement takes hold in medical care. Health Aff (Millwood). 1991;10:268­279.

102. Moses LE. Measuring effects without randomized trials? Options, problems, challenges. Med Care. 1995;33:AS8­AS14.

103. Mullan F, Jacoby I. The town meeting for technology: the maturation of Consensus Conferences. JAMA. 1985;254:1068­1072.

104. New York State Department of Health. Coronary artery bypass surgery in New York state, 1990­1992. Albany, NY: New York State Department of Health; 1993.

105. Nolan TW, Provost LP. Understanding Variation. Quality Progress. 1990;May, 1990:70­78.

106. O'Brien BJ, Buxton MJ, Ferguson BA. Measuring the effectiveness of heart transplant programs: Quality of life data and their relationship to survival analysis. J Chron Dis. 1987;40:137s­153s.

107. O'Brien BJ, Buxton MJ, Patterson DL. Relationship between functional status and health­related quality­of­life after myocardial infarction. Med Care. 1993;31:950­955.

108. O'Young J, McPeek B. Quality of life variables in surgical trials. J Chron Dis. 1987;40(6):513­522.

109. Oye RK, Landefeld CS, Jayes RL. Outcomes in SUPPORT. J Clin Epidemiol. 1990;43 Suppl:83S­87S.

110. Palmer RH, Strain R, Maurer J, Rothrock J, Thompson M. Quality assurance in eight adult Medicare practices. Med Care. 1984;22:632­643.

111. Park RE, Brook RH, Kosecoff J, et al. Explaining variations in hospital death rates: randomness, severity of illness, quality of care. JAMA. 1992;264:484­490.

112. Parkerson GR, Broadhead WE, Tse C-KJ. Comparison of the Duke Health Profile and the MOS Short­Form in healthy young adults. Med Care. 1991;29:679­683.

113. Parkerson GR, Broadhead WE, Tse C-KJ. The Duke Health Profile. A 17­item measure of health and dysfunction. Medical Care. 1990;28:1056­1072.

114. Parkerson GR, Connis RT, Broadhead WE, Patrick DL, Taylor TR, Tse C-KJ. Disease­specific versus generic measurement of health­related quality of life in insulin­dependent diabetic patients. Med Care. 1993;31:629­639.

115. Parkerson GR, Gehlbach SH, Wagner EH, James SA, Clapp NE, Muhlbaier LH. The Duke­UNC Health Profile: An adult health status instrument for primary care. Med Care. 1981;XIX No. 8:806­827.

116. Patton MD. Action research and the process of continual quality improvement in a cancer center. Oncol Nurs Forum. 1993;20:751­755.

117. Pauker SG, Kassirer JP. Decision analysis. New England Journal of Medicine. 1987;316:250­257.

118. Payer L. Medicine and culture. New York: Penguin Books; 1988.

119. Perry S. The NIH Consensus Development Program: a decade later. N Engl J Med. 1987;317:485­488.

120. Powe NR, Turner JA, Maklan CW, Ersek M. Alternative methods for a formal literature review and meta­analysis in AHCPR Patient Outcomes Research Teams. Med Care. 1994;32, Supplement:JS22­JS37.

121. Power EJ. Identifying health technologies that work. From the Congressional Office of Technology Assessment. JAMA. 1995;274:205

122. Raskin IE, Maklan CW. Medical treatment effectiveness research: a view from inside the Agency for Health Care Policy and Research. Evaluation and the Health Professions. 1991;14:161­186.

123. Richardson J. Cost utility analysis: what should be measured? Soc Sci Med. 1994;39:7­21.

124. Robbins JA, Bertakis KD, Helms LJ, Azari R, Callahan EJ, Creten DA. The influence of physician practice behaviors on patient satisfaction. Fam Med. 1993;25:17­20.

125. Ross CK, Steward CA, Sinacore JM. A comparative study of seven measures of patient satisfaction. Medical Care. 1995;33:392­406.

126. Ruta DA, Garratt AM, Leng M, Russell IT, MacDonald LM. A new approach to the measurement of quality of life: The patient­generated index. Med Care. 1994;32:1109­1126.

127. Safran DG, Tarlov AR, Rogers WH. Primary care performance in fee­for­service and prepaid health care system: Results from the Medical Outcomes Study. JAMA. 1994;271:1579­1586.

128. Schag CAC, Heinrich RL. Development of a comprehensive quality of life measurement tool:CARES. Oncology. 1990;4:135­138.

129. Schipper H. Guidelines and caveats for quality of life measurement in clinical practice and research. Oncology. 1990;4:51­57.

130. Schipper H, Clinch J, McMurray A, Levitt M. Measuring the quality of life of cancer patients: The Functional Living Index­Cancer:Development and validation. J Clin Oncol. 1984;2:472­483.

131. Scott IU, Schein OD, West S, Bandeen­Roche K, Enger C, Folstein MF. Functional status and quality of life measurement among ophthalmic patients. Arch Ophthalmol. 1994;112:329­335.

132. Simmer TL, Nerenz DR, Rutt WM, Newcomb CS, Benfer DW. A randomized, controlled trial of an attending staff service in general internal medicine. Med Care. 1991;29:JS31­JS40.

133. Sox HC, Blatt MA, Higgins MC, Marton KI. Medical Decision Making. Stoneham,MA: Butterworths; 1989.

134. Spierer M, Sims HW, Micklitsch CN, Lewis BE. Asessment of patient satisfaction as part of a physician performance evaluation: the Fallon Clinic experience. J Ambulatory Care Manage. 1994;17:1­7.

135. Spitzer WO, Dobson AJ, Hall J, et al. Measuring the quality of life of cancer patients. A concise QL­Index for use by physicians. J Chronic Dis. 1981;34:585­597.

136. Sprangers MAC, Cull A, Bjordal K. The European organization for research and treatment of cancer approach to quality of life assessment: Guidelines for developing questionnaire modules. Quality of Life Research. 1993;287:295

137. Stalfelt AM. Quality of life during induction treatment of acute myuloid leukemia: A comparison of three intensive chemotherapy regimens using three instruments for quality of life assessment. Acta Oncologica. 1994;33:477­485.

138. Steinberg EP, Bergner M, Sommer A, et al. Variations in cataract management: patient and economic outcomes. Health Serv Res. 1990;25:727­731.

139. Stewart AL, Hays RD, Ware JE. Methods of constructing health measures. In: Stewart AL, Ware JE, eds. Measuring functioning and well­being: The Medical Outcomes Study approach. Durham,N.C. Duke University Press; 1992:67­85.

140. Stewart AL, Hays RD, Ware JE. Health perceptions, energy/fatigue, and health distress measures. In: Stewart AL, Ware JE, eds. Measuring functioning and well­being: The Medical Outcomes Study approach. Durham,N.C. Duke University Press; 1992:143­172.

141. Street RL, Gold WR, McDowell T. Using health status surveys in medical consultations. Med Care. 1994;32:732­744.

142. Tanenbaum SJ. What physicians know. N Engl J Med. 1993;329:1268­1270.

143. Teasdale G, Jennett B. Assessment of coma and impaired consciousness: a practical scale. Lancet. 1974;2:81­84.

144. Tsvat J, Weeks JC, Guadagnoli E, et al. Using health­related quality of life information: Clinical encounters, Clinical Trials, and Health Policy. J Gen Internal Med. 1994;9:576­582.

145. Tunis SR, Hayward RS, Wilson MC. Internists' attitudes about clinical practice guidelines. Ann Intern Med. 1994;120:956­963.

146. Ware JE. Standards for validating health measures: Definition and content. J Chron Dis. 1987;40:473­480.

147. Ware JE, Davies AR, Rubin HR. Patients' assessments of their care. In: Office of Technology Assessment, ed. The Quality of Medical Care: Information for Consumers. Washington,D.C. US Congress, 1988:231­247.

148. Ware JE, Hays RD. Methods for measuring patient satisfaction with specific medical encounters. Med Care. 1988;26:393­402.

149. Ware JE, Sherbourne CD. The MOS 36 Item Short­Form Health Survey (SF­36): I. Conceptual framework and item selection. Med Care. 1992;30:473­483.

150. Ware JE, Snyder MK. Dimensions of patient attitudes regarding doctors and medical care servcies. Med Care. 1975;13:609­615.

151. Ware JE, Snyder MK, Wright WR, Davies AR. Defining and measuring patients satisfaction with medical care. Evaluation and Program Planning. 1983;6:247­2263.

152. Ware JE, Jr. Conceptualizing and measuring generic health outcomes. Cancer. 1991;67:774­779.

153. Weinstein M, Fineberg H. Clinical Decision Analysis. Philadelphia. Saunders. 1980.

154. Wen SW, Hernandez R, Naylor CD. Pitfalls in nonrandomized outcomes studies. The case of incidental appendectomy with open cholecystectomy. JAMA. 1995;274:1687­1691.

155. Wennberg J, Gittelsohn A. Small area variations in health care delivery. Science. 1973;182:1102­1107.

156. Wennberg JE, Barnes BA, Zubkoff M. Professional uncertainty and the problem of supplier­induced demand. Soc Sci Med. 1982;16:811­824.

157. Wennberg JE, Bunker JP, Barnes B. The need for assessing the outcomes of common medical practices. Ann Rev Public Health. 1980;1:277­295.

158. Wennberg JE, Roos N, Sola L, Schori A, Jaffe R. Use of claims data systems to evaluate health care outcomes. Mortality and reoperation following prostatectomy. JAMA. 1987;257(7):933­936.

159. Wensing M, Grol R, Smits A. Quality judgements by patients on general practice care: a literature analysis. Soc Sci Med. 1994;38:45­53.

160. Wilson SR. Patient and physician behavior models related to asthma care. Med Care. 1993;31:MS49­MS60.

161. Winkler JD, Kanouse DE, Brodsley L, et al. Popular press coverage of eight National Institutes of Health consensus development topics. JAMA. 1986;255:1323­1327.

162. Wray NP, Ashton CM, Kuykendall DH, et al. Selecting disease­outcomes pairs for monitoring the quality of hospital care. Med Care. 1995;33:75­89.

163. Yerushalmy J. Statistical problems in assessing methods of medical diagnosis, with special reference to x­ray techniques. Public Health Rep. 1947;62:1432­1449.

164. Zubrod CG, Schneiderman M, Frei E. Appraisal of methods for the study of chemotherapy of cancer in man: Comparative therapeutic trial of nitrogen mustards and triethylene thiophosphoramide. J Chronic Dis. 1960;11:7­30.

---

Otolaryngology-Specific Articles

1. Acoustic neuroma. Reprinted from NIH Consens Dev Conf Consens Statement Dec 11­13;9(4); 1991.

2. Early identification of hearing impairment in infants and young children. NIH Consens Statement Mar 1­3;11(1):1­24); 1993.

3. Baker CA. Factors associated with rehabilitation in head and neck cancer. Cancer Nurs. 1992;15:395­400.

4. Benninger MS, King F, Nichols RD. Management guidelines for improvement of otolaryngology referrals from primary care physicians. Otolaryngol Head Neck Surg. 1995;113:446­452.

5. Bjordal K, Kaasa S, Mastekaasa A. Quality of life in patients treated for head and neck cancer: a follow­up study 7­11 years after radiotherapy. Int J Radiation Oncol Biol Phys. 1994;28:847­856.

6. Browman GP, Levine MN, Hodson DI, et al. The Head and Neck Radiotherapy Questionnaire: a morbidity/quality­of­life instrument for clinical trials of radiation therapy in locally advanced head and neck cancer. J Clin Oncol. 1993;11:863­872.

7. Burns L, Chase D, Goodwin WJ. Treatment of patients with stage IV cancer: Do the ends justify the means? Otolaryngol Head Neck Surg. 1987;97:8­14.

8. Dana ST. Sinusitis panel develops reimbursement criteria. Out of committee. AAO­HNS Bulletin. 1993;12:11

9. Fielder H, Denholm SW, Lyons RA, et al. Measurement of health status in patients with vertigo. Clin Otolaryngol. 1996;21:124­126.

10. Gliklich RE, Hilinski JM. Longitudinal sensitivity of generic and specific health measures in chronic sinusitis. Qual Life Res. 1995;4:27­32.

11. Gliklich RE, Metson R. A comparison of sinus computed tomography (CT) staging systems for outcomes research. Am J Rhinol. 1994;8:291­297.

12. Gliklich RE, Metson R. Techniques for outcomes research in chronic sinusitis. Laryngoscope. 1995;105:387­390.

13. Gliklich RE, Metson R. The health impact of chronic sinusitis in patients seeking otolaryngologic care. Otolaryngol Head Neck Surg. 1995;113:104­109.

14. Gotay CC, Moore TD. Assessing quality of life in head and neck cancer. Quality of Life Research. 1992;1:5­17.

15. Grimby A, Rosenthall U. Health­related quality of life and dizziness in old age. Gerontology. 1995;41:286­298.

16. Grundfast KM. Management of otitis media and the new Agency for Health Care Policy and Research guideline. Arch Otolaryngol Head Neck Surg. 1994;120:797­798.

17. Hannley M. Academy enters outcomes research arena with new protocol: treatment outcomes for obstructive sleep apnea. News & views from the office of research development. AAO­HNS Bulletin. 1992;33:

18. Harris JP, Anderson JP, Novak R. An outcomes study of cochlear implants in deaf patients: Audiologic, economic, and quality­of­life changes. Arch Otolaryngol Head Neck Surg. 1995;121:398­404.

19. Hassan SJ, Weymuller EA, Jr. Assessment of quality of life in head and neck cancer patients. Head & Neck. 1993;15:485­496.

20. Hoffman SR, Dersarkissian RM, Buck SH, Stinziano GD, Buck GM. Sinus disease and surgical treatment: a results oriented quality assurance study. Otolaryngology ­ Head & Neck Surgery. 1989;100:573­577.

21. Hoffman SR, Mahoney MC, Chmiel JF, Stinziano GD, Hoffman KN. Symptom relief after endoscopic sinus surgery: an outcomes­based study. Ear, Nose, & Throat Journal. 1993;72:413­4, 419­20.

22. Isenberg SF, Rosenfeld RM. Problems and pitfalls in community­based outcomes research. Otolaryngol Head Neck Surg. 1996;(In Press)

23. Jacobson GP, Newman CW. The development of the dizziness handicap inventory. Arch Otolaryngol Head Neck Surg. 1990;116:424­427.

24. Johns ME. The medical appropriateness of tympanostomy tubes proposed for children younger than 16 years in the United States. Editorial Comment. Arch Otolaryngol Head Neck Surg. 1994;120:801­802.

25. Juniper EF, Guyatt GH. Development and testing of a new measure of health status for clinical trials in rhinoconjunctivitis. Clin Exp Allergy. 1991;21:77­83.

26. Kleinman LC, Kosecoff J, Dubois RW, Brook RH. The medical appropriateness of tympanostomy tubes proposed for children younger than 16 years in the United States. JAMA. 1994;271:1250­1255.

27. Kuk FK, Tyler RS, Russell D, et al. The psychometric properties of a tinnitus handicap questionnaire. Ear Hear. 1990;11:434­445.

28. List MA, Ritter­Sterr CA, Baker TM. Longitudinal assessment of quality of life in laryngeal cancer patients. Head Neck. 1996;18:1­10.

29. Mangham CA. Decision analysis of auditory brainstem responses and rotational vestibular tests in acoustic tumor diagnosis. Otolaryngol Head Neck Surg. 1987;96:22­29.

30. McNeil BJ, Weichselbaum R, Pauker SG. Speech and survival. Tradeoffs between quality and quantity of life in laryngeal cancer. N Engl J Med. 1981;305:982­987.

31. Meister S. Emerging risk: failure to detect hearing disability in newborns. The problem: lack of screening programs for normal newborns. QRC Advis. 1993;10:1­4.

32. Mulrow CD, Aguilar C, Endicott JE. Quality­of­life changes and hearing impairment: a randomized trial. Ann Int Med. 1990;113:188­194.

33. Mulrow CD, Aguilar C, Endicott JE, et al. Association between hearing impairment and the quality of life of elderly individuals. J Am Geriatr Soc. 1990;38:45­50.

34. O'Rourke IC, McNeil RJ, Walker PJ, Bull CA. Objective evaluation of the quality of palliation in patients with oesophageal cancer comparing surgery, radiotherapy and intubation. Aust N Z J Surg. 1992;62:922­930.

35. Piccirillo JF. Decision analysis in head and neck cancer. Archives of Otolaryngology ­­ Head & Neck Surgery. 1991;117:933

36. Piccirillo JF. The inclusion of comorbidity in a staging system for head and neck cancer. Oncology. 1995;9:831­836.

37. Piccirillo JF. The use of patient satisfaction data to assess the impact of continuous quality improvement efforts in an academic otolaryngology office. Archives of Otolaryngology ­­ Head & Neck Surgery. 1996;122:1045-1048

38. Piccirillo JF, Edwards D, Haiduk A, Yonan C, Thawley SE. Psychometric and clinimetric validity of the 31­item rhinosinusitis outcome measure (RSOM­31). Am J Rhinol. 1995;9:297­306.

39. Plante DA, Piccirillo JF, Sofferman RA. Decision analysis of treatment options in pyriform sinus carcinoma. Med Decis Making. 1987;7:74­83.

40. Rosenfeld RM. Pilot study of outcomes in pediatric rhinosinusitis. Arch Otolaryngol Head Neck Surg. 1995;121:729­736.

41. Rosenfeld RM, Mandel EM, Bluestone CD. Systemic steroids for otitis media with effusion in children. Arch Otolaryngol Head Neck Surg. 1991;117:984­989. P> 42. Rubin HR. Can patients evaluate the quality of hospital care? Med Care Rev. 1990;47:267­326.

43. Rubin HR, Wu AW. Patient satisfaction: its importance and how to measure it. In: Gitnick G, ed. The Business of Medicine: A Physician's Guide. New York, NY: Elsevier Science Publishing Co. 1991:397­409.

44. Sher AE, Schechtman KB, Piccirillo JF. The efficacy of surgical treatments for obstructive sleep apnea. Sleep. 1996;19:156­177.

45. Snyderman CH, D'Amico F. Outcome of carotid artery resection for neoplastic disease: a meta­analysis. Am J Otolaryngol. 1992;13:373­380.

46. The Otitis Media Guideline Panel. Clinical practice guideline, otitis media with effusion in young children. Rockville, MD: Agency for Health Care Policy and Research; July 1994:iv. AHCPR publication No. 94­0622; 1994.

47. Turner RG, Nielsen DW. Application of clinical decision analysis to audiological tests. Ear Hear. 1984;5:125­133.

48. U.S.Department of Health and Human Services. Managing otitis media with effusion in young children. Arch Otolaryngol Head Neck Surg. 1994;120:793­796.

49. Weiss MH, Harrison LB, Isaacs RS. Use of decision analysis in planning a management strategy for the stage N0 neck. [Review]. Archives of Otolaryngology ­­ Head & Neck Surgery. 1994;120:699­702.

We thank Drs. Edwin M. Monsell, Coordinator for Research, and Maureen Hannley, Associate Vice President for Research Development, American Academy of Otolaryngology Foundation, Inc., for their support of this work.