National Academies Press: OpenBook

Language and Machines: Computers in Translation and Linguistics (1966)

Chapter: Appendix 15. Evaluation by Science Editors and Joint Publications Research Service and Foreign Technology Division Translations

« Previous: Appendix 14. Translation Versus Postediting of Machine Translation
Suggested Citation:"Appendix 15. Evaluation by Science Editors and Joint Publications Research Service and Foreign Technology Division Translations." National Research Council. 1966. Language and Machines: Computers in Translation and Linguistics. Washington, DC: The National Academies Press. doi: 10.17226/9547.
×
Page 102
Suggested Citation:"Appendix 15. Evaluation by Science Editors and Joint Publications Research Service and Foreign Technology Division Translations." National Research Council. 1966. Language and Machines: Computers in Translation and Linguistics. Washington, DC: The National Academies Press. doi: 10.17226/9547.
×
Page 103
Suggested Citation:"Appendix 15. Evaluation by Science Editors and Joint Publications Research Service and Foreign Technology Division Translations." National Research Council. 1966. Language and Machines: Computers in Translation and Linguistics. Washington, DC: The National Academies Press. doi: 10.17226/9547.
×
Page 104
Suggested Citation:"Appendix 15. Evaluation by Science Editors and Joint Publications Research Service and Foreign Technology Division Translations." National Research Council. 1966. Language and Machines: Computers in Translation and Linguistics. Washington, DC: The National Academies Press. doi: 10.17226/9547.
×
Page 105
Suggested Citation:"Appendix 15. Evaluation by Science Editors and Joint Publications Research Service and Foreign Technology Division Translations." National Research Council. 1966. Language and Machines: Computers in Translation and Linguistics. Washington, DC: The National Academies Press. doi: 10.17226/9547.
×
Page 106

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix 15 Evaluation by Science Editors of Joint Publications Research Service and Foreign Technology Division Translations Five Joint Publications Research Service (JPRS) translations and five Foreign Technology Division (FTD) translations (four post- edited machine translations and one unedited rough-draft human translation) were sent to six science editors of the American As- sociation for the Advancement of Science and to one translation- agency owner. The translations were ranked according to their quality as scientific writings. The JPRS translations were, in general, ranked higher than the FTD translations. The agreement was almost unanimous that the worst translation of all was the FTD unedited rough-draft human translation. We requested that the Clearinghouse for Federal Scientific and Technical Information provide us with the six most recently ac- quired Russian-to-English translations from JPRS and FTD. When these arrived, we eliminated three translations—two because of length and one because we wanted to include an unedited rough- draft translation in the sample. The ten translations that formed the sample were keyed as follows: (A) Absorption of Radio Waves by Air Behind a Shock Wave, FTD AD605883, FTD-MT- 63- 74, by T. V. Bazhenova and Yu. S . Lobastov 9/62 (B) Translations on Soviet Construction and Building Materials Industry No. 65, USSR (Large-Scale Building Activity in Process Throughout the Soviet Union) JPRS: 27,2 67, TT: 64- 51522 1 1/6/64 (C) USSR Industrial Development, Soviet Chemical Industry, No. 188 JPRS: 27,271, TT: 64-51526 11/6/64 (D) Research on Heat Exchange in Vacuum by A. N. Devoyno, FTD-MT- 63-09 Edited Machine Translation, 20 Feb. 1964 (E) Testing and Ozokerite Bacillus Culture Liquid for Toxicity by Ch. B. Bayriyev- USSR- JPRS: 27,268, TT: 64-51523 11/6/64 (~) There is Such a Machine by Ye. Temchin, FTD-TT-64-1170/1 27 Oct. 1964 102

(H) (I) (J) Method of Detection and Identification of Remote Explosions by V. S. Voyutskiy, FTD-MT-64-407, Edited Machine Trans- lation, 6 Oct. 1964 Prevention of Brucellosis by I. N. Ivashurova - USSR JPRS: 27, 269 TT: 64-51524 1176/64 Investigation of Optical Oscillator on Ruby at Liquid Nitro- gen Temperature by V. K. Konyukhov, L. A. Kulevskiy, and A. M. Prokhorov, FTD-MT-63-100, 21 Oct. 1963 Translations on Soviet Agriculture No. 44, JPRS: 27,272, TT: 64- 51527 6 November 1964 The translations were then stripped of any identifying markers and photoreproduced. The samples were then sent to the science editors at the American Association for the Advancement of Science and to the owner of a commercial translation agency who did not read Russian but was experienced in the editing of translations . These editors were given the following instructions: What is needed is a rank-ordering of the enclosed materials with the best document being given the number "it, and the worst document number "10. " The basis for judgement would be the standards which you as a scientific editor normally apply. What we are after is your rating of excellence or lack of excellence of the writing in these documents. In other words, how does the stuff read? In addition to your rank-ordering of these items (which thus shows their standing relative to each other)? we would welcome your comments as to how they impress you on an absolute scale. That is, although number "1" will be the best of the total group, it still may be an example of poor scientific writing TABLE 7. Ranking of FTD (letters in parentheses) and JPRS Translations Best ~ Editor Number 1 (Commercial firm) 2 4 5 6 l H (G) _ 4 5 (D) C (I) C H J (G) E H C (G) E H C (G) J C H H E E (G) Rating ~ Worst _ 7 8 9 10_ E (A) B E (D) J (G) E (A) (G) B (D) C (I) (D) B (A) (A) (D) (I) J (D) B (A) (F) J B (A) (I) (I) (D) J (F) (F) (F) (F) (I) (A) (F) (0 (J) (F) 103

Results of the editors' ranking are given in Table 7. In order to obtain a numerical rating of the translations, those appearing in column 1 were given a score of 100; each column was scored 10 points lower so that those in column 10 were given a rating of 10. On this basis the numerical scores of the translations are as follows: Translation H 640 C 580 E G D B A F Score Translating Agency 550 530 360 310 270 270 260 80 JPRS JPRS JPRS FTD FTD JPRS FTD JPRS FTD FTD If both FTD and JPRS had had equal numbers of translations on either side of the median (55), their scores would each have been 1,925 (half of the total 3,850 points possible). Actually the JPRS translations scored 2,350 points and the FTD translations scored 1,500 points. Concerning the absolute merit of these translations, some com- ments of editors might be informative: Number 4. "I consider this (E-JPRS) a paper of average merit, which, from the standpoint of style and clarity, would be accept- able for publication in a technical scientific journal." Number 4. "tWhat is it all about ?' says paper F. What indeed! This one is hopeless." Number 3. "(E and H) could be published as is or with very little rephrasing. " Number 2. "As scientific writing, C is acceptable, H,J,G, and E are fair and could be fixed up with a little editing. The rest go from poor to very poor." Although the sample was too small to allow one to generalize with a great deal of confidence, the consensus of the editors con- cerning the relative worth (or worthlessness) of some of the translations (e.g., H and F) tends to increase one's confidence 104

in the findings of this study; i.e., the JPRS translations are some- what better than the postedited machine translation, arid the un- edited rough-draft human translation is the worst of all. This conclusion, when coupled with the report from the Govern- ment Printing Office (Exhibit 1) concerning the graphic arts quality of these samples, would tend to indicate that JPRS trans- lations are superior to FTD translations. Statistical reliability figures based on these ratings have been computed by Professor J. B. Carroll. They are as follows: Kendall's W., a coefficient of concordance, based on the JPRS- FTD comparison ratings, is 0.724, well beyond the 0.001 level, but not as high as 1.00, the figure indicating perfect reliability. The application of the Mann-Whitney U-test to the summed ratings gives a value of U=4.5. For the case where 5 values are being compared with 5 values, this is significant only with a probability between 0.096 and 0.15. This is not sufficiently signi- ficant to reject with any confidence the null hypothesis that the two sets of translation are drawn from the same population. The summed ranks on which the Mann-Whitney test was based are as follows: JPRS H 13 C 19 E 22 B 46 50 FTD (G) 24 (D) 41 (I) 50 (A) 51 (F) 69 105

EXHIBIT 1. Dr. A. Hood Roberts, Executive Secretary National Academy of Sciences National Research Council 2101 Constitution Avenue Washington, D.C. 20418 Dear Dr. Roberts: March 24, 1965 In answer to your request for an evaluation of the quality of the printing of the translated material which you left with me, we have arrived at the following breakdown: 1. F 2. B,C,H,J 3. G,E,D,A,I Rating Satisfactory Fair Poor Group 1: This is adequate perhaps only because it is double spaced and seems to be blacker than the rest of the submissions. Group 2: The printing of these is very poor, although not so bad but what the text can be read. The difficulty here seems to be that there has been no attempt to maintain good ink coverage, or good quality camera work and platemaking. The presswork is particularly bad where smudges are permitted to appear across the printing. Group 3: This group contains lithe illustrations. Most of them are evidently too many times removed from the original, or they were made from duplicator copies (Xerox, Ozalid, etc.) which always lose much of the detail. If the original copy had been used as camera copy, I am sure much better results could have been obtained. If the original copy was used, then the results are simply bad handling or inexperienced personnel. There seems to be little reason for repro- ductions as poor as this last group. Sincerely yours, JAMES L. HARRISON Public Printer By: Frank H. Mortimer Typography and Design Manager United States Government Printing Office 106

Next: Appendix 16. Government Support of Machine-Translation Research »
Language and Machines: Computers in Translation and Linguistics Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!