2. Corpus lexicography

So lexicography is one of the fields where corpus linguistics not only introduced new methods, but also extended the entire scope of research, however, without putting too much emphasis on the theoretical aspects of corpus-based lexicography.

One may say that Corpus lexicography is the process of compiling or revising a dictionary based on texts (of written and/or spoken language) collected in an electronic format (i.e., corpora).

It was John Sinclair who lead the way as initiator of the first strictly corpus-based dictionary of general language. British linguist, the founder of the COBUILD project at the University of Birmingham, oversaw the production of the first strictly corpus-based dictionary, Collins COBUILD English Language Dictionary (1987).

Britain was also the site of the first corpus-based collocation dictionaries.

Bilingual lexicography may also benefit from a corpus-oriented approach: a fact that is evident when comparing the traditional Le Robert & Collins English-French Dictionary edited by B.T.S. Atkins with Valerie Grundy and Marie-H´ el` ene Corr´ eard’sOxford-Hachette Dictionary which covers the same language pair. Here, the use of (monolingual) corpora lead to a remarkably greater number of multiword translation units (collocations, set phrases) and to context profiles that had been written with the target language in mind.

”Wörter und Wortgebrauchin Ost und West” (Words and Word Usage in East and West Germany) (1992) by Manfred W. Hellmann may serve as the only German example of that era, using the corpus for lemma selection rather than semantic description. Only recently, in 1997, did a true corpus-based dictionary appear: “Schlüsselwörterder Wendezeit” (Keywords during German Unification) by Dieter Herberg, Doris Steffens and Elke Tellenbach.

As of spring 2010, according to the website of Oxford Dictionaries, the Oxford English Corpus "contains over 2 billion words of real 21st century English. It is not only size that matters, though: it is the size of the corpus coupled with the careful selection and development of its contents . . .".

2. Corpora in Lexical Studies

Empirical data has been used in lexicography long before the discipline of corpus linguistics was invented. Samuel Johnson, for example, illustrated his dictionary with examples from literature, and in the 19th Century the Oxford Dictionary used citation slips to study and illustrate word usage. Corpora, however, have changed the way in which linguists can look at language.

What differs in practice for the modern lexicographer is the possibility to produce through text-processing software, contexts for the totality of words in the corpus ordered alphabetically with frequency counts attached: a rather different exercise from the manual assignment of slips to 1,029 alphabetically ordered pigeon-holes.

However, the issue of judgment remains: whether the lexicographer is working with ten slips for a headword or 300 lines of a concordance, the task of grouping quotations/contexts according to sense remains an interpretive exercise, albeit made easier by various processing options when working digitally. In principle, the exercise still consists of organising the quotations/examples ―that they might in due course be arranged and classified alphabetically and by meanings (1859 version) or ―that they might in due course be arranged and classified alphabetically and significantly (1879 version).

The main difference between manual and computer-aided lexicography seems to lie not principally in selection or in interpretation but in the superior processing capacity of machines to sort the data for human interpretation, compared to that of humans manually sorting the data for human interpretation. In a digital medium, for example, text-processing options such as left-sort can be used in order to identify formal patterns in which the potentially polysemous search word is the phrasal head; such patterns—grouped together automatically—can in themselves be indicative of different senses. The full context for the search word is also usually just a click away.

Nevertheless, despite praising the Cobuild English dictionaries for their innovative descriptive definitions, word-frequency bands and up-to-date entries, in a later edition of Translation Today, Peter argues that the value of corpus linguistics is overrated when it is claimed that a corpus represents the real language, since many mistakes and misunderstandings arise from contextual guestimates.

But it is not quite clear here how a qualitative distinction can be drawn between the interpretation of quotations on slips on the one hand and of a (probably) greater number of examples presented in a concordance on the other hand. Perhaps the key lies in the implied indiscriminate inclusion of deficient texts in a digital corpus. Such an interpretation of the raw materials is indeed consistent with Peter‘s summative comment that Cobuild‘s deficiency is that it fails to account for linguistic ignorance, carelessness and misinformation.

Hence, Peter concludes that the book and the up-to-date dictionary will always be indispensible. Indeed, but corpus-based lexicography (with the exception of scholarly historical studies) aims to be precisely that, i.e. up-to-date by making available to the lexicographer ample contemporary examples of how a particular word behaves in context.

A linguist who has access to a corpus, or other (non-representative) collection of machine readable text can call up all the examples of a word or phrase from many millions of words of text in a few seconds. Dictionaries can be produced and revised much more quickly than before, thus providing up-to-date information about language. Also, definitions can be more complete and precise since a larger number of natural examples are examined.

This example is taken from Atkins and Levin "Building on a corpus: a linguistic and lexicographical look at some near-synonyms", International Journal of Lexicography 8:2, 85-114. Atkins and Levin studied verbs in the semantic class of shake and quoted definitions from the following dictionaries:

• the Longman Dictionary of Contempary English

• the Oxford Advanced Learner's Dictionary

• the Collins COBUILD Dictionary

Two of the entries which Atkins and Levin disuss are quake and quiver. Both the Longman and COBUILD dictionaries list these verbs as being instransitive, while the Oxford dictionary lists quakeas being intransitive, but lists quiver as being transitive.

Looking at the occurences of these verbs in a corpus of 50,000,000 words, Atkins and Levin were able to discover examples of both quiver and quake in transitive constructions.

E.g. It quaked her bowels and quivering its wings.

In other words, the dictionaries had got it wrong - both verbs could be transitive as well as intransitive. This small example demonstrates how a sufficiently large and representative corpus can either supplement or refute the lexicographer's intuitions.

Examples extracted from corpora can be easily organised into more meaningful groups for analysis. Furthermore, because corpus data contains a rich amount of textual information - regional variety, author, date, genre, part-of-speech tags etc it is easier to tie down usages of particular words or phrases as being typical of particular regional varieties, genres and so on.

The open-ended (constantly growing) monitor corpus has its greatest role in dictionary building as it enables lexicographers to keep on top of new words entering the language, or existing words changing their meanings, or the balance of their use according to genre etc. However, finite corpora also have an important role in lexical studies - in the area of quantification. It is possible to rapidly produce reliable frequency counts and to subdivide these areas across various dimensions according to the varieties of language in which a word is used.

Finally, the ability to call up word combinations rather than individual words, and the existence of mutual information tools which establish relationships between co-occuring words mean that we can treat phrases and collocations more systematically than was previously possible. A phraseological unit may consitute a piece of technical terminology or an idiom, and collocations are important clues to specific word senses.

Notable English language corpora include the following:

• The Brown Corpus

• The American National Corpus (ANC)

• British National Corpus (BNC)

• The Corpus of Contemporary American English (COCA)

• The International Corpus of English (ICE)

The Brown University Standard Corpus of Present-Day American English (or just Brown Corpus) was compiled in the 1960s by H. Kucera and W. Nelson Francis at Brown University, Providence, Rhode Island as a general corpus (text collection) in the field of corpus linguistics. It contains 500 samples of English-language text, totaling roughly one million words, compiled from works published in the United States in 1961.

The Corpus is divided into 500 samples of 2000+ words each. Each sample begins at the beginning of a sentence but not necessarily of a paragraph or other larger division, and each ends at the first sentence ending after 2000 words. The samples represent a wide range of styles and varieties of prose. Verse was not included on the ground that it presents special linguistic problems different from those of prose. (Short verse passages quoted in prose samples are kept, however.) Drama was excluded as being the imaginative recreation of spoken discourse, rather than true written discourse. Fiction was included, but no samples were admitted which consisted of more than 50% dialogue.

The British National Corpus (BNC) is a 100-million-word text corpus of samples of written and spoken English from a wide range of sources. The corpus covers British English of the late 20th century from a wide variety of genres with the intention that it be a representative sample of spoken and writtenBritish English of that time.

The American National Corpus (ANC) is a text corpus of American English containing 22 million words written and spoken data produced since 1990. The ANC may at some point of time include a range of genres comparable to the British National Corpus. It is annotated for part of speech and lemma, shallow parse, and named entities.

The ANC in its current size of 22 million words is available from the Linguistic Data Consortium. A 15 million word subset of the corpus, called the Open American National Corpus (OANC), is freely available with no restrictions on its use from the ANC Website.

The corpus of contemporary American English

The Corpus of Contemporary American English (COCA) is the largest freely-available corpus of English, and the only large and balanced corpus of American English. The corpus was created by Mark Davies of Brigham Young University, and it is used by tens of thousands of users every month (linguists, teachers, translators, and other researchers). COCA is also related to other large corpora that we have created.

The corpus contains more than 450 million words of text and is equally divided among spoken, fiction, popular magazines, newspapers, and academic texts. It includes 20 million words each year from 1990-2012 and the corpus is also updated regularly. Because of its design, it is perhaps the only corpus of English that is suitable for looking at current, ongoing changes in the language.

The International Corpus of English (ICE) is a set of corpora representing varieties of English from around the world. Over twenty countries or groups of countries where English is the first language or an official second language are included.

ICE corpora contain 60% (600,000 words) of orthographically transcribed spoken English. The father of the project, Sidney Greenbaum, insisted on the primacy of the spoken word, following Randolph Quirk and Jan Svartvik's collaboration on the original London-Lund Corpus (LLC). This emphasis on word-for-word transcription marks out ICE from many other corpora, including those containing, e.g. parliamentary or legal paraphrases.