A Document Space Model for Automated Text Classification based on Frequency Distribution ac

A Document Space Model for Automated Text Classification

based on Frequency Distribution across Categories

Akshar Bharati

Kiran Varanasi Chaitanya Kamisetty

Rajeev Sangal S. M. Bendre

International Institute of Information Technology,

Hyderabad

Email : {kiran, kamisetty}@https://www.360docs.net/doc/0113936310.html,, {sangal,bendre}@https://www.360docs.net/doc/0113936310.html,

Abstract

The distribution of words on a 2D plane with the document frequencies of the word as axes, is an ideal space to define a weighing metric for terms in the categories. In this paper, a completely data driven approach is presented to compute a weight surface over this plane, taking the axes as the document frequency of the word in the category and that in the rest of the categories.

The statistical distribution of words across the categories is taken as the basis to build a document space model for representing the category vectors. This paper discusses how the model encompasses heuristics like the Inverse Document Frequency, and how it captures the semantics of the given categories. This paper also presents an evaluation of the model through various experiments.

1. Introduction

Automatic text classification deals with the task of assigning predefined categories to unclassified text documents. Each such category is usually defined by a collection of example text documents which we say belong to that category. They constitute the training data. The system is trained on these documents to capture the semantics of each category. When an unknown document is given to the system, it automatically assigns it to the category into which it most appropriately fits.

This classification of textual data has great practical significance in effective document management. In particular, as the amount of available online information increases, managing and retrieving these documents is difficult without proper classification. Text classification has practical utility in automatic filitering of email messages, redirecting of the messages to the appropriate personnel etc.

The theoretical model for a text classifier, like any pattern classification system, is the vector space model[1]. Each document is represented as a vector in a feature space. The features used are usually the words (terms) that make up the document. The component of a document vector along a feature dimension is called the weight of the term in that document.

In the document space, the categories are represented as clusters of their document vectors, usually as simple mean vectors of their document vectors. When a test document is given to the system, it computes different

similarity measures between the test vector and each of the category vectors. The system then assigns the document to the most similar category.

Some classifiers have more complex decision mechanisms. They use support vector machines [5], KNN classifiers, regression models, symbolic rule learning etc. But most of them have an underlying vector space model for the categories, and use a weighing formula for representing the category vectors. Several measures are used to assign weights, and each of them relies on a heuristic – inverse document frequency, maximizing information gain etc.[6] In this paper, we try to develop a more elaborate model for assigning the weights.

In text classification, there are a few interesting questions that need to be answered. (1) How true are each of the category’s documents to the representation given to the category ? (2) How well does computing a simple mean vector capture the essence of the category ? (3) How well does a category vector captures the semantics of what it means to be a category, say sports ? As we try to answer these questions, it becomes clear that giving a proper representation to the category through its documents is a non-trivial exercise. In this paper, a vector space model for representing the categories is presented. This is based on assigning weights proportional to the document frequencies of a word in the current category as against the rest of the categories. The paper discusses how this model can be used to answer the above questions.

In Section 2, a document space model is introduced. This provides the background for our research. In Section 3, representation of categories in the document space is discussed. It also presents a few popular measures – in particular the method of assigning weights proportional to inverse document frequency, in view of the proposed model. In Section 4, a new metric for assigning term weights developed from the model is presented. Section 5 gives an introduction to the corpus that is used in our experiments. Section 6 gives an evaluation of the proposed method based on the results of different experiments.

2 The Document Space Model

By observing how the training data is clustered in the document space, we can draw a few deductions about the possible success of the classification exercise. For example, if the categories are represented as well separated clusters of documents, they are easier to classify. If the documents in a category do not form a single solid cluster but form some loosely separated chunks in the document space, then we can say that such a category is not well defined. It is not a problem with the documents themselves, but with the logic used to group them into a single category which guides the success of classification. It is rational to predict that such categories obtain low recall in the classification process. Also, if two categories form highly overlapping clusters in the document space, then it might be difficult to classify documents amongst them which translates to obtaining a low precision in classification.

The main difficulty of a text classifier is the high dimensionality of the feature space. All the same, the number of dimensions (terms) in a category is also a valuable performance index. In all the terms we have, a few terms will be important in giving an identity to the category – these words have great power to discriminate the category from the rest of the categories. The other terms will have frequent occurrence in many categories and so are not very effective. Besides, there are a few words which have equally high frequencies in all the categories, and hence are irrelevant for the classification exercise. These words are called stop words. They usually have very high frequency in the total corpus, and are removed prior to classification. Also, some words occur very rarely and cannot be regarded as statistical evidence. These words can be removed prior to classification as rare words.

Several objective measures can be developed to measure the separability of a given document space.

Intra Cluster Distance

∑==n i i k k D C Dt Q 1)*,( (2.1) Inter Cluster Distance *)*,(,k j k j C C Dt IC =

(2.2) The average percentage of words per category which occur in only one document in the corpus

))|(),|(()(i j j l j l i j j C D k D t t Num D t D D t t Num Avg C Unique k i ∈∈??∧∈=∈ (2.3) The distance of the document most distant from the category centroid

))*,(()(i k C D k D C Dt Max C Extent k i ∈= (2.4) Space Density as proposed by Salton et al [2]

),(∑∑

=i j i j D D S SD (2.5) where D i : i th document vector S(x.y): a similarity measure C k * : Centroid of the k th category Dt(x,y) : Euclidean distance between vectors x, y

The above measures indicate how difficult it is to classify the documents of a given corpus into categories. We also derive heuristics from the above measures for computing the importance of a term for classification. 3 Representating the categories

Section 3.1 describes a procedure for computing document vectors based on inverse document frequency. The categories are given a representation from the document vectors. In section 3.2, we propose an alternative method of category representation based on frequency distribution of words across categories. The categories are represented directly, without any intermediate computation of document vectors.

3.1 Representation using Inverse Document Frequency

To represent a document as a vector in term dimensions, we need a function to assign the projection value of the document along each dimension. In other words, we need a formula to compute the weight of each word in the document.

It has been observed that the Inverse Document frequency is a good index of the usefulness of a word in classification[3][4]. That is to say, fewer the documents a word occurs in, greater is its ability to discriminate and identify the category of the document. The weights are assigned as the product of Term frequency and Inverse document frequency (TfIdf ) given by

)(/*),(),(j j i j i t DF NDOCS t D TF t D Wt = (3.1) where TF(D i ,t j ) : Frequency of term t j in document D i

NDOCS : Total number of documents in the corpus

DF(t j ) : Number of documents in which t j occurs

Now each document vector is normalized to unit length using cosine normalization which gives

∑=j j i j i j i t D Wt t D Wt t D Wt ),(),(),(*2 (3.2)

It was observed by Salton et al[2] that TfIdf vectors improve the space density measure more than the simple TF vectors, making the document vectors easier to classify. Further, the category vectors are represented as the mean vectors of their documents. We experimented by varying the standard TfIdf formula, with the objective of improving the accuracy. The variations considered are

1) Words having equal frequencies across the categories are considered to be stop words and are filtered out.

2) A log (and sqrt ) function is applied to the inverse document frequency term.

))(/log(*),(),(j j i j i t DF NDOCS t D TF t D Wt =

(3.3) 3) An additional factor is used to multiply the TfIdf weight to boost the words occuring more frequently in the present category

∑≠=k

l j l j k j k j k t C TF t C TF t C Wt t C Wt ),(),(*),(),(*2 (3.4) The precision of classification using weights in Eq (3.4) was 10% greater than the precision obtained using the orignal TfIdf weights (refer to Section 6 for the listing of results). This is because the document clusters have now become more separated.

3.2 Representation based on document frequency distribution across categories

The document frequencies of words in a corpus give valuable clues to answer the following questions : (1) Which are the words important in discriminating a category ? (2) Can we draw generalizations about their average frequency of occurrence in the concerned category and in the rest of the categories? For measuring the ability of a word to discriminate between two categories, one has to observe the projections of its frequencies on the line which joins the centroids of the two categories. The location of the projection on the line gives an idea of the extent to which the word falls in the opposite category. This signifies the relative importance of the word for the concerned categories.

The ability of a word to discriminate a category depends on how frequently it occurs in that category as against the rest of the categories. Hence, to index the significance of a word, we choose a plot over the 2D plane defined by – the Frequency in the current category (x-axis) and Average Frequency in the rest of the categories (y-axis). We call this plane the reference plane for indexing.

The reference plane between two categories looks as shown in Fig 1. A few statements can be readily made about certain areas in the figure :

1) Words occuring in the upper left corner [region A] have no ability to characterize the current

category. However a few of them might characterize some other categories

2) Words in the upper right corner [region B] do not characterize any category. These words can be

regarded as noise and can be removed as stop words

3) Words in the lower right area [region C] have great ability to characterize the present category. A few

of these words may get low weights in the TfIdf formula, but nevertheless, these are words with good discriminating ability.

4) Words in the lower left area [region D] have weak characterizing ability due to low statistical

evidence.

Table 1 gives examples of such words in a category - Computer Graphics (refer to Section 5 for details on the corpus) listed with frequency (DF), total term frequency in the category (TF) , average frequency over the rest of the categories (AvgRest) and a weight computed for the word using Eq (3.4).

Figure 1: Regions in the reference

plane

Figure 2: Example words in Table 1 on the reference plane

No Word DF TF AvgRest Weight

Type A words

1 scale 24

138 6.75 1.63

2 document 24 66 20.5 0.37

3 controller 5 32 47.2 0.01

Type B words

4 gif 94

402 39.71 5 motif 20

120 40.25 0.13 6 viewer 46 195 4 18.10

Type C words

7 3D 110

292 5 71.36 8 jpeg 48

753 0.5 69.63

9 raytracing 14 24 0 56.90

10 cjpeg 10 28 0 12.42

Type D words

11 gum 3 6 0 3.47

12 refinement 2 6 0 2.03

13 yuv 5 10 0 7.13

14 IEEE 9 17 1 0.86 Table 1 : Examples of types of words in

Computer Graphics

A distribution of these words over the reference plane is given in Fig 2. It can be observed that the words 3D

(7) and jpeg (8) occur in region C, in the lower right corners of the plane. They deserve the high weights assigned to them because they have good characterizing ability.

Each category has a certain set of words almost exclusive to it (belonging to region C). If such a word occurs in an unknown document, we can immediately identify it with that category. Identifying such words considerably improves the precision of the classification performance.

For the remaining words (regions B, D), the relative numerical values of the weights assigned across the categories is more important. Improving the weight function brings in a corresponding improvement in the recall of the classification performance. Thus, there are two goals in developing a representation for the categories : (I) Identifying the characterizing words and (II) Giving a proper weight function.

To realize (I), we separate out the characterizing words in region C. It is possible to delineate a geometric curve on the reference plane, above which no such characterizing words occur. We call this curve the low spread curve because the words demarcated by this curve occur frequently in only the current category and have low spread in other categories. Similarly it is possible to delineate another curve called the high spread

curve which demarcates the boundary of all the words worth considering for classification and separates them from the words which just add noise. An elaborate method is presented in Section 4.3 to define a proper weight function over the reference plane to accomplish (II).

4 A metric for assigning weights

As discussed in Section 3.2, it is the projection of the word on the line joining the centroids of two categories that affects its ability to discriminate between the two categories. Correspondingly, a method in this section to assign weights to terms. This is an empirical method which is based on the distribution of the corpus words on the reference plane. It does not assume any other theoretical model for the probability distribution of the categories. This method of assigning weights to terms (dimensions) is composed of 3 stages :

? Based on the document frequency distribution of words, choose certain points on the x-axis. We call

these points knots.

? Obtain the y-coordinate values of the low spread curve and the high spread curve at these knots. The

curves are then delineated as an interpolation of these values.

? Based on these 2 curves, define a weight surface over the reference plane.

4.1 Selecting the knots

Let k 1, k 2 … k n be the knots selected such that they form zones of equal number of words on the reference plane. We fix them in the same order. In particular, let

KW : The number of words occurring between the knots k i and k i+1

NW : The total number of words in the corpus

Then the knot k i+1 is positioned such that : NW n KW */1= (4.1) The value of n is not very important, n=8 works well for a corpus of 43000 words.

4.2 Fitting the curves

Out of the KW words occuring between the knots k i and k i+1, let L words occur below the low spread curve(l i ) and H occur above the high spread curve(h i ). i=1,…,n

We vary l i to obtain different values for L. We fix l i when we obtain an L which satisfies :

α=KW L / 10<<α (4.2)

And we fix h i when we obtain an H which satisfies the equation :

β=KW H / 10<<β

(4.3) It was observed on the corpora that 5% of the words in every category have very good characterizing ability. And about 30% of the words do not have any ability to identify the category. So, a choice of 05.0=αand 3.0=β works well. Infact, these values are a good choice for any corpus because observation on statistics of different corpora confirms that αβ> and these values are independent of the category. Once we fix {l i } and {h i } for i = 1,…,n we can obtain the low spread curve l(x) as a linear interpolation of the {l i } points. On similar lines, the high spread curve h(x) is drawn by interpolating the {h i } points. (refer to Fig 3)

4.3 Defining the weights

We define a weight function Wt(x,y) on the reference plane. For y = l(x), the weight function along the low spread curve l(x) is defined as (refer to Fig 4)

γμμ*)

1ln()100/*)1ln(()())(,(++==x x F x l x Wt (4.4) ZKHUH DUH FRQVWDQWV

Figure 3: Fitting the low spread and

high spread curves

Figure 4 : Weights along the low

spread curve

All the words that fall on and beyond the high spread curve are given zero weight. And along a vertical line at a constant x, the weight function is defined as (refer to Fig 5)

0),(=y x Wt )(x h y ≥? )(*)()()()(),(x F x l x h x l y x F y x Wt ???= )()(x h y x l <

the training data set. We used the values RI 1DQG 1

The values of {k i }, {l i } and {h i } that we obtain on the reference planes for different categories look appreciably similar (Rrefer to distributions on the reference plane : Fig 7 and Fig 8 in Section 5) This provides a sound evidence for modelling a single weight surface for all the categories. To compute the surface, we average over the values of {k i }, {l i } and {h i } that we obtain for the individual categories. The generated weight surface on the reference plane looks similar to Fig 6.

Figure 5 : Weights along a vertical line at constant DF

Figure 6 : The weight surface as a plot

on the reference plane

5 Experiments on the corpus

For all our experiments, we primarily use a corpus of 43000 words1. There are five mutually exclusive categories : Graphics, MS Windows, X Windows, IBM PC Hardware and Mac PC Hardware. Each category is a collection of 900 emails on that topic. For our classification purposes, we consider only the body section of each email and delete the subject and header sections. The basic statistics of these categories are given in Table 2. We use the measures defined in Section 2 to analyse the separability of the corpus. The values of the number of unique dimensions for different categories,as defined in Eq (2.3), are listed in the 5th column of the table 2.

Table 2 Basic Statistics of the Email Corpus

Category Number of words

having non zero

freq

Average document

frequency

Number of words

occuring > 40 docs

Percentage of

words occurring in

only one document

Number of words

unique to the

category

Graphics 13525 4.59 189 46

% 6190 MS Windows 9389 4.76 156 53 % 3471

X Windows 19640 3.50 214 65 % 10832

IBM PC 10723 4.54 175 58 % 4658

Mac PC 9638 4.67 168 54 % 3984

Table 3 Inter Cluster distance between the categories

The numbers “a(b)c” in each cell are :

(a) : Angle between the vectors using TfIdf weights assigned by Eq (3.3)

(b) : The angle between the vectors using weights assigned by Eq (3.4)

(c) : Euclidean distance between vectors using weights assigned by Eq (3.4)

Graphics MS Windows X Windows IBM PC Mac PC Graphics - 54.6 (89.3) 36.5 53.9 (89.6) 62.5 61.7 (89.7) 39.6 62.5 (89.8) 29.9 MS Windows 54.6 (89.3) 36.5 - 51.7 (89.5) 98.6 55.2 (89.4) 17.5 59.1 (89.5) 64.9 X Windows 53.9 (89.6) 62.5 51.7 (89.5) 98.6 - 65.2 (89.8) 101.1 65.6 (89.9) 36.1

IBM PC 61.7 (89.7) 39.6 55.2 (89.4) 17.5 65.2 (89.8) 101.1 - 48.2 (89.4) 67.9

Mac PC 62.5 (89.8) 29.9 59.1 (89.5) 64.9 65.6 (89.9) 36.1 48.2 (89.4) 67.9 - Classification amongst these categories is difficult due to the following reasons : (1) There is considerable overlap between the categories – for example, many words occur with equal frequencies in the categories of

IBM PC and Mac PC. (2) Some categories have very few characterizing words [ Region C ] (not many dimensions exclusive to the category) - MS Windows has only few words that occur with very high frequencies, hence these few words get exorbitantly high weights. If these words occur in some other category, then that document may be misclassified as MS Windows. (3) In contrast, some categories contain many characterizing words – but none of them get very high weights because of their relatively low frequency [bordering Region D]. For instance, X Windows is defined on many such words, so it overlaps with many other categories. Playmation is usually a good word for graphics, but is not good in discriminating graphics from X Windows (4) We assume that the relative order of occurrence of words in a document is unimportant

for classification. This may not be true – though visual and basic may both characterize the graphics category, visual basic as a string belongs to MS Windows. (5) There is a huge percentage of words in each document which do not occur elsewhere in the entire corpus. This means that we cannot use this percentage of

1The corpus is a subset of the 20 Newsgroups collection used as a testbed for the Rainbow classification system. Downloadable at https://www.360docs.net/doc/0113936310.html,/~mccallum/bow/rainbow

Figure 7: The distribution of words on Graphics

reference plane Figure 8 : Distribution of words on the MS Windows reference plane

information in each document for classification. Fig 7 and Fig 8 show the distribution of words on the reference planes of Graphics and MS Windows categories.

Inter cluster distances computed using Eq (2.2) are listed in Table 3. They give us more insight into the nature of the categories. In particular, the first metric (a) is a valuable indicator of the nearness of two categories –IBM and Mac are the categories closest to each other. The second metric (b) indicates that the categories become more separated after multiplying with the factor Freq / AvgRest. The third metric (c) says that Euclidean distance is not a good indicator because of the large number of dimensions. The number of words of overlap influence the separability more than the length of overlap along each word.

6 Results and Evaluation

The results of different experiments that we conducted on the corpus are listed in Table 4. The accuracy of classification using the weighing metric (d) as defined in Section 3.2, is comparable to the classical TfIdf heuristic (a) which gave 77 % accuracy. If the weighing metric is defined on total term frequencies (c) rather than document frequencies as we discussed (d), the classification accuracy decreased by 7 %. The weighted TfIdf formula (b) as defined in Eq (3.4), gave the best accuracy of 85 %. The classical performance metrics –precision and recall can be computed from the table. It is to be noted that, for a text classification system, precision (accuracy) is more important than recall, because it is usually possible to provide human supervison to classify the unclassified documents.

Graphics MS Windows X Windows IBM PC Mac PC Graphics 69, 77, 63, 71 15, 8, 4, 5 4, 6, 12, 9 6, 2, 5, 3 3, 4, 7, 5 MS Windows 4, 7, 5, 6 88, 87, 55, 72 3, 4, 9, 6 10, 13, 17, 12 4, 3, 15,5

X Windows 12, 8, 13, 11 12, 4, 2, 2 75, 88, 78, 81 0, 0, 0, 1 1, 0, 2, 1 IBM PC 1, 1, 2, 1 12, 10, 6, 4 1, 1, 6, 2 83, 84, 71, 75 3, 4, 6, 9 Mac PC 3, 0, 1, 0 3, 3, 2, 2 1, 1, 0, 0 20, 5, 7, 5 72, 90, 84, 87 Table 4 : Classification Results on the email corpus

(for 100 test documents in each category)

The numbers (a, b, c, d) in each cell are :

(a) : TfIdf as TermFrequency * sqrt(InverseDocument Frequency) as in Eq(3.3)

(b) : TfIdf multiplied by Freq / AvgRest as in Eq(3.4)

(c) : Weight Surface defined over TF(C k, t j) i.e, term frequencies of words

(d) : Weight Surface defined over DF(C k, t j) i.e, Document frequencies of words

Table 5 : Classification Results on Reuters Corpus

for 24 test documents in each category

The weights used are (b,d) as defined in Table 4

Gold Coffee Sugar Interest Trade Gold 24, 24 0, 0 0, 0 0, 0 0, 0

Coffee 0, 0 24, 24 0, 0 0, 0 0, 0

Sugar 0, 0 1, 1 22, 23 1, 0 0, 0

Interest 0, 0 1, 0 0, 0 23, 24 0, 0

Trade 0, 0 0 , 0 0, 0 1, 2 23, 22

We conducted experiments on a second corpus, the Reuters corpus2 which is a collection of newswire articles on different topics. The results of classification using the formulae (b) and (d) are listed in Table 5.

We chose 5 categories each containing 100 very short articles to train the system. They are articles on Sugar, Coffee, Gold, Interest and Trade. We used 24 testing documents in each category to test the system. The system classified the documents with perfect accuracy. Similar investigation using separability measures shows that these categories have very minor overlap and are thus easily separated. Hence we obtain good accuracy in classification. Also, as the training corpus becomes smaller in size the characterizing power as defined by the weight surface performs better than the weighted TfIdf formula. This is because Inverse Document Frequency is not a very good heuristic for a small corpus. The measures (b) and (d) give almost the same ranking to words in terms of weights in a category, for both the corpora. This is because, the inverse document frequency, in a weaker sense, captures the same heuristic as the document space model in Section 2.

6 Conclusion

In this paper we discussed a document space model for assigning weights to terms based on their frequencies across different categories. The frequency of the word in the current category as opposed to that in the rest of the categories is used to index its relative discriminating ability. To further improve the performance, we are experimenting with modifications to the weight surface. We plan to use this as the ground model instead of the TfIdf method and analyze the performance of various classifiers based on support vector machines, boosting , k-nearest neighbour etc.

References

[1] L. Guthrie and E. Walker Document Classification by Machine: Theory and Practice Proceedings of COLING 94. 1059-1063. (1994)

[2] G.Satlon, A.Wong, C.S.Yang, Cornell University A Vector Space Model for Automatic Indexing, Association of Computing Machines (1975)

[3]Kazuhide Yamamoto, Shigeru Masuyama, Shozo Naito Automatic Text Classification Method with Simple Class-Weighting Approach (1995)

[4] Takenobu Tokunaga, Makoto Iwayama Text categorization based on weighted inverse document frequency (1994)

[5] Thorsten Joachims Text Categorization with Support Vector Machines: Learning with Many Relevant Features Proceedings of ECML-98, 10th European Conference on Machine Learning (1998)

[6] Yiming Yang, Jan O. Pedersen A Comparative Study on Feature Selection in Text Categorization Proceedings of ICML-97, 14th International Conference on Machine Learning (1997)

2The Reuters 21578 data collection is a standard corpus for comparing the performance of different classification systems. Downloadable at https://www.360docs.net/doc/0113936310.html,/databases/reuters21578/reuters21578.html