Document Description

1 Document Analysis

Figure 1 The process of building a document description.

1. The lexical analysis of textual streams.
2. The treatment of the stopwords.
3. The selection of the words through thesauri (controlled vocabularies).
4. The stemming of the remaining words.
5. The selection of the concepts used to build the description.

2 File Extraction

2.1 Semantic Information

• the XML technologies are based on XML schemas proposing semantic models composed from semantic concepts (the tags);
• the combination of some tags and their content provides useful information to represent XML documents (for example the well-known “metadata tags” of the Dublin Core MetaData Initiative (DCMI) [5]);
• all the tags aren’t needed to catch the main semantic information contained in a XML document, since some of them serve structuring purposes only (they regroup several child tags related to a same kind of information) or give information such as the language used.

<bibliography>

• <entry>
  

  • <author>Pascal Francq</author>
    <title>Internet : La construction d’un mythe</title>
    <editor>EME Intercommunication</editor>
    

  </entry>
  

</bibliography>

2.2 XML Metadata Detection

Figure 2 Heuristic to detect metadata tags

1. A “metadata tag” is “high” in the XML hierarchy (for example it has a depth of maximum 3 starting with depth 0 for the root node).
2. A “metadata tag” doesn’t appear too often in the whole document. In practice, the heuristic may limit the number of occurrences to an absolute value (for example less than 5 occurrences) or to a relative value (less than 5% of the total number of different tags in the document).
3. A “metadata tag” has no child tags.
4. A “metadata tag” contains a maximal number of terms (for example 20).

2.3 Document Divisions Detection

<h1>Bourgeois and Proletarians</h1>
<p>

• The history of all hitherto existing
  society is the history of class
  struggles.
  

</p>

<chapter>

• <title>Bourgeois and Proletarians</title>
  <para>
  

  • The history of all hitherto existing
    society is the history of class
    struggles.
    

  </para>
  

</chapter>

<h1>Title 1</h1>
<p>This a first paragraph</p>
<h3>Title 1.1</h3>
<p>This is a second paragraph.</p>

2.4 A complete example

<?xml version="1.0"?>
<doc xmlns="http://mytags.org/" xmlns:dc="/">

• <dc:title>Document Title</dc:title>
  <dc:creator>John Cleese</dc:creator>
  <dc:creator>Michael Palin</dc:creator>
  <part>
  

  • <para>Monty Python’s Flying Circus.</para>
    <para>And now, something completely different!</para>
    <support>DVD</support>
    

  </part>
  

</doc>

1. The tags <dc:title> and <dc:creator> are defined as a metadata by the Dublin Core MetaData Initiative XML schema.
2. The tags <part> and <para> propose textual contents.
3. We may suppose that the <support> tag doesn’t provide any interesting information (at least regarding the knowledge embodied in the document).

1. A vector for the XML schema of the Dublin Core MetaData Initiative.
2. A vector for the XML schema of the “mytags” XML application.
3. A vector for the metadata “title”.
4. A vector for the metadata “creator”.
5. A vector for the textual content of the document.

3 Lexical Analysis

3.1 Token Categories

3.2 Computational Linguistics

3.2.1 Syntactic and Semantic Labeling

3.2.2 Syntagm Variant Recognition

3.2.3 Anaphora Resolution

• The rock band has recorded a new album. It plans to make a new world tour.
• Different stemming algorithms for words exist. Implementations of these algorithms can be found on the Web.
• The formula one has win the race. This car is really the best in the paddock.

3.3 Tokenizing Process

1. A textual unit cannot start with a space or a punctuation character (defined by the classical C ispunct function).
2. A sequence of punctuation characters followed by a space cannot end a textual unit. So, “pascal.francq” is a textual unit while “pascal. francq” are two textual units (“pascal” and “francq”).
3. A textual unit always ends with a space.

Figure 3 Tokenizing using a finite-state machine.

1. A textual unit is considered as a token if it is made up of letters only. This option ensures that tokens are definitively words. It has also the advantage to limit the “sizes” of the descriptions (thus less memory used and a reduced computational speed).
2. Some textual units containing non-alphabetical characters may pick up structured information like norm designations or email addresses. To eliminate those who are not useful, a textual unit must respect two constraints:
   1. A given character cannot occur consecutively more than a given number of times (for example 3). This rule avoids to consider as token the textual unit “loool”.
   2. The textual unit must contained a minimum percentage of letters (for example 30%). This rule eliminates the textual unit “B-1070” (16,6%) but not “pascal.francq@world.earth” (88%).
   3. A given token cannot contain more than a maximum number of non-letter characters (for example 5). This rule recognizes as token the textual unit “iso6002” but not the token “0123456789a”

4 Stopword Treatment

4.1 Stopword Removal

4.2 Document Language Detection

1. The document must contain a reasonable number of different stopwords. If a document contains only the stopwords “no” and “yes” for example, it is probably not an English-language document. This can be expresses with the $R_{l,m}$ ratio:
   $$R_{l,m}\geq Min_{R}$$
   where $\mathit{Min}_{\mathrm{R}}$ is a given threshold
2. Since the stopwords constitute a large amount of the words found in a document, we may suppose that the correct language is the one having the most stopwords in the document:
   $$\begin{eqnarray*} d_{m} & \leftarrow & l=\arg\max_{l\in\mathcal{\mathcal{\mathfrak{R}}}_{l,m}}|\mathcal{\mathfrak{S}}_{l,m}| \end{eqnarray*}$$
   where $\mathfrak{R}_{l,m}$ represents the set of all languages respecting the first condition for document $d_{m}$ .

5 Thesauri

• to provide a standard vocabulary for a given system for searching and indexing;
• to assist users to find the correct formulation for their interest by extending their queries to include related terms;
• to provide a classification of words and to introduce a notion of neighborhood between them.

The adjective "genetic" has 4 senses in the online WordNet [D]  [D] http://wordnetweb.princeton.edu/perl/webwn, WordNet Search 3.1, consulted on January 12, 2012. in 2012.

• 1. (adj) familial, genetic, hereditary, inherited, transmitted, transmissible (occurring among members of a family usually by heredity) "an inherited disease"; "familial traits"; "genetically transmitted features"
  2. (adj) genic, genetic, genetical (of or relating to or produced by or being a gene) "genic combinations"; "genetic code"
  3. (adj) genetic (pertaining to or referring to origin) "genetic history reconstructs the origins of a literary work"
  4. (adj) genetic, genetical (of or relating to the science of genetics) "genetic research"

• it is a freely accessible online to anyone;
• it contains millions of concepts (articles) in different languages with links between them [F]  [F] When a Wikipedia page is consulted, in the menu part (on the left), the net surfers can see the corresponding pages in other languages if their exist.;
• it is updated with a high frequency.

6 Stemming

If a word ends in “ies” but not “eies” or “aies”, 
     replace “ies” by “y”.
If a word ends in “es” but not “aes”, “oes”, or “aes”, 
     replace “es” by “e”.
If a word ends in “s” but not “us” or “ss”, 
     remove “s”.

7 Concept Selection

7.1 Concept Filtering

7.2 Concept Space Transformation

8 Implementation

1. Extract textual streams associated with semantic and division information from the document to analyze.
2. Extract textual tokens from the textual streams.
3. Analyze the textual tokens to decide which ones must be removed (for example stopwords), replaced (such as stems replacing words) or added (during a concept space transformation for example), i.e. choose the textual tokens that will become concepts describing the document.

Figure 4 Document Analysis Implementation.

1. The GFilter class provides a generic plug-in that extracts textual streams from a local file. In practice, each plug-in provides a filter that can handle a set of MIME types (for example “text/email” or “application/msword”). The GDocAnalyze class searches the filter corresponding to the MIME type of the document to analyze, and passes it the local file name (if no filter is found, the document cannot be analyzed). The GDocAnalyze class provides several methods that can be used by a filter, the most important one being:
   ExtractBody Extract some textual content and associate it to the default textual content corresponding to the meta-concept “body”.
   ExtractDMCI Extract some textual content and associate it a metadata defined by the Dublin Core MetaData Initiative (DCMI) [5].
   ExtractTextual Extract some textual content and associate it to a meta-concept to specify.
   AddConcept Associate a concept to a meta-concept to specify.
2. The GTokenizer provides a generic tokenizer plug-in that extracts tokens from textual streams. The GDocAnalyze class passes the textual contents to the current tokenizer plug-in which communicates the tokens extracted trough the method GDocAnalyze::AddToken (if not tokenizer is selected, a document cannot be analyzed).
3. The GAnalyzer provides a generic plug-in to analyzes the textual tokens. In practice, The GDocAnalyze class calls all the active analyzers in a specific order (for example to ensure that the plug-in dealing with stopwords and determining the document language is called before the one that does the stemming which is language-dependent). At least one analyzer should associate to the tokens a concept through the GTextualToken::SetConcept method [H]  [H] This association can be done anytime during the analysis, but for efficiency it is recommended to do so as late as possible. In fact, if we do a stemming, thus replacing all the words with the corresponding stems, it isn’t useful to create concepts for these words since they will never be used.. This can be done with the simple following code:

   GConceptCat* TextCat(Session->GetInsertConceptCat("Text"));
   GConceptType* TermsSpace(Session->GetInsertConceptType(TextCat,"Terms","Terms"));
   GConcept* Concept(TermsSpace->GetInsertConcept(Token->GetToken()));
   Token->SetConcept(Concept);