Calliope: work flow, engine

This is the Calliope work flow: Input text files feed the "Preformat" module, to yield a set of XML work files "Extract" module permits to create, load, edit dedicated lexicons, which contents must be reviewed and tagged by the User. Work files are then automatically "indexed" by validated terms. "Process" module applies dedicated algorithms (coword analysis, trend analysis) "Viewer" module displays the outputs: thematic maps, clusters, and trend curves. Allows to track documents linked to those clusters, to link to other fields. With the same working set of files, it is possible to have more than a single work field.

Evolution

Next version "Calliope 2011" will output data sets feeding market graphers, such as Gephi (open source software http://gephi.org/)

Calliope Extract operation

The Calliope Extract module extracts meaningful terms from textual corpus of files. This extraction is a multi-step process.

• With Calliope Preformat, the raw source files containing the many documents are transformed into a single XML file, maintaining the logical structure of the documents. This preprocessing step is simple and nearly automatic, with no parameter to specify.
• The extraction of the terms contained in the documents is automated and is based upon statistical and linguistic processings (dump meaningless terms, process genders, plurals...). Once this is done, you have a lexicon of extracted terms. Though this list is purged from empty words, some of the terms are synonyms, some others are too generic, or meaningless in the study context. To increase the relevance and efficiency of the next module Calliope Process, you have to either validate these terms or use already existing lexicons.
  The only parameter is the minimal frequency of occurrence of terms.
• If no validation lexicon exists, you have to build one. In this step, you are supported by Calliope which permits you to retrieve the terms within their context, which assists you in getting sorted lists. Among all the Calliope operating steps, this one is for sure the only one to be long and tedious, requiring from you a certain investment. A magnitude order for building from scratch a new lexicon would be 4 to 5 hours for 150 pages of text. When this phase is completed, you have 3 lexicons: a reference lexicon, containing all the controlled and meaningful vocabulary, an "antilexicon" containing rejected terms of no interest within the work context, and finally a lexicon of "synonyms".These lexicons can be reused later for working with other files, sparing time. You will just have to enrich them with newly validated terms.
  There is no parameter to indicate. After this extraction and indexation through Calliope Extract, every plain text document points towards a list of validated terms describing their content.

Calliope Process operation

Calliope Process is actually the heart of Calliope. Its outputs are displayed by the Calliope Viewer module.  Calliope Process leads two essential actions:

1. an automated analysis of the whole content of a file of documents, which represent a certain domain or collection of documents, at a given period of time,
2. an analysis of the dynamics of the evolution of above contents, above several files of documents, periodically collated.

• Phase 1, and for each period: Calliope Process runs classification algorithms to build up networks of terms, which are thematically homogeneous. The algorithms are based upon the coword analysis, which state that two words regularly co-occurring within a corpus are linked by the same topic. The main underlying topics are then automatically extracted by Calliope Process. The resulting cartography is displayed in Calliope Viewer.
  Two computing schemes are possible: a fully automatic mode, asking for no parameter, where Calliope Process optimizes itself through iterative trials, and an "expert" mode letting the choice of parameters to the User. One may compare these modes to picture taking:
  1. you click on the button and the picture is taken in optimal conditions of sharpness, contrats, luminosity;
  2. you wish to play with aperture and speed, to favour certain aspects of the picture, to higlight a certain zone, to give more focus on another one…
• Phase 2, and from 2 or more periods, Calliope Process automatically quantifies the relative importance of terms within a period (a corpus) by computing their "attraction power", i.e. their "participation ratio" in a network of terms build during phase 1. If this attraction power increases over periods, the term is supposed to gain importance and becomes "emerging"; on the contrary, it becomes "aging" or "declining", and if it remains globally constant, it is said "stable". Calliope Viewer displays the resulting trend curves of different terms. It allows to sort them into categories: emerging, stable, declining.
  No user parameter is required..