XQuery/Filtering Words

Motivation Sometimes you have a text body and you want to filter out words that are on a given list, often called a stoplist. Screen Image center|frame|Screen Image Sample Program xquery version "1.0"; (: Test to see if a word is in a list :) declare namespace exist = "http://exist.sourceforge.net/NS/exist"; declare option exist:serialize "method=xhtml media-type=text/html indent=yes omit-xml-declaration=yes"; (: A list of words :) let $stopwords := <words> <word>a</word> <word>and</word> <word>in</word> <word>the</word> <word>or</word> <word>over</word> </words> let $input-text := 'a quick brown fox jumps over the lazy dog' return <html> <head> <title>Test of is a word on a list</title> </head> <body> <h1> Test of is a word on a list</h1> <h2>WordList</h2> <table border="1"> <thead> <tr> <th>StopWord</th> </tr> </thead> <tbody>{ for $word in $stopwords/word return <tr> <td align="center">{$word}</td> </tr> }</tbody> </table> <h2>Sample Input Text</h2> <p>Input Text: <div style="border:1px solid black">{$input-text}</div></p> <table border="1"> <thead> <tr> <th>Word</th> <th>On List</th> </tr> </thead> <tbody>{ for $word in tokenize($input-text, '\s+') return <tr> <td>{$word}</td> <td>{ if ($stopwords/word = $word) then(<span style="color:green;">true</span>) else(<span style="color:red;">false</span>) }</td> </tr> }</tbody> </table> </body> </html> Discussion The input string is split into words using the tokenize function which accepts two parameters, the string to be parsed and a separator expressed as a regular expression. Here words are separated by one or more spaces. The result is a sequence of words. This program uses XPath generalized equality to compare the sequence $stopwords/word with the sequence (of one item) $word. This is true if the two sequences have items in common, that is if the stoplist contains the word. Alternative coding You can also use a quantified expression to perform a stopword lookup using the some...satisfies – see XQuery/Quantified Expressions expression such as: some $word in $stopwords satisfies ($word = $thisword) There are other alternatives; the stop words as a sequence of strings, or a long string and use contains() or a element in the database. There are however significant differences in performance. There is a set of tests which show the differences in a number of alternatives. Unit Tests What these tests reveal is that, on the eXist db platform, both the suggested implementations are far from optimal. Testing against a sequence of strings takes about a fifth of the time to compare with elements. Generalised equality is equally superior to the use of a qualified expression. Recommended Practice It would appear that the preferable approach is: let $stopwords := ("a","and","in","the","or","over") let $input-string := 'a quick brown fox jumps over the lazy dog' let $input-words := tokenize($input-string, '\s+') return for $word in $input-words return $stopwords = $word If the stop words are held as an element, it is better to convert to a sequence of atoms first: let $stopwords := <words> <word>a</word> <word>and</word> <word>in</word> <word>the</word> <word>or</word> <word>over</word> </words> let $stopwordsx := $stopwords/word/string(.) let $input-string := 'a quick brown fox jumps over the lazy dog' let $input-words := tokenize($input-string, '\s+') return for $word in $input-words return $stopwordsx = $word Note that referencing the stop list in the database slightly improved performance.