Recoll indexing can be performed along two main modes:

Recoll supports defining multiple indexes, each defined by its own configuration directory. A configuration directory contains several files which describe what should be indexed and how.

When recoll or recollindex is first executed, it creates a default configuration directory. This configuration is the one used for indexing and querying when no specific configuration is specified. It is located in $HOME/.recoll/ for Unix-like systems and %LOCALAPPDATA%\Recoll on Windows (typically C:\Users\[me]\Appdata\Local\Recoll).

All configuration parameters have defaults, defined in system-wide files. Without further customisation, the default configuration will process your complete home directory, with a reasonable set of defaults. It can be adjusted to process a different area of the file system, select files in different ways, and many other things.

In some cases, it may be useful to create additional configuration directories, for example, to separate personal and shared indexes, or to take advantage of the organization of your data to improve search precision.

In order to do this, you would create an empty directory in a location of your choice, and then instruct recoll or recollindex to use it by setting either a command line option (-c /some/directory), or an environment variable (RECOLL_CONFDIR=/some/directory). Any modification performed by the commands (e.g. configuration customisation or searches by recoll or index creation by recollindex) would then apply to the new directory and not to the default one.

Once multiple indexes are created, you can use each of them separately by setting the -c option or the RECOLL_CONFDIR environment variable when starting a command, to select the desired index.

It is also possible to instruct one configuration to query one or several other indexes in addition to its own, by using the External index function in the recoll GUI, or some other functions in the command line and programming tools.

A plausible usage scenario for the multiple index feature would be for a system administrator to set up a central index for shared data, that you choose to search or not in addition to your personal data. Of course, there are other possibilities. for example, there are many cases where you know the subset of files that should be searched, and where narrowing the search can improve the results. You can achieve approximately the same effect with the directory filter in advanced search, but multiple indexes may have better performance and may be worth the trouble in some cases.

A more advanced use case would be to use multiple index to improve indexing performance, by updating several indexes in parallel (using multiple CPU cores and disks, or possibly several machines), and then merging them, or querying them in parallel.

See the section about configuring multiple indexes for more detail

Recoll knows about quite a few different document types. The parameters for document types recognition and processing are set in configuration files.

Most file types, like HTML or word processing files, only hold one document. Some file types, like email folders or zip archives, can hold many individually indexed documents, which may themselves be compound ones. Such hierarchies can go quite deep, and Recoll can process, for example, a LibreOffice document stored as an attachment to an email message inside an email folder archived in a zip file...

recollindex processes plain text, HTML, OpenDocument (Open/LibreOffice), email formats, and a few others internally.

Other file types (ie: postscript, pdf, ms-word, rtf ...) need external applications for preprocessing. The list is in the installation section. After every indexing operation, Recoll updates a list of commands that would be needed for indexing existing files types. This list can be displayed by selecting the menu option File → Show Missing Helpers in the recoll GUI. It is stored in the missing text file inside the configuration directory.

After installing a missing handler, you may need to tell recollindex to retry the failed files, by adding option -k to the command line, or by using the GUI File → Special indexing menu. This is because recollindex, in its default operation mode, will not retry files which caused an error during an earlier pass. In special cases, it may be useful to reset the data for a category of files before indexing. See the recollindex manual page. If your index is not too big, it may be simpler to just reset it.

By default, Recoll will try to index any file type that it has a way to read. This is sometimes not desirable, and there are ways to either exclude some types, or on the contrary define a positive list of types to be indexed. In the latter case, any type not in the list will be ignored.

Excluding files by name can be done by adding wildcard name patterns to the skippedNames list, which can be done from the GUI Index configuration menu. Excluding by type can be done by setting the excludedmimetypes list in the configuration file (1.20 and later). This can be redefined for subdirectories.

You can also define an exclusive list of MIME types to be indexed (no others will be indexed), by settting the indexedmimetypes configuration variable. Example:

        indexedmimetypes = text/html application/pdf
      

It is possible to redefine this parameter for subdirectories. Example:

      [/path/to/my/dir]
      indexedmimetypes = application/pdf
    

(When using sections like this, don't forget that they remain in effect until the end of the file or another section indicator).

excludedmimetypes or indexedmimetypes, can be set either by editing the configuration file (recoll.conf) for the index, or by using the GUI index configuration tool.

Indexing may fail for some documents, for a number of reasons: a helper program may be missing, the document may be corrupt, we may fail to uncompress a file because no file system space is available, etc.

The Recoll indexer in versions 1.21 and later does not retry failed files by default, because some indexing failures can be quite costly (for example failing to uncompress a big file because of insufficient disk space). Retrying will only occur if an explicit option (-k) is set on the recollindex command line, or if a script executed when recollindex starts up says so. The script is defined by a configuration variable (checkneedretryindexscript), and makes a rather lame attempt at deciding if a helper command may have been installed, by checking if any of the common bin directories have changed.

In the rare case where the index becomes corrupted (which can signal itself by weird search results or crashes), the index files need to be erased before restarting a clean indexing pass. Just delete the xapiandb directory (see next section), or, alternatively, start the next recollindex with the -z option, which will reset the database before indexing. The difference between the two methods is that the second will not change the current index format, which may be undesirable if a newer format is supported by the Xapian version.

Xapian versions usually support several formats for index storage. A given major Xapian version will have a current format, used to create new indexes, and will also support the format from the previous major version.

Xapian will not convert automatically an existing index from the older format to the newer one. If you want to upgrade to the new format, or if a very old index needs to be converted because its format is not supported any more, you will have to explicitly delete the old index (typically ~/.recoll/xapiandb), then run a normal indexing command. Using recollindex option -z would not work in this situation.

The Recoll index does not hold complete copies of the indexed documents (it almost does after version 1.24). But it does hold enough data to allow for an almost complete reconstruction. If confidential data is indexed, access to the database directory should be restricted.

Recoll will create the configuration directory with a mode of 0700 (access by owner only). As the index data directory is by default a sub-directory of the configuration directory, this should result in appropriate protection.

If you use another setup, you should think of the kind of protection you need for your index, set the directory and files access modes appropriately, and also maybe adjust the umask used during index updates.

This only needs concern you if your index is going to be bigger than around 5 GBytes. Beyond 10 GBytes, it becomes a serious issue. Most people have much smaller indexes. For reference, 5 GBytes would be around 2000 bibles, a lot of text. If you have a huge text dataset (remember: images don't count, the text content of PDFs is typically less than 5% of the file size), read on.

The amount of writing performed by Xapian during index creation is not linear with the index size (it is somewhere between linear and quadratic). For big indexes this becomes a performance issue, and may even be an SSD disk wear issue.

The problem can be mitigated by observing the following rules:

Multiple Recoll indexes can be created by using several configuration directories which are typically set to index different areas of the file system.

A specific index can be selected by setting the RECOLL_CONFDIR environment variable or giving the -c option to recoll and recollindex.

The recollindex program, used for creating or updating indexes, always works on a single index. The different configurations are entirely independant (no parameters are ever shared between configurations when indexing).

All the search interfaces (recoll, recollq, the Python API, etc.) operate with a main configuration, from which both configuration and index data are used, and can also query data from multiple additional indexes. Only the index data from the latter is used, their configuration parameters are ignored. This implies that some parameters should be consistent among index configurations which are to be used together.

When searching, the current main index (defined by RECOLL_CONFDIR or -c) is always active. If this is undesirable, you can set up your base configuration to index an empty directory.

Index configuration parameters can be set either by using a text editor on the files, or, for most parameters, by using the recoll index configuration GUI. In the latter case, the configuration directory for which parameters are modified is the one which was selected by RECOLL_CONFDIR or the -c parameter, and there is no way to switch configurations within the GUI.

See the configuration section for a detailed description of the parameters

Some configuration parameters must be consistent among a set of multiple indexes used together for searches. Most importantly, all indexes to be queried concurrently must have the same option concerning character case and diacritics stripping, but there are other constraints. Most of the relevant parameters affect the term generation.

Using multiple configurations implies a small level of command line or file manager usage. The user must explicitly create additional configuration directories, the GUI will not do it. This is to avoid mistakenly creating additional directories when an argument is mistyped. Also, the GUI or the indexer must be launched with a specific option or environment to work on the right configuration.

mkdir /path/to/my/new/config

recoll -c /path/to/my/new/config

someEditor /path/to/my/new/config/recoll.conf

recollindex -c /path/to/my/new/config

recoll -c /path/to/my/new/config

As of Recoll version 1.18 you have a choice of building an index with terms stripped of character case and diacritics, or one with raw terms. For a source term of Résumé, the former will store resume, the latter Résumé.

Each type of index allows performing searches insensitive to case and diacritics: with a raw index, the user entry will be expanded to match all case and diacritics variations present in the index. With a stripped index, the search term will be stripped before searching.

A raw index allows using case and diacritics to discriminate between terms, e.g., returning different results when searching for US and us or resume and résumé. Read the section about search case and diacritics sensitivity for more details.

The type of index to be created is controlled by the indexStripChars configuration variable which can only be changed by editing the configuration file. Any change implies an index reset (not automated by Recoll), and all indexes in a search must be set in the same way (again, not checked by Recoll).

Recoll creates a stripped index by default if indexStripChars is not set.

As a cost for added capability, a raw index will be slightly bigger than a stripped one (around 10%). Also, searches will be more complex, so probably slightly slower, and the feature is relatively little used, so that a certain amount of weirdness cannot be excluded.

One of the most adverse consequence of using a raw index is that some phrase and proximity searches may become impossible: because each term needs to be expanded, and all combinations searched for, the multiplicative expansion may become unmanageable.

The Recoll indexing process recollindex can use multiple threads to speed up indexing on multiprocessor systems. The work done to index files is divided in several stages and some of the stages can be executed by multiple threads. The stages are:

You can also read a longer document about the transformation of Recoll indexing to multithreading.

The threads configuration is controlled by two configuration file parameters.

The following example would use three queues (of depth 2), and 4 threads for converting source documents, 2 for processing their text, and one to update the index. This was tested to be the best configuration on the test system (quadri-processor with multiple disks).

          thrQSizes = 2 2 2
          thrTCounts =  4 2 1
        

The following example would use a single queue, and the complete processing for each document would be performed by a single thread (several documents will still be processed in parallel in most cases). The threads will use mutual exclusion when entering the index update stage. In practise the performance would be close to the precedent case in general, but worse in certain cases (e.g. a Zip archive would be performed purely sequentially), so the previous approach is preferred. YMMV... The 2 last values for thrTCounts are ignored.

          thrQSizes = 2 -1 -1
          thrTCounts =  6 1 1
        

The following example would disable multithreading. Indexing will be performed by a single thread.

          thrQSizes = -1 -1 -1
        

Most parameters for a given index configuration can be set from a recoll GUI running on this configuration (either as default, or by setting RECOLL_CONFDIR or the -c option.)

The interface is started from the Preferences → Index Configuration menu entry. It is divided in four tabs, Global parameters, Local parameters, Web history (which is explained in the next section) and Search parameters.

The Global parameters tab allows setting global variables, like the lists of top directories, skipped paths, or stemming languages.

The Local parameters tab allows setting variables that can be redefined for subdirectories. This second tab has an initially empty list of customisation directories, to which you can add. The variables are then set for the currently selected directory (or at the top level if the empty line is selected).

The Search parameters section defines parameters which are used at query time, but are global to an index and affect all search tools, not only the GUI.

The meaning for most entries in the interface is self-evident and documented by a ToolTip popup on the text label. For more detail, you will need to refer to the configuration section of this guide.

The configuration tool normally respects the comments and most of the formatting inside the configuration file, so that it is quite possible to use it on hand-edited files, which you might nevertheless want to backup first...

The rclpdf.py script in Recoll version 1.23.2 and later can extract XMP metadata fields by executing the pdfinfo command (usually found with poppler-utils). This is controlled by the pdfextrameta configuration variable, which specifies which tags to extract and, possibly, how to rename them.

The pdfextrametafix variable can be used to designate a file with Python code to edit the metadata fields (available for Recoll 1.23.3 and later. 1.23.2 has equivalent code inside the handler script). Example:

import sys
        import re

        class MetaFixer(object):
            def __init__(self):
                pass

            def metafix(self, nm, txt):
                if nm == 'bibtex:pages':
                    txt = re.sub(r'--', '-', txt)
                elif nm == 'someothername':
                    # do something else
                    pass
                elif nm == 'stillanother':
                    # etc.
                    pass
        
                return txt
            def wrapup(self, metaheaders):
                pass
        

If the 'metafix()' method is defined, it is called for each metadata field. A new MetaFixer object is created for each PDF document (so the object can keep state for, for example, eliminating duplicate values). If the 'wrapup()' method is defined, it is called at the end of XMP fields processing with the whole metadata as parameter, as an array of '(nm, val)' pairs, allowing an alternate approach for editing or adding/deleting fields.

If pdftk is installed, and if the the pdfattach configuration variable is set, the PDF input handler will try to extract PDF attachements for indexing as sub-documents of the PDF file. This is disabled by default, because it slows down PDF indexing a bit even if not one attachment is ever found (PDF attachments are uncommon in my experience).

The initial default search mode is Query language. Without special directives, this will look for documents containing all of the search terms (the ones with more terms will get better scores), just like the All terms mode. Any term will search for documents where at least one of the terms appear. File name will exclusively look for file names, not contents

All search modes allow terms to be expanded with wildcards characters (*, ?, []). See the section about wildcards for more details.

In all modes except File name, you can search for exact phrases (adjacent words in a given order) by enclosing the input inside double quotes. Ex: "virtual reality".

The Query Language features are described in a separate section.

When using a stripped index (the default), character case has no influence on search, except that you can disable stem expansion for any term by capitalizing it. Ie: a search for floor will also normally look for flooring, floored, etc., but a search for Floor will only look for floor, in any character case. Stemming can also be disabled globally in the preferences. When using a raw index, the rules are a bit more complicated.

Recoll remembers the last few searches that you performed. You can directly access the search history by clicking the clock button on the right of the search entry, while the latter is empty. Otherwise, the history is used for entry completion (see next). Only the search texts are remembered, not the mode (all/any/file name).

While text is entered in the search area, recoll will display possible completions, filtered from the history and the index search terms. This can be disabled with a GUI Preferences option.

Double-clicking on a word in the result list or a preview window will insert it into the simple search entry field.

You can cut and paste any text into an All terms or Any term search field, punctuation, newlines and all - except for wildcard characters (single ? characters are ok). Recoll will process it and produce a meaningful search. This is what most differentiates this mode from the Query Language mode, where you have to care about the syntax.

You can use the Tools → Advanced search dialog for more complex searches.

The File name search mode will specifically look for file names. The point of having a separate file name search is that wild card expansion can be performed more efficiently on a small subset of the index (allowing wild cards on the left of terms without excessive cost). Things to know:

After starting a search, a list of results will instantly be displayed in the main window.

By default, the document list is presented in order of relevance (how well the system estimates that the document matches the query). You can sort the result by ascending or descending date by using the vertical arrows in the toolbar.

Clicking the Preview link for an entry will open an internal preview window for the document. Further Preview clicks for the same search will open tabs in the existing preview window. You can use Shift+Click to force the creation of another preview window, which may be useful to view the documents side by side. (You can also browse successive results in a single preview window by typing Shift+ArrowUp/Down in the window).

Clicking the Open link will start an external viewer for the document. By default, Recoll lets the desktop choose the appropriate application for most document types. See further for customizing the applications.

You can click on the Query details link at the top of the results page to see the query actually performed, after stem expansion and other processing.

Double-clicking on any word inside the result list or a preview window will insert it into the simple search text.

The result list is divided into pages (the size of which you can change in the preferences). Use the arrow buttons in the toolbar or the links at the bottom of the page to browse the results.

The Preview and Open edit links may not be present for all entries, meaning that Recoll has no configured way to preview a given file type (which was indexed by name only), or no configured external editor for the file type. This can sometimes be adjusted simply by tweaking the mimemap and mimeview configuration files (the latter can be modified with the user preferences dialog).

The format of the result list entries is entirely configurable by using the preference dialog to edit an HTML fragment.

As an alternative to the result list, the results can also be displayed in spreadsheet-like fashion. You can switch to this presentation by clicking the table-like icon in the toolbar (this is a toggle, click again to restore the list).

Clicking on the column headers will allow sorting by the values in the column. You can click again to invert the order, and use the header right-click menu to reset sorting to the default relevance order (you can also use the sort-by-date arrows to do this).

Both the list and the table display the same underlying results. The sort order set from the table is still active if you switch back to the list mode. You can click twice on a date sort arrow to reset it from there.

The header right-click menu allows adding or deleting columns. The columns can be resized, and their order can be changed (by dragging). All the changes are recorded when you quit recoll

Hovering over a table row will update the detail area at the bottom of the window with the corresponding values. You can click the row to freeze the display. The bottom area is equivalent to a result list paragraph, with links for starting a preview or a native application, and an equivalent right-click menu. Typing Esc (the Escape key) will unfreeze the display.

Apart from the Open and Open With operations, which allow starting an application on a result document (or a temporary copy), based on its MIME type, it is also possible to run arbitrary commands on results which are top-level files, using the Run Script entry in the results pop-up menu.

The commands which will appear in the Run Script submenu must be defined by .desktop files inside the scripts subdirectory of the current configuration directory.

Here follows an example of a .desktop file, which could be named for example, ~/.recoll/scripts/myscript.desktop (the exact file name inside the directory is irrelevant):

          [Desktop Entry]
          Type=Application
          Name=MyFirstScript
          Exec=/home/me/bin/tryscript %F
          MimeType=*/*
        

The Name attribute defines the label which will appear inside the Run Script menu. The Exec attribute defines the program to be run, which does not need to actually be a script, of course. The MimeType attribute is not used, but needs to exist.

The commands defined this way can also be used from links inside the result paragraph.

As an example, it might make sense to write a script which would move the document to the trash and purge it from the Recoll index.

The default format for the result list entries and the detail area of the result table display an icon for each result document. The icon is either a generic one determined from the MIME type, or a thumbnail of the document appearance. Thumbnails are only displayed if found in the standard freedesktop location, where they would typically have been created by a file manager.

Recoll has no capability to create thumbnails. A relatively simple trick is to use the Open parent document/folder entry in the result list popup menu. This should open a file manager window on the containing directory, which should in turn create the thumbnails (depending on your settings). Restarting the search should then display the thumbnails.

There are also some pointers about thumbnail generation in the Recoll FAQ.

The preview window opens when you first click a Preview link inside the result list.

Subsequent preview requests for a given search open new tabs in the existing window (except if you hold the Shift key while clicking which will open a new window for side by side viewing).

Starting another search and requesting a preview will create a new preview window. The old one stays open until you close it.

You can close a preview tab by typing Ctrl-W (Ctrl + W) in the window. Closing the last tab, or using the window manager button in the top of the frame will also close the window.

You can display successive or previous documents from the result list inside a preview tab by typing Shift+Down or Shift+Up (Down and Up are the arrow keys).

A right-click menu in the text area allows switching between displaying the main text or the contents of fields associated to the document (ie: author, abtract, etc.). This is especially useful in cases where the term match did not occur in the main text but in one of the fields. In the case of images, you can switch between three displays: the image itself, the image metadata as extracted by exiftool and the fields, which is the metadata stored in the index.

You can print the current preview window contents by typing Ctrl-P (Ctrl + P) in the window text.

Selecting the Tools → Query Fragments menu entry will open a window with radio- and check-buttons which can be used to activate query language fragments for filtering the current query. This can be useful if you have frequent reusable selectors, for example, filtering on alternate directories, or searching just one category of files, not covered by the standard category selectors.

The contents of the window are entirely customizable, and defined by the contents of the fragbuts.xml file inside the configuration directory. The sample file distributed with Recoll (which you should be able to find under /usr/share/recoll/examples/fragbuts.xml), contains an example which filters the results from the WEB history.

Here follows an example:

<?xml version="1.0" encoding="UTF-8"?>
<fragbuts version="1.0">

  <radiobuttons>
    <!-- Actually useful: toggle WEB queue results inclusion -->
    <fragbut>
      <label>Include Web Results</label>
      <frag></frag>
    </fragbut>

    <fragbut>
      <label>Exclude Web Results</label>
      <frag>-rclbes:BGL</frag>
    </fragbut>

    <fragbut>
      <label>Only Web Results</label>
      <frag>rclbes:BGL</frag>
    </fragbut>

  </radiobuttons>

  <buttons>

    <fragbut>
      <label>Example: Year 2010</label>
      <frag>date:2010-01-01/2010-12-31</frag>
    </fragbut>

    <fragbut>
      <label>Example: c++ files</label>
      <frag>ext:cpp OR ext:cxx</frag>
    </fragbut>

    <fragbut>
      <label>Example: My Great Directory</label>
      <frag>dir:/my/great/directory</frag>
    </fragbut>

  </buttons>

</fragbuts>
        

Each radiobuttons or buttons section defines a line of checkbuttons or radiobuttons inside the window. Any number of buttons can be selected, but the radiobuttons in a line are exclusive.

Each fragbut section defines the label for a button, and the Query Language fragment which will be added (as an AND filter) before performing the query if the button is active.

The advanced search dialog helps you build more complex queries without memorizing the search language constructs. It can be opened through the Tools menu or through the main toolbar.

Recoll keeps a history of searches. See Advanced search history.

The dialog has two tabs:

Click on the Start Search button in the advanced search dialog, or type Enter in any text field to start the search. The button in the main window always performs a simple search.

Click on the Show query details link at the top of the result page to see the query expansion.

Recoll automatically manages the expansion of search terms to their derivatives (ie: plural/singular, verb inflections). But there are other cases where the exact search term is not known. For example, you may not remember the exact spelling, or only know the beginning of the name.

The search will only propose replacement terms with spelling variations when no matching document were found. In some cases, both proper spellings and mispellings are present in the index, and it may be interesting to look for them explicitly.

The term explorer tool (started from the toolbar icon or from the Term explorer entry of the Tools menu) can be used to search the full index terms list. It has three modes of operations:

Note that in cases where Recoll does not know the beginning of the string to search for (ie a wildcard expression like *coll), the expansion can take quite a long time because the full index term list will have to be processed. The expansion is currently limited at 10000 results for wildcards and regular expressions. It is possible to change the limit in the configuration file.

Double-clicking on a term in the result list will insert it into the simple search entry field. You can also cut/paste between the result list and any entry field (the end of lines will be taken care of).

See the section describing the use of multiple indexes for generalities. Only the aspects concerning the recoll GUI are described here.

A recoll program instance is always associated with a specific index, which is the one to be updated when requested from the File menu, but it can use any number of Recoll indexes for searching. The external indexes can be selected through the external indexes tab in the preferences dialog.

Index selection is performed in two phases. A set of all usable indexes must first be defined, and then the subset of indexes to be used for searching. These parameters are retained across program executions (there are kept separately for each Recoll configuration). The set of all indexes is usually quite stable, while the active ones might typically be adjusted quite frequently.

The main index (defined by RECOLL_CONFDIR) is always active. If this is undesirable, you can set up your base configuration to index an empty directory.

When adding a new index to the set, you can select either a Recoll configuration directory, or directly a Xapian index directory. In the first case, the Xapian index directory will be obtained from the selected configuration.

As building the set of all indexes can be a little tedious when done through the user interface, you can use the RECOLL_EXTRA_DBS environment variable to provide an initial set. This might typically be set up by a system administrator so that every user does not have to do it. The variable should define a colon-separated list of index directories, ie:

export RECOLL_EXTRA_DBS=/some/place/xapiandb:/some/other/db

Another environment variable, RECOLL_ACTIVE_EXTRA_DBS allows adding to the active list of indexes. This variable was suggested and implemented by a Recoll user. It is mostly useful if you use scripts to mount external volumes with Recoll indexes. By using RECOLL_EXTRA_DBS and RECOLL_ACTIVE_EXTRA_DBS, you can add and activate the index for the mounted volume when starting recoll. Unreachable indexes will automatically be deactivated when starting up.

Documents that you actually view (with the internal preview or an external tool) are entered into the document history, which is remembered.

You can display the history list by using the Tools/Doc History menu entry.

You can erase the document history by using the Erase document history entry in the File menu.

The documents in a result list are normally sorted in order of relevance. It is possible to specify a different sort order, either by using the vertical arrows in the GUI toolbox to sort by date, or switching to the result table display and clicking on any header. The sort order chosen inside the result table remains active if you switch back to the result list, until you click one of the vertical arrows, until both are unchecked (you are back to sort by relevance).

Sort parameters are remembered between program invocations, but result sorting is normally always inactive when the program starts. It is possible to keep the sorting activation state between program invocations by checking the Remember sort activation state option in the preferences.

It is also possible to hide duplicate entries inside the result list (documents with the exact same contents as the displayed one). The test of identity is based on an MD5 hash of the document container, not only of the text contents (so that ie, a text document with an image added will not be a duplicate of the text only). Duplicates hiding is controlled by an entry in the GUI configuration dialog, and is off by default.

When a result document does have undisplayed duplicates, a Dups link will be shown with the result list entry. Clicking the link will display the paths (URLs + ipaths) for the duplicate entries.

Both simple and advanced query dialogs save recent history, but the amount is limited: old queries will eventually be forgotten. Also, important queries may be difficult to find among others. This is why both types of queries can also be explicitly saved to files, from the GUI menus: File → Save last query / Load last query

The default location for saved queries is a subdirectory of the current configuration directory, but saved queries are ordinary files and can be written or moved anywhere.

Some of the saved query parameters are part of the preferences (e.g. autophrase or the active external indexes), and may differ when the query is loaded from the time it was saved. In this case, Recoll will warn of the differences, but will not change the user preferences.

You can customize some aspects of the search interface by using the GUI configuration entry in the Preferences menu.

There are several tabs in the dialog, dealing with the interface itself, the parameters used for searching and returning results, and what indexes are searched.

User interface parameters: 

Result list parameters: 

Search parameters: 

External indexes: This panel will let you browse for additional indexes that you may want to search. External indexes are designated by their database directory (ie: /home/someothergui/.recoll/xapiandb, /usr/local/recollglobal/xapiandb).

Once entered, the indexes will appear in the External indexes list, and you can chose which ones you want to use at any moment by checking or unchecking their entries.

Your main database (the one the current configuration indexes to), is always implicitly active. If this is not desirable, you can set up your configuration so that it indexes, for example, an empty directory. An alternative indexer may also need to implement a way of purging the index from stale data,

<a href="F%N">%P</a>

            "<table class=\"respar\">\n"
            "<tr>\n"
            "<td><a href='%U'><img src='%I' width='64'></a></td>\n"
            "<td>%L &nbsp;<i>%S</i> &nbsp;&nbsp;<b>%T</b><br>\n"
            "<span style='white-space:nowrap'><i>%M</i>&nbsp;%D</span>&nbsp;&nbsp;&nbsp; <i>%U</i>&nbsp;%i<br>\n"
            "%A %K</td>\n"
            "</tr></table>\n"
            

            <u><b><a href="P%N">%T</a></b></u><br>
            %A<font color=#008000>%U - %S</font> - %L
            

            <img src="%I" align="left">%L <font color="#900000">%R</font>
            &nbsp;&nbsp;<b>%T&</b><br>%S&nbsp;
            <font color="#808080"><i>%U</i></font>
            <table bgcolor="#e0e0e0">
            <tr><td><div>%A</div></td></tr>
            </table>%K
            

The Recoll KIO slave allows performing a Recoll search by entering an appropriate URL in a KDE open dialog, or with an HTML-based interface displayed in Konqueror.

The HTML-based interface is similar to the Qt-based interface, but slightly less powerful for now. Its advantage is that you can perform your search while staying fully within the KDE framework: drag and drop from the result list works normally and you have your normal choice of applications for opening files.

The alternative interface uses a directory view of search results. Due to limitations in the current KIO slave interface, it is currently not obviously useful (to me).

The interface is described in more detail inside a help file which you can access by entering recoll:/ inside the konqueror URL line (this works only if the recoll KIO slave has been previously installed).

The instructions for building this module are located in the source tree. See: kde/kio/recoll/00README.txt. Some Linux distributions do package the kio-recoll module, so check before diving into the build process, maybe it's already out there ready for one-click installation.

As a sample application, the Recoll KIO slave could allow preparing a set of HTML documents (for example a manual) so that they become their own search interface inside konqueror.

This can be done by either explicitly inserting <a href="recoll://..."> links around some document areas, or automatically by adding a very small javascript program to the documents, like the following example, which would initiate a search by double-clicking any term:

<script language="JavaScript">
        function recollsearch() {
        var t = document.getSelection();
        window.location.href = 'recoll://search/query?qtp=a&p=0&q=' +
        encodeURIComponent(t);
        }
        </script>
        ....
        <body ondblclick="recollsearch()">

        

Recoll 1.24 and later support range clauses on fields which have been configured to support it. No default field uses them currently, so this paragraph is only interesting if you modified the fields configuration and possibly use a custom input handler.

A range clause looks like one of the following:

myfield:small..big
myfield:small..
myfield:..big
        

The nature of the clause is indicated by the two dots .., and the effect is to filter the results for which the myfield value is in the possibly open-ended interval.

See the section about the fields configuration file for the details of configuring a field for range searches (list them in the [values] section).

Some characters are recognized as search modifiers when found immediately after the closing double quote of a phrase, as in "some term"modifierchars. The actual "phrase" can be a single term of course. Supported modifiers:

All words entered in Recoll search fields will be processed for wildcard expansion before the request is finally executed.

The wildcard characters are:

You should be aware of a few things when using wildcards.

Two characters are used to specify that a search hit should occur at the beginning or at the end of the text. ^ at the beginning of a term or phrase constrains the search to happen at the start, $ at the end force it to happen at the end.

As this function is implemented as a phrase search it is possible to specify a maximum distance at which the hit should occur, either through the controls of the advanced search panel, or using the query language, for example, as in:

"^someterm"o10

which would force someterm to be found within 10 terms of the start of the text. This can be combined with a field search as in somefield:"^someterm"o10 or somefield:someterm$.

This feature can also be used with an actual phrase search, but in this case, the distance applies to the whole phrase and anchor, so that, for example, bla bla my unexpected term at the beginning of the text would be a match for "^my term"o5.

Anchored searches can be very useful for searches inside somewhat structured documents like scientific articles, in case explicit metadata has not been supplied (a most frequent case), for example for looking for matches inside the abstract or the list of authors (which occur at the top of the document).

It is surprisingly convenient to be able to show or hide the Recoll GUI with a single keystroke. Recoll comes with a small Python script, based on the libwnck window manager interface library, which will allow you to do just this. The detailed instructions are on this wiki page.

This is probably obsolete now. Anyway:

The Recoll source tree contains the source code to the recoll_applet, a small application derived from the find_applet. This can be used to add a small Recoll launcher to the KDE panel.

The applet is not automatically built with the main Recoll programs, nor is it included with the main source distribution (because the KDE build boilerplate makes it relatively big). You can download its source from the recoll.org download page. Use the omnipotent configure;make;make install incantation to build and install.

You can then add the applet to the panel by right-clicking the panel and choosing the Add applet entry.

The recoll_applet has a small text window where you can type a Recoll query (in query language form), and an icon which can be used to restrict the search to certain types of files. It is quite primitive, and launches a new recoll GUI instance every time (even if it is already running). You may find it useful anyway.

Recoll simple handlers are usually shell-scripts, but this is in no way necessary. Extracting the text from the native format is the difficult part. Outputting the format expected by Recoll is trivial. Happily enough, most document formats have translators or text extractors which can be called from the handler. In some cases the output of the translating program is completely appropriate, and no intermediate shell-script is needed.

Input handlers are called with a single argument which is the source file name. They should output the result to stdout.

When writing a handler, you should decide if it will output plain text or HTML. Plain text is simpler, but you will not be able to add metadata or vary the output character encoding (this will be defined in a configuration file). Additionally, some formatting may be easier to preserve when previewing HTML. Actually the deciding factor is metadata: Recoll has a way to extract metadata from the HTML header and use it for field searches..

The RECOLL_FILTER_FORPREVIEW environment variable (values yes, no) tells the handler if the operation is for indexing or previewing. Some handlers use this to output a slightly different format, for example stripping uninteresting repeated keywords (ie: Subject: for email) when indexing. This is not essential.

You should look at one of the simple handlers, for example rclps for a starting point.

Don't forget to make your handler executable before testing !

If you can program and want to write an execm handler, it should not be too difficult to make sense of one of the existing handlers.

The existing handlers differ in the amount of helper code which they are using:

There is a sample trivial handler based on rclexecm.py, with many comments, not actually used by Recoll. It would index a text file as one document per line. Look for rcltxtlines.py in the src/filters directory in the online Recoll Git repository (the sample not in the distributed release at the moment).

You can also have a look at the slightly more complex rclzip which uses Zip file paths as identifiers (ipath).

execm handlers sometimes need to make a choice for the nature of the ipath elements that they use in communication with the indexer. Here are a few guidelines:

In any case, the main goal is that it should be easy for the handler to extract the target document, given the file name and the ipath element.

execm handlers will also produce a document with a null ipath element. Depending on the type of document, this may have some associated data (e.g. the body of an email message), or none (typical for an archive file). If it is empty, this document will be useful anyway for some operations, as the parent of the actual data documents.

There are two elements that link a file to the handler which should process it: the association of file to MIME type and the association of a MIME type with a handler.

The association of files to MIME types is mostly based on name suffixes. The types are defined inside the mimemap file. Example:

            .doc = application/msword
          

If no suffix association is found for the file name, Recoll will try to execute a system command (typically file -i or xdg-mime) to determine a MIME type.

The second element is the association of MIME types to handlers in the mimeconf file. A sample will probably be better than a long explanation:

          [index]
          application/msword = exec antiword -t -i 1 -m UTF-8;\
          mimetype = text/plain ; charset=utf-8

          application/ogg = exec rclogg

          text/rtf = exec unrtf --nopict --html; charset=iso-8859-1; mimetype=text/html

          application/x-chm = execm rclchm
        

The fragment specifies that:

Both the simple and persistent input handlers can return any MIME type to Recoll, which will further process the data according to the MIME configuration.

Most input filters filters produce either text/plain or text/html data. There are exceptions, for example, filters which process archive file (zip, tar, etc.) will usually return the documents as they are found, without processing them further.

There is nothing to say about text/plain output, except that its character encoding should be consistent with what is specified in the mimeconf file.

For filters producing HTML, the output could be very minimal like the following example:

          <html>
            <head>
              <meta http-equiv="Content-Type" content="text/html;charset=UTF-8"/>
            </head>
            <body>
              Some text content
            </body>
          </html>
          

You should take care to escape some characters inside the text by transforming them into appropriate entities. At the very minimum, "&" should be transformed into "&", "<" should be transformed into "<". This is not always properly done by external helper programs which output HTML, and of course never by those which output plain text.

When encapsulating plain text in an HTML body, the display of a preview may be improved by enclosing the text inside <pre> tags.

The character set needs to be specified in the header. It does not need to be UTF-8 (Recoll will take care of translating it), but it must be accurate for good results.

Recoll will process meta tags inside the header as possible document fields candidates. Documents fields can be processed by the indexer in different ways, for searching or displaying inside query results. This is described in a following section.

By default, the indexer will process the standard header fields if they are present: title, meta/description, and meta/keywords are both indexed and stored for query-time display.

A predefined non-standard meta tag will also be processed by Recoll without further configuration: if a date tag is present and has the right format, it will be used as the document date (for display and sorting), in preference to the file modification date. The date format should be as follows:

          <meta name="date" content="YYYY-mm-dd HH:MM:SS">
          or
          <meta name="date" content="YYYY-mm-ddTHH:MM:SS">
        

Example:

          <meta name="date" content="2013-02-24 17:50:00">
        

Input handlers also have the possibility to "invent" field names. This should also be output as meta tags:

          <meta name="somefield" content="Some textual data" />
        

You can embed HTML markup inside the content of custom fields, for improving the display inside result lists. In this case, add a (wildly non-standard) markup attribute to tell Recoll that the value is HTML and should not be escaped for display.

          <meta name="somefield" markup="html" content="Some <i>textual</i> data" />
        

As written above, the processing of fields is described in a further section.

Persistent filters can use another, probably simpler, method to produce metadata, by calling the setfield() helper method. This avoids the necessity to produce HTML, and any issue with HTML quoting. See, for example, rclaudio in Recoll 1.23 and later for an example of handler which outputs text/plain and uses setfield() to produce metadata.

The indexer will interpret ^L characters in the handler output as indicating page breaks, and will record them. At query time, this allows starting a viewer on the right page for a hit or a snippet. Currently, only the PDF, Postscript and DVI handlers generate page breaks.

The Recoll Python programming interface can be used both for searching and for creating/updating an index. Bindings exist for Python2 and Python3 (Jan 2021: python2 support will be dropped soon).

The search interface is used in a number of active projects: the Recoll Gnome Shell Search Provider , the Recoll Web UI, and the upmpdcli UPnP Media Server, in addition to many small scripts.

The index update section of the API may be used to create and update Recoll indexes on specific configurations (separate from the ones created by recollindex). The resulting databases can be queried alone, or in conjunction with regular ones, through the GUI or any of the query interfaces.

The search API is modeled along the Python database API version 2.0 specification (early versions used the version 1.0 spec).

The recoll package contains two modules:

There is a good chance that your system repository has packages for the Recoll Python API, sometimes in a package separate from the main one (maybe named something like python-recoll). Else refer to the Building from source chapter.

As an introduction, the following small sample will run a query and list the title and url for each of the results. The python/samples source directory contains several examples of Python programming with Recoll, exercising the extension more completely, and especially its data extraction features.

#!/usr/bin/python3

from recoll import recoll

db = recoll.connect()
query = db.query()
nres = query.execute("some query")
results = query.fetchmany(20)
for doc in results:
    print("%s %s" % (doc.url, doc.title))

You can also take a look at the source for the Recoll WebUI, the upmpdcli local media server, or the Gnome Shell Search Provider.

A few elements in the interface are specific and and need an explanation.

Two specific configuration variables: pyloglevel and pylogfilename allow overriding the generic values for Python programs. Set pyloglevel to 2 to suppress default startup messages (printed at level 3).

The update API can be used to create an index from data which is not accessible to the regular Recoll indexer, or structured to present difficulties to the Recoll input handlers.

An indexer created using this API will be have equivalent work to do as the the Recoll file system indexer: look for modified documents, extract their text, call the API for indexing it, take care of purging the index out of data from documents which do not exist in the document store any more.

The data for such an external indexer should be stored in an index separate from any used by the Recoll internal file system indexer. The reason is that the main document indexer purge pass (removal of deleted documents) would also remove all the documents belonging to the external indexer, as they were not seen during the filesystem walk. The main indexer documents would also probably be a problem for the external indexer own purge operation.

While there would be ways to enable multiple foreign indexers to cooperate on a single index, it is just simpler to use separate ones, and use the multiple index access capabilities of the query interface, if needed.

There are two parts in the update interface:

[MBOX]
fetch = /path/to/recoll/src/python/samples/rclmbox.py fetch
makesig = path/to/recoll/src/python/samples/rclmbox.py makesig

The following code fragments can be used to ensure that code can run with both the old and the new API (as long as it does not use the new abilities of the new API of course).

Adapting to the new package structure:

try:
from recoll import recoll
from recoll import rclextract
hasextract = True
except:
import recoll
hasextract = False

Adapting to the change of nature of the next Query member. The same test can be used to choose to use the scroll() method (new) or set the next value (old).

rownum = query.next if type(query.next) == int else query.rownumber

The following prerequisites are described in broad terms and not as specific package names (which will depend on the exact platform). The dependancies should be available as packages on most common Unix derivatives, and it should be quite uncommon that you would have to build one of them.

If you do not need the GUI, you can avoid all GUI dependancies by disabling its build. (See the configure section further).

The shopping list:

Check the Recoll download page for up to date version information.

Recoll has been built on Linux, FreeBSD, Mac OS X, and Solaris, most versions after 2005 should be ok, maybe some older ones too (Solaris 8 used to be ok). If you build on another system, and need to modify things, I would very much welcome patches.

Configure options: 

Normal procedure (for source extracted from a tar distribution):

          cd recoll-xxx
          ./configure
          make
          (practices usual hardship-repelling invocations)
        

When building from source cloned from the git repository, you also need to install autoconf, automake, and libtool and you must execute sh autogen.sh in the top source directory before running configure.

Use make install in the root of the source tree. This will copy the commands to prefix/bin and the sample configuration files, scripts and other shared data to prefix/share/recoll.

The Python interface can be found in the source tree, under the python/recoll directory.

As of Recoll 1.19, the module can be compiled for Python3.

The normal Recoll build procedure (see above) installs the API package for the default system version (python) along with the main code. The package for other Python versions (e.g. python3 if the system default is python2) must be explicitly built and installed.

The python/recoll/ directory contains the usual setup.py. After configuring and building the main Recoll code, you can use the script to build and install the Python module:

          cd recoll-xxx/python/recoll
          pythonX setup.py build
          sudo pythonX setup.py install
        

We did not test building the GUI on Solaris for recent versions. You will need at least Qt 4.4. There are some hints on an old web site page, they may still be valid.

Someone did test the 1.19 indexer and Python module build, they do work, with a few minor glitches. Be sure to use GNU make and install.

This file contains information about dynamic fields handling in Recoll. Some very basic fields have hard-wired behaviour, and, mostly, you should not change the original data inside the fields file. But you can create custom fields fitting your data and handle them just like they were native ones.

The fields file has several sections, which each define an aspect of fields processing. Quite often, you'll have to modify several sections to obtain the desired behaviour.

We will only give a short description here, you should refer to the comments inside the default file for more detailed information.

Field names should be lowercase alphabetic ASCII.

Here follows a small example of a personal fields file. This would extract a specific email header and use it as a searchable field, with data displayable inside result lists. (Side note: as the email handler does no decoding on the values, only plain ascii headers can be indexed, and only the first occurrence will be used for headers that occur several times).

[prefixes]
        # Index mailmytag contents (with the given prefix)
        mailmytag = XMTAG

        [stored]
        # Store mailmytag inside the document data record (so that it can be
        # displayed - as %(mailmytag) - in result lists).
        mailmytag = 

        [queryaliases]
        filename = fn
        containerfilename = cfn

        [mail]
        # Extract the X-My-Tag mail header, and use it internally with the
        # mailmytag field name
        x-my-tag = mailmytag
        

mimemap specifies the file name extension to MIME type mappings.

For file names without an extension, or with an unknown one, a system command (file -i, or xdg-mime) will be executed to determine the MIME type (this can be switched off, or the command changed inside the main configuration file).

All extension values in mimemap must be entered in lower case. File names extensions are lower-cased for comparison during indexing, meaning that an upper case mimemap entry will never be matched.

The mappings can be specified on a per-subtree basis, which may be useful in some cases. Example: okular notes have a .xml extension but should be handled specially, which is possible because they are usually all located in one place. Example:

[~/.kde/share/apps/okular/docdata]
        .xml = application/x-okular-notes

The recoll_noindex mimemap variable has been moved to recoll.conf and renamed to noContentSuffixes, while keeping the same function, as of Recoll version 1.21. For older Recoll versions, see the documentation for noContentSuffixes but use recoll_noindex in mimemap.

The main purpose of the mimeconf file is to specify how the different MIME types are handled for indexing. This is done in the [index] section, which should not be modified casually. See the comments in the file.

The file also contains other definitions which affect the query language and the GUI, and which, in retrospect, should have been stored elsewhere.

The [icons] section allows you to change the icons which are displayed by the recoll GUI in the result lists (the values are the basenames of the png images inside the iconsdir directory (which is itself defined in recoll.conf).

The [categories] section defines the groupings of MIME types into categories as used when adding an rclcat clause to a query language query. rclcat clauses are also used by the default guifilters buttons in the GUI (see next).

The filter controls appear at the top of the recoll GUI, either as checkboxes just above the result list, or as a dropbox in the tool area.

By default, they are labeled: media, message, other, presentation, spreadsheet and text, and each maps to a document category. This is determined in the [guifilters] section, where each control is defined by a variable naming a query language fragment.

A simple exemple will hopefully make things clearer.

[guifilters]

Big Books = dir:"~/My Books" size>10K
My Docs = dir:"~/My Documents"
Small Books = dir:"~/My Books" size<10K
System Docs = dir:/usr/share/doc
        

The above definition would create four filter checkboxes, labelled Big Books, My Docs, etc.

The text after the equal sign must be a valid query language fragment, and, when the button is checked, it will be combined with the rest of the query with an AND conjunction.

Any name text before a colon character will be erased in the display, but used for sorting. You can use this to display the checkboxes in any order you like. For exemple, the following would do exactly the same as above, but ordering the checkboxes in the reverse order.

[guifilters]

d:Big Books = dir:"~/My Books" size>10K
c:My Docs = dir:"~/My Documents"
b:Small Books = dir:"~/My Books" size<10K
a:System Docs = dir:/usr/share/doc
        

As you may have guessed, The default [guifilters] section looks like:

[guifilters]
text = rclcat:text
spreadsheet = rclcat:spreadsheet
presentation = rclcat:presentation
media = rclcat:media
message = rclcat:message
other = rclcat:other
        

mimeview specifies which programs are started when you click on an Open link in a result list. Ie: HTML is normally displayed using firefox, but you may prefer Konqueror, your openoffice.org program might be named oofice instead of openoffice etc.

Changes to this file can be done by direct editing, or through the recoll GUI preferences dialog.

If Use desktop preferences to choose document editor is checked in the Recoll GUI preferences, all mimeview entries will be ignored except the one labelled application/x-all (which is set to use xdg-open by default).

In this case, the xallexcepts top level variable defines a list of MIME type exceptions which will be processed according to the local entries instead of being passed to the desktop. This is so that specific Recoll options such as a page number or a search string can be passed to applications that support them, such as the evince viewer.

As for the other configuration files, the normal usage is to have a mimeview inside your own configuration directory, with just the non-default entries, which will override those from the central configuration file.

All viewer definition entries must be placed under a [view] section.

The keys in the file are normally MIME types. You can add an application tag to specialize the choice for an area of the filesystem (using a localfields specification in mimeconf). The syntax for the key is mimetype|tag

The nouncompforviewmts entry, (placed at the top level, outside of the [view] section), holds a list of MIME types that should not be uncompressed before starting the viewer (if they are found compressed, ie: mydoc.doc.gz).

The right side of each assignment holds a command to be executed for opening the file. The following substitutions are performed:

In addition to the predefined values above, all strings like %(fieldname) will be replaced by the value of the field named fieldname for the document. This could be used in combination with field customisation to help with opening the document.

ptrans specifies query-time path translations. These can be useful in multiple cases.

The file has a section for any index which needs translations, either the main one or additional query indexes. The sections are named with the Xapian index directory names. No slash character should exist at the end of the paths (all comparisons are textual). An exemple should make things sufficiently clear

          [/home/me/.recoll/xapiandb]
          /this/directory/moved = /to/this/place

          [/path/to/additional/xapiandb]
          /server/volume1/docdir = /net/server/volume1/docdir
          /server/volume2/docdir = /net/server/volume2/docdir