search_query_stats_spans.md 20.1 KB

Edit Raw Blame History



Statistics - spans

To get statistics on the occurrence of a span within a set of documents, besides the parameter to enable statistics, the following parameter should be provided.


Parameter
Value
Obligatory


mtas.stats.spans
true
yes


Multiple statistics on the occurrence of a span can be produced within the same request. To distinguish them, a unique identifier has to be provided for  each of the required statistics. Furthermore, statistics for the occurrence of multiple spans can be produced. Spans are described by a query, and to distinguish multiple spans, also a query identifier has to be provided. 


Parameter
Value
Info
Obligatory


mtas.stats.spans.<identifier>.key
<string>
key used in response
no


mtas.stats.spans.<identifier>.field
<string>
mtas field
yes


mtas.stats.spans.<identifier>.query.<identifier query>.type
<string>
query language: cql

yes


mtas.stats.spans.<identifier>.query.<identifier query>.value
<string>
query: cql

yes


mtas.stats.spans.<identifier>.query.<identifier query>.prefix
<string>
default prefix
no


mtas.stats.spans.<identifier>.query.<identifier query>.ignore
<string>
ignore query: cql

no


mtas.stats.spans.<identifier>.query.<identifier query>.maximumIgnoreLength
<integer>
maximum number of succeeding occurrences to ignore
no


mtas.stats.spans.<identifier>.type
<string>
required type of statistics

no


mtas.stats.spans.<identifier>.minimum
<double>
minimum number of occurrences span
no


mtas.stats.spans.<identifier>.maximum
<double>
maximum number of occurrences span
no


The key is added to the response and may be used to distinguish between multiple statistics on the occurrence of spans, and should therefore be unique. The optional minimum and maximum can be used to focus only on documents satisfying a condition on the number of occurrences of the spans. When multiple queries are provided, the provided boundary will hold on the sum of occurrences of the resulting spans.


Variables

The query may contain one or more variables, and the value(s) of these variables have to be defined 


Parameter
Value
Info
Obligatory


mtas.stats.spans.<identifier>.query.<identifier query>.variable<identifier variable>.name
<string>
name of variable
yes


mtas.stats.spans.<identifier>.query.<identifier query>.variable<identifier variable>.value
<string>
comma separated list of values
yes


Functions

To compute statistics for values based on the occurrence of one or multiple spans, optionally functions can be added. The parameters for these functions are the number of occurrences $q0, $q1, ... for each span and the number of positions $n in a document. Statistics on the value computed for each document in the set are added to the response.


Parameter
Value
Info
Obligatory


mtas.stats.spans.<identifier>.function.<identifier function>.key
<string>
key used in response
no


mtas.stats.spans.<identifier>.function.<identifier function>.expression
<string>
see functions

yes


mtas.stats.spans.<identifier>.function.<identifier function>.type
<string>
required type of statistics

no


Again, the key is added to the response and may be used to distinguish between multiple functions, and should therefore be unique.


Examples


Basic : basic statistics on the occurrence of a word.

Minimum and Maximum : statistics on the occurrence of a word with restrictions on the number of occurrences.

Subset : statistics on the occurrence of a word within a subset of documents.

Multiple : statistics on the occurrence of multiple words.

Prefix : default prefix for query

Ignore : query with ignore

Ignore and maximumIgnoreLength : query with ignore and maximumIgnoreLength

Functions : statistics using functions.

Multiple and Functions : statistics using functions on the occurrence of multiple words.


Basic

Example

Total and average number of occurrences of the word "de" and the number of documents.

CQL

[t="de"]

Request and response

q=*%3A*&mtas=true&mtas.stats=true&mtas.stats.spans=true&mtas.stats.spans.0.field=text&mtas.stats.spans.0.query.0.type=cql&mtas.stats.spans.0.query.0.value=%5Bt%3D%22de%22%5D&mtas.stats.spans.0.key=example - basic&mtas.stats.spans.0.type=n%2Csum%2Cmean&rows=0&wt=json&indent=true
"mtas":{
    "stats":{
      "spans":[{
          "key":"example - basic",
          "mean":10.484745312881392,
          "sum":21648986,
          "n":2064808}]}}


Minimum and Maximum

Example

Full statistics on the number of occurrences of the word "de" for documents with a minimum of 100 occurrences, for documents with a maximum of 200 occurrences, and for documents with between 100 and 200 occurrences.

CQL

[t="de"]

Request and response

q=*%3A*&mtas=true&mtas.stats=true&mtas.stats.spans=true&mtas.stats.spans.0.field=text&mtas.stats.spans.0.query.0.type=cql&mtas.stats.spans.0.query.0.value=[t%3D"de"]&mtas.stats.spans.0.key=example - minimum&mtas.stats.spans.0.type=all&mtas.stats.spans.0.minimum=100&mtas.stats.spans.1.field=text&mtas.stats.spans.1.query.0.type=cql&mtas.stats.spans.1.query.0.value=[t%3D"de"]&mtas.stats.spans.1.key=example - maximum&mtas.stats.spans.1.type=all&mtas.stats.spans.1.maximum=200&mtas.stats.spans.2.field=text&mtas.stats.spans.2.query.0.type=cql&mtas.stats.spans.2.query.0.value=[t%3D"de"]&mtas.stats.spans.2.key=example - minimum and maximum&mtas.stats.spans.2.type=all&mtas.stats.spans.2.minimum=100&mtas.stats.spans.2.maximum=200&rows=0&wt=json&indent=true
"mtas":{
    "stats":{
      "spans":[{
          "key":"example - minimum",
          "sumsq":8.697716821E9,
          "populationvariance":419246.79216222034,
          "max":18192.0,
          "sum":4531791.0,
          "kurtosis":164.0070630212961,
          "standarddeviation":647.5106543337658,
          "n":18029,
          "quadraticmean":694.5712655602209,
          "min":100.0,
          "median":136.0,
          "variance":419270.04747574165,
          "mean":251.36119585112766,
          "geometricmean":160.51059457862394,
          "sumoflogs":91557.75154264584,
          "skewness":10.551742594336096},
        {
          "key":"example - maximum",
          "sumsq":7.37120919E8,
          "populationvariance":271.7584436494812,
          "max":200.0,
          "sum":1.9094863E7,
          "kurtosis":31.747278519903332,
          "standarddeviation":16.485101621383148,
          "n":2061622,
          "quadraticmean":18.90883830898543,
          "min":0.0,
          "median":4.0,
          "variance":271.75857546732925,
          "mean":9.26205822405864,
          "geometricmean":0.0,
          "sumoflogs":"-Infinity",
          "skewness":4.742017950067824},
        {
          "key":"example - minimum and maximum",
          "sumsq":2.73659206E8,
          "populationvariance":684.261700791398,
          "max":200.0,
          "sum":1977668.0,
          "kurtosis":-0.4734932618605048,
          "standarddeviation":26.15927758668186,
          "n":14843,
          "quadraticmean":135.78262099542508,
          "min":100.0,
          "median":127.0,
          "variance":684.3078038570759,
          "mean":133.23910260728962,
          "geometricmean":130.83058078303992,
          "sumoflogs":72343.34534205312,
          "skewness":0.7178184624731176}]}}

  
Subset

Example

Total and average number of occurrences of the word "de" and the number of documents for a subset of documents.

CQL

[t="de"]

Request and response

q=text:koe&rows=0&mtas=true&mtas.stats=true&mtas.stats.tokens=true&mtas.stats.tokens.0.field=text&mtas.stats.tokens.0.key=example - subset&mtas.stats.tokens.0.type=sum,mean,n&wt=json&indent=true
"mtas":{
    "stats":{
      "tokens":[{
          "key":"example - subset",
          "mean":49644.49556868538,
          "sum":134437294,
          "n":2708}]}}

  
Multiple

Example

Total and average number of occurrences of the word "de" and "het", and the number of documents.

CQL  


combined cql: [t="de"|t="het"]


combined regexp: [t="(de|het)"]


two queries: [t="de"] [t="het"]


Request and response

q=*%3A*&mtas=true&mtas.stats=true&mtas.stats.spans=true&mtas.stats.spans.0.field=text&mtas.stats.spans.0.query.0.type=cql&mtas.stats.spans.0.query.0.value=[t%3D"de"|t%3D"het"]&mtas.stats.spans.0.key=multiple+-+combined+cql&mtas.stats.spans.0.type=n%2Csum%2Cmean&mtas.stats.spans.1.field=text&mtas.stats.spans.1.query.0.type=cql&mtas.stats.spans.1.query.0.value=[t%3D"(de|het)"]&mtas.stats.spans.1.key=multiple+-+combined+regexp&mtas.stats.spans.1.type=n%2Csum%2Cmean&mtas.stats.spans.2.field=text&mtas.stats.spans.2.query.0.type=cql&mtas.stats.spans.2.query.0.value=[t%3D"de"]&mtas.stats.spans.2.query.1.type=cql&mtas.stats.spans.2.query.1.value=[t%3D"het"]&mtas.stats.spans.2.key=multiple+-+two+queries&mtas.stats.spans.2.type=n%2Csum%2Cmean&rows=0&wt=json&indent=true
"mtas":{
    "stats":{
      "spans":[{
          "key":"multiple - combined cql",
          "mean":15.173083405333571,
          "sum":31329504,
          "n":2064808},
        {
          "key":"multiple - combined regexp",
          "mean":15.173083405333571,
          "sum":31329504,
          "n":2064808},
        {
          "key":"multiple - two queries",
          "mean":15.173083405333571,
          "sum":31329504,
          "n":2064808}]}}}

  
Prefix

Example

Total and average number of occurrences of the word "de" followed by an adjective.

CQL

"de" [pos="ADJ"]

Request and response

q=*%3A*&mtas=true&mtas.stats=true&mtas.stats.spans=true&mtas.stats.spans.0.field=text&mtas.stats.spans.0.query.0.type=cql&mtas.stats.spans.0.query.0.value="de" [pos%3D"ADJ"]&mtas.stats.spans.0.query.0.prefix=t_lc&mtas.stats.spans.0.key=example - prefix&mtas.stats.spans.0.type=n%2Csum%2Cmean&rows=0&wt=json&indent=true
"mtas":{
    "stats":{
      "spans":[{
          "key":"example - prefix",
          "mean":2.1725308115815127,
          "sum":4485859,
          "n":2064808}]}}

  
Ignore

Example

Total and average number of occurrences of an article followed by a noun, ignoring adjectives.

CQL

[pos="LID"][pos="N"]

Ignore 
[pos="ADJ"]

Request and response

q=*%3A*&mtas=true&mtas.stats=true&mtas.stats.spans=true&mtas.stats.spans.0.field=text&mtas.stats.spans.0.query.0.type=cql&mtas.stats.spans.0.query.0.value=[t_lc%3D"de"]&mtas.stats.spans.0.key=functions+-+de&mtas.stats.spans.0.type=n%2Csum%2Cmean&mtas.stats.spans.0.function.0.expression=%24q0%2F%24n&mtas.stats.spans.0.function.0.key=relative+frequency&mtas.stats.spans.0.function.0.type=mean%2Cstandarddeviation%2Cdistribution(start%3D0%2Cend%3D0.1%2Cnumber%3D10)&mtas.stats.spans.0.function.1.expression=%24n&mtas.stats.spans.0.function.1.key=number+of+words&mtas.stats.spans.0.function.1.type=n%2Csum&rows=0&wt=json&indent=true
"mtas":{
    "stats":{
      "spans":[{
          "key":"functions - de",
          "mean":12.34790062804871,
          "sum":25496044,
          "n":2064808,
          "functions":{
            "number of words":{
              "sum":337230767,
              "n":2064808},
            "relative frequency":{
              "distribution(start=0,end=0.1,number=10)":{
                "[0.000,0.010)":950500,
                "[0.010,0.020)":80369,
                "[0.020,0.030)":115695,
                "[0.030,0.040)":139752,
                "[0.040,0.050)":162877,
                "[0.050,0.060)":168598,
                "[0.060,0.070)":145493,
                "[0.070,0.080)":109117,
                "[0.080,0.090)":77214,
                "[0.090,0.100)":51243},
              "mean":0.030196372045937097,
              "errorList":{"division by zero":691633},
              "standarddeviation":0.03428066513492476,
              "errorNumber":691633}}}]}}

  
Ignore and maximumIgnoreLength

  
Functions

Example

Statistics for the relative frequency of the word "de" and the total number of words in documents containing this word.

CQL

[t="de"]

Functions

$q0/$n

$n

Request and response

q=*%3A*&mtas=true&mtas.stats=true&mtas.stats.spans=true&mtas.stats.spans.0.field=text&mtas.stats.spans.0.query.0.type=cql&mtas.stats.spans.0.query.0.value=[t_lc%3D"de"]&mtas.stats.spans.0.key=functions+-+de&mtas.stats.spans.0.type=n%2Csum%2Cmean&mtas.stats.spans.0.function.0.expression=%24q0%2F%24n&mtas.stats.spans.0.function.0.key=relative+frequency&mtas.stats.spans.0.function.0.type=mean%2Cstandarddeviation%2Cdistribution(start%3D0%2Cend%3D0.1%2Cnumber%3D10)&mtas.stats.spans.0.function.1.expression=%24n&mtas.stats.spans.0.function.1.key=number+of+words&mtas.stats.spans.0.function.1.type=n%2Csum&rows=0&wt=json&indent=true
"mtas":{
    "stats":{
      "spans":[{
          "key":"functions - de",
          "mean":12.34790062804871,
          "sum":25496044,
          "n":2064808,
          "functions":{
            "number of words":{
              "sum":337230767,
              "n":2064808},
            "relative frequency":{
              "distribution(start=0,end=0.1,number=10)":{
                "[0.000,0.010)":950500,
                "[0.010,0.020)":80369,
                "[0.020,0.030)":115695,
                "[0.030,0.040)":139752,
                "[0.040,0.050)":162877,
                "[0.050,0.060)":168598,
                "[0.060,0.070)":145493,
                "[0.070,0.080)":109117,
                "[0.080,0.090)":77214,
                "[0.090,0.100)":51243},
              "mean":0.030196372045937097,
              "errorList":{"division by zero":691633},
              "standarddeviation":0.03428066513492476,
              "errorNumber":691633}}}]}}

  
Multiple and Functions

Example

Statistics for the absolute and relative frequency of the words "de", "het" and "een", for part of speech type "LID" and the total number of words in documents containing this word.

CQL

[t="de"]

[t="het"]

[t="een"]

[pos="LID"]

Functions

$q0/$n

$q1/$n

$q2/$n

$q3/$n

$q0/$q3

$q1/$q3

$q2/$q3

($q0+$q1+$q2)/$q3  

Request and response

q=*%3A*&mtas=true&mtas.stats=true&mtas.stats.spans=true&mtas.stats.spans.0.field=text&mtas.stats.spans.0.query.0.type=cql&mtas.stats.spans.0.query.0.value=[t_lc%3D"de"]&mtas.stats.spans.0.query.1.type=cql&mtas.stats.spans.0.query.1.value=[t_lc%3D"het"]&mtas.stats.spans.0.query.2.type=cql&mtas.stats.spans.0.query.2.value=[t_lc%3D"een"]&mtas.stats.spans.0.query.3.type=cql&mtas.stats.spans.0.query.3.value=[pos%3D"LID"]&mtas.stats.spans.0.key=multiple+and+functions+-+de%2Bhet%2Been+and+LID&mtas.stats.spans.0.type=n&mtas.stats.spans.0.minimum=1&mtas.stats.spans.0.function.0.expression=%24q0&mtas.stats.spans.0.function.0.key=de+-+absolute&mtas.stats.spans.0.function.0.type=n%2Csum&mtas.stats.spans.0.function.1.expression=%24q1&mtas.stats.spans.0.function.1.key=het+-+absolute&mtas.stats.spans.0.function.1.type=n%2Csum&mtas.stats.spans.0.function.2.expression=%24q2&mtas.stats.spans.0.function.2.key=een+-+absolute&mtas.stats.spans.0.function.2.type=n%2Csum&mtas.stats.spans.0.function.3.expression=%24q3&mtas.stats.spans.0.function.3.key=LID+-+absolute&mtas.stats.spans.0.function.3.type=n%2Csum&mtas.stats.spans.0.function.4.expression=%24q0%2F%24n&mtas.stats.spans.0.function.4.key=de+-+relative+to+positions&mtas.stats.spans.0.function.4.type=n%2Cmean&mtas.stats.spans.0.function.5.expression=%24q1%2F%24n&mtas.stats.spans.0.function.5.key=het+-+relative+to+positions&mtas.stats.spans.0.function.5.type=n%2Cmean&mtas.stats.spans.0.function.6.expression=%24q2%2F%24n&mtas.stats.spans.0.function.6.key=een+-+relative+to+positions&mtas.stats.spans.0.function.6.type=n%2Cmean&mtas.stats.spans.0.function.7.expression=%24q3%2F%24n&mtas.stats.spans.0.function.7.key=LID+-+relative+to+positions&mtas.stats.spans.0.function.7.type=n%2Cmean&mtas.stats.spans.0.function.8.expression=%24q0%2F%24q3&mtas.stats.spans.0.function.8.key=de+-+relative+to+LID&mtas.stats.spans.0.function.8.type=n%2Cmean&mtas.stats.spans.0.function.9.expression=%24q1%2F%24q3&mtas.stats.spans.0.function.9.key=het+-+relative+to+LID&mtas.stats.spans.0.function.9.type=n%2Cmean&mtas.stats.spans.0.function.10.expression=%24q2%2F%24q3&mtas.stats.spans.0.function.10.key=een+-+relative+to+LID&mtas.stats.spans.0.function.10.type=n%2Cmean&mtas.stats.spans.0.function.11.expression=(%24q0%2B%24q1%2B%24q2)%2F%24q3&mtas.stats.spans.0.function.11.key=de%2Bhet%2Been+-+relative+to+LID&mtas.stats.spans.0.function.11.type=n%2Cmean&rows=0&wt=json&indent=true
"mtas":{
    "stats":{
      "spans":[{
          "key":"multiple and functions - de+het+een and LID",
          "n":1889646,
          "functions":{
            "een - relative to LID":{
              "mean":0.26176457027078637,
              "errorList":{"division by zero":24165},
              "n":1889646,
              "errorNumber":24165},
            "LID - absolute":{
              "sum":44062088,
              "n":1889646},
            "de+het+een - relative to LID":{
              "mean":1.0864042218694616,
              "errorList":{"division by zero":24165},
              "n":1889646,
              "errorNumber":24165},
            "het - relative to LID":{
              "mean":0.27408299432800154,
              "errorList":{"division by zero":24165},
              "n":1889646,
              "errorNumber":24165},
            "een - relative to positions":{
              "mean":0.014397108127121947,
              "errorList":{"division by zero":631875},
              "n":1889646,
              "errorNumber":631875},
            "een - absolute":{
              "sum":10616743,
              "n":1889646},
            "het - relative to positions":{
              "mean":0.014874612933292992,
              "errorList":{"division by zero":631875},
              "n":1889646,
              "errorNumber":631875},
            "de - absolute":{
              "sum":25496044,
              "n":1889646},
            "het - absolute":{
              "sum":11527080,
              "n":1889646},
            "LID - relative to positions":{
              "mean":0.05786145893684233,
              "errorList":{"division by zero":631875},
              "n":1889646,
              "errorNumber":631875},
            "de - relative to LID":{
              "mean":0.5505566572707496,
              "errorList":{"division by zero":24165},
              "n":1889646,
              "errorNumber":24165},
            "de - relative to positions":{
              "mean":0.03299544495178249,
              "errorList":{"division by zero":631875},
              "n":1889646,
              "errorNumber":631875}}}]}}