Probing datasets for Polish and English
A probing task is a classification problem that focuses on simple linguistic properties of sentences. The objective for training a classifier is to check whether a liguistic property is retained in vector representations of sentences (i.e. sentence embeddings). A classifier can be trained on a probing dataset that contains the pairs of sentences and their categories (i.e. linguistic properties).
The proposed probing datasets were automatically extracted from a corpus of dependency parsed sentences. The extraction procedure was based on a set of rules compatible with the Universal Dependencies annotation schema and is thus universal for languages with the UD style dependency treebanks. Probing datasets consist of 90k examples each (75k for training, 7.5k for validation and testing). The details are presentend in the article Empirical Linguistic Study of Sentence Embeddings published in the proceedings of ACL 2019.
WC (word content)
Classification of sentences containing exactly one of pre-selected 750 target words.
SentLen (sentence length)
Classification of sentences by their length.
TreeDepth (dependency tree depth)
Classification of sentences based on the depth of the corresponding dependency trees.
TopDeps (top dependency schema)
Classification of sentences based on the multiset of the dependency types labelling the relations between the top-most node (the ROOT’s only dependent) and all its children, barring
Passive (passive voice)
Classification of passive voice sentences and active sentences.
Tense (grammatical tense)
Classsification of sentences by the grammatical tense of their main predicates.
SubjNum (grammatical number of subjects)
Classification of sentences by the grammatical number of nominal subjects (marked with the UD label
nsubj) of main predicates.
ObjNum (grammatical number of objects)
Classification of sentences by the grammatical number of direct objects (marked with the UD label
obj) of main predicates.
SentType (sentence type)
classification of sentences by their types:
inter for interrogatve sentences (e.g. Do you like him?),
imper for imperative sentences (e.g. Get out of here!), and
other for declarative sentences (e.g. He likes her.) and exclamatory sentences (e.g. What a liar!).
The research presented in this paper was supported by SONATA 8 grant no 2014/15/D/HS2/03486 from the National Science Centre Poland. The computing was performed at Poznań Supercomputing and Networking Center.
- Katarzyna Krasnowska-Kieraś and Alina Wróblewska (2019) Empirical Linguistic Study of Sentence Embeddings In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5729–5739 Florence, Italy, July 28-August 2, 2019.