SemEval-2020 Task 3: Graded Word Similarity in Context

For this tasks we ask participants to build systems that try to predict the effect that context has in human perception of similarity of words. We have seen very interesting work that uses local context to predict discrete changes in meaning: the different senses of a polysemous word. However context also has more subtle, continuous (graded) effects on meaning, even for words not necessarily considered polysemous. In order to be able to look at these effects we are building several datasets where we ask annotators to score how similar a pair of words are after they have read a short paragraph (which contains the two words). Each pair is scored within two of these paragraphs, allowing us to look at changes in similarity ratings due to context. CodaLab was used to run this task, you can see the dedicated website and the results of the participants at: https://competitions.codalab.org/competitions/20905