Selected Papers on Linear Word Analogies in Embeddings

Early Papers on Word Analogies

• Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 746–751.
• Levy, O., & Goldberg, Y. (2014). Neural Word Embedding as Implicit Matrix Factorization. Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, 2177–2185.
• Levy, O., & Goldberg, Y. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. Proceedings of the Eighteenth Conference on Computational Natural Language Learning, 171–180.

Critiques

• Linzen, T. (2016). Issues in Evaluating Semantic Spaces Using Word Analogies. Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, 13–18.
• Gladkova, A., & Drozd, A. (2016). Intrinsic Evaluations of Word Embeddings: What Can We Do Better? Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, 36–42.
• Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-Based Detection of Morphological and Semantic Relations with Word Embeddings: What Works and What Doesn’t. In J. Andreas, E. Choi, & A. Lazaridou (Eds), Proceedings of the NAACL Student Research Workshop (pp. 8–15). Association for Computational Linguistics.
• Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word Embeddings, Analogies, and Machine Learning: Beyond king - man + woman = queen. In Y. Matsumoto & R. Prasad (Eds), Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). The COLING 2016 Organizing Committee.
• • Rogers, A., Drozd, A., & Li, B. (2017). The (too Many) Problems of Analogical Reasoning with Word Vectors. In N. Ide, A. Herbelot, & L. Màrquez (Eds), Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017) (pp. 135–148). Association for Computational Linguistics.

Explanations

• Finley, G., Farmer, S., & Pakhomov, S. (2017). What Analogies Reveal about Word Vectors and their Compositionality. In N. Ide, A. Herbelot, & L. Màrquez (Eds), Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017) (pp. 1–11). Association for Computational Linguistics.
• Gittens, A., Achlioptas, D., & Mahoney, M. W. (2017). Skip-Gram—Zipf + Uniform = Vector Additivity. In R. Barzilay & M.-Y. Kan (Eds), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 69–76). Association for Computational Linguistics.
• Ethayarajh, K. (2019). Rotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space. In K. Inui, J. Jiang, V. Ng, & X. Wan (Eds), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3503–3508). Association for Computational Linguistics.
• Ethayarajh, K., Duvenaud, D., & Hirst, G. (2019). Towards Understanding Linear Word Analogies. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3253–3262.
• Allen, C., Balazevic, I., & Hospedales, T. (2019). What the Vec? Towards Probabilistically Grounded Embeddings. Advances in Neural Information Processing Systems, 32.
• Allen, C., & Hospedales, T. (2019). Analogies Explained: Towards Understanding Word Embeddings. Proceedings of the 36th International Conference on Machine Learning, 223–231.
• Fournier, L., Dupoux, E., & Dunbar, E. (2020). Analogies Minus Analogy Test: Measuring Regularities in Word Embeddings. In R. Fernández & T. Linzen (Eds), Proceedings of the 24th Conference on Computational Natural Language Learning (pp. 365–375). Association for Computational Linguistics.
• Chiang, H.-Y., Camacho-Collados, J., & Pardos, Z. (2020). Understanding the Source of Semantic Regularities in Word Embeddings. In R. Fernández & T. Linzen (Eds), Proceedings of the 24th Conference on Computational Natural Language Learning (pp. 119–131). Association for Computational Linguistics.
• Fournier, L., & Dunbar, E. (2021). Paraphrases Do Not Explain Word Analogies. In P. Merlo, J. Tiedemann, & R. Tsarfaty (Eds), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (pp. 2129–2134). Association for Computational Linguistics.
• Ri, N., Lee, F.-T., & Verma, N. (2023). Contrastive Loss is All You Need to Recover Analogies as Parallel Lines. In B. Can, M. Mozes, S. Cahyawijaya, N. Saphra, N. Kassner, S. Ravfogel, A. Ravichander, C. Zhao, I. Augenstein, A. Rogers, K. Cho, E. Grefenstette, & L. Voita (Eds), Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023) (pp. 164–173). Association for Computational Linguistics.
• Hernandez, E., Sharma, A. S., Haklay, T., Meng, K., Wattenberg, M., Andreas, J., Belinkov, Y., & Bau, D. (2023, October 13). Linearity of Relation Decoding in Transformer Language Models. The Twelfth International Conference on Learning Representations.
• Torii, T., Maeda, A., & Hidaka, S. (2024). Distributional Hypothesis as Isomorphism Between Word-Word Co-Occurrence and Analogical Parallelograms. PLOS ONE, 19(10), e0312151.
• Korchinski, D. J., Karkada, D., Bahri, Y., & Wyart, M. (2025). On the Emergence of Linear Analogies in Word Embeddings (arXiv:2505.18651).
• Park, K., Choe, Y. J., Jiang, Y., & Veitch, V. (2024, October 4). The Geometry of Categorical and Hierarchical Concepts in Large Language Models. The Thirteenth International Conference on Learning Representations.
• Karkada, D., Simon, J. B., Bahri, Y., & DeWeese, M. R. (2025, October 29). Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models. The Thirty-ninth Annual Conference on Neural Information Processing Systems.