Pauline Van Nies - Llms And Puzzle Agents Solving Cryptic Crosswords
Outline
- 0-5 min: Introduction Language models and Cryptic Crosswords, benchmarking different LLMs
- 5 - 10 min: Is it possible to improve the LLM performance by expert guided few-shot learning using prompt optimization?
- 10 -15 min: Is it possible to improve the LLM performance by knowledge distillation using finetuning?
- 15 - 20 min: LLMs + tools: Agent framework choice and setup
- 20 -25 min: Showcase Cryptic Crosswords solving by agents & Comparison LLMs
- 25 - 30 min: Lessons learned and explanation how transferable is the framework to business related tasks and even more complex puzzles (AIVD Christmas puzzle?)
We are interested in evaluating and optimizing language models on a complex task that requires linguistic insight and creativity. Cryptic crosswords contain clues with misleading surface reading, whose solutions require disambiguation of wordplay. Hence, a language model needs to conduct multi-step (iterative) reasoning by extracting and/or identifying the relevant components of a cryptic clue that lead to the final answer. Our aim is to get a better understanding of the capabilities and bottlenecks of language models, as well as an interesting insight in how language models solve a task that in humans is described as a feeling of ‘the solution just falls in place’.
Thus, we set up an evaluation pipeline for language models to test their ability to solve cryptic clues (Dutch and English datasets). In this presentation, we will share the results of comparing different LLMs and optimization approaches on performing this specific task. We apply standard optimization search methods like adjusting the temperature and using Chain-of-thought, adding domain expertise and the few-shot approach. Additionally, we apply automated prompt engineering using the DSPy framework. We will showcase this approach of systematically experimenting, tracking results (mlflow) and optimizing a language model and prompt.
Next, we will show if an LLM can learn to solve a cryptic clue, by either knowledge distillation from expert reasoning traces as a guided few-shot approach for in-context learning or by transferring the behaviour of an LLM teacher to an (initially worse performing) LLM student with finetuning.
Finally, we simulate how LLM agents can solve a cryptic crossword together, using different strategies that humans would do naturally. In the agent framework, tools such as look up of anagrams and synonyms are provided to the react agent to help solving the clues. The implementation of the puzzle agent framework and the choice for its technical framework (LangGraph over Google ADK) is elaborated on and the challenges to overcome. We will show with a demo how the puzzle agent framework (depending on the LLM it is powered by) in real-time solves a cryptic crossword.
We summarize the conclusions and comment shortly on how these insights relate to business projects. We also give an outlook on the question: how transferable is this puzzle agent framework to even more complex puzzles? In the Netherlands the holy grail of puzzle solving are the AIVD Christmas puzzles which contains a.o. language puzzles. We will comment on the possibility to solve those with the puzzle agent framework and future approaches.
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.
00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.
Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps