The Domestic LLM

One of the reasons I built my own computer recently was to have a machine available considerably more powerful than my laptop so that I can learn about and experiment with current technology. I am now playing around with large language models (aka LLMs) which is the key technology behind ChatGPT and its rivals. As widely recognized, these state-of-the-art systems consume enormous amounts of resources to build and keep running. What’s less widely known is that smaller versions of these same models are continually being released as freely available downloads for community experimentation, research, and development. Many of these open models are still far too large to run at home, but many others will run happily on ordinary home computers (albeit the more powerful your graphics card is, the better off you are). I’ve been learning a great deal experimenting with these. Some of the things I’ve learned are:

• There is an enormous amount of development going on across this whole space. There are tools and approaches available now that would have been really useful to me professionally a year-and-a-half ago.
• Nobody really understands how this technology works. An example: I asked my local LLM to write a poem in iambic tetrameter about the cat sleeping on his cat bed in my office. After some nudging (the first version was in iambic pentameter, but I told it to try again), it succeeded in producing some doggerel with the correct scansion. How did the system do this? We have no idea. We cannot point to a “metrical poetry” module within the system — rather, we are seeing emergent behavior.
• It is straightforward to set up an LLM system (even one at home) to let you conversationally ask questions and get natural language responses about a body of documents. (My test corpus has been a set of 80 or so conservation management plans from California). What is not at all straightforward is getting responses that are reliably accurate. Enormous amounts of engineering effort across every domain is being expended right now to build reliable conversational systems, but for now this is both very challenging and expensive.
• It is not clear what the important real-world applications of local LLM systems (i.e. ones you can run on a laptop or desktop) are going to be. There are a great deal of privacy benefits to them, since you can avoid shipping your sensitive documents off to Meta/Google/OpenAI etc. for LLM-based analyses, but will the local systems be powerful enough to conduct the analyses? One application of much interest to me is extracting structured information from unstructured or semi-structured text documents. This has been a challenge I’ve been pondering for quite a number of years, and LLM-based techniques for doing this are just starting to emerge.