Cloud Hosting

Experimenting with Large Language Models for free (Part 2)Photo by Glib Albovsky, Unsplash

In the first part of the story, we used a free Google Colab instance to run a Mistral-7B model and extract information using the FAISS (Facebook AI Similarity Search) database. In this part, we will go further, and I will show how to run a LLaMA 2 13B model; we will also test some extra LangChain functionality like making chat-based applications and using agents. In the same way, as in the first part, all used components are based on open-source projects and will work completely for free.

Lets get into it!

A LLaMA.CPP is a very interesting open-source project, originally designed to run an LLaMA model on Macbooks, but its functionality grew far beyond that. First, it is written in plain C/C++ without external dependencies and can run on any hardware (CUDA, OpenCL, and Apple silicon are supported; it can even work on a Raspberry Pi). Second, LLaMA.CPP can be connected with LangChain, which allows us to test a lot of its functionality for free without having an OpenAI key. Last but not least, because LLaMA.CPP works everywhere, it's a good candidate to run in a free Google Colab instance. As a reminder, Google provides free access to Python notebooks with 12 GB of RAM and 16 GB of VRAM, which can be opened using the Colab Research page. The code is opened in the web browser and runs in the cloud, so everybody can access it, even from a minimalistic budget PC.

Before using LLaMA, lets install the library. The installation itself is easy; we only need to enable LLAMA_CUBLAS before using pip:

For the first test, I will be using a 7B model. Here, I also installed a huggingface-hub library, which allows us to automatically download a Llama-27b-Chat model in the GGUF format needed for LLaMA.CPP. I also installed a LangChain

The rest is here:

LLMs for Everyone: Running the LLaMA-13B model and LangChain in Google Colab - Towards Data Science