Overview
As part of our funding from the Patrick J. McGovern Foundation, we've been exploring how machine learning approaches might be useful in our democracy programme of work.
This report is working notes on the application of vector search to a few of our ongoing projects.
This covers:
- Basics of vector search
- Practically storing vectors in a postgres/django setup.
- Three use cases:
- Improving transcript parsers using vector analysis to find variations on significant phrases.
- Creating a similarity dataset that could be used alongside a traditional database search.
- A lightweight vector alert process to find declarations of interests.
Vector search is a very useful byproduct of large language models that uses how they convert text into numbers to find similar concepts in different sets of text.
When language models are trained on large amounts of text, this changes the internal shape of the model so that text with similar meanings ends up being ‘closer' to each other inside the model. A vector (or embedding) is a series of numbers that represent this location. By looking at the distance between embeddings, we can identify groups of similar terms with similar meanings. While a more basic text similarity approach would say that ‘bat' and ‘bag' are very similar, a model that sorts based on meaning would identify that ‘bat' and ‘owl' are more similar.
This means that without needing to re-train the model on a specific dataset, you can explore the similarities between concepts. Different models will produce different embeddings, and larger models may produce more accurate "closeness" - but also there are big efficiencies to be found in smaller models (e.g. DistilBERT is almost as good as BERT, but much smaller and faster).
A vector database is a way of storing these numeric embeddings with an associated index (generally a HNSW graph) that helps speed up nearest neighbour searches.
Running a vector search has three components:
- Creating the vectors from the documents to be searched
- Storing these vectors in a database to be queried
- Creating a vector of a search term, and querying the database for nearest matches.
- This might be used as the primary search, or as a way of reranking results from a standard text search.
The idea of stand-alone "vector stores" is part of the commercial AI offering - but there are increasingly bolt-ons available for the kind of relational databases we already deploy. Postgres (the database provider we use for everything *except* TheyWorkForYou) already has a plugin available that can provide the right field types and indexes. There are similar projects in motion for mariadb, sqlite, and duckdb. This in general means we should be cautious about solutions involving new stores or databases - because it's pretty likely the techniques will be available in an easier way soon.
To find out how hard it is to work with the kind of tools we're already using, I made a vector explorer based on postgres and django (that could, for instance, replace the approach we have in CAPE relatively straightforwardly).
This work can be seen in the twfy-vector-explorer repo. In the Dockerfile, I'm using a different source for postgres that already has the pg_vector plugin loaded.
With that, in django you just need to bring in an extra package and you have access to the VectorField field type, and the HNSW index type. In practice, for loading very large amounts of data I was dropping and recreating this index.
The next step is to load TheyWorkForYou's transcript data into the database.
To do this we go through all speeches in the raw data (I used the last few years for devolved parliaments, and the last decade for the Commons), and convert them at the paragraph level into an embedding (in this case, a 364 length vector) and load them into the database (using the django bulk_create method).
Practically, this took about a day to get the data in - but that included some back and forth about the best way to create indexes. I think to fully convert the full TheyWorkForYou history would be the work of a week once the process was stabilised.
This enables: as demonstrated by this notebook, to take a search query, convert it to an embedding - and query the database for similar phrases that are *not* direct text matches.
In this case, a search for mental health is also returning paragraphs about ‘mental and physical health' and ‘mental illness care'.
Uses
Once I had this working store, I've started to apply it to various problems we're working with at moment:
- Better data on what people are voting on
- Improved political monitoring and search
- WhoFundsThem - work around the register of members interests
One of the problems I'm looking at is correctly identifying possible motions in parliamentary transcripts, and assigning those to decisions made.
This makes it easier for us to display in one place both what was being voted on, and how people voted.
In principle, generative AI tools like ChatGPT are word calculators, and should be good at exactly this kind of problem. What I found was that it was relatively good at extracting a motion if given text with a motion - but was much less effective when given a range of speeches that may or may not contain a motion.
I think this could be much improved by creating a good dataset to give more examples or training data - but there are also fundamental disadvantages of this approach: it's slow and when it goes wrong, it won't always be clear *why*.
The approach I took was to use this vector search to augment a traditional approach.
Parliamentary speech is quite structured - with form and procedures for how actions like moving a motion will happen. This means it's a good candidate for building parsers that look for text that signifies "motion starts here" "motion ends here" and extracting the bit in the middle.
However, while speech is structured, it's not consistent - with lots of variations on similar forms of words. Here's where we can use vector search to help us out.
By searching for the motion start phrases we know about, we can find similar phrases, search for those and so on. Each one needs to be checked that it really is doing the same job - but it's a useful way to find variations at the start of the process - rather than encountering them as you expand the process.
To bring other machine learning tools into the mix - a really useful tool in building parsers is regular expressions (regex) - these are expressions used for pattern matching that describe abstract shapes of text ("a phrase followed by a bracket followed by text followed by close bracket followed by colon") and can then match and extract content from text it's given.
This is a really powerful tool - but you have to both understand in the abstract what you're trying to match, and how regex would express that. Once you get the hang of it, reading what expressions are doing is easy enough, but I always find creating them fiddly.
So when finding variations like "Amendment proposed (31 January):" I used ChatGPT to propose what the regular expression should be, and then give it a test to see if it's doing what it should. My general finding was that it was pretty good - but not 100% - and about one-in-five times needed some extra manual tweaking. But - this was a speed up,and it was easier to fix something that was 90% there than start from scratch.
With CAPE, we have previously experimented with a halfway house approach to machine learning enhanced search. Here we (working with a fellow as part of the Faculty support provided as part of Quadrature funding) generated vectors from terms used in the CAPE corpus, and used it to identify terms that were similar. These terms are then loaded into the CAPE database. During a search, when a known term is used, related terms are injected as ‘OR' into the query.
The advantages of this approach is we do not need to store historic embeddings or generate new ones for every search. What the fuzzy matching is doing is also visible to the user - with some related search terms highlighted. The magic is behind the scenes, but not a black box.
Based on that work, I explored a similar approach for the TheyWorkForYou corpus. Here we're trying to learn things from vectors of the whole corpus, to create a dataset that we can strategically apply to queries elsewhere.
Specifically: we want to help people make better email alerts that capture more references to the concept, rather than the exact phrase they're after.
To do this I took ten years of the Commons corpus and created ngrams of length 2-5 (counts of the unique occurrences of two word sequences, three word sequences, etc).
I then applied various thresholds to exclude phrases that are very uncommon (and help the dataset fit into memory), and created embeddings and loaded it into the vector explorer database.
I could now search through not just the sentences in the corpus, but find similarities between specific phrases. I added a few extra filters to catch where phrases were ending or starting "or, and", or duplicate longer phrases ("mental health" and "with mental health"). Experimenting with the threshold - a cosine similarity of about 0.15 seemed a good amount of relevant matches.
For instance, querying "mental health", would suggest:
- Mental ill health
- Mental illnesses
- Mental and physical health
- Health and mental
- Mental illness
- Mental ill-health
- Mental or physical health.
If you wanted to have a keyword search for mental health, having the option to add some of the above would be helpful in capturing references you might otherwise miss.
From this I took the search log from 2020+, reduced it to searches that have been done 50 times, and then to searches that were also in the ngram list to end up with about 1000 common search queries. Many of these were for politician names - and so less useful - but there's some good illustrations of the concept.
The first word here is an actual search that has been made - the rest are extra checkboxes we could then prompt with:
- foreign aid: overseas aid, international aid
- Freedom of information: right to information, information rights
- air pollution: air quality, air pollutants
- offshore wind: offshore renewable energy, offshore renewable, offshore energy
- gambling harm: gambling-related harm, gambling addiction
- contaminated blood: blood contamination, infected blood
- Gay rights: lgbt rights, rights of lgbt, transgender rights
- free school meals: free school dinners, free school lunches, school meals, school meal provisions
When speaking in a debate that touches on one of their outside financial interests, MPs are supposed to mention this. One of the interesting things I found in the initial experiment was that the vector search was kicking up lots of variations in this approach.
For WhoFundsThem - we want to be able to track and give information to volunteers about these kinds of declarations. This information helps us either create guidance, or datasets that can be used directly to answer the questions.
But if we had a comprehensive list of these as they were said - we could do some regular analysis and comms around good and bad ways this information is disclosed.
This seemed a good candidate to try and create a vector alert tool. This can be seen in https://github.com/mysociety/mini-transcript-search.
To do searches of recent transcripts - you don't need to build a big database or index at all. Instead, I pulled out the embedding generating code into a little repository - which then converted the last few days to vectors to calculate the vector similarity against a specific query in about a minute.
I've built this as a tool that could be used from a notebook or the command line.
Running this today:
uvx --from git+https://github.com/mysociety/mini-transcript-search transcript-search "register of members financial interests" --threshold 0.4 --n 5 --dest yesterday.json
- Installs the underlying requirements and embeddings model.
- Downloads transcripts
- Generates embeddings locally.
- Does a search - and dumps the results in a file.
This gave me a file that contained:
From this we can click through to the register on TheyWorkForYou to find what he's talking about is a donation from the Prison Officers Association.
Now, this is to be fair something that would work pretty well on a normal search. Extending it back for all of September gives us a few more you might not otherwise pick up:
A search for the normal form of words would miss this (slightly more rhetorical) of declaring interests - but if trying to understand how MPs talk positively about their other interests - it's really useful to have this.