Hi,

I think the architecture you suggest is a nice way to integrate a LLM running externally. I just have a few comments:

• "send request to LLM service (request JSON if possible?)": asking the LLM for JSON format is unlikely to be a strong enough guarantee. I would simply have a validation step in the LLMExtractor class, and return an exception / 500 status when it fails.
• the env var logic you describe sounds quite rigid: as I read it, if LLM_API_URL is defined, you cannot run the "traditional" approach without changing the environment. Why not add an optional URL parameter to the API to decide which backend should be used?
• currently, the Meteor backend can be installed as a python module (https://github.com/NationalLibraryOfNorway/meteor#installing-the-python-module). It is not necessary for running Meteor as a service, but is a nice feature to have, and ideally your changes should preserve it.

We look forward to seeing the results of your work!

Thanks for your comments @pierrebeauguitte !

"send request to LLM service (request JSON if possible?)": asking the LLM for JSON format is unlikely to be a strong enough guarantee. I would simply have a validation step in the LLMExtractor class, and return an exception / 500 status when it fails.

Yes, you are right. In any case the JSON response from the LLM needs to be parsed within LLMExtractor because it has to be transformed to the format Meteor expects. If anything goes wrong in that process, returning a 500 status code makes sense.

the env var logic you describe sounds quite rigid: as I read it, if LLM_API_URL is defined, you cannot run the "traditional" approach without changing the environment. Why not add an optional URL parameter to the API to decide which backend should be used?

Yes, why not. There could be a parameter backend=llm / backend=finder for the / (HTML), /json and /file methods. What would the default be? Use LLMExtractor if available, fall back to Finder if not?

To make it easy to try this out, there could be an element in the web UI for selecting the backend.

currently, the Meteor backend can be installed as a python module (https://github.com/NationalLibraryOfNorway/meteor#installing-the-python-module). It is not necessary for running Meteor as a service, but is a nice feature to have, and ideally your changes should preserve it.

Sure, no reason to break this. Thanks for the reminder! Also I think it's important to keep the unit tests working and ideally add new tests for all the new functionality.

The current status is that we have an initial, very rough but working fine-tuned LLM based on Zephyr-7B, and it has been uploaded to HuggingFace Hub both in original HF format and in GGUF quantized format for CPU-only inference (e.g. for local development). We still need to take a closer look at inference platforms. After setting up one or more of those we can start development of the Meteor part.

Hi again! There has been some progress on this lately: not yet the Meteor implementation part, but getting closer. Here are some news from FinGreyLit:

1. We have refined the data set quite a lot, adjusted the document categorization (now there are five types, not four) and verified a lot of the metadata so that they correspond more closely to what is actually stated in the documents themselves to make a proper ground truth data set.
2. We switched the input and output formats used when talking with the LLM. Now the input is in JSON (not ad-hoc text) and the metadata schema is also a simple form of JSON that doesn't pretend to be Dublin Core anymore. The schema is pretty close to what Meteor produces, although we've added a few fields such as doi, p-isbn, p-issn and type_coar that Meteor doesn't support, and intend to add some more in the future.
3. The evaluation code has been overhauled a bit. It now has field-specific rules (including some reusable generic ones) for the comparisons and metrics. It probably still needs more work and we need to get back to the discussion on shared metrics. But at least the code is now clean and it's easy to see what the current logic is.
4. We've continued fine-tuning LLMs, now with the updated data set, and the results are slowly but steadily improving. There are two main LLMs: a larger model based on Mistral-7B and a much smaller one based on Qwen2-0.5B. The larger one needs a GPU to be practical, but the smaller one can be run on a laptop CPU using llama.cpp. The current evaluation scores are roughly 0.92 for the larger LLM, 0.85 for the smaller LLM, 0.64 for Meteor and 0.39 for the null baseline (always predicting nothing, which is surprisingly often the correct answer!).

But the most relevant part when it comes to adding LLM support in Meteor is the new Jupyter notebook for metadata extraction using a LLM API service. It needs an API service such as llama.cpp running locally. That is really easy to install and run at least on Linux - see the instructions at the beginning of the notebook. So you may want to give it a try! It also shows how the text is extracted from the PDF and converted to the simple JSON format that the fine-tuned LLM expects.

The same logic (text extraction, conversion to JSON, performing API calls to the LLM, parsing the results from the LLM) now just needs to be integrated into Meteor itself - as stated in the implementation plan above - and that's what I intend to try next!