Audience
Academic researchers and developers seeking a solution offering tools for implementing and evaluating listwise reranking with large language models
About RankLLM
RankLLM is a Python toolkit for reproducible information retrieval research using rerankers, with a focus on listwise reranking. It offers a suite of rerankers, pointwise models like MonoT5, pairwise models like DuoT5, and listwise models compatible with vLLM, SGLang, or TensorRT-LLM. Additionally, it supports RankGPT and RankGemini variants, which are proprietary listwise rerankers. It includes modules for retrieval, reranking, evaluation, and response analysis, facilitating end-to-end workflows. RankLLM integrates with Pyserini for retrieval and provides integrated evaluation for multi-stage pipelines. It also includes a module for detailed analysis of input prompts and LLM responses, addressing reliability concerns with LLM APIs and non-deterministic behavior in Mixture-of-Experts (MoE) models. The toolkit supports various backends, including SGLang and TensorRT-LLM, and is compatible with a wide range of LLMs.
Other Popular Alternatives & Related Software
RankGPT
RankGPT is a Python toolkit designed to explore the use of generative Large Language Models (LLMs) like ChatGPT and GPT-4 for relevance ranking in Information Retrieval (IR). It introduces methods such as instructional permutation generation and a sliding window strategy to enable LLMs to effectively rerank documents. It supports various LLMs, including GPT-3.5, GPT-4, Claude, Cohere, and Llama2 via LiteLLM. RankGPT provides modules for retrieval, reranking, evaluation, and response analysis, facilitating end-to-end workflows. It includes a module for detailed analysis of input prompts and LLM responses, addressing reliability concerns with LLM APIs and non-deterministic behavior in Mixture-of-Experts (MoE) models. The toolkit supports various backends, including SGLang and TensorRT-LLM, and is compatible with a wide range of LLMs. RankGPT's Model Zoo includes models like LiT5 and MonoT5, hosted on Hugging Face.
Learn more
Amazon Personalize
Amazon Personalize enables developers to build applications with the same machine learning (ML) technology used by Amazon.com for real-time personalized recommendations – no ML expertise required. Amazon Personalize makes it easy for developers to build applications capable of delivering a wide array of personalization experiences, including specific product recommendations, personalized product re-ranking, and customized direct marketing. Amazon Personalize is a fully managed machine learning service that goes beyond rigid static rule based recommendation systems and trains, tunes, and deploys custom ML models to deliver highly customized recommendations to customers across industries such as retail and media and entertainment. Amazon Personalize provisions the necessary infrastructure and manages the entire ML pipeline, including processing the data, identifying features, using the best algorithms, and training, optimizing, and hosting the models.
Learn more
ColBERT
ColBERT is a fast and accurate retrieval model, enabling scalable BERT-based search over large text collections in tens of milliseconds. It relies on fine-grained contextual late interaction: it encodes each passage into a matrix of token-level embeddings. At search time, it embeds every query into another matrix and efficiently finds passages that contextually match the query using scalable vector-similarity (MaxSim) operators. These rich interactions allow ColBERT to surpass the quality of single-vector representation models while scaling efficiently to large corpora. The toolkit includes components for retrieval, reranking, evaluation, and response analysis, facilitating end-to-end workflows. ColBERT integrates with Pyserini for retrieval and provides integrated evaluation for multi-stage pipelines. It also includes a module for detailed analysis of input prompts and LLM responses, addressing reliability concerns with LLM APIs and non-deterministic behavior in Mixture-of-Experts.
Learn more
MonoQwen-Vision
MonoQwen2-VL-v0.1 is the first visual document reranker designed to enhance the quality of retrieved visual documents in Retrieval-Augmented Generation (RAG) pipelines. Traditional RAG approaches rely on converting documents into text using Optical Character Recognition (OCR), which can be time-consuming and may result in loss of information, especially for non-textual elements like graphs and tables. MonoQwen2-VL-v0.1 addresses these limitations by leveraging Visual Language Models (VLMs) that process images directly, eliminating the need for OCR and preserving the integrity of visual content. This reranker operates in a two-stage pipeline, initially, it uses separate encoding to generate a pool of candidate documents, followed by a cross-encoding model that reranks these candidates based on their relevance to the query. By training a Low-Rank Adaptation (LoRA) on top of the Qwen2-VL-2B-Instruct model, MonoQwen2-VL-v0.1 achieves high performance without significant memory overhead.
Learn more
Pricing
Starting Price:
Free
Free Version:
Free Version available.
Company Information
Castorini
Canada
github.com/castorini/rank_llm/
Videos and Screen Captures
Other Useful Business Software
MongoDB Atlas runs apps anywhere
MongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
Product Details
Platforms Supported
Cloud
Training
Documentation
Support
Online
