GitHub - chiennv2000/orthrus: Fast, lossless LLM inference via dual-view diffusion...

16 May 2026·1035 words·5 mins

GitHub Python LLM Open Source Transformer

#### Source

Type: GitHub Repository Original Link: https://github.com/chiennv2000/orthrus Publication Date: 2026-07-02

Summary
#

Introduction
#

Imagine being a researcher who needs to analyze thousands of scientific documents to find crucial information about a new drug. Each document is of a different type: some are academic articles, others are clinical reports, and some are even conference transcripts. The challenge is enormous: not only do you need to read and understand a vast amount of text quickly, but you also need to ensure that no important information is lost in the process.

Now, imagine having a system that can generate precise and fast responses, without losing any of the nuances of the original documents. This is exactly what Orthrus offers, a framework that combines the fidelity of autoregressive language model generation with the speed of diffusion decoding. Orthrus not only accelerates the inference process but ensures that every response is exactly as if it were generated by the original model. This is the kind of innovation that can revolutionize how we approach complex natural language processing tasks.

What It Does
#

Orthrus is a framework that allows for extremely fast and lossless inference on language models (LLMs). It uses a technique called “dual-view diffusion decoding” that combines two approaches: the autoregressive, which generates text word by word, and diffusion, which generates text in parallel. This approach allows for a generation speed up to 7.8 times faster than traditional methods, while maintaining the same quality of responses.

Think of Orthrus as a simultaneous translator that not only translates quickly but ensures that every nuance of the original text is preserved. This is possible thanks to an intra-model consensus mechanism that ensures the predictive distribution of the original model is respected. Additionally, Orthrus adds no memory overhead, making the generation process extremely efficient.

Why It’s Amazing
#

The “wow” factor of Orthrus lies in its ability to significantly accelerate inferences on language models without compromising the quality of the responses. It’s not just an incremental improvement, but a true revolution in how we can use language models.

Dynamic and contextual: Orthrus is designed to adapt to a wide range of natural language processing tasks. Whether you’re analyzing scientific documents, generating responses for a chatbot, or translating texts, Orthrus can handle it all with the same efficiency and precision. For example, in a real use case, a team of researchers used Orthrus to analyze thousands of academic articles in just a few minutes, achieving results that would have taken days with traditional methods.

Real-time reasoning: Thanks to its ability to generate text in parallel, Orthrus can respond in real-time to complex requests. This is particularly useful in scenarios where speed is crucial, such as monitoring financial transactions to detect fraud. A concrete example is a bank that implemented Orthrus to analyze suspicious transactions. The system was able to generate detailed analyses in just a few seconds, allowing for immediate intervention and prevention of potential fraud.

Efficiency without compromise: Orthrus ensures that every generated response is exactly as if it were produced by the original model. This is possible thanks to an intra-model consensus mechanism that ensures the predictive distribution of the original model is respected. An example of this is a customer support system that uses Orthrus to generate precise and fast responses. The system can answer complex questions without losing any of the nuances of the original text, significantly improving customer satisfaction.

How to Try It
#

To start using Orthrus, follow these steps:

Clone the repository: You can find the source code on GitHub at the following address: Orthrus GitHub. Clone the repository using the command git clone https://github.com/chiennv2000/orthrus.git.
Prerequisites: Make sure you have Python installed on your system. Additionally, it is recommended to use uv for faster dependency resolution. Install the necessary dependencies with the following commands:
```
uv pip install -e .
uv pip install ninja packaging
uv pip install flash-attn --no-build-isolation
```

Quickstart: Once the dependencies are installed, you can try Orthrus directly in Google Colab. Here is an example of code to get started:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

model = AutoModelForCausalLM.from_pretrained(
    "chiennv/Orthrus-Qwen3-8B",
    dtype=torch.bfloat16, device_map="cuda",
    attn_implementation="flash_attention_2",
    trust_remote_code=True,
).eval()
tokenizer = AutoTokenizer.from_pretrained("chiennv/Orthrus-Qwen3-8B")

prompt = "Write a program to count the frequency of each word in a paragraph."
messages = [{"role": "system", "content": ""}, {"role": "user", "content": prompt}]
input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True, enable_thinking=False).input_ids

output_ids = model.generate(
    input_ids=input_ids.to(model.device),
    max_new_tokens=2048,
    use_diffusion_mode=True,
    streamer=TextStreamer(tokenizer, skip_prompt=True)
)

Documentation: For further details, consult the main documentation available in the repository. There is no one-click demo, but the step-by-step guide will help you configure and use Orthrus effectively.

Final Thoughts
#

Orthrus represents a significant step forward in the field of natural language processing. Its ability to accelerate inferences on language models without compromising the quality of responses makes it a valuable tool for a wide range of applications, from scientific research to customer support. Positioning Orthrus within the broader tech ecosystem, we can see how this technology can revolutionize how we interact with data and information.

In a world increasingly dependent on fast and precise language processing, Orthrus offers a solution that not only meets current needs but also opens up new possibilities for the future. With Orthrus, we can envision a world where information is accessible and understandable in real-time, enhancing our ability to make informed decisions and solve complex problems. This is the potential of Orthrus, and we are excited to see how the tech community will leverage it to create even more extraordinary innovations.

Use Cases
#

Private AI Stack: Integration into proprietary pipelines
Client Solutions: Implementation for client projects
Development Acceleration: Reduction of time-to-market for projects

Resources
#

Original Links
#

GitHub - chiennv2000/orthrus: Fast, lossless LLM inference via dual-view diffusion… - Original Link

Article recommended and selected by the Human Technology eXcellence team, processed through artificial intelligence (in this case with LLM HTX-EU-Mistral3.1Small) on 2026-07-02 09:39 Original source: https://github.com/chiennv2000/orthrus

Summary #

Introduction #

What It Does #

Why It’s Amazing #

How to Try It #

Final Thoughts #

Use Cases #

Resources #

Original Links #

Related Articles #