Microsoft drops 'MInference' demo, challenges status quo of AI processing

MInference technology is a method developed by Microsoft to accelerate the processing speed of large language models. It focuses on improving the "pre-filling" stage of language model processing, which becomes a bottleneck when dealing with long text inputs. MInference can reduce processing time by up to 90% for inputs of one million tokens while maintaining accuracy.

MInference improves processing speed by using dynamic sparse attention to optimize the inference process for large language models. It identifies three distinct attention patterns (A-shape, Vertical-Slash, and Block-Sparse) and builds sparse indices for these patterns during inference, significantly reducing latency without altering pre-training or fine-tuning. This method effectively reduces inference latency by up to 10x for pre-filling on an A100 GPU while maintaining accuracy.