Introducing Falcon2: Next-Gen Language Model by TII

The Falcon-2-11B Vision Language Model (VLM) is a groundbreaking model that adds image understanding and multi-modularity to the Falcon language model2. This allows the model to handle both text and image inputs, making it capable of tasks such as image-to-text conversation. The VLM has been trained on a large-scale dataset and is able to generate detailed responses based on the visual content of an image, in addition to understanding textual prompts. This makes it a powerful tool for a variety of applications, including those in healthcare, finance, e-commerce, education, and legal sectors. The model is available under the TII Falcon License 2.0, a permissive Apache 2.0-based software license that promotes the responsible use of AI.

Setting up a Python virtual environment to use the Falcon-2 models for inference involves several steps. Here's a step-by-step guide:

1. Create a new Python virtual environment:

   • Open your terminal or command prompt.
   • Navigate to the directory where you want to create your Python project2.
   • Run the following command to create a new virtual environment:

     python -m venv venv
     

   • This command creates a new virtual environment named "venv" in the current directory.

2. Activate the virtual environment:

   • To activate the virtual environment, use the appropriate command for your operating system:
     • On Windows: venv\Scripts\activate
     • On macOS and Linux: source venv/bin/activate
   • After activating the virtual environment, your terminal prompt will be prefixed with the name of the virtual environment (e4.g., (venv)).

3. Install required libraries and dependencies:

   • With the virtual environment activated, you can install the required libraries and dependencies using the Python package manager (pip).
   • Create a "requirements.txt" file with the following content:

     accelerate # For distributed loading
     bitsandbytes # For Quantization
     torch # Used by HuggingFace
     transformers # To load pipelines and models
     Pillow # Basic Loading and Image Processing
     requests # Downloading image from URL
     

   • Save the "requirements.txt" file in your project directory.
   • Run the following command to install all the dependencies:

     pip install -r requirements.txt
     

4. Load the Falcon-2 model:

   • With the virtual environment activated and the required libraries installed, you can now load the Falcon-2 model.
   • Run the following code to load the model:

     from transformers import LlavaNextForConditionalGeneration, LlavaNextProcessor
     
     processor = LlavaNextProcessor.from_pretrained(
         "tiiuae/falcon-11B-vlm",
         tokenizer_class='PreTrainedTokenizerFast'
     )
     model = LlavaNextForConditionalGeneration.from_pretrained(
         "tiiuae/falcon-11B-vlm",
         quantization_config=quantization_config,
         device_map="auto"
     )
     

Now you have set up a Python virtual environment and loaded the Falcon-2 model for inference. You can start using the model to generate responses.