TiTok: An Innovative AI Method for Tokenizing Images into 1D Latent Sequences

Modern image generation models utilize standard image tokenizers and de-tokenizers in a similar manner to how generative language models use tokenizers and de-tokenizers. The tokenizers compress high-dimensional images into low-dimensional latent representations, which are then decoded back into images by the de-tokenizers. This process helps reduce computational demands compared to directly processing pixels and enhances the effectiveness and efficiency of the generation process.

In the context of image generation, the tokenizers transform images into latent representations, and the de-tokenizers decode these representations back into images. This process is based on the assumption that the latent space should retain a 2D structure to maintain a direct mapping for locations between the latent tokens and image patches. However, this design can limit the tokenizer's ability to effectively leverage the redundancy inherent in images to cultivate a more compressed latent space.

One example of a standard image tokenizer and de-tokenizer is the Vector-Quantized (VQ) tokenizer. It is widely used due to its broad applicability across various domains, including image and video generation, large-scale pretraining, and multimodal models. A typical VQ model contains three key components: an encoder, a decoder, and a codebook with a predefined number of embeddings. The encoder maps the input image into a latent space, and the decoder reconstructs the image from the latent space. The codebook, consisting of a set of discrete latent codes, plays a crucial role in preserving the semantic information of the input image.

In summary, modern image generation models utilize standard image tokenizers and de-tokenizers to transform images into latent representations and decode them back into images, enabling more efficient and effective image generation.

The paper discusses three existing methods in the realm of image processing and understanding:

1. Image Tokenization: This is a fundamental approach that has been used since the early days of deep learning. Autoencoders are utilized to compress high-dimensional images into low-dimensional latent representations and then decode them back. This method has been widely used for tasks such as image compression, denoising, and colorization.

2. Tokenization for Image Understanding: This approach is used for image understanding tasks such as image classification, object detection, segmentation, and multimodal large language models (MLLMs). It involves tokenizing images into 2D grids of tokens, which are then processed by transformer models for various downstream tasks. This method has shown great success in tasks such as image classification, object detection, and segmentation.

3. Image Generation: Methods for image generation have evolved from sampling variational autoencoders (VAEs) to utilizing generative adversarial networks (GANs), diffusion models, and autoregressive models. These methods have shown remarkable progress in generating high-quality, realistic images. However, they typically require a large number of image tokens, which can make the generation process computationally expensive.