Systems and methods for training and inference of large multimodal models

Embodiments described herein provide a method of performing a vision-language task by a neural network multimodal model in response to multiple input images, the method comprising: receiving, via a data interface, a text input and an image input; generating text tokens based on the text input; generating a plurality of image patches, wherein each image patch of the plurality of image patches includes a portion of the image input at substantially a same resolution as the image input; generating a downsized image based on a downsizing of the image input; generating vision tokens based on the plurality of image patches and the downsized image; generating, via a neural network based language model, an output based on the text tokens and the vision tokens; and updating parameters of the neural network based language model based on a loss objective based on the output.