01

Gmail

10:23 AM

71%

Jeannine Meyers

Done

q

w

e

r

t

y

u

i

o

p

a

s

d

f

g

h

j

k

l

z

x

c

v

b

n

m

123

space

return

cached

prefill

decode

decode

KV vectors

The

quick

brown

fox

jumps

over

prompt

output

The quick brown fox

jumps

look

eats

jumps

fox

brown

quick

The

fox

brown

quick

The

jumps

fox

brown

quick

The

over

VLA

ViT

Projector

LLM

Action Head

Emb

LLM

LLM

Emb

VLM

ViT

Projector

LLM

Emb

Image patches

Language input

Input state

Output actions

Typical VLA Architecture

ViT

Projector

LLM

Action Head

Emb

Image patches

Language input

Input state

Output actions

Notable VLA Models

RT-2 (Google)

PaLM-E backbone, web-scale pre-training, 55B parameters

OpenVLA

Open-source 7B VLA, Prismatic ViT + Llama 2 backbone

π₀ (Physical Intelligence)

Flow matching action head, pre-trained across 7 robot types

GR00T (NVIDIA)

Foundation model for humanoid robots, multimodal I/O

Octo (Berkeley)

Transformer-based, generalizes across embodiments

SmolVLA (LeRobot)

Sub-billion foundation VLA model

02

Edge AI runs machine learning models directly on local devices – smartphones, robots, cameras – instead of relying on remote cloud servers.

Cloud AI

kW to MW per rack

100s of GB HBM

10B to 1T params

BF16, FP8

vLLM, CUDA, XLA

Training, LLM serving

Edge AI

5 to 300 W

4 to 64 GB LPDDR

0.1B to 10B params

INT8, INT4, ternary

TensorRT, ggml, NPU

Robotics, ADAS, AR/VR

TinyML

0.1 to 100 mW

32 KB to 2 MB SRAM

under 1M params

INT8, binary, no FPU

TFLM, CMSIS-NN

Wake word, anomaly

SGLang

Remember the prefix

A serving engine built around RadixAttention: an LRU radix tree over KV blocks that shares cached prefixes across requests, not just within one.

vLLM

The throughput default

A GPU serving engine whose core idea, PagedAttention, treats KV cache like virtual memory so continuous batching can pack hundreds of concurrent requests without fragmentation waste.

llama.cpp

Inference any where

Took large-model inference off the GPU and onto commodity CPUs through quantized kernels, putting local LLMs within reach of anyone with a laptop.

Unsloth

Low-cost training

Cut fine-tuning memory and time by a large factor with custom kernels and PEFT, so a single consumer card can train models that once needed a cluster.

No dedicated engine for VLA deployment (yet)

Optimizing LLM and optimizing VLA share common techniques

LLM

KV cache grows.

Weights don’t fit.

Decode is serial.

VLA

Hard real time.

Errors compound

Batch size is 1.

Not all LLM techniques work with VLA

03

4x

Compression

Quantization

Reduce precision from FP32 to INT8/INT4. Up to 4x smaller models with minimal accuracy loss. Post-training & quantization-aware training.

50– 90%

Sparsity

Pruning

Remove redundant weights and neurons. Structured pruning removes entire channels for hardware-friendly speedup.

10x

Smaller Model

Knowledge Distillation

Train a small ‘student’ model to mimic a large ‘teacher’. Captures complex behaviors in a lightweight architecture.

Faster

Implementation

Kernel Optimization

Kernel optimization is more hardware-constrained and usually speeds up the computation without changing the math.

Quantization reduces the numerical precision of model weights and activations — from 32-bit floats to 8-bit, 4-bit, or even 1.58-bit ternary — shrinking model size and accelerating computation on edge hardware.

S

E

E

E

E

E

E

E

E

M

M

M

M

M

M

M

M

M

M

M

M

M

M

M

M

M

M

M

M

M

M

M

IEEE FP32 (single)

IEEE FP16 (half)

Google BF16

NVIDIA FP8 (E4M3)

NVIDIA FP4 (E1M2) INT4

Continuous Signal

Quantized Signal

Signal

Quantization Error

time

Pruning identifies and removes weights, neurons, or entire channels that contribute least to model output. The result is a sparser, smaller network that runs faster — especially when combined with hardware-aware structured pruning.

Unstructured Pruning

Structured Pruning

Train Full model

Score Rank importance

Prune Remove low-rank

Fine-tune Recover accuracy

A large, accurate “teacher” model transfers its learned knowledge to a smaller, faster “student” model. The student trains on the teacher’s soft probability outputs (logits), capturing richer information than hard labels alone. The goal of knowledge distillation is to align the class probability distributions from teacher and student networks.

Input

Logits

Logits

Label

Distillation Loss

Classification Loss

Teacher Network

Student Network

Teacher not only gives answer, but also teaches knowledge.

Kernel optimization is more hardware-constrained and usually speeds up the computation without changing the math. Common techniques: tiling, grid layout, memory access

=

+

*

+

*

+

*

+

*

C[0:2,0:2]

C[0:2,0:2]

A[0:1,:]

B[:,0:1]

A[1:2,:]

B[:,0:1]

A[1:2,:]

B[:,1:2]

A[0:1,:]

B[:,1:2]

Example of MAC:

C = C+A*B

(A,B,C are 4x4 matrices)

Kernel optimization is more hardware-constrained and usually speeds up the computation without changing the math. Common techniques: tiling, grid layout, memory access

=

+

*

+

*

C[0:2,0:2]

C[0:2,0:2]

A[0:2,0:2]

B[0:2,0:2]

A[2:4,0:2]

B[0:2,2:4]

Example of MAC: (block matrix-multiplication)

C = C+A*B

(A,B,C are 4x4 matrices)

04

Github: https://github.com/VinRobotics/vla.cpp

Visualization: https://fai-modelopt-tech.github.io/learn-vla-cpp/

HuggingFace: https://huggingface.co/collections/vrfai/vlacpp-model-bundles

Paper: https://arxiv.org/abs/2606.08094

Star for us!

>10

Architecture

Support various SOTA VLAs

From foundation models like Gr00t families, Pi families, SmolVLA to efficient policies like VLA-Adapter, Evo1, BitVLA

>5

Platforms

From Consumer GPU to Edge

NVIDIA products: RTX 30x, 50x, AGX Orin, Orin Nano. non-NVIDIA products: Intel Arc GPU, Apple Mac Mini.

25%

Compression

New quantization schemes

Reduce precision from FP32 to INT8/INT4. 25% smaller models with minimal accuracy loss. New quantization scheme: Per-row Quantization

CPU

Deployment

Extreme deployment

Deploying VLA with CPU is possible with vla.cpp. Tested with MAC M4, Ultrabook with AMD Ryzen7 and Laptop with Intel Core I9

Robot Demonstrations

FOCA adaptation

Explicit future learning

Implicit future learning

Finetuning

Video World Model (optional)

Context Decoder

Latent Embedding Prediction

Projector

Synthetic

New rollouts

VLM Layer

VLM Layer i

Self-Attention

Cross-Attention

Self-Attention

Cross-Attention

Action Expert

x N

DiT Blocks

Pooling

Concatenate

Projector

Projector

Vision Encoder

Vision Encoder

Text Tokenizer

Representative explicit tokens

Representative implicit tokens

Robot State

Noised Action

“put the bowl ....”

Patch Embedding

focavla.github.io

Internal Behavior of Pruned model vs Base Model

Distribution of hidden state PCA

Vision-language-action

model

CKA-guided Pruning

33%-66% Transformer layers

30%-50% Training and inference time

~30% FLOPs

Block K

Layer N

Layer i

Layer 2

Layer 1

Layer N

Layer i

Layer 2

Layer 1

Layer K

Layer 1

Layer K

Layer 1

Fine-tune pruned model

High CKA similarity block

Before Finetuning

After Finetuning

Base model

CKA-guided pruned model

Base model

CKA-guided pruned model

banana to pot

pull napkin

Single-arm ALOHA

SIMULATION

UR5

UR10e

Screwdriver to basket

Bimanual ALOHA

3 simulations 10 real-world tasks 4 robot embodiments

fold short

pick up ...

Pull napkin ...

Batched data

Sampled

clpvla.github.io

github.com/anindex/polystep

Visualization by: https://vietngth.github.io/polystep-visualization/

VLA

Gr00t-N1.6-3B

5Hz

Inference

INT8

Quantization

95%

Quality

Finetuned by IsaacGr00t. Deploy with ALOHA with Jetson AGX Orin

95%

Success Rate

17GB

Training VRAM

480M

Parameter

57ms

Inference Time

1-NFE INFERENCE (DEPLOYMENT)

1-Step Denoising

Gaussian Noise X₁

Predicted Action X̂₀

forward pass. No iteration needed.

2.5x speedup

Tested with SmolVLA. Figure borrowed from SnapFlow

05

Star –> fork –> do the exercises -> commit

05

Contact: https://www.linkedin.com/in/khanhnd185