sharding strategies (tensor, layer, context, expert)

GPU all-reduce communication

Each sharding axis determines which parameters reside on which device and therefore the required inter-GPU communication pattern.

1 source videos