The IBD happens only after identifying the most work chain, which involves downloading all historical headers. So that does include a cumulative offset at every height.
In the future we could support a compressed format for a range of headers. That would include only one cumulative offset for an arbitrarily large range of headers.