IOMMU

IOMMU: Strategies for Mitigating the IOTLB Bottleneck

- from Sixth Annual Workshop on the Interaction between Operating Systems and Computer Architecture

- Nadav Ammit (@techion), Muli Ben-Yhhuda(@ibm), Ben-Ami Yassour(@ibm)

The computational cost of the mapping and unmapping of memory regions in the DMAR units is therefore considered the main bottleneck.

The IOMMU translates the virtual I/O address to the machine(host physical) address. To efficiently perform this task, an I/O Translation Lookaside Buffer(IOTLB) is included in the IOMMU. However, every IOTLB cache-miss presents high latency as it requires physical address resolution, which is performed by a page-walk through the DMAR paging hierarchy in the main memory. Thus, IOTLB implementation and its usage by operating systems may have significant impact on I/O throughput.

Translation Data Structures Definitions

Request ID (RID) - label identifying the source of a transaction

Address Translation Cache (ATC) - Local or remote coherent copy of address translations

I/O Translation Look aside Buffer (IOTLB) - A remote ATC that exists in a device associated with an IOMMU

Address Translation Services (ATS) - Extensions supporting remote caching of address translations

Page Directory Entry (PDE) - Translation table entry that points at a table

Page Table Entry (PTE) - Translation table entry that contains a translation

Root Translation table - Translation table at the top of translation hierarchy

Device Table - Maps Requester ID to root translation table

Process Address Space Identifier (PASID) - DMA requests with virtual address (gva) are tagged with a PASID value that identifies the targeted virtual address space. 

DMAR - DMA remapping
DRHD - DMA Engine Reporting Structure
RMRR - Reserved memory Region Reporting Structure
ZLR  - Zero length reads from PCI devices
IOVA - IO Virtual address.

process map /proc/$id/maps

ref: http://sunshout.tistory.com/1649

QEMU process number is 4254.
physical address of VM is mapped in 2aaaaac00000 - 2aab0ac00000 in QEMU's virtual address.

root@cnode24-m:/proc/4254# pwd /proc/4254 

root@cnode24-m:/proc/4254# more maps

2aaaaac00000-2aab0ac00000 rw-p 00000000 00:17 13954 /hugepage/libvirt/qemu/qemu_back_mem.pc.ram.XPQHB2 (deleted)

>>> int(0x2aab0ac00000) - int(0x2aaaaac00000)

1610612736 => Physical memory size of VM

>>> 1572864 * 1024

1610612736

<domain type="kvm">

  <name>dpdk</name>

  <memory>1572864</memory>