HashiCorp Vault Disk Inode Is Full!

Recently, I had very interesting issue with Vault that, it stopped functioning two hours after integration with Redfish Bare-Metal Host Monitoring which uses Approle to get iLO user and credential.

Vault audit logs shows that Approle for redfish monitoring requested heavly in a second. It might be still normal for around 80 Physical servers polling, but something is still not quite okay. After some investigation and tests, it was found that File system Inode for Vault file backend run out of space.

df -hi /vault/storage
Filesystem                   Inodes  IUsed   IFree  IUse%    Mounted on
/dev/mapper/rootvg-lv_vault   88K      88K    0      100%    /vault/storage

Root Cause(Incorrect Token Type)

It was simply usage of incorrect token type. Creation of new client tokens(service token) were so quickly than the removal of expired tokens.

Solution(Batch Token)

Tokens are core method for authentication within the Vault. There are two token types in Vault. Service and Batch tokens. Service tokens are persistent, they must be replicated across the primary and secondary clusters. On the other hand, batch tokens are neither persisted to disks nor live in memory. Usage of batch tokens reduce stress on the Vault storage backend and improve the performance.

vault write auth/approle/role/monitor policies="redfish" \
    token_type="batch" \
    token_ttl="1m" \
    secret_id_ttl=0 \
    secret_id_num_uses=0