HashiCorp Vault Disk Inode Is Full!
Recently, I had very interesting issue with Vault that, it stopped functioning two hours after integration with Redfish Bare-Metal Host Monitoring which uses
Approle to get iLO user and credential.
Vault audit logs shows that Approle for redfish monitoring requested heavly in a second. It might be still normal for around 80 Physical servers polling, but something is still not quite okay. After some investigation and tests, it was found that File system Inode for Vault file backend run out of space.
df -hi /vault/storage Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/rootvg-lv_vault 88K 88K 0 100% /vault/storage
Root Cause(Incorrect Token Type)
It was simply usage of incorrect token type. Creation of new client tokens(service token) were so quickly than the removal of expired tokens.
Tokens are core method for authentication within the Vault. There are two token types in Vault.
Batch tokens. Service tokens are persistent, they must be replicated across the primary and secondary clusters. On the other hand, batch tokens are neither persisted to disks nor live in memory. Usage of batch tokens reduce stress on the Vault storage backend and improve the performance.
token_typeis not specified, vault creates
servicetype token by default.
vault write auth/approle/role/monitor policies="redfish" \ token_type="batch" \ token_ttl="1m" \ secret_id_ttl=0 \ secret_id_num_uses=0