Recently, I had very interesting issue with Vault that, it stopped functioning two hours after integration with Redfish Bare-Metal Host Monitoring which uses Approle to get iLO user and credentials.
Vault audit logs shows that Approle for redfish monitoring requested heavy in a second. It might be still normal for around 80 Physical servers polling, but something is still not quite okay. After some investigation and tests, it was found that File system Inode for Vault file backend run out of space.
df -hi /vault/storage
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/rootvg-lv_vault 88K 88K 0 100% /vault/storageRoot Cause(Incorrect Token Type)
It was simply usage of incorrect token type. Creation of new client tokens(service token) were so quickly than the removal of expired tokens.
Solution(Batch Token)
Tokens are core method for authentication within the Vault. There are two token types in Vault. Service and Batch tokens. Service tokens are persistent, they must be replicated across the primary and secondary clusters. On the other hand, batch tokens are neither persisted to disks nor live in memory. Usage of batch tokens reduce stress on the Vault storage backend and improve the performance.
Note: if token_type is not specified, vault creates service type token by default.
vault write auth/approle/role/monitor policies="redfish" \
token_type="batch" \
token_ttl="1m" \
secret_id_ttl=0 \
secret_id_num_uses=0
Leave a Reply