Working in a highly regulated corporate environment can often feel daunting, especially when it comes to connecting to the internet. Strict regulations, mandatory security inspections, proxy requirements, and self-signed certificates can all introduce unexpected challenges.
In such environment, I ran into a problem during the execution of the following pipeline on a self-hosted agent. One part of the pipeline is to fetch a secret from a specified Azure Key Vault.
Unfortunately, what should have been a straightforward task turned into a 30-minute troubleshooting session.
trigger:
- main
pool:
name: self-managed-dev
steps:
- task: AzureKeyVault@2
inputs:
azureSubscription: 'svc-azpipeline-vault-reader'
KeyVaultName: '<REDACTED>'
SecretsFilter: 'webhooktest'
RunAsPreJob: false
...Azure Pipeline Definition
...(omitted)
Downloading secret value for: webhooktest.
##[error]Error while trying to get OIDC token: Error: unable to get local issuer certificate
##[error]Error while trying to get OIDC token: Error: unable to get local issuer certificate
##[error]Error while trying to get OIDC token: Error: unable to get local issuer certificate
##[error]Error while trying to get OIDC token: Error: unable to get local issuer certificateAzure Pipeline Execution Output
To make matters worse, the error logs were not very informative. Unless you explicitly enable diagnostic logs, the output doesn’t provide much detail about what’s really happening under the hood.
Nevertheless, there’s a high chance that issues like this are related to TLS verification.
After same search that Azure pipeline agent uses nodejs technology and it could the be the reason that nodejs does NOT use system CA certificate bundle, which is /etc/pki/tls/certs/ca-bundle.crt for RHEL based hosts.
Following snippet of code also confirmed that, nodejs is not aware of any system ca-bundle location.
<Agentfolder>./node -e 'const tls = require("tls"); console.log(tls.rootCertificates);'
undefinedAccording to nodejs documentation there is environment variable to define additional ca-bundle certificates.
NODE_EXTRA_CA_CERTS=/etc/pki/tls/certs/ca-bundle.crtAfter updated the systemd service file definition and restart the service, on the self-hosted agent, it fetched the secrets successfully.
/etc/systemd/system/azure-pipeline-agent.service
[Unit]
Description=Azure Pipeline Agent
After=network-online.target
Wants=network-online.target
[Service]
User=1003
ExecStart=/home/user1/azure-agent/run.sh
Environment=NODE_EXTRA_CA_CERTS=/etc/pki/tls/certs/ca-bundle.crt
Environment=http_proxy="http://your-proxy:8080"
Environment=https_proxy="http://your-proxy:8080"
Environment=no_proxy="localhost,127.0.0.1"
[Install]
WantedBy=multi-user.targetEnvironment=NODE_EXTRA_CA_CERTS=/etc/pki/tls/certs/ca-bundle.crtsystemctl daemon-reload
systemctl restart azure-pipeline-agent.serviceStarting: AzureKeyVault
==============================================================================
Task : Azure Key Vault
Description : Download Azure Key Vault secrets
Version : 1.259.2
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-key-vault
==============================================================================
SubscriptionId: <REDACTED>
Key vault name: <REDACTED>
Downloading secret value for: webhooktest.
Finishing: AzureKeyVault
Leave a Reply