It's been a long holiday break!
I'm back with a new experience on DevSecOps. As you can see from the other posts, we will build up the elastic environments with Ansible script via Jenkins pipeline. So, the process with seed the initialize data such as credential for the admin or integration, this will raise a need to store those secret credential into a secure tool.
So I propose to use HashiCorp Vault to store the secret as a basic requirement but then we can use it for other purposes such as an internal Certificate Authority ...
sudo apt-get install unzip ca-certificates openssh jq
cd /tmp
curl -L https://releases.hashicorp.com/vault/1.0.2/vault_1.0.2_linux_amd64.zip -O
unzip vault_1.0.2_linux_amd64.zip
sudo mv vault /usr/local/bin/
vault -v
vault -autocomplete-install
complete -C /usr/local/bin/vault vault
We will setup Vault running as a systemd service with non-root user so we have to enable mlock syscall for non-root user
sudo setcap cap_ipc_lock=+ep /usr/local/bin/vault
I want to store the data into my service user home directory so I setup the Vault service that allow to access to /home with a server file in /etc/systemd/system/vault.service
[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
[Service]
User=<service_user>
Group=<service_user>
ProtectSystem=full
#ProtectHome=read-only
ProtectHome=off
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitIntervalSec=60
StartLimitBurst=3
[Install]
WantedBy=multi-user.target
Create the service directory and add permission to service user
sudo mkdir /etc/vault.d
sudo chown --recursive <service_user>:<service_user> /etc/vault.d
If you want to use TLS on your Vault server for REST API, then create a certificate for it. My case, I use self-sign certificate by using (create 10 years certificate :P)
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout /etc/vault.d/ssl/vault-selfsigned.key -out /etc/vault.d/ssl/vault-selfsigned.crt
Create the vault configuration file in the service directory
ui = true
storage "file" {
path = "/home/<service_user>/data/vault/data/" # make sure this directory exists
}
listener "tcp" {
address = "127.0.0.1:8200"
tls_disable = 1
}
listener "tcp" {
address = "<vault_FQDN>:8200"
tls_cert_file = "/etc/vault.d/ssl/vault-selfsigned.crt"
tls_key_file = "/etc/vault.d/ssl/vault-selfsigned.key"
}
api_addr = "https://<vault_FQDN>:8200"
Make sure service user has proper permission on the configuration file
sudo chmod 640 /etc/vault.d/vault.hcl
Then start and enable vault service at boot time
sudo systemctl enable vault
sudo systemctl start vault
systemctl status vault
To init the vault, with Shamir Secret Sharing mechanism, with ONE key share and ONE threshold (more than one key share will need more than one threshold). The unseal key and root token is generated once per machine/installation so you have to keep this information for later use in a secret place (this is why they use the Shamir's Secret Sharing to properly keep this initial data). Then use the unseal key to unseal the Vault
export VAULT_ADDR=http://127.0.0.1:8200
vault operator init -key-shares=1 -key-threshold=1
vault operator unseal
Then you will use the root token to do the initialization. To logout with root token, delete the token helper at ~/.vault-token
vault login <root_token>
To seal the Vault, lock down all access and clean up all encrypted data in memory. After the vault service restart it also seal the Vault
vault operator seal
Below is my simple solution of using Vault
Note, with the TLS enable for http request with self-sign certificate, you will have to do the configuration differently based on the client.
Below session will show you how to deal with self-sign certificate
To retrieve the root/public certificate from Vault server to vault-ss.crt
echo quit | openssl s_client -connect <vault_server>:8200 > vautl-ss.crt
Depend on your client, you have to setup the self-sign differently in order to be in trusted store for the handshake transaction
Bash
I'm back with a new experience on DevSecOps. As you can see from the other posts, we will build up the elastic environments with Ansible script via Jenkins pipeline. So, the process with seed the initialize data such as credential for the admin or integration, this will raise a need to store those secret credential into a secure tool.
So I propose to use HashiCorp Vault to store the secret as a basic requirement but then we can use it for other purposes such as an internal Certificate Authority ...
Installation
The guideline from HashiCorp is quite good, you can follow its detail with the summary as below:- Install unzip, ca-certificates, openssh, jq
- Download Vault from Download page then unzip it
- Move the extracted executable file to the system $PATH, such as: /usr/local/bin
- Test the vault executable with: vault -v
- Use autocomplete installation
sudo apt-get install unzip ca-certificates openssh jq
cd /tmp
curl -L https://releases.hashicorp.com/vault/1.0.2/vault_1.0.2_linux_amd64.zip -O
unzip vault_1.0.2_linux_amd64.zip
sudo mv vault /usr/local/bin/
vault -v
vault -autocomplete-install
complete -C /usr/local/bin/vault vault
Configuration
I only setup the system with simple configuration so please research more in case you need to have a comprehensive configuration.We will setup Vault running as a systemd service with non-root user so we have to enable mlock syscall for non-root user
sudo setcap cap_ipc_lock=+ep /usr/local/bin/vault
I want to store the data into my service user home directory so I setup the Vault service that allow to access to /home with a server file in /etc/systemd/system/vault.service
[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
[Service]
User=<service_user>
Group=<service_user>
ProtectSystem=full
#ProtectHome=read-only
ProtectHome=off
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitIntervalSec=60
StartLimitBurst=3
[Install]
WantedBy=multi-user.target
Create the service directory and add permission to service user
sudo mkdir /etc/vault.d
sudo chown --recursive <service_user>:<service_user> /etc/vault.d
If you want to use TLS on your Vault server for REST API, then create a certificate for it. My case, I use self-sign certificate by using (create 10 years certificate :P)
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout /etc/vault.d/ssl/vault-selfsigned.key -out /etc/vault.d/ssl/vault-selfsigned.crt
Create the vault configuration file in the service directory
ui = true
storage "file" {
path = "/home/<service_user>/data/vault/data/" # make sure this directory exists
}
listener "tcp" {
address = "127.0.0.1:8200"
tls_disable = 1
}
listener "tcp" {
address = "<vault_FQDN>:8200"
tls_cert_file = "/etc/vault.d/ssl/vault-selfsigned.crt"
tls_key_file = "/etc/vault.d/ssl/vault-selfsigned.key"
}
api_addr = "https://<vault_FQDN>:8200"
Make sure service user has proper permission on the configuration file
sudo chmod 640 /etc/vault.d/vault.hcl
Then start and enable vault service at boot time
sudo systemctl enable vault
sudo systemctl start vault
systemctl status vault
Initialization
With the configuration above, we turned off the TLS on local call so we have to set the environment variable $VAULT_ADDR=http://127.0.0.1:8200 because this variable has default value to https. Permanent solution is to add this variable into ~/.profile.To init the vault, with Shamir Secret Sharing mechanism, with ONE key share and ONE threshold (more than one key share will need more than one threshold). The unseal key and root token is generated once per machine/installation so you have to keep this information for later use in a secret place (this is why they use the Shamir's Secret Sharing to properly keep this initial data). Then use the unseal key to unseal the Vault
export VAULT_ADDR=http://127.0.0.1:8200
vault operator init -key-shares=1 -key-threshold=1
vault operator unseal
Then you will use the root token to do the initialization. To logout with root token, delete the token helper at ~/.vault-token
vault login <root_token>
To seal the Vault, lock down all access and clean up all encrypted data in memory. After the vault service restart it also seal the Vault
vault operator seal
Solution
The Vault itself has provide all the concept for storing secret in secure mode, however, we have to define a way to use it because you have to be a trusted entity in order to access or query the Vault even to get the token to access it. Refer to this discussion for more infoBelow is my simple solution of using Vault
- Secret data will be stored in key/value secret engine (version 1)
- Authenticate to Vault server with AppRole authentication method
- Use token_bound_cidrs to bind the app server IP ranges to the AppRole (so the approle token can be used on those valid IPs)
- Use response wrapping for all responses from Vault to app server.
So the component diagram will be
Implementation
Setup Data
Create secret data into default kv engine at secret path with CLI
vault kv put secret/<role_name>/<secret1> <key1>=<value1>
vault kv put secret/<role_name>/<secret2> <key1>=<value1> <key2>=<value2>
You can delete a secret item with
vault kv delete secret/<role_name>/<secret1>
PS: I setup in path secret/<role_name>/<item> to categorize the secrets of multiple roles, you can define the path base on your need.
Setup policy, role
Then now, create the role access to these secret. Our assumption that this role is to read the secret only so we will create a read-only policy to the <role_name> path then create an AppRole with that policy.
Create a policy file (role-read-policy.hcl) in /etc/vault.d
path "secret/<role_name>/*" {
capabilities = ["read", "list"]
}
Format the policy file to hcl syntax then create Policy and verify it
vault policy fmt /etc/vault.d/<role>-read-policy.hcl
vault policy write <role>-read /etc/vault.d/<role>-read-policy.hcl
vault policy list
vault policy read <role>-read
Create an AppRole with that policy
vault auth enable approle
vault write auth/approle/role/<role_name> policies="<role>-read,default" -token_bound_cidrs=0.0.0.0/24
vault read auth/approle/role/devops/role-id
The Vault CLI is a wrapper for the API so you can find more detail with the API call.
In order to have the secretID of the created AppRole, you have to be a trusted entity to Vault, so we will create a "public-trusted" token to assign to the app server as a signature of trusted. So we will create a policy that can read the roleID and write/create secretID of the <role_name> then create a "long-live" token for this policy and put it into the app servers. Because the AppRole <role_name> is bound with with the app server IP so even people can get the "public" token they can't use the AppRole token from other machine, except they use it on app server (this case the app server is stealth)
Create a policy file (role-base-policy.hcl) in /etc/vault.d
path "auth/approle/role/<role_name>/secret-id" {
capabilities = ["create", "read", "update", "list"]
}
path "auth/approle/role/<role_name>/*" {
capabilities = ["read", "list"]
}
Then create a public token with this policy
vault token create -policy=<role>-base -period=8760h
Then we will push this public token to the app server
Usage
From the app server, we will use REST API to trigger the call to Vault server
- Use the public token to retrieve roleID and secretID
- Use the roleID and secretID to login with AppRole to retrieve the wrapped token
- Unwrap the token then use it for secret query
curl -X GET --header "X-Vault-Token: <base_token>" "https://<vault_server>:8200/v1/auth/approle/role/<role_name>/role-id"
curl -X POST --header "X-Vault-Token: <base_token>" "https://<vault_server>:8200/v1/auth/approle/role/<role_name>/secret-id"
curl -X POST --header "X-Vault-Wrap-TTL: 2m" --data '{"role_id": "<roleID>", "secret_id": "<secretID>"}' "https://<vault_server>:8200/v1/auth/approle/login"
curl -X POST --header "X-Vault-Token: <wrapped_token>" "https://<vault_server>:8200/v1/sys/wrapping/unwrap"
curl -X GET --header "X-Vault-Token: <approle_token>" "https://<vault_server>:8200/v1/secret/<role_name>/<secret>"
Note, with the TLS enable for http request with self-sign certificate, you will have to do the configuration differently based on the client.
Below session will show you how to deal with self-sign certificate
To retrieve the root/public certificate from Vault server to vault-ss.crt
echo quit | openssl s_client -connect <vault_server>:8200 > vautl-ss.crt
Depend on your client, you have to setup the self-sign differently in order to be in trusted store for the handshake transaction
Bash
- Copy the root certificate to /usr/local/share/ca-certificates
- Update ca-certificate
- Or use curl with specific certificate
sudo cp vault-ss.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
# or
curl --cacert -X POST ...
Python
There are many http module, which I will use requests module as the sample below
httpResponse = requests.get(vaultAPI, headers=headers, verify=vaultCert)
Java
Java application will use the trusted keystore at $JAVA_HOME/jre/lib/security/cacerts
- Read JAVA_HOME
- List all key from keystore
- Import keystore
# to find the $JAVA_HOME
readlink -f $(which java)
# to view all certificates in trusted store
keytool -list -keystore $JAVA_HOME/jre/lib/security/cacerts
# to add certificate into trusted store
keytool -import -noprompt -trustcacerts -alias <Alias> -file <cert_file> -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit # by default storepass is changeit
Please note, I have not succeeded to use this keystore for Jenkins application. Because for my case, I use this implementation from Jenkins and the build console output will print out all the command so it will list the <base_token> into the console which is not good at all. So I use a python script which is an out-process from Jenkins then I can control which will be in the console output.
Alright, this is my 3 weeks studying, hope I can save you a bit time and have a workable sample here.
PS: This is another full load sample which is similar with the guideline from HashiCorp with a bit more in depth
Comments
Post a Comment