It’s an age-old problem in IT, you have a bunch of databases in various environments, and you need to be able to securely store the parameters required to access those databases. Now I am sure everyone has at some point stored such information in a spreadsheet or a text file or somewhere equally as insecure. There really is no excuse these days to store this anywhere other than a secure storage area. Now passwords are well covered with the advent of Password vaults we can securely store passwords and even have them change periodically without human intervention. But what about those other parameters, Database name, Username etc. where do we store these? 

Well, if you are on AWS there is a built-in solution to this problem called AWS Secrets Manager. 

AWS Secrets Manager is a secure password and parameter vault. It allows you to store passwords, keys, and other parameters. As with everything AWS there is a price attached, but with it costing $0.40 per secret and $0.05 per 10,000 API calls using it is not going to break the bank. 

Storing your secrets 

Creating a new secret store on AWS is simple, first, you navigate to the Secrets Manager service and once on that page, you click the “Store a new secret” button.  Then you can select the “Other type of secret” from the list of Secret Types. This then allows you to create and store Key/Value pairs in Secrets Manager. 

You enter your Key/values pairs, then click Next. On the next page, you need to give your new secret a name. Other than that, you can keep everything else on the default settings.  

Accessing the secret 

We have now created our secret; how do we access it? 

Well, there are a few different ways but for our purposes, we are going to use the AWS command line interface.  

In this utility we use it as follows:  

The AWS CLI returns the secret, assuming you have access, in json format. 

As you can see the output of the command is a JSON string wrapped within a JSON string. The part we are interested in is the SecretString as this contains the secret entries, we entered in Secrets Manager. 

You could parse this output using the programming language of your choice. We have used Ab Initio for our solution as ultimately, we want to add these parameters into a Parameter Set (pset) to be used by our Ab Initio code. One of the quickest ways in Ab Initio to create DML that describes JSON is to use the Ab Initio built-in utility json-to-dml. You pass in a file with the json in it and it will write out DML to describe the json: 

Creating a Pset from our Secret 

To make use of our parameters in an Ab Initio graph or plan we must create a Parameter Set (pset). In this example, I will discuss how you can create a Sandbox pset or a Configuration pset using the key/value pairs stored in our secret.    

When we add parameters to a Configuration or Sandbox pset we are overriding the value of a parameter that already exists. You must bear in mind that we are not creating the parameter and if we attempt to set the value of a parameter that does not exist, we will get an error and the parameter override will not get created. The parameter must then already exist in the project or one of the common projects included in the project to be overridden. 

The choice of whether to use a Configuration pset or the Sandbox pset comes down to where your secrets are stored. If you are storing your secrets in your development environment, then the Configuration Pset option could be preferable. The main difference between a Configuration pset and a Sandbox pset is that a Configuration pset is intended to be checked into the EME and promoted to other environments. Whereas a Sandbox pset cannot be checked in or promoted but must be authored in each environment.  

No matter which type of pset you want to create the method of creating them is the same, you must use the Ab Initio “air sandbox parameter” command for example: 

In conclusion 

This document has given a brief overview of how you can use the AWS Secrets Manager to securely store and retrieve essential parameters and then use them in Ab Initio. If you need any assistance with this or anything else Cloud and Ab Initio related, then please do not hesitate to contact us at info@datavium.com.