A combination of Ab Initio for data manipulation and complementary cloud tooling allowed for exit from a legacy data platform.

Legacy data storage appliances take up valuable data centre space and demand even more already-stretched support staff, leading many to consider cloud data storage for large volumes of data that must be retained for regulatory purposes. 

AWS Snowball was used to migrate on-prem trading systems data. When data reaches the cloud the containerised Ab Initio process consumes that data, performs data quality validation, and writes back out in a modern big data format (Parquet). This bulk movement of ‘stock’ data allowed for swift exit of the existing platform, generating significant cost savings. 

Post stock movement, the ‘flow’ data begins – now data is moved using typical secure transfer mechanisms and validated before adding to the stores built in S3. Controls built into the Ab Initio process provide extra levels of risk control by ensuring that the data is both complete and accurate.

Utilising a generic containerised Ab Initio application enables the scaling of the data ingestion, not only from a context of data volume within feed but also across feeds. Metadata capture for feed description as well as eventual data lineage with data quality overlays are all built once and deployed many times.