Triad improve the resilience and flexibility of BEIS Corporate Services Data Management System
Working with BEIS Corporate Services, our engineering practices improved the resilience of their Data Management System to serve a wide variety of organisations, with issues being identified and fixed up to five times quicker than previously.
About the client
BEIS Corporate Services (previously BEIS Digital) now operates across the Department for Energy Security and Net Zero (DESNZ) and Department for Science, Innovation and Technology (DSIT).
The challenge
Various organisations, including Local Authorities and energy suppliers, use the DMS (Data Management System) website. It allows them to upload data that can be processed to produce reports for policy teams. The client recognised it needed a resilient best practices website that could be easily managed. Our brief was to improve the engineering practices and code base to resolve two major problems:
- Reduce the time and skill set required to onboard new data ingestion schemes.
- Add the ability to detect and reduce the time to resolve issues and improve the resilience of the deployment of the code base.
The solution
System issues were hard to detect, so we added application monitoring and alerting. This enabled the DMS team to be alerted of any issues by email to resolve them quickly and proactively.
We then created modern continuous integration and continuous deployment (CI/CD) pipelines with staged deployment. This allowed us to redeploy the system to an earlier version, if required.
The system wasn’t capable of ingesting large quantities of data. We solved this by adding a new ingestion mechanism using SAS Tokens for a Storage Account so that large files can be uploaded automatically. This enabled schemes to programmatically upload their data as multiple files on a schedule, which was not previously possible.
We changed the code base to be metadata driven to improve the time taken to create new schemes. Creating a new ingestion scheme is now simply a matter of adding new metadata, not writing new code. All UI configuration is now stored in the database and is used to tailor the number of files to upload and their validation.
The result
We have created a best practice system that is fit for purpose:
- Application monitoring and alerting have enabled the team to resolve issues within hours rather than days.
- We couldn’t previously upload large data sets. This is now easily managed.
- New schemes can be added over five times quicker, taking hours rather than weeks.
- Since no new code needs to be written, the quality of the code base can be guaranteed when a new scheme is introduced, reducing the test time. This has enabled multiple schemes that ingest Energy Price Guarantee data from Energy Suppliers to be brought online in very tight timescales.
- An Azure infrastructure was designed using Terraform from Azure DevOps pipelines described in YAML files for rapid deployment.
“I have been impressed with Triad’s proactive approach. They have brought a “can-do” attitude. Critically, they recognised the importance of keeping development on track whilst ensuring that new schemes are still delivered.”
Gavin Dollin, Deputy Director, D3, ICS Digital
