This page provides some background to how we calculate the probability of triads throughout the winter period.
We have been working on the triad forecasting methodology since 2009 when a client asked for our help. However, our experience of triads goes all the way back to 1990 when the when the British electricity companies were privatised. The methodology we developed in 2009 and for which we retain the intellectual property rights, made use of Excel, an Access database, VBA and an open source emailing application. Excel was used as it has a very powerful calculation engine, good graphs and, with VBA, is easily linked to databases. It is also simple to automate.
The methodology makes use of dynamic data from the internet and history of electricity demands. Each forecast estimates the peak demands for every day in the triad period using the best available data type for the day. Data types are associated with variances derived from historical information and using Monte Carlo methods, the probability of a day being a triad is calculated for every day in the period.
Whether a day should be ‘called’ as a likely triad depends to an extent on experience and the values in a set of parameters within the application.
In late autumn 2014 we reviewed the application and decided to produce an experimental stand-alone version not using Excel. We considered using Java and Microsoft .NET technology but eventually selected Python as the language, Sqlite3 as the database and the Raspberry Pi as the platform. Compared with the Excel/VBA version, a lot more calculation is carried out via queries on the database or in attached temporary tables in memory.
Having tried Java and written some parts in VB.NET perviously, Python (version 3) gave a much faster development time. It also has some valuable features:
Close binding to the Sqlite3 database giving good performance;
Very easy internet access;
FTP for web page updating;
Dictionary and Set data structures;
Simple to sort data in memory;
Mathematical functions such as normally distributed random numbers;
It is available for Windows, Linux, and Mac.
There are some downsides:
Security is not a strong point and Python code has to come from a reliable source;
Code is not easy to protect from copying and alteration;
Variables are not declared ahead of use and typos in variable names can cause run-time errors, especially when on the right-hand side of an expression;
It is an interpreted language and thus performance is less than a truly compiled language such as C or FORTRAN;
It requires the language and any additional libraries to be installed on the host machine (as do Java and VB.Net)
In this particular application, the security and performance issues are not issues. Indeed, the performance is roughly equivalent to Excel/VBA. As mentioned above, the major deciding factor was the speed of development.
Why the Raspberry Pi?
We developed and tested the Python code in a Windows environment and then ported it to Linux. No changes were needed. The only restriction was the version of Sqlite3 used (3.7 rather than 3.8) to fit with the Python build for the Raspberry Pi. (Sqlite3 version 3.8 has common table expressions for complicated queries but for version 3.7 additional permanent views are used instead.)
The Raspberry Pi was something of a solution looking for a problem! However, using it did mean that the application could be left running without worrying about system upgrades, other activities running on the machine etc.. The Pi is used in ‘headless’ mode with no screen nor keyboard - just a power lead and a LAN cable. Communication is via remote login over the LAN. Obviously it is behind a firewall and is not visible externally! It is very low power. The standard Linux feature of CRONTAB made job scheduling easy.
Fow security purposes, it does have a shared NAS drive mounted so that its database and running logs are backed up from its internal SD Card storage.
This embedded database was a natural choice with Python. The file structure is identical for all platforms and its performance is good. It has views and triggers with the SQL being close to standard.
A Windows ODBC driver is available and we have linked Excel to the database to provide some summary data and graphs.
Managing the database can be done via a simple command line application or via a GUI add-in to the Firefox browser!
If you would like more information, do contact us.