Improved reliability of real-time data

Real time data displayed on this website have seen interruptions to the updating of the data on a number of occasions. These ongoing issues should now be solved with the introduction of an automated process to deal with this. Most recently an 8 hour outage occurred during the overnight hours. Measures are in place to deal with these issues, including automated monitoring of data reaching the website in a timely manner.

These issues started with the introduction of a Weatherlink Live data capture and streaming device used to collect the data from our on-site sensors measuring the weather. This device was brought into use in order to expand the data in collect and publish, which was not possible for technical reasons with its predecessor.

Due to this system relying on our residential home network (including wi-fi connectivity) to a greater deal than the previous implementation of the weather system there are occasional temporary dropped connections between the Weatherlive Live device and the Raspberry Pi computer used to collate, store and upload the data within our network. As a consequence of this, sometimes the Raspberry Pi computer loses connection to the Weatherlink Live device, and doesn’t regain a successful connection.

The impact of this is the software continues to run and upload data that have not updated correctly. In the most recent outage the data rolled over to the new day, but continued to used data from the previous day.

Usually, however a reboot of the Raspberry Pi computer fixes the issue. However if the Raspberry Pi has lost connection to the network, then it makes more difficult to reboot it, as the Raspberry Pi operates in a headless configuration with no keyboard, mouse or monitor. Usually cutting power to the computer and turning it back on has been the only option to get the system back up and running.

So in view of this, a bash script was created to manage these issues in an automated manner. The script handles a) the data collection software is not running because it crashed for some reason, and b) the data collection software is running but the data is not current because it lost connection within the network. When these situations occur the software is shut down, and the system rebooted. The script is ran as a scheduled cron task every 5 minutes on the Raspberry Pi computer. The Raspberry Pi computer on boot up is configured to auto-start all of the required systems, thereby getting the data back online. The scripts allows for the time periods for these two time periods to be defined, and to write the actions taken to a log file when triggered in the script.

The bash script is as the following:

Whilst these measures will need monitoring over time, and with possible tweaks/ or improvements, this should help with increasing the availability of the data on this website.

If you find this type of blog post of value, please free free to comment or share on this post. You can also contact us if you have any questions directly related to the information on this website.

Want to share this?

Website after 3 years

On this day three years ago the Ferny Grove Weather website was launched. It is with great appreciation and amazement of the growth of the continued support of everyone who have visited this website during this time. I would especially like to thank those who have showed a continued interest in the website who have visited many times and especially during times of severe weather.

In the past year a total of 5,629 visits have been made, which is 59 % of all visits made to the website in the past three years. In the past three years there has been a total of 9508 visits that were made by 3713 visitors of which 58% have previously visited the website. Those who have visited this website most frequently (in top 1% of all return visits made) have brought 22.1 % of all visits which were made by 20 unique visitors. This is a large drop compared to last year, as a result of a larger audience who are using this website. A total of 13,343 page views have been made.

It has brought great satisfaction of how this website is used more during periods of severe weather. This was especially the case during the significant weather event in February 2022 which saw a massive increase in visitor activity with 1491 visits by 520 visitors (2294 page views) and over double the previously most active month. This is quite obvious in the information contained on the Website Analytics page.

There is very strong interest in the real time data on this website which received a large proportion of all activity on the website (47.7 % for the home page and 14.8 % for the gauges out of all of the page views). This has however dropped by 27 % with other pages on the website being of more interest than it was previously, mostly notable the Today and Yesterday page.

This website came about based on the desire to upgrade the weather station used for improved accuracy and to shift towards measuring the weather in real-time. This website was built as a personal project to view the data that was easy to access, be updated in an automated nature as nearly to real-time as possible and no matter what screen size I am using. This project have evolved to also experiment with various data presentation methods and to explore programming techniques and technologies.

In the past year new additions to the website to the website include:

  • UV Index and UV Dose (Standard Erythermal Dose)
  • Upgraded automated process to update the website analytics by rainfall in Website Analytics
  • Thermal graphs to Historic Graphs showing Growing Degree Days of daily average temperatures and Temperature Sum of daily minimum, maximum and average temperature
  • Heat Map charts that shows logged historical data which may be viewed by hour and day across a given year for a selected metric.
  • Wind Rose charts for wind speed for various periods across the entire station period including specific month and years and various recent time periods. This shows the distribution of wind speed by wind direction.
  • Various small additions and tweaks

Moving forwards there is much more planned to improve and add to the website and continues with a plan to publish data that to date is not viewable on the website, some of which is currently under development. As this is a personal project and with the complexities involved, it does takes time to complete these additions with the accuracy and use friendliness of the website being important components to get right.

But if there are any specific things on the website you would like to see for consideration or if you have feedback, you are always welcome to reply here or to Contact Us with your thoughts.

Want to share this?

June 2021 weather review

The review of the weather at Ferny Grove during June 2021 is now available.

In June at Ferny Grove overnight and daytime temperatures were close to average. Daytime temperatures were the lowest in 5 years. Minimum temperatures during June were predominately below average with some periods of above average temperatures throughout the month. Maximum temperatures in June were near average or slightly above average during most days with some cool days.

Rainfall for June was below average but close to the long term median with a total of 23.9 mm (38.6% of the long term average and 61.8% of the site median). Light rain on the 3rd and light showers early on the 9th. A mild mid-morning thunderstorm with brief moderate falls and then light rain later in morning on the 15th. Light showers on the 23rd and 24th. Light rain at times on the 24th and 26th. Light showers on the 29th and 30th with some moderate falls. After some rainfall at the start of the month, about one-third of the rainfall was measured on the 16th, and another third later in the month. Most rainfall during the month was light.

The long period rainfall totals continued to remain above average for rainfall periods up to 18 month but generally remained near average at the longer time spans, though 36 month remained below average. As compared with May 2021 the 6 monthly rainfall dropped back by 57.7 mm to 96.4 mm above average, 9 month rainfall dropped back by 15.1 mm to 94.9 mm and 12 monthly slipped by 20.5 mm to 75.6 mm above average.18 month rainfall remained above average at 212.1 mm above average after improving by 46.4 mm. 24 month rainfall slipped slightly by 18 mm from 9.8 mm above average to 8.1 mm below average. 36 month rainfall continued to remain the only period that was significantly below average that was 129.8 mm below average but improved by 35.2 mm from May 2021. 48 monthly rainfall slipped slightly from 2.2 mm above average to 23.4 mm above average. Compared with the same period in 2020 the long period rainfall totals improved or dropped slightly.

The summary containing the key information can be found here.

The full report of more detailed analysis is available here.

Want to share this?

Additions to website

Several additions to the website in the latest update (v1.12.0) has been made with several new graphs now available. Various other changes have been made and the seasonal weather summary for the summer of 2020 to 2021 has been published in Reviews.

The new graphs available are as the following:

  • Website Analytics: 2 new graphs of cumulative number of visits and page views of all visitors (Cumulative Visits) and return visitors (Cumulative Return Visits)
  • Historic Graphs: New graphs for annual thermal energy (shown under Thermal). This is shown for Growing Degree Days and Temperature Sum that utilises daily temperatures compared to base temperatures to track plant growth and seasonal progress
  • Heatmap charts: This is a new page that shows our logged historical data for the past 10 years by hour and day across a given year for a selected metric
  • Wind Rose charts: This is another new page that charts the distribution of wind speed and wind direction for various periods. The time periods include recent data and any year, month or all data during the last 10 years

To provide further information about how the new graphs have been added, that php scripts are used to run queries against the database and insert that data into JSON format arrays in a format that the Highcharts graphs can use. The link to the js file containing the Javascript code for the highcharts can be viewed from page source code, such as analyticsSQL.js for the Website Analytics graphs. From the Javascript code the code of the php scripts used as the data input can be accessed by appending a ?view=sce to the url.

Adding the new website analytics graphs as mentioned above came about because it is of interest to plot the month to month trend in the visitor numbers. From this it can clearly seen that the trend have been generally linear even though there are variations in individual months. The direct links to the php scripts are as per the following: Cumulative Visits and Cumulative Return Visits

Explaining the Growing Degree Days (GDD) in more detail this provides an easy to use simple indication of crop or plant development over a period of time as result of the accumulation of heat. Plant growth is often dependent on the amount of heat experienced over time. So to quantify the amount of thermal energy available over time, GDD is a cumulative total of daily temperatures that can give an indication of how the season/s is progressing for plant growth and provide a future indication to when a required amount of GDD would be reached.

Note that other factors are not included such time integrated warmth over each day, transpiration, soil moisture and rainfall. The daily GDD contribution is calculated as the difference of the daily average temperature (average of minimum and maximum temperatures) for each day from base temperatures. The daily GDD is set to zero if the average temperature is below the base temperature (not possible in our climate). Daily maximum temperatures over 30 C are capped at 30 C as many plants don’t grow as quickly in hotter weather.

The temperature sum is similar and provides an indication of the accumulative effect of the thermal energy available over a period of time based on the temperature. However that while the Growing Degree Days (GDD) provides an indication for plant growth, Temperature Sum provides a more general gauge on the progress of the total amount of coolth or warmth of the season/s. Similar to the GDD base temperatures are also employed.

The temperature for each day for the temperature sum is calculated as the difference of the daily average temperature (average of minimum and maximum temperatures) for each day from base temperatures. The daily temperatures used for sum that are below the base temperature do reduce the cumulative temperature sum, in contrast to this not affecting the cumulative temperature in GDDs.

The php scripts employed for the GDD and temperature sum are parameterised meaning that values are appended to the end of the url to control the output received in the query. This allows for the base temperatures and starting month to be defined as well as the capping of temperatures at 30 C in GDD or the metric used in the temperature sum. The header in the php script explains in more detail about the parameters the script accepts. The temperature sum is show for daily average, minimum and maximum temperatures (metric used is set in the url parameter) because there are differences between these that show the coolness and warmness of the season in different ways depending on the base temperatures chosen. The direct link to the php scripts is the following: Growing Degree Days and Temperature Sum.

And if of interest this is the main SQL query for the annual Growing Degree Days for each year as contained in the php script which uses both SQL and php variables.

And this is the main SQL query for annual temperature sum for each year as contained in the php script which has both SQL and php variables.

Let us know if you find this content interesting or if you have any questions or comments. I hope that you find the new data helpful or of interest.

Want to share this?

May 2021 weather review

The review of the weather at Ferny Grove during May 2021 is now available.

In May 2021 overnight and daytime temperatures were close to average. Daytime temperatures was the lowest for May in 6 years. Minimum temperatures in May were above average during most days in the first half of the month before near or below average temperatures during the remainder of the month. May maximum temperatures were above average during most days in the first half of the month and then most days during the rest of the month were near or below average.

Rainfall for May was close to average but above the long term median with a total of 99.5 mm (96.6% of the long term average and 220.4% of the site median). On the 1st and 2nd were showers with some moderate to heavy falls. Further rainfall occurred on the 4th and 5th with light showers tending to rain after a mild thunderstorm with brief heavy falls on the 5th. Overnight on the 11th and 12th was light to moderate thundery rain with a thunderstorm. In the afternoon of the 12th was a severe thunderstorm with fierce electrical activity with 20 mm of rainfall occurring in 12 minutes. Almost all of the rainfall fell in the first half of the month.

Long period rainfall totals remained above average for the rainfall periods up to 18 months and improved for some of the longer periods, though 36 monthly rainfall remained significantly below average. 6 monthly rainfall improved by 82.3 mm in April to 154.1 mm above average, 9 monthly rainfall improved by 23.8 mm to 110 mm above average and 12 monthly rainfall improved by 88.8 mm to 96.1 mm above average. 18 month rainfall improved by 72.6 mm to 165.6 mm above average. 24 monthly rainfall returned to being in the positive from 14 mm below average to 9.8 mm above average. 36 month was the only period that was below average being 165 mm below average which was the same as the previous month. 48 monthly rainfall slipped slightly from 24.9 mm above average to 2.2 mm above average. Compared with the same period in 2020 the long period rainfall totals improved and significantly for some periods but have worsened for 36 and 48 monthly totals.

The summary containing the key information can be found here.

The full report of more detailed analysis is available here.

Want to share this?

April 2021 weather review

The review of the weather at Ferny Grove during April 2021 is now available.

In April 2021 overnight temperatures were below average. Daytime temperatures were close to average but was the coolest for April in 11 years. April minimum temperatures were near or above average in the first 10 days of the month before an extended period of below average temperatures during most of the remaining of the month. Maximum temperatures during April were near average or below average throughout the month with some above average temperatures.

April rainfall was the highest for April in 6 years and was above average with a total of 130.1 mm (138.6% of the long term average and 308.1% of the site median). Light showers with some moderate to heavy falls occurred at the start of the month tending to drizzle and light rain from the 4th and cleared on the 7th with moderate falls on the 6th. Light to moderate showers on the 8th and light showers on the 17th with some moderate to heavy falls. On the 30th were light to moderate thundery showers. Most of the rainfall during the month fell in the first week where the monthly rainfall total on the 7th had reached 114.4 mm.

Long period rainfall totals improved for most periods as compared with February 2021. Many of the long period totals remained slightly above average or above average, though the longer time periods of 24 and 36 month have remained with negative anomalies. 6 monthly rainfall dropped slightly by 13.4 mm to 71.8 mm above average and 9 monthly rainfall improved slightly by 20.2 mm to 86.2 mm above average. 12 monthly rainfall improved by 113.9 mm to 7.3 mm above average. 18 month rainfall improved by 69.5 mm and was 93 mm above average. 24 month and 36 month remained below average which improved by 109.9 mm to 14 mm below average for 24 month rainfall and improved by 60.1 mm to 165 mm below average for 36 month rainfall. 48 month rainfall improved by 135.2 mm to 24.9 mm above average. Compared with the same period in 2020 the long period rainfall totals improved and significantly for some periods but have worsened for 36 and 48 monthly totals.

The summary containing the key information can be found here.

The full report of more detailed analysis is available here.

Want to share this?

March 2021 weather review

The review of the weather at Ferny Grove during March 2021 is now available.

In March 2021 overnight temperatures were close to average. Daytime temperatures were close to average but was the coolest for March in 8 years. Minimum temperatures in March were near average or above average before below average temperatures in the last week of the month. Maximum temperatures during March saw a mixture of below and above average temperatures throughout the month, with a number of near average or below average maximum temperatures in the middle of the month.

Rainfall in March was well above average with a total of 297.3 mm (239.4% of the long term average and 248.5% of the site median). Many days in the first half of the month saw light showers and rain, with a gusty thunderstorm with brief heavy rainfall on the 2nd. The 14th to 20th saw periods of drizzle and light rain with light showers before increasing wet conditions on the 21st to 23rd with drizzle, moderate to heavy rain periods and a thunderstorm. In the last few days of the month were some light showers. Most of the total rainfall in the month occurred between the 14 March and 24 March when 263.3 mm of rainfall fell and 203.3 mm was measured between the 22 and 24 March.

Long period rainfall totals until the end of March 2021 improved for all periods as compared with February 2021. Many of the long period totals became slightly above average, though the longer time periods have remained with negative anomalies. 6 monthly rainfall improved by 179.9 mm to 85.2 mm above average and 9 monthly rainfall improved by 174.6 mm to 66 mm above average. 12 monthly rainfall improved by 149.8 mm but remained 106.6 mm below average. 18 month rainfall saw an improvement of 191.2 mm and was 23.4 mm above average. 24 month, 36 month and 48 month all remained below average but have improved by 188.1 mm to 123.9 mm below average for 24 month rainfall, improved by 106.9 mm to 237.7 mm below average for 36 month rainfall and by 250 mm to 110.3 mm for 48 month rainfall. Compared with the same period in 2020 the long period rainfall totals were mixed between improving and worsening for 6, 9, 12 and 18 monthly but have worsened for 24, 36 and 48 monthly totals. However, many periods have now see lower rainfall deficiencies as compared with the start of summer in December 2020

The summary containing the key information can be found here.

The full report of more detailed analysis is available here.

Want to share this?

February 2021 weather review

The review of the weather at Ferny Grove during February 2021 is now available.

In February 2021 overnight temperatures were close to average. Daytime temperatures were close to average and the lowest in 6 years. In February minimum temperatures were close to or below average during the month with some warm nights in the middle of the month and in the last week of the month. Maximum temperatures in February were below average during many days of the month and were above average during some days mostly occurring late in the month.

Rainfall in February was close to average with a total of 126.2 mm (78% of the long term average and 82.1 % of the site median). In the first few days of the month were showers with moderate to heavy falls before mostly light showers and rain until the middle of the month on some days. Further moderate to heavy falls from showers in the middle and later in the month along with some thunderstorm activity on the 19th, 22nd and 23rd.

Long period rainfall totals compared with January 2021 reduced further for most periods and were below average or well below average. 6 monthly rainfall reduced slightly by 20 mm below average to 94.7 mm below average and 9 monthly rainfall slipped by 45 mm to 108.6 mm below average. 12 monthly rainfall was slightly above average in January but fell to 256.4 mm below average in February. 18 month rainfall fell slightly to 167.8 mm below average while the 24 month, 36 month and 48 month rainfall totals fell by a large amount. The deficiencies compared to the long term average for 24 month rainfall fell from -163.9 mm to -312 mm, 36 month rainfall fell from -442.6 mm to -344.6 mm and 48 month rainfall fell from -68.8 mm to -360.3 mm. Compared with the same period in 2020 rainfall deficiencies have deteriorated, in contrast to January where the rainfall totals were less deficient than in January 2020. When compared with the start of summer rainfall totals worsened for all periods except for 24 month rainfall that improved by 54 mm but was still well below average.

The summary containing the key information can be found here.

The full report of more detailed analysis is available here.

Want to share this?

January 2021 weather review

The review of the weather at Ferny Grove during January 2021 is now available.

In January 2021 overnight temperatures were above average. Daytime temperatures were below average and the lowest in 9 years. In January minimum temperatures saw a number of slightly warm nights throughout the month accompanied with some below average temperature throughout much of the month. Maximum temperatures in January were predominately below average that saw very few warm days during the month. Daytime temperatures on the 7th to 12th were below average, falling as low of 8 ºC below average on the 8th.

Rainfall in January was close to average with a total of 95.8 mm (66.1% of the long term average). During the first 10 days of the month was an active period of showers, thunderstorms, and light rain. During the middle of the month were some thunderstorms with some moderate falls. A moderate to heavy shower in the evening of the 16th produced 8 mm in 6 minutes. During the rest of the month were light showers on many of these days.

Most of the long period rainfall totals continue to remain below or well below average that reduced in January. As compared with the previous month the 6 monthly rainfall worsened a little to 74.7 mm below average, 9 monthly rainfall dropped from 52.7 mm below average to 153.5 mm below average and 12 month rainfall slipped by 86 mm to 8.2 mm above average. 18-month rainfall dropped significantly from 529.5 mm above average to 150.4 mm below average. 24- and 48-month rainfall improved while remaining below average a 163.9 mm below average and 68.8 mm below average respectively, while 36-month rainfall fell by 806 mm to 442.6 mm below average. Compared with the same period in the previous year these totals have generally improved except for 36-month rainfall falling from 124.6 mm above average to 442.6 mm below average.

The summary containing the key information can be found here.

The full report of more detailed analysis is available here.

Want to share this?

UV radiation data

The website have been updated (v1.11.0) with new data that is now available on multiple pages to incorporate Ultraviolet (UV) radiation data following the start of recording this data from 21 March 2021. This addition is part of changes to measure solar radiation and air quality and to switch to a Weatherlink Live receiving unit between 19 and 26 December 2020.

The following pages have been updated with additional data:

  • The home page: the UV Index have been added to the real time updates including the UV Index exposure categories in the international colours with sun protection suggestions when hovering over the UV Index category
  •  Gauges: A gauge for UV Index was added
  •  Recent Graphs, Past Months Graphs and Historical Graphs: updated with new graphs containing UV Index (with banding for the exposure categories) and UV Dose data, as well as adding the UV Index to the existing Solar radiation graphs
  •  Today and Yesterday: Extremes and total for UV Index and UV Dose added
  •  This Month and Year: Extremes for UV Index added
  •  Annual Data Summary: Data for UV Index added
  • All-Time Records: Extremes for UV Index added
  •  Top 10 Records: UV Index records added
  •  Recent Extremes Since: UV Index extremes added for today and yesterday
  • About: updated with new information regarding the addition of the UV sensor that have been installed
  •  System Status: Reworked to align with the system changes and to provide additional system information now available with this change
  •  Glossary: Updated with additional terms for UV Index and UV Dose, including details of the formula used to calculate the UV Dose.

 Other changes made with this update include:

  • The database script that updates the Website Analytics page with current rainfall data by website visitors is now automated at the start of each month. This is a very expensive series of calculations that is only updated monthly.
  • On the Website Analytics page an issue fixed with the showing values greater than 90% on the Daily Rain by Visits graph
  • Fix an issue with the “New records in last 10 days” on the home page that didn’t show the month for a new monthly rain rate record
  • Added a note of discouragement to the link page about not requesting links to be added for commerical or spam purposes.

The UV Dose was added as to quantify the amount of UV radiation received at this location for each hour and over the course of day. This is because it is known that it is both the magnitude and length of time that determines how much radiation exposure is being received for sun protection purposes. This is calculated for each hour and shown on several pages. Whilst the Glossary page provides more information about this measure and how it calculated, for those interested below is the SQL statement to calculate the UV Dose from the real-time data for the Recent Graphs page.

And similarly the SQL statement for UV Dose on the Past Months Graphs page:

Feel free to let us know of your thoughts or suggestions regarding these additions.

Want to share this?