Improved reliability of real-time data – Ferny Grove Weather Blog

Real time data displayed on this website have seen interruptions to the updating of the data on a number of occasions. These ongoing issues should now be solved with the introduction of an automated process to deal with this. Most recently an 8 hour outage occurred during the overnight hours. Measures are in place to deal with these issues, including automated monitoring of data reaching the website in a timely manner.

These issues started with the introduction of a Weatherlink Live data capture and streaming device used to collect the data from our on-site sensors measuring the weather. This device was brought into use in order to expand the data in collect and publish, which was not possible for technical reasons with its predecessor.

Due to this system relying on our residential home network (including wi-fi connectivity) to a greater deal than the previous implementation of the weather system there are occasional temporary dropped connections between the Weatherlive Live device and the Raspberry Pi computer used to collate, store and upload the data within our network. As a consequence of this, sometimes the Raspberry Pi computer loses connection to the Weatherlink Live device, and doesn’t regain a successful connection.

The impact of this is the software continues to run and upload data that have not updated correctly. In the most recent outage the data rolled over to the new day, but continued to used data from the previous day.

Usually, however a reboot of the Raspberry Pi computer fixes the issue. However if the Raspberry Pi has lost connection to the network, then it makes more difficult to reboot it, as the Raspberry Pi operates in a headless configuration with no keyboard, mouse or monitor. Usually cutting power to the computer and turning it back on has been the only option to get the system back up and running.

So in view of this, a bash script was created to manage these issues in an automated manner. The script handles a) the data collection software is not running because it crashed for some reason, and b) the data collection software is running but the data is not current because it lost connection within the network. When these situations occur the software is shut down, and the system rebooted. The script is ran as a scheduled cron task every 5 minutes on the Raspberry Pi computer. The Raspberry Pi computer on boot up is configured to auto-start all of the required systems, thereby getting the data back online. The scripts allows for the time periods for these two time periods to be defined, and to write the actions taken to a log file when triggered in the script.

The bash script is as the following:

#!/bin/bash
#Declare script variables
CMX_INSTALL_FOLDER='/home/pi/CumulusMX'
CMX_DATA_FOLDER='/data/'
CMX_DIAGS_FOLDER='/MXdiags'
CMX_DATA_STOPPED_AGE_TRIGGER_MINUTES=5
CMX_PROGRAM_STOPPED_AGE_TRIGGER_MINUTES=15
SCRIPT_LOG_FILE='/home/pi/.local/CMXdataStopCheckLog.txt'

#PROD_{START;END}_TIME defines the time of data to permit running the script.
#Restarting CMX near the daily rollover can causes data integrity issues
PROD_START_TIME="001500"
PROD_END_TIME="234500"

#Determine if script should run based on the time range
CURR_TIME=`date +"%H%M%S"`
if [[ $CURR_TIME -ge $PROD_START_TIME && $CURR_TIME -le $PROD_END_TIME ]];
then
      CONT_WITH_SCRIPT=1
else
        exit
fi

#Define current month/ year and file path to log file
CURR_MONTH=$(date +%b)
CURR_YEAR=$(date +%y)
LOG_FILE_NAME="${CURR_MONTH}${CURR_YEAR}log.txt"
CURR_LOG_FILE="${CMX_INSTALL_FOLDER}${CMX_DATA_FOLDER}${LOG_FILE_NAME}"
DIAGS_LOG_FILE="${CMX_INSTALL_FOLDER}${CMX_DIAGS_FOLDER}"

#Determine epoch time
CURR_TIME_SINCE_EPOCH_SECONDS=$(date +%s)

#Determine age of diags file since Epoch time in seconds
DIAG_FILE_MODIFIED_SINCE_EPOCH_SECONDS_FLOAT=$(find "$DIAGS_LOG_FILE" -type f -printf "%T@\n" | sort | tail -1)
DIAG_FILE_MODIFIED_SINCE_EPOCH_SECONDS=$(printf "%.*f\n" "$p" $DIAG_FILE_MODIFIED_SINCE_EPOCH_SECONDS_FLOAT)
DIAG_FILE_LAST_MODIFIED_SECONDS=$(($CURR_TIME_SINCE_EPOCH_SECONDS-$DIAG_FILE_MODIFIED_SINCE_EPOCH_SECONDS))


#Determine if CMX is not running AND have been stopped for too long -  shut down and restart the system
if [ $DIAG_FILE_LAST_MODIFIED_SECONDS -gt $(($CMX_PROGRAM_STOPPED_AGE_TRIGGER_MINUTES*60)) ]
then
        echo "Not running for at least $CMX_PROGRAM_STOPPED_AGE_TRIGGER_MINUTES minute/s. Shutting down CMX and restarting system"
	if pidof mono >/dev/null
	then
	        MONO_RUNNING_FLAG=1
	else
		echo "Kill mono"
		$(killall mono) 	
	fi
printf "$(date +'%d/%m/%Y %H:%M:%S'): Shutting down CMX and restarting system \n" >> $SCRIPT_LOG_FILE
echo "Reboot"
$(sudo reboot)

fi


#Determine if CMX is running
if [ $DIAG_FILE_LAST_MODIFIED_SECONDS -gt $(($CMX_DATA_STOPPED_AGE_TRIGGER_MINUTES*60)) ]
then
       	#echo "Not running"
	CMX_STOPPED_FLAG=1
	exit
else 	#only proceed if CMX is running - can't check for data stopped if CMX is not running!
	CMX_STOPPED_FLAG=0
	#echo "Running OK"
fi

#determine if data has stopped by the age of modifed date of log file since Epoch time in seconds
LOG_FILE_MODIFIED_SINCE_EPOCH_SECONDS=$(date -r $CURR_LOG_FILE "+%s")
LOG_FILE_LAST_MODIFIED_SECONDS=$(($CURR_TIME_SINCE_EPOCH_SECONDS-$LOG_FILE_MODIFIED_SINCE_EPOCH_SECONDS))

#Set Data stopped flag according to age of last modifed date
#if data has stopped shut down CMX and restart the system
if [ $CMX_STOPPED_FLAG -eq 0 ] && [ $LOG_FILE_LAST_MODIFIED_SECONDS -gt $(($CMX_DATA_STOPPED_AGE_TRIGGER_MINUTES*60)) ] 
then 
	#DATA_STOPPED_FLAG=1
	echo "CMX data have stopped but CMX is running. Shutting down CMX and restarting system"
        printf "$(date +'%d/%m/%Y %H:%M:%S'): CMX have stopped updating data for at least $CMX_DATA_STOPPED_AGE_TRIGGER_MINUTES minutes. CMX is running. \n" >> $SCRIPT_LOG_FILE
        if pidof mono >/dev/null
	then
		MONO_RUNNING_FLAG=1
	else
		echo "kill mono"
		$(killall mono)
        fi
	printf "$(date +'%d/%m/%Y %H:%M:%S'): Shutting down CMX and restarting system \n" >> $SCRIPT_LOG_FILE
        echo "reboot"
	$(sudo reboot)

else 
	#Data updating OK
	DATA_STOPPED_FLAG=0
	#echo "Data updating OK"
	#exit
fi

#!/bin/bash

#Declare script variables

CMX_INSTALL_FOLDER='/home/pi/CumulusMX'

CMX_DATA_FOLDER='/data/'

CMX_DIAGS_FOLDER='/MXdiags'

CMX_DATA_STOPPED_AGE_TRIGGER_MINUTES=5

CMX_PROGRAM_STOPPED_AGE_TRIGGER_MINUTES=15

SCRIPT_LOG_FILE='/home/pi/.local/CMXdataStopCheckLog.txt'

#PROD_{START;END}_TIME defines the time of data to permit running the script.

#Restarting CMX near the daily rollover can causes data integrity issues

PROD_START_TIME="001500"

PROD_END_TIME="234500"

#Determine if script should run based on the time range

CURR_TIME=`date +"%H%M%S"`

if [[ $CURR_TIME -ge $PROD_START_TIME && $CURR_TIME -le $PROD_END_TIME ]];

then

CONT_WITH_SCRIPT=1

else

exit

#Define current month/ year and file path to log file

CURR_MONTH=$(date +%b)

CURR_YEAR=$(date +%y)

LOG_FILE_NAME="${CURR_MONTH}${CURR_YEAR}log.txt"

CURR_LOG_FILE="${CMX_INSTALL_FOLDER}${CMX_DATA_FOLDER}${LOG_FILE_NAME}"

DIAGS_LOG_FILE="${CMX_INSTALL_FOLDER}${CMX_DIAGS_FOLDER}"

#Determine epoch time

CURR_TIME_SINCE_EPOCH_SECONDS=$(date +%s)

#Determine age of diags file since Epoch time in seconds

DIAG_FILE_MODIFIED_SINCE_EPOCH_SECONDS_FLOAT=$(find "$DIAGS_LOG_FILE" -type f -printf "%T@\n" | sort | tail -1)

DIAG_FILE_MODIFIED_SINCE_EPOCH_SECONDS=$(printf "%.*f\n" "$p" $DIAG_FILE_MODIFIED_SINCE_EPOCH_SECONDS_FLOAT)

DIAG_FILE_LAST_MODIFIED_SECONDS=$(($CURR_TIME_SINCE_EPOCH_SECONDS-$DIAG_FILE_MODIFIED_SINCE_EPOCH_SECONDS))

#Determine if CMX is not running AND have been stopped for too long - shut down and restart the system

if [ $DIAG_FILE_LAST_MODIFIED_SECONDS -gt $(($CMX_PROGRAM_STOPPED_AGE_TRIGGER_MINUTES*60)) ]

then

echo "Not running for at least $CMX_PROGRAM_STOPPED_AGE_TRIGGER_MINUTES minute/s. Shutting down CMX and restarting system"

if pidof mono >/dev/null

then

MONO_RUNNING_FLAG=1

else

echo "Kill mono"

$(killall mono)

printf "$(date +'%d/%m/%Y %H:%M:%S'): Shutting down CMX and restarting system \n" >> $SCRIPT_LOG_FILE

echo "Reboot"

$(sudo reboot)

#Determine if CMX is running

if [ $DIAG_FILE_LAST_MODIFIED_SECONDS -gt $(($CMX_DATA_STOPPED_AGE_TRIGGER_MINUTES*60)) ]

then

#echo "Not running"

CMX_STOPPED_FLAG=1

exit

else #only proceed if CMX is running - can't check for data stopped if CMX is not running!

CMX_STOPPED_FLAG=0

#echo "Running OK"

#determine if data has stopped by the age of modifed date of log file since Epoch time in seconds

LOG_FILE_MODIFIED_SINCE_EPOCH_SECONDS=$(date -r $CURR_LOG_FILE "+%s")

LOG_FILE_LAST_MODIFIED_SECONDS=$(($CURR_TIME_SINCE_EPOCH_SECONDS-$LOG_FILE_MODIFIED_SINCE_EPOCH_SECONDS))

#Set Data stopped flag according to age of last modifed date

#if data has stopped shut down CMX and restart the system

if [ $CMX_STOPPED_FLAG -eq 0 ] && [ $LOG_FILE_LAST_MODIFIED_SECONDS -gt $(($CMX_DATA_STOPPED_AGE_TRIGGER_MINUTES*60)) ]

then

#DATA_STOPPED_FLAG=1

echo "CMX data have stopped but CMX is running. Shutting down CMX and restarting system"

printf "$(date +'%d/%m/%Y %H:%M:%S'): CMX have stopped updating data for at least $CMX_DATA_STOPPED_AGE_TRIGGER_MINUTES minutes. CMX is running. \n" >> $SCRIPT_LOG_FILE

if pidof mono >/dev/null

then

MONO_RUNNING_FLAG=1

else

echo "kill mono"

$(killall mono)

printf "$(date +'%d/%m/%Y %H:%M:%S'): Shutting down CMX and restarting system \n" >> $SCRIPT_LOG_FILE

echo "reboot"

$(sudo reboot)

else

#Data updating OK

DATA_STOPPED_FLAG=0

#echo "Data updating OK"

#exit

Whilst these measures will need monitoring over time, and with possible tweaks/ or improvements, this should help with increasing the availability of the data on this website.

If you find this type of blog post of value, please free free to comment or share on this post. You can also contact us if you have any questions directly related to the information on this website.

Want to share this?

Leave a Reply Cancel Reply