Davis Van ISS Weather Station With Raspbe

In this project I'm going to document my journey trying to set up a Davis Vantage Vue weather station continuously updating data on the internet with the help of a Raspberry Pi 4. You may say: there are a lot of projects like that on the internet! And you may be wrong :)

In this project we're going to use ONLY the Integrated Sensor Suite (ISS) of the Vantage Vue (which is sold separately for about 100 euros less than the kit including the console) and a RTL-SDR type receiver to get the sensor data. This way we'll be able to save a lot of money as a normal setup would need to have at least the console (to receive the data) and the USB datalogger from Davis (to interface with any software).

We're going to use Weewx (https://www.weewx.com/) to manage the weather station data. It is an amazing piece of open-source software which lets you connect basically every weather station that you can find on the market. Notably, it lets you use an RTL-SDR dongle to intercept data transmitted from a Davis Station (but really you could also get any data transmitted by a "normal" weather station) on the 868 MHz band (I'm based in Europe but it will work in a similar way for people in the US).

• NooElec NSDR Nano2+ RTL receiver - 35 EUR (but you can use whatever you have available)
• Davis Vantage Vue ISS - 269 EUR
• Raspberry Pi 4 4GB, Power Adapter, SD Card - I paid 80 EUR for a complete set also with case
• BME280 Pressure and temperature sensor (optional) - 15 EUR
• Lots of patience

NOTE: this assumes that you have a working installation of Debian or Ubuntu on the Raspberry PI. If that's not the case you can just check one of the thousand guides that explain how to perform a first setup. I will mainly work for the terminal so you can perform the following operations either connected via SSH or directly on the raspberry terminal.

Installing WeeWx is fairly easy. I followed the instructions for Debian (https://www.weewx.com/docs/debian.htm) and used the installation method with apt. Take note of the installation folders because we'll need that later.

executable:	/usr/bin/weewxd
configuration file:	/etc/weewx/weewx.conf
skins and templates:	/etc/weewx/skins
sqlite databases:	/var/lib/weewx/
generated web pages and images:	/var/www/html/weewx/
documentation:	/usr/share/doc/weewx/
examples:	/usr/share/doc/weewx/examples/
utilities:	/usr/bin/wee_*

You can use the "Simulator" station to start with while we set up the rtldavis executable, which will take most of our time.

Once finished the installation a daemon will already be running in the background: you can check that by doing

systemctl status weewx

In order to reach the webpage of the station you either need to configure a web server/reverse proxy (to access directly the web page to the same machine where weewx is running) or just set up a copy of the files into a FTP server (optional step). The HTML pages created by weewx are nothing but pure static content created on a user-defined interval so you can put them wherever you want.

I only had to install lighttpd to see the webpage

sudo apt install lighttpd

it will look for files into /var/www/html and, as weewx copies its files into /var/www/html/weewx, you'll be able to reach the webpage at raspberry-pi-local-ip/weewx.

You'll see the data update after the first 5 minutes. Don't worry, we'll change this ugly skin in a minute :)

This step really depends on the system that you're using, but in theory you should only need to install some packages from apt and make some correction to the udev rules. The final goal is to have a working installation of RTL-SDR which can recognize your dongle.

NOTE: you don't have to use the Noeelec NANO: any RTL-SDR dongle with the RTL2832U chip will work. This happened to be the smallest and cheapest receiver that I could find in Germany.

Start by installing the drivers.

sudo apt install rtl-sdr

Now we have to set some rules.

echo 'blacklist dvb_usb_rtl28xxu' | sudo tee – append /etc/modprobe.d/blacklist-dvb_usb_rtl28xxu.conf

Then go into /etc/udev/rules.d and create a new file (with sudo) called rtl-sdr.rules, with the content from this file https://osmocom.org/projects/rtl-sdr/repository/rtl-sdr/revisions/master/entry/rtl-sdr.rules.

It should be everything; if you now execute rtl_test into the command line you should be able to see something like this

$ rtl_test
Found 1 device(s):
 0: Realtek, RTL2838UHIDIR, SN: 00000001

Using device 0: Generic RTL2832U OEM
Found Rafael Micro R820T tuner
Supported gain values (29): 0.0 0.9 1.4 2.7 3.7 7.7 8.7 12.5 14.4 15.7 16.6 19.7 20.7 22.9 25.4 28.0 29.7 32.8 33.8 36.4 37.2 38.6 40.2 42.1 43.4 43.9 44.5 48.0 49.6
[R82XX] PLL not locked!
Sampling at 2048000 S/s.

Info: This tool will continuously read from the device, and report if
samples get lost. If you observe no further output, everything is fine.

Reading samples in async mode...

If you have troubles make sure to follow the RLT-SDR blog installation here https://www.rtl-sdr.com/tag/install-guide/ (scroll to Install on Linux)

This is the step that took me the longest, as it is poorly documented. rtldavis is a program initially created by

Douglas Hall (https://github.com/bemasher/rtldavis) who unfortunately hasn't been able to maintain it. Luckily

Luc Heijst forked the repo (https://github.com/lheijst/rtldavis) and added some new corrections and the European frequencies, which we're going to need. That's the one that we're going to use, together with the driver for weewx (weewx-rtldavis), which is needed to translate the data read by rtldavis into a LOOP packet that weewx can use.

We'll start by installing rtldavis and testing it .

First of all install golang

sudo apt-get install golang git cmake

Then install librtlsdr-dev manually: this is a requirement, otherwise rtldavis will not work.

cd /home/pi
git clone https://github.com/steve-m/librtlsdr.git
cd librtlsdr
mkdir build
cd build
cmake ../ -DINSTALL_UDEV_RULES=ON
make
sudo make install
sudo ldconfig

Change some lines into your profile file

sudo vim ~/.profile 

  export GOROOT=/usr/lib/go
  export GOPATH=$HOME
  export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
  
source ~/.profile

Now get the rtldavis package

cd /home/pi
go get -v github.com/lheijst/rtldavis

and compile the source

cd $GOPATH/src/github.com/lheijst/rtldavis
git submodule init
git submodule update
go install -v .

If everything goes well you should find a new folder in your home with the executable inside /home/pi/bin/rtldavis.

Make sure to install librtlsdr-dev as showed before; if you use the one packaged in your system you'll not see an error, but instead a message (allocating zero copy buffer) and the executable will not produce anything.

We're going to test the executable alone to see that everything's working fine. Most importantly we NEED to find the right frequencies. The go package has some EU frequencies hardcoded (868077250, 868197250, 868317250, 868437250, 868557250) but those will likely not work for you out of the box because of some discrepancies.

Go into /home/pi/bin/ and tun this command

./rtldavis -startfreq 868000000 -endfreq 868600000 -stepfreq 10000

This will scan the entire frequency spectrum where the station should transmit (you already turned the ISS sensors on, haven't you?). It will take a while but you should see some frequencies where the transmission is

10:10:31.123820 rtldavis.go VERSION=0.15
10:10:31.124130 tr=1 fc=0 ppm=0 gain=0 maxmissed=51 ex=0 receiveWindow=300 actChan=[0] maxChan=1
10:10:31.124179 undefined=false verbose=false disableAfc=false deviceString=0
10:10:31.124216 TEST: startFreq=868000000 endFreq=868600000 stepFreq=10000
10:10:31.124687 BitRate: 19200
10:10:31.124738 SymbolLength: 14
10:10:31.124773 SampleRate: 268800
10:10:31.124807 Preamble: 1100101110001001
10:10:31.124841 PreambleSymbols: 16
10:10:31.124875 PreambleLength: 224
10:10:31.124909 PacketSymbols: 80
10:10:31.124943 PacketLength: 1120
10:10:31.124977 BlockSize: 512
10:10:31.125010 BufferLength: 2048
Found Rafael Micro R820T tuner
10:10:31.525099 Hop: {ChannelIdx:0 ChannelFreq:868077250 FreqError:0 Transmitter:0}
Exact sample rate is: 268800.001367 Hz
10:10:31.698617 GetTunerGain: 0 Db
10:10:31.698697 SetFreqCorrection 0 ppm Successful
10:10:31.702138 Init channels: wait max 17 seconds for a message of each transmitter
10:11:07.580455 TESTFREQ 1: Frequency 868000000: NOK
10:11:25.519440 TESTFREQ 2: Frequency 868010000: NOK
10:11:43.458467 TESTFREQ 3: Frequency 868020000: NOK
10:12:01.397429 TESTFREQ 4: Frequency 868030000: NOK
10:12:19.337436 TESTFREQ 5: Frequency 868040000: NOK
10:12:37.276475 TESTFREQ 6: Frequency 868050000: NOK
10:12:55.215505 TESTFREQ 7: Frequency 868060000: NOK
10:13:13.154774 TESTFREQ 8: Frequency 868070000: NOK
10:13:31.093651 TESTFREQ 9: Frequency 868080000: NOK
10:13:49.032647 TESTFREQ 10: Frequency 868090000: NOK
10:14:06.971697 TESTFREQ 11: Frequency 868100000: NOK
10:14:24.910614 TESTFREQ 12: Frequency 868110000: NOK
10:14:27.099174 TESTFREQ 13: Frequency 868120000 (freqCorr=0): OK, msg.data: 80014D272907E6A4
10:14:39.912796 TESTFREQ 14: Frequency 868130000 (freqCorr=1031): OK, msg.data: 90058B09F39525CB
10:14:57.851600 TESTFREQ 15: Frequency 868140000: NOK
10:15:15.790642 TESTFREQ 16: Frequency 868150000: NOK
10:15:33.729646 TESTFREQ 17: Frequency 868160000: NOK
10:15:51.668677 TESTFREQ 18: Frequency 868170000: NOK
10:16:09.607740 TESTFREQ 19: Frequency 868180000: NOK
10:16:27.546769 TESTFREQ 20: Frequency 868190000: NOK
10:16:45.486136 TESTFREQ 21: Frequency 868200000: NOK
10:17:03.425139 TESTFREQ 22: Frequency 868210000: NOK
10:17:21.364168 TESTFREQ 23: Frequency 868220000: NOK
10:17:39.303185 TESTFREQ 24: Frequency 868230000: NOK
10:17:46.975373 TESTFREQ 25: Frequency 868240000 (freqCorr=1170): OK, msg.data: E0045ACB030E7D35
10:17:59.788122 TESTFREQ 26: Frequency 868250000 (freqCorr=2130): OK, msg.data: 50075FFF730024FC
10:18:17.727103 TESTFREQ 27: Frequency 868260000: NOK
10:18:35.666147 TESTFREQ 28: Frequency 868270000: NOK
10:18:53.605043 TESTFREQ 29: Frequency 868280000: NOK
10:19:11.544189 TESTFREQ 30: Frequency 868290000: NOK
10:19:29.483189 TESTFREQ 31: Frequency 868300000: NOK
10:19:47.422126 TESTFREQ 32: Frequency 868310000: NOK
10:20:05.361277 TESTFREQ 33: Frequency 868320000: NOK
10:20:23.300354 TESTFREQ 34: Frequency 868330000: NOK
10:20:41.239652 TESTFREQ 35: Frequency 868340000: NOK
10:20:59.178850 TESTFREQ 36: Frequency 868350000: NOK
10:21:06.849820 TESTFREQ 37: Frequency 868360000 (freqCorr=2310): OK, msg.data: 80046326B9066F1D
10:21:06.851418 TESTFREQ 38: Frequency 868370000 (freqCorr=3070): OK, msg.data: 80046326B9066F1D
10:21:24.789923 TESTFREQ 39: Frequency 868380000: NOK
10:21:42.729097 TESTFREQ 40: Frequency 868390000: NOK
10:22:00.668286 TESTFREQ 41: Frequency 868400000: NOK
10:22:18.607461 TESTFREQ 42: Frequency 868410000: NOK
10:22:36.546700 TESTFREQ 43: Frequency 868420000: NOK
10:22:54.484625 TESTFREQ 44: Frequency 868430000: NOK
10:23:12.423892 TESTFREQ 45: Frequency 868440000: NOK
10:23:30.363085 TESTFREQ 46: Frequency 868450000: NOK
10:23:48.302206 TESTFREQ 47: Frequency 868460000: NOK
10:24:06.241397 TESTFREQ 48: Frequency 868470000: NOK
10:24:13.915688 TESTFREQ 49: Frequency 868480000 (freqCorr=3727): OK, msg.data: A00552752B01A3E9
10:24:26.727222 TESTFREQ 50: Frequency 868490000 (freqCorr=4422): OK, msg.data: E00455CB010B9F1C
10:24:44.666628 TESTFREQ 51: Frequency 868500000: NOK
10:25:02.605734 TESTFREQ 52: Frequency 868510000: NOK
10:25:20.544182 TESTFREQ 53: Frequency 868520000: NOK
10:25:38.483333 TESTFREQ 54: Frequency 868530000: NOK
10:25:56.422456 TESTFREQ 55: Frequency 868540000: NOK
10:26:14.361597 TESTFREQ 56: Frequency 868550000: NOK
10:26:32.300674 TESTFREQ 57: Frequency 868560000: NOK
10:26:50.240205 TESTFREQ 58: Frequency 868570000: NOK
10:27:08.179294 TESTFREQ 59: Frequency 868580000: NOK
10:27:26.118451 TESTFREQ 60: Frequency 868590000: NOK
10:27:33.790629 TESTFREQ 61: Frequency 868600000 (freqCorr=3955): OK, msg.data: 500565FF720677CA
10:27:33.790832 Test reached endfreq; test ended

Results will probably vary for you, in my case I immediately saw that there was a frequency shift with respect to the "official" EU frequency of about 50000 Hz, and this was the reason why running rtldavis directly didn't work.

Anyway, after finding your offset you can re-rung rtldavis in receiving mode

./rtldavis -fc 56000 -tf EU -tr 1 -ppm 1

and you should see messages being received. You won't see any data from the sensors yet, but you should see something being printed like this

10:31:27.364138 rtldavis.go VERSION=0.15
10:31:27.365227 tr=1 fc=56000 ppm=1 gain=0 maxmissed=51 ex=0 receiveWindow=300 actChan=[0] maxChan=1
10:31:27.365752 undefined=false verbose=false disableAfc=false deviceString=0
10:31:27.367250 BitRate: 19200
10:31:27.367777 SymbolLength: 14
10:31:27.368256 SampleRate: 268800
10:31:27.368757 Preamble: 1100101110001001
10:31:27.369228 PreambleSymbols: 16
10:31:27.369697 PreambleLength: 224
10:31:27.370163 PacketSymbols: 80
10:31:27.370701 PacketLength: 1120
10:31:27.371174 BlockSize: 512
10:31:27.371642 BufferLength: 2048
Found Rafael Micro R820T tuner
10:31:27.811079 Hop: {ChannelIdx:0 ChannelFreq:868077250 FreqError:0 Transmitter:0}
Exact sample rate is: 268800.001367 Hz
10:31:27.988707 GetTunerGain: 0 Db
10:31:28.049017 SetFreqCorrection 1 ppm Successful
10:31:28.053112 Init channels: wait max 17 seconds for a message of each transmitter
10:31:32.105955 TRANSMITTER 0 SEEN
10:31:32.106099 Hop: {ChannelIdx:2 ChannelFreq:868317250 FreqError:0 Transmitter:0}
10:31:34.667998 300567FFC1807FAA 2 0 0 0 0 msg.ID=0
10:31:34.668201 Hop: {ChannelIdx:4 ChannelFreq:868557250 FreqError:0 Transmitter:0}
10:31:37.229970 E00367CB03079EA7 3 0 0 0 0 msg.ID=0
10:31:37.230392 Hop: {ChannelIdx:1 ChannelFreq:868197250 FreqError:0 Transmitter:0}
10:31:39.793944 50035FFF7101DBB9 4 0 0 0 0 msg.ID=0
10:31:39.794112 Hop: {ChannelIdx:3 ChannelFreq:868437250 FreqError:0 Transmitter:0}
10:31:42.355950 80035E2679071BD3 5 0 0 0 0 msg.ID=0
10:31:42.356254 Hop: {ChannelIdx:0 ChannelFreq:868077250 FreqError:274 Transmitter:0}

You're ready to go to the next step

Now we can install the rtldavis driver for weewx. Make sure to stop weewx before proceeding further

sudo systemctl stop weewx
wget -O weewx-rtldavis-master.zip https://github.com/lheijst/weewx-rtldavis/archive/master.zip
sudo wee_extension --install weewx-rtldavis-master.zip
sudo wee_config --reconfigure --driver=user.rtldavis --no-prompt

We first need to do some modifications to the hardcoded folder of the binary:

sudo vim /usr/share/weewx/user/rtldavis.py

and modify all the occurrences of /home/pi/work/bin/rtldavis with /home/pi/bin/rtldavis.

Now try to run the driver

sudo PYTHONPATH=/usr/share/weewx python /usr/share/weewx/user/rtldavis.py

and you should see some output.

Now let's configure the driver in weewx.

sudo vim /etc/weewx/weewx.conf

[Rtldavis]
  cmd = /home/pi/bin/rtldavis -fc 55500 -ppm 1
  transceiver_frequency = EU

  iss_channel = 1
  # The values below only apply for Vantage Pro or Pro2
  anemometer_channel = 0
  leaf_soil_channel = 0
  temp_hum_1_channel = 0
  temp_hum_2_channel = 0
  # rain bucket type (0: 0.01 inch, 1: 0.2 mm)
  rain_bucket_type = 1

  # Print debug messages
  # 0=no logging; 1=minimum logging; 2=normal logging; 3=detailed logging
  debug_parse = 0
  debug_rain = 0
  debug_rtld = 1  # rtldavis logging: 1=inf; 2=(1)+data+chan; 3=(2)+pkt

  # The pct_good per transmitter can be saved to the database
  # This has only effect with 2 transmitters or more
  save_pct_good_per_transmitter = True
  log_unknown_sensors = True
  log_unmapped_sensors = True

  # The driver to use:
  driver = user.rtldavis

Remember to change the station type to rtldavis in [Station] before rebooting weewx. You can also turn debug=1 and debug_rtld=3 to check that everything is working properly.

The ISS does not have sensors for pressure, nor it can measure inside temperature and humidity (obviously), so to replicate the original Davis console we need to add these data manually. While Weewx can easily continue running without inside temperature and humidity, pressure is such an important weather variable that it would be a pity not having it into the DB. Luckily you can use a cheap BME280 sensor to enhance the database with pressure, inside temperature and humidity.

First of all we need to turn of the inside temperature and humidity captured by rtldavis

sudo vim /usr/share/weewx/user/rtldavis.py

look for DEFAULT_SENSOR_MAP and comment these lines

    #'inTemp': 'temp_in', # temperature of optional BMP280 module
    #'inHumidity': 'humidity_in',

Then connect the BME280 sensor with the right pins (see attached image) and make sure that I2C is enabled in raspi-config (https://www.raspberrypi-spy.co.uk/2014/11/enabling-the-i2c-interface-on-the-raspberry-pi/).

Now you can install i2cdetect

sudo apt install i2c-tools

if everything went well you should be able to run the following command and see the output

i2cdetect -y 1
   0 1 2 3 4 5 6 7 8 9 a b c d e f
00:             -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- 1a -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- 77

77 is the address of our sensor, which we'll need later.

Now that the sensor is working we need to (you say it) install a weewx driver that can parse this data. Luckily there's already a driver ready to be downloaded: https://gitlab.com/mfraz74/bme280wx/ (note, I use a fork of the original project that reduces logging).

Download and install the driver with

sudo systemctl stop weewx
wget https://gitlab.com/mfraz74/bme280wx/-/archive/master/bme280wx-master.zip
sudo wee_extension --install bme280wx-master.zip

Then go into the configuration file /etc/weewx/weewx.conf and modify some lines

[Bme280wx]
  i2c_port = 1
  i2c_address = 0x77
  usUnits = US
  temperatureKeys = inTemp
  temperature_must_have = ""
  pressureKeys = pressure
  pressure_must_have = outTemp
  humidityKeys = inHumidity
  humidity_must_have = ""

you'll notice we use the same address found with i2cdetect: in case you have a different address you of course need to change it. You can leave the rest as it is. Just make sure that the Engine / Services part of the configuration file has data_services = user.bme280wx.Bme280wx (should be added automatically by the installation).

Restart weewx and you should see the internal temperature and humidity, together with the pressure, being updated.

I used the skin shown in the picture which includes data and plots of different periods.

You can install it easily by following the instructions here https://neoground.com/docs/neowx-material/installation

Once you installed the skin make sure you change your weewx.conf to only produce HTML pages for the neowx skin:

[StdReport]
 ...
  [[SeasonsReport]]
    skin = Seasons
    enable = false

  [[SmartphoneReport]]
    skin = Smartphone
    enable = false

  [[MobileReport]]
    skin = Mobile
    enable = false

  [[StandardReport]]
    skin = Standard
    enable = false

  [[Neow]]
    skin = neowx-material
    enable = true

Restart again weewx and everything should work as expected.

You can modify settings of the skin here /etc/weewx/skins/neowx-material. Note that if you press on the "telemetry" link (lower left) you can see the frequency errors, which clearly depend on the temperature, so that you can check if the frequency offset that you chose at the beginning needs to be tuned. You can even check if the battery of the capacitor in the Sensor suite is low or not.

See the screenshot for the values that I have,

Unfortunately weewx does not give you the possibility to modify the data directly in the web interface.

However, the data is stored into a simple database file that you can open, view and modify: it can be found in /var/lib/weewx/

here is a sample Python script to read the data

import sqlite3
import pandas as pd
con = sqlite3.connect('/var/lib/weewx/weewx.sdb')
df = pd.read_sql_query("SELECT * from archive", con,
                       index_col='dateTime')#.dropna(axis=1)

df['datetime'] = pd.to_datetime(df.index, unit='s', utc=True)
df['outTemp'] = df['outTemp'].apply(lambda x: (x-32.) * (5./9.) )
df['inTemp'] = df['inTemp'].apply(lambda x: (x-32.) * (5./9.) )
df['appTemp'] = df['outTemp'].apply(lambda x: (x-32.) * (5./9.) )

df['heatindex'] = df['heatindex'].apply(lambda x: (x-32.) * (5./9.) )
df['humidex'] = df['humidex'].apply(lambda x: (x-32.) * (5./9.) )
df['barometer']=df['barometer'].apply(lambda x:  x * 33.86389 )
df['pressure']=df['pressure'].apply(lambda x:  x * 33.86389 )
df['altimeter']=df['altimeter'].apply(lambda x:  x * 33.86389 )
df['dewpoint'] = df['dewpoint'].apply(lambda x: (x-32.) * (5./9.) )
df['inDewpoint'] = df['dewpoint'].apply(lambda x: (x-32.) * (5./9.) )

To modify the data see the article https://github.com/weewx/weewx/wiki/Cleaning-up-old-'bad'-data

You can use the [[FTP]] skin to upload the data to your website so that you can access the data also on the go, not only in your local LAN.

Just set the settings accordingly

  [[FTP]]
    skin = Ftp
    enable = true
    user = ...
    password = ....
    server = ....  # The ftp server name, e.g, www.myserver.org
    path = ....  # The destination directory, e.g., /weather
    secure_ftp = False
    port = 21
    passive = 1
    ftp_encoding = latin-1

Note that I used ftp_encoding = latin-1 because some servers only accept this encoding. if that's not the case for you just comment this line.

Coming soon...