HappyDay Neighbors Web Service Side – Let’s Give Raspberry Pi a Twirl!

I’m getting my head around the collecting and visualizing of power readings.  As in, it’s not hard to collect a lot of power readings.  But what should be displayed and how should it be displayed?  This post is about coming up with a self contained, easy to use data collecting and visualization software that moves my thoughts farther in the direction of what concerns I need to think about before I figure out the best way to build an awesome experience.  I find it is first best to try stuff out.  This way I gain knowledge on what’s the best way to build what I want to deploy to others.  In my previous attempt at building a prototype back end for collecting and visualizing energy data, I used AWS Services. 

Frustrated

Bumbling through the AWS services to do what I want felt like I was wading through something that is gooey, muddy, and slimy.  In fact, I bumbled around so successfully that apparently in just one day I had blown through my free level and started to get charged.  Amazon’s charging capability – seemingly down to the micro transaction – is a true testimonial in their ability to suck up every penny on the table.  Here’s part of a mail I got from AWS billing when I asked about charges: Although your account is eligible for the Free Tier, your usage charges were based on activity outside the specifications of the offer. Please note that the AWS Free Tier provides up to 750 hours per month of a single-AZ t2.micro.elasticsearch or t2.small.elasticsearch instance and 10GB per month of optional Amazon EBS storage (Magnetic or General Purpose), while you were exceeding the limitation of hours.  Hmmm…..I just want to collect energy data and draw a line chart here, a bar chart there….

 

 

So I changed direction on technologies to use for collecting and visualizing energy data:

In this prototype exploration, I  use a Raspberry Pi 3 to collect readings coming in from the energy monitor using mqtt.  The Raspberry Pi will also act as a web server to display visualizations of the energy data.  I’ll use Python as the programming language and hook together:

  • mosquitto – An mqtt Broker.
  • peewee – Makes database inserts/queries easy.
  • Dash – Puts together “the best of” Flask and plotly to make web pages with data visualization.

The Goal

The goal of this post is to build what I call a “sacrificial draft” of the Collect and Visualize component of the HappyDay Neighborhood project.  I will capture mqtt messages from a Monitor reading the current used by a lamp as I turn the lamp off and on.  This is a simple prototype that will give me an idea how all the “goo” fits together.

Step1:Install the Raspbian OS

Ooh – Raspberry Pi…I feel like I get to play with the cool kids.  I haven’t played around with one.I’ll use a Raspberry Pi as the “command device.”  It will receive mqtt messages from energy monitors and store them into a database.  My browser will access a web page on the Raspberry Pi to look at the power readings.  I got a Raspberry Pi in the mail.  Next I need to install an OS on it.  

NewImage

I have a natural instinct at this point to start installs at the Adafruit Learning Center.  I wasn’t disappointed.  I found and followed along.  I decided to give Raspbian Stretch Lite a twirl.  I made an image…and then…hmmm…how do I work on a headless Raspberry Pi? I don’t want to use a separate monitor and keyboard…but still the Raspberry Pi wouldn’t boot.  The challenge was the original format of the SD card.  I bought the cheapest SD micro I could find on Amazon.  I didn’t format prior to using Etcher to burning the Raspbian image.  What worked for me was first formatting the card with SDFormatter (overwrite).

Step 2: Read/Write Files to the Raspberry Pi

I am not fluent with “Unixy” copying and saving files between a Raspberry Pi and a Mac.  I wanted to use my Mac editors on files stored on the Raspberry Pi.  The way to do this is by extending the Mac file system “goo” to include the Raspberry Pi’s drive.

To be able to drag/drop/open files on the Raspberry Pi from within my Mac’s Finder, I:

    • Installed SSHFS from here.
    • Created a directory named ‘mount’ within my home directory.  I.e.: in my case this would be /Users/margaret/mount .
    • Opened a Terminal window and ran the command:
sshfs pi@192.168.1.17: /Users/margaret/mount

Where the IP address is the unique IP address of the Raspberry Pi within my internal network.

Note: The ssh connection will break when the connection is lost – say because the Raspberry Pi or the Mac was rebooted.  So the sshfs command will be run quite often.

Now the Raspberry Pi’s storage appears as a disk within the Mac’s Finder.

Pasted Image 10 9 17 3 32 AM

pi@raspberrypi:~ $ pwd

/home/pi

Looks like the directory on the Raspberry Pi associated with the mount folder on my mac is /home/pi.

Step 3: Install Mosquitto

Moving right along, I need an mqtt broker on the Raspberry Pi to grab the energy monitor mqtt messages.  For this I’ll use mosquitto.  Mosquitto seems to be THE mqtt software to use on the Raspberry Pi.

Run the following on the Raspberry Pi’s command line:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo apt-get install mosquitto mosquitto-clients
pip install paho-mqtt 

Step 4: Test mqtt

OK.  Mosquitto is installed…but does it work?  I tried a few different scenarios to check this out.

Test 1 – command line

There are two command line utilities, mosquitto_pub and and mosquitto_sub.  As with all the other Linux commands I’ve used – get help on each by typing

~ $ mosquitto_pub –help

  • Open a terminal window, logon to the Raspberry PI and type the command:

 $ mosquitto_sub -t test/test1  -d

I.e.: subscribe to mqtt messages brokered on localhost sent to the test/test1 topic.  Debugging info (-d):

Client mosqsub/17589-raspberry sending CONNECT

Client mosqsub/17589-raspberry received CONNACK

Client mosqsub/17589-raspberry sending SUBSCRIBE (Mid: 1, Topic: test/test1, QoS: 0)

Client mosqsub/17589-raspberry received SUBACK

Subscribed (mid: 1): 0

  • Open a second terminal window, logon to the Raspberry Pi and type the command:

$ mosquitto_pub -t test/test1 -m “test message” -d

Client mosqpub/17627-raspberry sending CONNECT

Client mosqpub/17627-raspberry received CONNACK

Client mosqpub/17627-raspberry sending PUBLISH (d0, q0, r0, m1, ‘test/test1’, … (12 bytes))

Client mosqpub/17627-raspberry sending DISCONNECT

The terminal that was running mosquitto_sub received an mqtt message:

Client mosqsub/17589-raspberry received PUBLISH (d0, q0, r0, m0, ‘test/test1’, … (12 bytes))

 

test message

Test 2 – python

———————————-

As an aside, I just checked /usr/bin on my Raspberry Pi and did ls -l python* :

lrwxrwxrwx 1 root root 9 Jan 24 2017 /usr/bin/python -> python2.7
lrwxrwxrwx 1 root root 9 Jan 24 2017 /usr/bin/python2 -> python2.7
-rwxr-xr-x 1 root root 3162224 Jan 19 2017 /usr/bin/python2.7
lrwxrwxrwx 1 root root 36 Jan 19 2017 /usr/bin/python2.7-config -> arm-linux-gnueabihf-python2.7-config
lrwxrwxrwx 1 root root 16 Jan 24 2017 /usr/bin/python2-config -> python2.7-config
lrwxrwxrwx 1 root root 9 Jan 20 2017 /usr/bin/python3 -> python3.5
-rwxr-xr-x 2 root root 3976264 Jan 19 2017 /usr/bin/python3.5
-rwxr-xr-x 2 root root 3976264 Jan 19 2017 /usr/bin/python3.5m
lrwxrwxrwx 1 root root 10 Jan 20 2017 /usr/bin/python3m -> python3.5m
lrwxrwxrwx 1 root root 16 Jan 24 2017 /usr/bin/python-config -> python2.7-config

Jeepers! I had no idea so many versions of Python were installed. For now, I defaulted to using python2.7

———————————-

I tested out running mosquitto commands in python using the script on this page.  Here are some of the results I was getting:

$SYS/broker/connection/Usuario-PC.bridgeConnection/state 1

$SYS/broker/connection/Lucas-PC.bridgeConnection/state 1

$SYS/broker/connection/Servidor-PC.bridgeConnection/state 1

$SYS/broker/connection/caixa02.bridgeConnection/state 1

after awhile I got this error:

  File “mqtt_test.py”, line 25, in <module>

 

    client.loop_forever()

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 1481, in loop_forever

 

    rc = self.loop(timeout, max_packets)

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 1003, in loop

 

    rc = self.loop_read(max_packets)

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 1284, in loop_read

 

    rc = self._packet_read()

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 1849, in _packet_read

 

    rc = self._packet_handle()

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 2305, in _packet_handle

 

    return self._handle_publish()

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 2500, in _handle_publish

 

    self._handle_on_message(message)

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 2647, in _handle_on_message

 

    self.on_message(self, self._userdata, message)

 

  File “mqtt_test.py”, line 13, in on_message

 

    print(msg.topic+” “+str(msg.payload))

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 360, in topic

 

    return self._topic.decode(‘utf-8’)

 

  File “/usr/lib/python2.7/encodings/utf_8.py”, line 16, in decode

 

    return codecs.utf_8_decode(input, errors, True)

 

UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0xe7 in position 31: invalid continuation byte

I’m thinking there are some characters in the topic name that can’t be parsed.  For now I’ll ignore and consider the test successful.

Test 3: Mac to Raspberry Pi Test

Can I publish an mqtt message on my Mac and have the Raspberry Pi get it? Easy-Peasy!

import paho.mqtt.publish as publish
# Based on this example: https://github.com/eclipse/paho.mqtt.python/blob/master/examples/publish_single.py
publish.single(“test/topic”, “Hello Pi”, hostname=”192.168.1.17″)

Whoa.  That was too easy!

Step 5: Store Incoming Energy Messages

Store mqtt messages into a database on the Raspberry Pi.  FINALLY some fun, yah?  According to this post which I’ll take as fact until I learn better: SQLite is built into Python but is only built for access by a single connection at a time.  For now, I’ll use the built in SQLite.  When I scale to include other houses, I’ll have to change to a different database.  I should have a better feel with what data to store, how to store it, and what visualizations work the best.

The other piece of software I’ll  use for database access is the peewee ORM.  You might want to take a read at the material at this link –  if you are new to using the peewee ORM.  I chose to use the peewee ORM because of it’s simplicity.  

Install peewee

$ pip install peewee

Create the EnergyData database

For this step, I wrote EnergyReadingModel.py.  It contains the definition of the power and time variables and creates the EnergyReading.db file. I used Atom as my editor and saved the file to the EnergyDB directory I created (within the mounted directory).

~/EnergyDB $ python EnergyReadingModel.py

Created the EnergyReading.db file. This will be the database where the Energy readings will be stored.

Send Power Readings

The Feather Huzzah with the ATM90 FeatherWing is running the SendEnergyReadingsUsingMQTT.ino Arduino code.  For this part, I needed an mqtt library for the Arduino.  I am using what seems to be the most popular –  pubsubclient.

SendEnergyReadingsUsingMQTT.ino – 

  • Reads the power, current, power factor, and voltage.
  • Sends the energy Readings as an mqtt message formatted in JSON.  I ended up using a very simple name/value format.
    • {“sysStatus”:2,”voltage”:124.5,”current”:10.5,”power”:1278,”pf”:0.75}
    •  I used an online JSON parser to understand how JSON strings are parsed…
NewImage

I use a 100 byte buffer to store the message.  This is lazy of me – what if the message is longer than 100 bytes?  I decided on 100 bytes after calculating the number of bytes in the above JSON string using this letter counting calculator:

NewImage

Since this sample was 67 characters, I assume the JSON string will never be longer than 100 bytes.

Add Power Readings to the Energy DB 

I run the addPowerReadingToEnergyDB.py python app on the Raspberry Pi.  addPowerReadingToEnergyDB.py subscribes to the EnergyReading/# topic.  When it receives a message, it puts it into the energy db.

Note: while writing the python app, I learned about timestamps…which made sense to me to use as the time the energy reading was created.  I did it this way because the Raspberry Pi syncs its clock with ntp.  This means I can use the clock on the Raspberry Pi and not add an additional complexity of a RTC to the Energy monitor. 

>>> import time

>>> ts = time.time()

>>> print(ts)

1507718863.44

>>> import datetime

>>> st = datetime.datetime.fromtimestamp(ts).strftime(‘%Y-%m-%d %H:%M:%S’)

Step 6: Make an HTML5 Web Page

I originally started down a path of Flask + plotly.

Then I figured out Dash was a better choice for visualizing energy readings on a web page.  I found this video gave me a good feel why I decided on Dash.  Dash builds on top of Flask and plotly so I get all the great features of these frameworks with the added benefit of doing everything in Python.  I  installed Dash using the directions in Getting Started.  Here is an image of the plot of energy readings I got after taking energy readings for our microwave over time:

NewImage
I wrote and ran the visualizeEnergy.py python code.  Whoa.  Pretty darn simple.  At least getting started is not hard.  I’ll find out how easy it is to do more complex visualizations soon enough! 🙂
 
Note:
if __name__ == __main__:
app.run_server(debug=True,port=9999,host=0.0.0.0)
 

Since I’m running this on the Raspberry Pi, I set the host to 0.0.0.0 instead of the default 127.0.0.1 (localhost). 

That’s it for now 

My-oh-my it took me a bit of bumbling to get to this point.  The learning part was thrilling.  I’m sure I’ll have many changes to my plans as I continue.  Next, I want to run the python script that adds mqtt messages to the Energy DB as a background service that starts up on boot.  Also, for some reason after “some amount” of time, the Feather starts getting 0 for all the energy readings…hmmm…. I never expect stuff to “just work.” 

 

 

Thanks for reading this far.  Please find many things to smile about.

 

 

 

Advertisements

Trying out the ATM90E26 FeatherWing

I’m excited! I just received Whatnick’s ATM90E26 FeatherWing that I ordered on Tindie.  I got this while I wait for the ones I ordered from Crowd Supply.  I guess I got a bit order happy.  I wanted to support Tisham’s work.

By the Way – I ran across Tisham’s discussion of his work at the Adalaide Maker Faire.  It helped give me perspective.

Spoiler alert – It works great.

The Goal

The goal of this post is to run the Arduino sketch provided by WhatNick on a Feather Huzzah with the ATM90E26 FeatherWing and get expected results for the readings.  I will use the UART option to communicate between the Huzzah and the ATM90 based on this comment made on WhatNick’s Tindie pageThe ATM90E26 has the flexibility to be accessed both over SPI and UART. However the SPI mode it supports is only Mode 3, which is an unsupported mode of the ESP8266 Arduino stack. So I ported my ATM90E26 Arduino interface code over to UART mode with CRC check and everything after a few days of head scratching.

The Arduino sketch I will run is ATM90E26_UART.ino found at this GitHub location.  

Thanks To Those That Went Before

  • To Whatnick for building the ATM90 FeatherWing and writing a very simple to use library for reading/writing to it’s registers using the Arduino IDE.  So far I am very impressed with the quality of work.  From the choice of components, PCB material, soldering, and design.  What a delight.  A HUGE THANK YOU to Tisham.  Without Tisham’s help, I would not have been successful.  THANK YOU.  Oh – and THANK YOU.
  • To Home Depot for carrying just what I need for my DIY projects (except the stuff I need from Adafruit…By the Way – can the two of you merge?) – ESPECIALLY FOR KEN. Ken used to head an electronics shop with 1,000’s of employees.  He works in the electronics section of Home Depot several mornings.  Ken is EXTREMELY knowledgable, helpful, and has stories/no stuff that is just interesting to listen to.  So much so that Home Depot is my favorite place to go. 

Materials

Test Tools

  • Kill A Watt – for comparison current and voltage measurements.  I’ve used this many times.
  • Current Clamp – another tool to compare readings.  This works in the same way the CT works.  

In order to test the ATM90 with the clamp, I made a line splitter.  With this little beauty of a tool, I can check the current use of any device.  I don’t have to go to the breaker box and muck about and mess up the electricity in a way that bothers other family members.

NewImage

 

Making the line splitter was fun/simple.   

  • 1’ green, white, black 14AWG wire $.23 * 3 = $.69
  • 3 wire female plug $2.79
  • 3 wire male plug $3.29
Total cost: $6.77.
 
Just follow the color coding.  Reminds me of coloring pictures that contained info on what color I should use (and by the way – I did not appreciate being told what colors to choose.  I would choose what I wanted.  Did a picture of the sky HAVE to be blue?).
 
Coloring info from this web site:
 
NewImage

Adjusting the Gain

Before I can start accurately measuring voltage and current, I need to calibrate by setting the UGain and IgainL.  This is discussed in section 3.2.4.5 (p. 14) of Atmel’s M90E26 app note.  Since I don’t know what I am doing, I asked Tisham about this.  His reply:

  • There are some videos that may be helpful (yes they are).  If I remember correctly, the first video was great at showing what values Tisham got when running test code.  He also mentions Ugain and Igain.
  • Do not use the calculation for Ugain that is in the app note.  Rather: new_Ugain = existing_Ugain * (expected_Voltage/current_Voltage)
  • calculate new_gain, update register, then make sure to update checksum 2.  If checksum 2 is not updated, the results are bad.
  • After the voltage is calibrated you can use a purely resistive load with known power usage or another meter to calibrate the current gain. The value already in the script should be pretty close.  
    • new_Igain = existing_Igain * (expected_power / current_Power)

Calculate Ugain

new_Ugain = existing_Ugain * (expected_Voltage/current_Voltage)

  • existing_Ugain is set in energyic_UART.cpp CommEnergyIC(0,Ugain,0xD464); //Voltage rms gain
  • expected_Voltage was measured at 124.5V
  • current_Voltage = 192.93.  I got this value from running ATM90E26_UART.ino:
Sys Status:2
Meter Status:801
Voltage:192.93
Current:0.09
Active power:11.00
p.f.:0.65

new_Ugain = 0xD464 * (124.5/192.93) = 54372 * (124.5/192.93) = 35086.891618722 = 35087 = 0x890F

Update Checksum 2

After changing the Ugain setting in energyic_UART.cpp to 0x890F, I reran the Arduino sketch.  Checksum 2 will be off, but as the data sheet points out: The readout value of the CS2 register is the calculation by the M90E26…

The library already writes out the values for the checksums and lets me know if there is an error.  There is indeed an error for Checksum 2.  The new value for Checksum 2 that I read: 0x428C.  I modified this line in energyic_UART.cpp from 

CommEnergyIC(0,CSTwo,0xD294); //Write CSTwo, as self calculated

CommEnergyIC(0,CSTwo,0x428C); //Write CSTwo, as self calculated 

Check Voltage Reading

Here are the results I got after running the Arduino sketch;

Sys Status:2
Meter Status:801
Voltage:123.60
Current:0.10
Active power:7.00
p.f.:0.66

The voltage I measured was 124.5V.  The voltage I’m getting back from the ATM90 is 123.6V.  My thoughts:

  • The reading I am getting back from the ATM90 is “close enough” to the reading I get from the Kill-A-Watt and multimeter.
  • The calculations will not be precise.
  • At this point, I’m not spending any time evolving my knowledge of the amount of error.

IgainL

I compared current readings from several devices.  The current measurements – for the ATM90 compared with the clamp were the same, if not extremely close.  For this reason, I am not adjusting the IgainL.

And…

 

ChallengesBringOpportunites YIPPEE………..IT WORKS!!!!!

But…

WhyCat hmmm… this works for one cable…but I have two:

IMG 6215

hmmm…..

Prototyping the HappyDay Neighbors Energy Monitoring Project

Tags

I’m thinking through the client side of my neighborhood energy monitoring project.  I’m starting to call this project HappyDay Neighbors.  Don’t know if that will be the final name…Naming is hard.  

The energy monitoring experience includes collecting energy readings from our breaker boxes, sending the readings using mqtt to a data store in DAH CLOUD.  The home owner then views the readings.  Through viewing the readings, the home owner will understand how much energy is being consumed and think of ways to decrease energy use.  I also want to explore social engagement using gaming/facebook/twitter…The social part is TBD.  My current focus is on reading and visualizing the data.

The Goal

The goal of this post is to explore a path to visualizing the energy readings.  I’ll do this by building a prototype that:

  • grabs 8,000 energy readings from Tisham’s data on Thingspeak and sends it to a cloud data store using mqtt.
  • shows a histogram of the energy readings in a web browser.

Thanks To Those That Went Before

  • The folks behind the abundant resources and excellent community of the OpenEnergyMonitor project.  The folks in the community as well as the resources are very accessible.  There are many of us who benefit greatly from their ongoing efforts.  I’m excited!  Just today I got an emonPi in the mail.  Hopefully the CTs I have on order will arrive soon.  Then I’ll be able to give the OEM experience a twirl.
  • A HUGE shout out to Tisham Dhar.  Tisham is the creator of the ATM90E26 Single-Phase Energy Monitor Dev Kits.   I have been impressed with the intelligence and care Tisham has taken to Open Source his work.  Tisham also kindly answered all the questions I peppered him with.  His work looks super sharp to me.  

Background Info

My exploration took me on a path that relied on AWS services.  I went down this path because:

  • eventually I want to scale collecting and analyzing energy readings throughout our neighborhood and beyond.
  • I wanted to build as quick and dirty as possible.  This meant relying on a system of services – mqtt broker, data base, graphing – that were loosely coupled together, had a lot of folks using it so that there were a lot of videos and docs on use cases.
  • I am using the prototype to better define the final experience.
When I started this post, I had no idea what AWS services to use.  My learning process was to jump right on in.  I’ll write a bit about what I’ve learned at the end of this post.

There was a lot of goo…um…technology…that I was unfamiliar with. I bumbled my way through relying on mostly these resources:

Open Source

These files are used in the post:

Part 1: Sending mqtt Messages

I wanted to divide and conquer taking energy readings using sensors/hardware from visualization.  This way, I can start  collecting energy readings prior to having the hardware reading.  It will also allow me to focus on the why’s and how’s of the data visualization step.  I’ll read power readings from Tisham’s Thingspeak feed. I started discussing Tisham’s excellent work in this post.

Step 1: Build a File of Power Readings

I did this in my first Jupyter notebook.  As I started putting together the workflow of this prototype, it became apparent that date conversions are hard to get right (…yah, no kidding 🙂 ).  The updated notebook – ReadTishamDataIntoCSV.ipynb.

Sending date/time strings are tricky. Computing loves numbers, humans love strings.  I convert the date/time string to it’s epoch value (a long datatype).

The format TIsham uses to store the date/time format:

%Y-%m-%dT%H:%M:%SZ

Sending date/time strings are tricky. Computing loves numbers, humans love strings.  I convert the date/time string to it’s epoch value (a long datatype).  Here’s the Python script I ended up using to take in the dates in the ThingSpeak feed and convert to an epoch format:

dt = datetime.datetime.strptime(entry[u’created_at’],’%Y-%m-%dT%H:%M:%SZ’)
epoch = datetime.datetime(1970, 1, 1)
epochTime = int((dt – epoch).total_seconds())

NewImage

Entries in the CSV file look like this:

176,1506532776
177,1506532795

i.e.: Energy reading in watts, epoch time when measurement was taken.

E.g.: converting an epoch time format to a human readable format using Python:

>>> import time

>>> time.strftime(‘%Y-%m-%d %H:%M:%S’, time.localtime(1506532776))

‘2017-09-27 10:19:36’

Step 2: Send the Power Readings as mqtt messages 

In order to send mqtt messages, I needed to figure out what mqtt broker to use.  My requirements for a broker included robust availability, storage and visualization of the energy messages.  I ended up choosing AWS services.

I put the code I used into the Jupyter notebook SendMqttToTishamDataTopic.ipynb (located at this GitHub location).  It sends the mqtt energymqtt messages to the topic /clientPrototype/data/Tisham.

To do this I needed boto3: and AWS_CLI.

  • Install the AWS CLI:  From the web page – The AWS CLI is an open source tool built on top of the AWS SDK for Python (Boto) that provides commands for interacting with AWS services. hmmm…I have the Anaconda variant of Python 3.6 installed.  So this seems to mean I’ll be need to use the conda package manager (instead of following Amazon’s instructions, I followed instructions from stack overflow):

$ conda install -c conda-forge awscli

$ conda install -c anaconda boto3

 
NewImage
I needed an access key and secret access key.  I followed the direction to get the access key and secret access key from this Quick Configuration Link.  Next I ran aws configure and copy / pasted the keys.  Here’s what the mqtt messages look like:
{'power': '176', 'time': '1506532776'}
{'power': '177', 'time': '1506532795'}

Part 2: Data Store and Visualization

For data store and visualization I used:

  • the AWS Elastic Search Service as the message store.
  • an AWS IoT rule that triggered when an mqtt message with the /clientPrototype/data/# topic come into the broker.
  • AWS Kibana for visualization.
It is not my goal to give tutorials on the AWS services I used.  While overwhelming at times, once I focused on each one I was able to bumble through using the available videos and tutorials.  I’ll just go through the steps I used after many starts and stops.

AWS Elastic Search Service

  • I created a test domain.
  • If the clientPrototype index (database) exists (from prior testing) delete:
curl -XDELETE search-test-XXXXXXXXXXXXXXXXXXXXXXX.us-west-2.es.amazonaws.com/clientprototype
  • Create the clientprototype index and powerReading type:
 
curl -i -X PUT
   -d
'{
  "mappings": {
    “powerReading": {
      "properties": {
        "time": {
          "type": “date",
          “format": “epoch_second"
        },
        “power": {
          "type": “long"
        }
      }
    }
  }
}
' 'https://search-test-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.es.amazonaws.com/clientPrototype'

NewImage

I followed the example in the tutorial:

  • As I mentioned above, I’m using the epoch date format.   Stuff that took me time to figure out:
    • Removing line breaks from the curl command:
      • Copy/pasted the curl command to create the powerReadingtype within the clientPrototype index of the test domain into TextWrangler to edit.  
      • Remove line breaks using the Remove Line Break option on TextWrangler under the Text menu.  
    • Figuring out the epoch date formatting – both on the Python side as well as the format to use in Kibana. 

 AWS IoT Rule

Here’s the rule I created in the AWS IoT Rule UI:

Filter

NewImage

when a message comes into the /clientPrototype/data/<anything> topic, send it on to Elastic Search.  The topic I set up in the SendMqttToTishamDataTopic.ipynb notebook, /clientPrototype/data/Tisham , will trigger this rule when the mqtt message comes into the AWS broker.  

Action

The action taken:

Pasted Image 9 29 17 10 39 AM

Role

Rules run in the system using a role set up within the IAM UI. 

NewImage

I gave the following permissions:

NewImage

Access to the CloudWatch logs is important for debugging.  I’ll get to that.  The other policy allows the rule to publish into ElectricSearch:

NewImage

 

Sending mqtt messages and publishing into ElasticSearch can now be testing.

Test

The test will:

  • use the AWS IoT test UI to see if the AWS broker is receiving the mqtt messages.
  • evaluate logs in the log file to verify power readings are published into Elastic Search.

AWS IoT Test

NewImage

Evaluate Logs

The AWS services – IoT (message broker) and ElasticSearch can be configured to log debug information.  Monitoring is done through the AWS CloudWatch service.

I couldn’t get it to work until I went through the AWS_CLI command:

aws iot set-logging-options –logging-options-payload roleArn=”arn:aws:iam::XXXXXXXXXXXX:role/service-role/pub_to_es”,logLevel=”DEBUG”

Here are the log entries that are created when the mqtt message is sent:

2017-09-30 17:17:13.659 TRACEID:f872acf1-ccaf-b6f0-6df4-f94a7e91c3c5 PRINCIPALID:XXXXXXXXXX/XXXXXXXXX/AIDAIHBTGHRY27IMGPW2M [INFO] EVENT:PublishEvent TOPICNAME:/clientPrototype/data/Tisham MESSAGE:PublishIn Status: SUCCESS


2017-09-30 17:17:13.659 TRACEID:f872acf1-ccaf-b6f0-6df4-f94a7e91c3c5 PRINCIPALID:XXXXXXXXXX/XXXXXXXXX/AIDAIHBTGHRY27IMGPW2M [INFO] EVENT:PublishEvent MESSAGE: IpAddress: 50.46.122.171 SourcePort: 51507


2017-09-30 17:17:13.712 TRACEID:f872acf1-ccaf-b6f0-6df4-f94a7e91c3c5 PRINCIPALID:393111621300/AIDAIHBTGHRY27IMGPW2M/AIDAIHBTGHRY27IMGPW2M [INFO] EVENT:MatchingRuleFound TOPICNAME:/clientPrototype/data/Tisham CLIENTID:N/A MESSAGE:Matching rule found: ClientPrototypeRule

2017-09-30 17:17:13.712 TRACEID:f872acf1-ccaf-b6f0-6df4-f94a7e91c3c5 PRINCIPALID:393111621300/AIDAIHBTGHRY27IMGPW2M/AIDAIHBTGHRY27IMGPW2M [DEBUG] EVENT:ElasticsearchActionStart TOPICNAME:/clientPrototype/data/Tisham CLIENTID:N/A MESSAGE:Starting execution of ElasticsearchAction on topic /clientPrototype/data/Tisham

2017-09-30 17:17:13.843 TRACEID:f872acf1-ccaf-b6f0-6df4-f94a7e91c3c5 PRINCIPALID:393111621300/AIDAIHBTGHRY27IMGPW2M/AIDAIHBTGHRY27IMGPW2M [INFO] EVENT:ElasticsearchActionSuccess TOPICNAME:/clientPrototype/data/Tisham CLIENTID:N/A MESSAGE:Successfully indexed document in ES. Message arrived on: /clientPrototype/data/Tisham, Action: elasticsearch, Endpoint: https://search-test-tkpccxbid4y7vlelgo6jq55kmi.us-west-2.es.amazonaws.com, Index: clientprototype, type: powerReading, id: 008edff0-11f1-46c3-9e84-76500a5a4771

The log entries are a HUGE help during debugging.  It took me many (many) attempts to get through the workflow.  The log files made it obvious what the problem was.  The problems typically were 1) ill-formed JSON 2) lack of security clearance.

Kibana Visualization

A link to Kibana is available in ElasticSearch once I created the test domain:

Pasted Image 9 29 17 1 02 PM

Once I wrapped my head around Kibana’s main use case – visualizing a gazillion data points..like millions upon millions (e.g.: all the flight data from the past few years…).  I was able to wrap my head around making this bar chart:

NewImage

For the Y-Axis, I ended up using the MAX Aggregation figuring when the data is zoomed in or out this becomes the most interesting for a bar chart given the constraints on setting the Y-Axis.  I.e.: unlike Excel – data stores and visualizers that …well um…excel with data sets in the 10,000’s but are not focused on BIG BIG DATA – that plot individual points, Kibana uses aggregates (hence one of the suggested videos is about Kibana aggregation).

Lessons Learned 

Whew.  Well, it works.  I learned:

  • AWS services are both powerful and overwhelming.  It gets confusing what is free and what ends up being charged. For example, one path I took ended up costing about $14.  This alarmed me because I had no idea I was getting charged…things “just worked.”  I need to get a better feel of free versus costs.
  • Using AWS services is an easy way to scale to many, many energy monitors.
  • ElasticSearch and Kibana are optimized for finding a needle in a haystack. In this case, the haystack is gazillions of data points.  The needle being a behavior that is in the data but needs to be searched out.  Their scenario is one of BIG DATA.  While this is a scenario I am interested in eventually, the primary interest is for evaluating energy readings within a fairly small time frame – a day, a week, a month and/or real time.  Other AWS services for the data store (e.g.: S3) might be a better solution.  This is dependent on the eventual scenarios.  Kibana is not the best visualization because it assumes the scenario is one of a gazillion data points.  Kibana’s context is one of putting data points into buckets to display.  
  • AWS IoT rules are awesome.  It is so easy to trigger an AWS service (like storage of the payload) and set up roles.
  • mqtt works well for sending energy readings from the energy monitor that will be located in the breaker box.
 
Thanks for reading this far…..    Please find many things to smile about.

 

 

For the prototype – where my goal is to tweak out all aspects – hardware, mqtt message passing, storage, visualization, scaling to our neighborhood (and beyond) – I feel the path I took works out.  There are many, many choices from different vendors…and many choices within the AWS family to get to the finish line.  For now, I’m happy with this workflow.

 

 

The Wiring in our House

I continue to learn how our house is wired.  Here’s where all the breaker switches are located:

IMG 6214

We started out with the original box.  Then when we added a generator, the other panel was added.  I have been adding energy monitoring hardware goo to wires in the newer panel.  For example, I live to tell about my installation of a Sense monitor.

IMG 6215

I bought and installed the Sense because I wanted to experience a consumer ready experience and the idea of the Sense figuring out what appliances were in my house intrigued me.  How good was it at doing this?  What I enjoyed most was how easy it was for me to start tracking our energy use.  I didn’t have to figgle around with config files, go through any complex setups – it just worked.  Now I figure I can DIY the hardware and software for my HappyDay Neighbors project.

When I started exploring what CTs to use, I noted:  I measured the wire gauge of the two wires where the Sense CTs are located.  They are  1/0 AWG.  This surprised me since most cable examples I had seen on the Internet used the wider 4/0 cable.

Hold on.  Wait a minute.  It became clear to me I didn’t understand our wiring.  In particular, where the heck is the mains breaker switch?  How is the electricity really flowing in from the transformer?  What Amps? 100A? 200A?..I dunno…

Bumbling back to the original breaker box, here are the innards:

IMG 6225

Hmmm….couple of things I am not sure about:

  • I expected to see a neutral wire in the center.
  • I expected to find a mains breaker.

But I didn’t.  Is this because of the installation of the generator and the newer panel?  Where is the mains breaker?  Why isn’t there a neutral wire?

Getting the Hardware Together – the Current Transformers

I’m (still?) getting up to speed on how electricity flows through our house and how use is commonly measured. ..

The Goal

The goal of this post is to:

  • explain why I chose to buy two SCT-013-000 CTs.
  • write down my understanding of how 240V electricity comes into our house.

Our House

All the energy monitors I know about use Current Transformers (CTs) to read the voltage and current used by appliances within a house.

Our house is wired with a three-wire single phased system.  As the Wikipedia article notes: This three-wire single phase system is common in North America for residential and light commercial applications. In this case, the voltages are 120 V line to neutral (grounded center tap of transformer) and 240 V line to line. This system allows 120 volts to be supplied for lighting and convenience outlets, and 240 volts for higher-demand appliances such as space heating, air conditioning, kitchen stoves or water heaters.

I copied this image from Home Depot’s Basics of Your Home’s Electrical System video:

Pasted Image 9 20 17 3 50 AM

The Learn section of OpenEnergyMonitor has a great article on North American Power Supply characteristics.  It includes this diagram:

As is true with my house: Standard practice among US electricians is to connect the circuits to the breakers in numerical sequence, with odd numbered breakers connected to one leg and even numbered breakers connected to the other leg. Thus, half of them will be on one leg, half on the other leg, with the aim being to balance the load evenly.

The implications of North American Power Supply characteristics to the firmware are briefly covered on the Use In North America/OpenEnergyMonitor’s setup guide.

Ordering CTs

How Many

Given the split of the 240V into two 120V lines, I will need two CTs, one for each line.

What Size

I used my caliper and measured the diameter of each of the 120V cables to be .32” or 8mm.

IMG 6222

According to the AWG wire sizes, this makes the cables 1/0 AWG.  This surprised me since I most cable examples I had seen on the Internet that had similar wiring to our house was 4/0, which is significantly wider.  Perhaps this is because the panel is actually a sub-panel that was put in place when we had a generator installed.  I didn’t open the original (main) panel.  Perhaps the cables for the 2 120V lines are 4/0 in the original panel.

IMG 6214

YIPPEE! This means I can use the common SCT-013-000 (data sheet).  YHDC sells several different CTs.  The SCT-013-000 is the one sold by OpenEnergyMonitor.  I would have ordered from their store.  Unfortunately, I would have to pay to have it delivered from the UK.  Instead I ordered two from Newegg.com.  

 

…so much for that…

Jupyter Notebooks are Awesome

I made my first Jupyter Notebook.  I had installed Jupyter awhile back after listening to Adafruit’s Python Playgrounds LIVE! Youtube video.  The video got me totally inspired by the Python community and the usefulness of Jupyter Notebooks.  The blog post documents my first experience creating a Jupyter Notebook.

Thanks To Those Before

  • A HUGE shout out to Tisham Dhar.  Tisham is the creator of the ATM90E26 Single-Phase Energy Monitor Dev Kits.  I just found out about Tisham and his work.  I have been impressed with the intelligence and care Tisham has taken to Open Source his work.  Tisham also kindly answered all the questions I peppered him with.  His work looks super sharp to me.  I have already learned a lot from what he has Open Sourced.
  • Adafruit for their Python Playgrounds Youtube video.  I learn quite a bit from Adafruit’s learning sections and videos.  This inspirational video got me yearning to play with Python and Jupyter Notebooks.

On To The Show

A few days back, I found out about Tisham’s fantastic work. In particular, the ATM90E26 Single-Phase Energy Monitor Dev Kits.   YIPPEE!!!…but..bummer…I won’t receive them until December 5th.

So I emailed Tisham if there was a way for me to get my hands on some of the data output (power readings over time) so that I can start working on the end user side of the Neighborhood Energy Monitoring Project I decided to start up.  Tisham kindly gave me access to his Jupyter Notebook that pulls data from his feed. 

Not letting the fact that I have an advanced beginner knowledge of Python and no knowledge of how Jupyter Notebooks work, I am happy to say after a few hours, I was able to create a Jupyter Notebook that fetched energy data from Tisham’s ThingSpeak feed, plotted the data, and wrote the data to a CSV file.

 

ChallengesBringOpportunites Talk about exhilarating.  I can see why I am hearing more about Jupyter Notebooks.  What a fun environment.  It reminds me of the times I used Swift Playgrounds when I was programming on the iOS.  A great thing about Jupyter Notebooks are the incredible amount of extremely powerful libraries available – for AI, machine learning, image recognition, plotting…tons of stuff I don’t understand….

 

Now that I have sample energy data to simulate what would be coming from DAH cloud to the web browser and/or smart phones, I can prototype this software as I wait for Tisham’s dev kits to arrive.

New Project: Neighborhood Energy Monitoring

Tags

, , ,

I decided for my next project to build an energy monitor experience for our neighborhood.  My goal is to change our behavior from wasting to saving energy.  

Hardware will be placed in our breaker boxes that monitor the voltage and current being provided to the house.  The hardware includes:

  • CTs on the mains lines (the white circle with clamps on the breakout box image)
  • Sensors, micro controller, and wifi to read in, digitize the current and the voltage, and send the readings to dah Cloud.
  • A web browser or smart phone app knows how to access the current and voltage readings that are stored in dah Cloud.  The readings are displayed as a graph.  This way, we can tell not only how much electricity is being used, but also what appliances are using the most electricity.  Using this information, we can adjust our electricity usage to bring the amount we use down.
  • A social app (on Facebook?  Twitter?  Text messages?) will reward members of our neighborhood with knowledge of how much CO2 they have saved from being released into the atmosphere – as well as other social game techniques.  Social gaming will be a method to get a social bond around energy conservation.  Perhaps from this we can decide as a group to install solar panels on our houses, etc.
I plan to use these blog posts as a way to document my progress.

 

Installing the Sense Monitor

I bought a Sense Monitor.  

NewImage                  NewImage

I am thrilled to see technology that uses a non-invasive method to monitor our home’s power usage than figures out what device is using what power.  I have this thought it would be most excellent to build one as a DIY project….As their product page notes: “Over the next few weeks, Sense will start automatically identifying devices, reveal trends in your power use, and keep getting smarter!”  OOH – I just had to have one.

Thanks To Those That Went Before

  • Ken at Home Depot.  Ken is a major reason to shop at Home Depot over Amazon.  Ken is a retired mega-electrician-manager (truly exceptional) who works at my Home Depot most mornings.  Here is just one of the many examples of why I am a HUGE fan of Ken’s.  Which makes me a HUGE fan of Home Depot.  The Sense installation requires a 2 pole breaker.  Easy enough to Google and figure out about 2 pole breakers.  Easy enough to check in my breaker box and make room for one.  BUT…even after carefully measuring the dimension of the 1 pole breakers, I still would have not know they were QOs. After all, there is no marking on them and until I knew about QO’s I didn’t know to Google QO.  If I did, I would have gotten this Square D Wikipedia page and enlightenment would follow.  Prior to going to Home Depot, I removed a 1 pole breaker from our box.  (aside – silly me – I thought no devices were using this circuit.  Imagine my surprise when I open the refrigerator and it’s power is off…ooopssss….now the time crunch began or my left overs were going to go bad.  Not happening on my watch!).  I bumble my way to the Home Depot electronics aisle, truly delighted to see Ken (another aside – Ken is a really nice person although he wasn’t as enthusiastic to see me.  Perhaps he was being professional? 🙂 ).  I show Ken the 1-Pole switch.  Ken looks at it and starts walking to this bin I hadn’t noticed.  Since there are oh – I dunno – about a gazillion different circuit breakers spanning several bins (another aside – it’s about as confusing to me as the cereal section is in the grocery store.  I mean – how many variants of cereals do we need?  Seriously…but I diverge).  Then Ken pulls out the 2-pole QO and explains the story of QO versus the HOM model. The two different models look similar to me and the display I picked circuit breakers from were the HOM models – which wouldn’t work.  But I’m leaving out the best part.  The best part is the story of why our breaker box has the QO models.  Ken says they are the older model.  More recently, Square D started making the HOM model – for well, homes.  Now, yes.  Any electrician would have known this.  And indeed I could have had an electrician do this.  But what’s the fun in that?  I would have missed talking with Ken and hearing his narratives on why electronic components are the way they are.  Can Amazon provide that?  I think not.  Would I ask the electrician that came to install?  No….  So THANK YOU KEN.

Installation

I followed the instruction in the installation guide, ignoring the part…Installing the Sense monitor is easy, but should only be done by a qualified electrician.  

NewImage

the key to my living through the installation was using one of these:   

IMG 6211

What a wonderful tool. A must for this project.  All I need to do is touch the point to a wire in the break box.  If the wire is hot/live, the tip flashes red and the tool starts beeping. What a lifesaver (literally).

Turning off Power

Here’s the electricity setup in our garage:

IMG 6214     IMG 6213

In between the active panel and the older panel is the “big gray” transfer power from the company that installed the generator.  We’ll make this area even more complicated when we install solar.  But that is in the future.

It was a bit of a search to find the mains switch.  Many years ago we installed a generator that kicks on when there is a power outage.  After googling a bit, I found terms like “sub-panel” that helped me realize the mains breaker switch was on the other panel labeled “sub-panel.”   

Connecting the Power

Connecting the power requires the use of a double pole breaker.  Luckily, we have room in our breaker box.

IMG 6187

I told the story of getting a 2 Pole breaker earlier.  Following the rest of the directions that came with the installation info with the Sense is easy.  

IMG 6215

 

All the pieces fit nicely into the breaker box.  I thought the design was well done.  The antenna fits nicely.  To insert it, I just removed a knockout.  The CTs fit easily around the main cables.  As I noted, installing the 2 pole breaker and connecting the three wires (red, black, and white) was easier than working with a breadboard.

Wait

So now I wait for the Sense to detect our devices.  The software says to give it a week…so I wait…I wait….but YIPPEE!!! I’m excited to see how well device detection is and what conclusions can be made about our current energy consumption.  Will what we find out change our behavior?  I come into this experiment assuming we waste a significant amount of energy we don’t even realize we are wasting.  I’d like to stop that.

 

 

Thanks for reading this far.  Please find many things to smile about.

Summer Project: Automatic Irrigation

The summer is pretty much over so I decided to finish up my summer auto irrigation project with a post.  I’ve had a few posts about the progress I have made on my summer project to implement an automatic irrigation system.  I have not had a summary post describing the different components.

The Goal

The goal of this post is to document the project.  Included:

  • a high level documentation of the auto irrigation system I implemented.
  • links to the code and anything else needed to rebuild the system.

Thanks To Those That Went Before

  • Adafruit for great products and services.  Their prototype stuff and Adafruit.io made this project possible.  What an excellent company.  THANK YOU.
  • Ray’s DIY Electronics Hobby Projects – An exceptional open sourced open sprinkler hardware and software project.  In particular, Ray provides thoughtful blog posts on stuff like how to use a 24VAC irrigation directed power source to also power a microcontroller.  THANK YOU.

Open Source

The Arduino sketches for the Controller, Moisture Puck and Watering Puck are located at this GitHub location.

High Level Diagram

The Job of the Auto Irrigation System is to water the plants when the soil is dry.  It is “smart” about this by not overwatering and watering at the best time of day to do so.  The  Auto Irrigation System consists of three pieces of hardware:
  • The Moisture Puck whose job is to send moisture info to the Controller.  I covered aspects of the Moisture Puck in this post and in this post.
  • The Controller whose job is to coordinate between the Moisture Puck, Watering Puck, and Adafruit IO.
  • The Watering Puck whose job is to turn on/off the irrigation valves.  More on the Watering Puck in this post.

A Bit About the Hardware

Since I didn’t know what I was doing when I started, all hardware is prototype.  It is either too expensive, too cobbled together, or not robust enough to be “mass produced.”  However, I am happily using the system.  I like to think and evolve these type of projects.  My hope is next year I will feel inspired to improve….until one day…who knows?  Every member of my family is using this system :-).

Interaction Between

The Moisture Puck, Controller, and Watering Puck talk to one another using the RFM69 wireless protocol at 915 MHZ.  The Moisture and Watering Puck use the Adafruit Feather M0 RFM69HCW.  

The three pieces of hardware wirelessly communicate with one another using the RFM69 wireless protocol at 915 MHz.

  • When the Moisture Puck is powered on, it requests and receives the current time (i.e.: current hour, minute, second).  This gives the Moisture Puck the knowledge to know when it is 4AM.  4AM is the time I determined is best to water the plants if they need watering.
  • At 4AM every day the Moisture Puck takes a moisture reading and sends the reading along with readings for the battery level and chip temperature of the RFM69 to the Controller.  The Controller checks to see if the reading is below the threshold for determining if the soil is dry.  If it is, the Controller tells the Watering Puck to water the soil.

Interaction With The Internet

The Controller talks not only to the pucks, but also to Adafruit.io over…DAH INTERNET….So the “heavy lifter” is the Adafruit Feather ESP8266 Huzzah.  Sitting on top of the Huzzah is the RFM69 FeatherWing.  The Huzzah yaps it up with Adafruit.io and runs the Arduino sketch.  The FeatherWing yaps it up with the Pucks.

Keeping Private WiFi Password, SSID and Adafruit.io AIO_USERNAME, AIO_KEY

I used the WiFiManager library to keep the WiFi stuff:

#define WLAN_SSID       “XXXXX"
#define WLAN_PASS       “XXXXX"

private.

Then there is the Adafruit.io private stuff:

#define AIO_USERNAME    “XXXXX"
#define AIO_KEY         “XXXXXXXXXXXXXXXXXX"

I put these into an include file (AdafruitIOLogin.h).  I added AdafruitIOLogin.h to .gitignore when pushing the sketches to GitHub.

Adafruit.io

I created a dashboard on Adafruit.io:

NewImage

and several feeds:

NewImage

I ended up not using the water feed.  The water feed is fed by an IFTT channel.  The channel lets me know when sunrise and sunset occur.  I was going to base watering on these values.  However, after thinking through watering, I decided it best to water at 4AM each morning.  I like knowing how the sunset and sunrise change over the days so I leave the feed in.

The feeds I pay the most attention to are the ControlInput and Log feeds.  

Control Input Feed

The ControlInput feed accepts commands that are sent to the Controller.  The challenge with using commands through Adafruit.io to get to the Controller is about 1/3 of the time I have to enter the commands multiple times.   To make sure a command completes, I open two Adafruit.io web pages – one focused on the ControlInput feed and the other focused on the Log Feed.  More on the log feed in a bit.

The commands include:

  • threshold=<value>  where the default value is 215.  Notice in the above images the moisture reading was 246.  The Controller checks this reading against the threshold value to figure out if the watering puck should water.  When the threshold = 215… 246 > 215 so watering does not happen.  If I enter the command threshold=247, 246 < 247 so watering would occur.  The Controller remembers the threshold value and will use it as long as the Controller is not rebooted.
NewImage
  • zones= <zone color> where zone color can be blue, green, yellow or a combination.  For example zones=yellow,green will tell the Controller to only ask the watering puck to  turn on the valves that are labeled yellow and green.  
IMG 6138

I put in three valves and color coded them.

NewImage
Notice the zone=yellow in the ControlInput feed.  The command that the Controller recognizes is zones= (or ZONES = …or ZoNeS=…).  The log feed gives me feedback on whether command is correct.  In this case, I received:
NewImage
  • start commands the Controller to tell the watering puck to start watering.  This command is useful when I want to force watering which was the case when I added an irrigation line to water our trees.
  • stop commands the Controller to tell the watering puck to stop watering.

Log Feed

I use the log feed to know whether a command got to the Controller and as a way of debugging.  I broke up the Controller’s activities into states.  The below logged the Moisture Puck sending data to the Controller and the Controller checking the moisture reading against the threshold.  In this case, the reading was above the threshold so no watering.

NewImage

Here is a section of the log feed when the start command is entered:

NewImage

IFTT

To explore IFTT integration with Adafruit.io, I explored these feeds:

NewImageNewImage  NewImage       Pasted Image 9 12 17 4 56 AM

The only one I am using to control the system is the battery level notification that goes to my cell phone.  I am having a challenge with the LiPo battery.  All of a sudden, readings went from somewhere around 4V to 3V on the next reading. The readings started waffling around 3.2V.  I think (not completely sure) the battery got damaged from being out in the garden when temperatures on the battery’s surface got around 100˚F.  If this keeps happening, I’ll look into it more.

And That Is That

The great news is everything works and I learned a lot.  Plus, there were issues with our plumbing that this project surfaced and we then fixed.  There is more I can do to make the system more robust and all that good stuff.  But for now, I am saving a lot of time not having to hand water the different areas of the house.

On To The Watering Puck

I got automatic watering working almost a month ago.  Then I broke it:

  • Turns out there was a problem with the water valves in the house.  These had to be fixed.
  • As I got more familiar with RF69 traffic / library, I realized RF69 messaging is best done to completion within (a tight) loop().  I had spread out the RF69 messages into functions in which I would loop through expecting a specific message.  The challenge with this is the Controller sent/received messages in an unpredictable way because the Controller is dealing with both MQTT traffic and commands as well as RF69 traffic.

The Goal

The goal of this post is to document the implementation of the first beta of the Watering Puck. 

Thanks To Those That Went Before

The Valves

I decided it made the most sense to use off-the-shelf solenoids from Orbit and a 24VAC power supply.  24VAC is the standard used in irrigation systems.  

Orbit Valve Box Base (YouTube video description

 I also got a “professional looking” enclosure, base and 24VAC Power (all Orbit products).  I watched this video on getting the wiring straight.

The Relays

I am using these three relays.  The cost was  ($13 for 4)/4 + $8 shipping (I ordered 4)  = $11.25 (high shipping cost).

Always looking for a good explanation on how relays work.  I found dial2fast’s How Does a Relay Work Youtube video to be very clear.  The DC current flows through copper coils inside the relay.  This generates a magnetic field which moves the switch to the other side.  NO (Normally Open) moves the arm from the open circuit to closing the circuit and letting the 24VAC flow through the valve.  

This image from 12VoltPlanet works for me:

NewImage

Since I am not smarter than a 5th grader, I thoroughly enjoyed these videos on how electricity can be used to create a magnetic field as is done within this relay.  In UniServeScienceVIDEO’s Magnetic Field Demonstrations Youtube video, there are three demonstrations that are fun to watch.  Jeff Regester’s Magnetic Field of a Coil of Wire Youtube video demonstrates a compass spinning when electricity goes through the coil.  I like both. UniServe’s is great in that it shows the motion of the magnetic field.  Jeff;s is awesome because it shows the standard physics class example.

Powering the Feather using the 24VAC

I want to run the Feather off the same 24VAC Power supply being used by the valves.  I’ll do this by adding a way to smooth out the AC waves into a flat DC line as well as a way to down shift the power from 24VDC to 5 VDC and connect the Feather through the USB port.  

First Attempt

My first attempt used this setup:

  • Rectifier circuit to convert AC to DC.  I’m thrilled to see the need for the rectifier circuit…of course, I couldn’t leave home without it if I’m ever taking AC to DC…I spent time understanding peak detection/rectifier circuits when I designed and built the Ladybug Shield.  Based on this experience, and the great advice in Dave’s excellent EEVBlog #490 Peak Detector Circuit.  I am going to use the very simple diode + capacitor peak detection circuit Dave talks about at the beginning of his video.  I am not doing any precision measuring as was the case with the EC circuit of the Ladybug Shield.  Rather, I just want to get the signal ready as input to the DC-DC Buck Converter.  While the diode will eat up about .7V, I don’t need to adjust anything because I’m not interested in the actual DC value.  Sure this method of rectification leaks current and drifts over time.  However, for this scenario that is ok.
  • DC-DC Buck Converter – The VDC output after using the rectifier circuit measured 40V. 
My first attempt “sort of” worked.  The DC signal was very noisy.  I was having trouble at this point with the electronics of the watering puck so I decided to do a better job with going from 24VAC to 3.3VDC.

Second Attempt

  I decided to use the schematic I found on RAYSHOBBY.NET:

NewImage

Since I easily forget how different components – such as diodes – do their magic, I thought Sparkfun’s diode tutorial was helpful. 

Parts I need:
  • D2 – 1N5819 Schottky Diode (Digikey)
  • C1 – 100µF / 50V Capacitor (Digikey)
  • L1 – 150 µH / 800mA Inductor (Digikey)  …argh I forgot to order this, so I ran down to our local electronics store.
  • IC1 – MC34063 (Digikey)

Parts I have:

  • D1 – Diode
  • C2 – 220µF/6.3V Capacitor
  • RT – 30K Resistor
  • RB – 10K Resistor
  • CT – 100pF Capacitor
  • RSC – 2 1 Ω Resistors
As I put this together on a prototype board, I realize my skill at layout of these simples skills could use a lot more practice. hmm.. I went put together two prototypes….
IMG 6101          NewImage
…and yet…sadly…I did get to see the inside of a capacitor…hmmm…
on to my third attempt.

Third Attempt

This time I’m going back to the first attempt – that is using the DC-DC Buck Converter.  However, the rectifier design I’ll use will be the more effective full wave rectifier.  Afrotechmods did an excellent YouTube video on this type of rectifier.

 

NewImage

The above image shows the difference between using a half-wave and full-wave rectifier.  The full-wave rectifier uses both the positive and negative parts of the AC to recharge the capacitor.  Much better.  I’ll use that technique.

Capacitor Size

What size of capacitor should I use?  Afrotechmods shows a design with a 1µF capacitor when there is no load:

Pasted Image 9 2 17 4 27 AM

The voltage will be 3.3V, the amount of current drawn will be about 35mA.  I guesstimate 35mA based on the RFM69 tests I did in a previous post.  Using the formula noted in this post:

NewImage

I = .035A

t = 1/60 Hz  = .0167s (NOTE: See DOH! below…)

for the ripple I’ll use 1V.

C = .035 * .0167 / 1 = .0005845  = 585 µF.  I need a capacitor that is at least 585µF that can handle around 50V.  I found this 680µF/50V capacitor at my local electronics shop.  The voltage ripple I should see V = .035*.0167/.00068 = about .86V.  Even better!

So crafty me…I built one:

IMG 6116

but…

Simpson DOHum… I screwed up for the time period.  Full wave is at 120 Hz, not 60 Hz.  So:

t = 1/120 = .0083

C = .035 * .0083 / 1 = .0002905  = 291 µF.  

DOH-DEE-DOO..I’m leaving the 680µF capacitor.  Overkiil, but from what I can tell should mean even less ripple.

 

The Firmware 

Arduino Sketch

The firmware code, WateringPuck2017_v3.ino, is located at this GitHub location.

Logic Pins

I needed to make sure I used available logic pins…Looking at the Feather m0’s logic pins

 I’m going to NOT use:

  • any pin between A0 and TX1
  • pin 13 since it is used by the LED
  • the SCL and SDA pin
ChallengesBringOpportunites And YIPPEE!!!! It all works…