New! Improved! The Energy Monitor is Running Circuit Python

Tags

This post is about evolving the energy monitor firmware to use Python instead of the Arduino IDE.  It’s been awhile since I discussed the pieces of this project.  Most of this post is about the firmware running on the Energy Monitor.

Here is the “bigger picture” of the HappyDay Neighbors prototype:

The prototyping environment Circuit Python enables makes me smile.  It is for this reason I evolved the environment such that the Energy Monitor:

The Goal

The goal of this post is to run a Circuit Python script on the Huzzah that publishes energy readings to an mqtt broker.  I want to document what I have done to ensure I and others can have the opportunity to not repeat my bumblings or gain a spot of insight that shortens the amount of time to get a prototype out.

Open Source

Circuit Python Libraries

We need access to the ATM90e26 and mqtt.  Here’s how I did it:

We also need Adafruit’s spi_device.mpy

Python Scripts

There is a Python Script running continually on the Huzzah and on the Raspberry Pi.  The Huzzah’s script is using mqtt to publish energy readings.  The Raspberry Pi subscribes to the mqtt messages and then appends them to a local file.

Huzzah

  • The python script running on the energy monitor’s Huzzah is main.py.  NOTE: I don’t like storing my WLAN ssid and password on GitHub.  So to run main.py, there also needs to be a file network_credentials.info in the same directory as main.py.  There is one line in the file:  ssid,pwd:
def init_network():
  full_path = os.path.realpath(__file__)
  dir_path = os.path.dirname(full_path)
  full_filename = dir_path+/network_credentials.info
  ssid =
  pwd =
  with open(full_filename,r) as file:
  for line in file:
  ssid,pwd = line.split(,)
  wlan = network.WLAN(network.STA_IF)
  wlan.active(True)
  wlan.connect(ssid,pwd)
  while not wlan.isconnected():

Stop…STOP!

NOTE: The challenge at this point is stopping the While True: loop so we can fix the code as needed.  The code does nothing to allow us to cut in.  Rather, it runs continually…sleeping for some amount of time and then taking another reading.  When I was trying to access the Huzzah with main.py running using ampy, I got:

ampy.pyboard.PyboardError: could not enter raw repl

What to do? What to do? ( First, I thank @deshipu on Adafruit’s circuit python).

$ screen /dev/tty.SLAB_USBtoUART 115200

ctl-b into REPL

>>>import os

>>>os.rename(‘main.py’,’someothername.py’)

Yippee!  there is no longer a main.py…so now I can get into ampy! 

Raspberry Pi

The Raspberry Pi is running the mosquitto mqtt broker.

This will create and add energy readings to the HappyDay_Readings.csv file in the same directory as the python script.  
 
It is easy to get the CSV file onto another file store by using sshfs.  I discuss how to set up sshfs in this post.

Readings

Here is a link to a Google spreadsheet that contains the readings. A previous post describes taking energy readings in North America and why there are two voltage, current, etc. readings.  This is why there is a v1 and a v2…etc.

That’s It

ChallengesBringOpportunites A very HAPPY time.  It works…. at least for now :-).  Thanks for reading this far.  Please find many things to smile about.

 

 

Advertisements

Using mqtt on Circuit Python

Now that I have a Circuit Python library for the ATM90e26, I need an mqtt library.  I didn’t find one within Circuit Python, but I did find Micropython’s umqtt.simple library.  How hard can it be to use the umqtt.simple on the Huzzah to send mqtt messages from a Circuit Python App?

The Goal

The goal of this post is to document the steps I took to send an mqtt message from a Feather Huzzah to a Raspberry Pi.

Open Source

Get umqtt.simple Library Up-n-Running

The umqtt.simple library is located here.   I copied the contents to my local folder as umqtt_simple.py on my Mac for editing with Atom.

MQTT address and port

The DNS name of my Raspberry Pi is ‘raspberrypi.home’ …I found this by looking at the IP addresses in Angry IP.

I had installed the Mosquitto broker on the Raspberry Pi.  To better understand traffic, I started mosquitto in the verbose mode.

  • Stop the currently running instance of Mosquitto:  sudo killall mosquito
  • Restart in verbose mode: mosquito -v


I left the window I started mosquitto in open so I could watch the mosquitto messages fly by.

1509813288: Opening ipv4 listen socket on port 1883.

1509813288: Opening ipv6 listen socket on port 1883.

ooh! Looky looky…port 1883!…I guess this means mqtt connections come in as SSL….

but OOPS….the code in simple.py

if port == 0:
  port = 8883 if ssl else 1883

Isn’t right…so I replaced it with:

if port == 0: ####!!! BUG IN SIMPLE.PY
     if ssl:
           port = 1883
     else:
           port = 8883

Connecting Huzzah to WAN

I seemed to only have to do this once…I would think I should try connecting every time I start an mqtt conversation…I thought this python code was a “nice and robust” way to include connecting to the wifi.

Anywho, I assume you know how to connect to a wireless network with micropython/circuitpython…

import network
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect(’ssid’,’password’) 

Test

I started up the mqtt client on the Raspberry Pi:

 $ mosquitto_sub -t test/test1 -d

 
 
I then used ampy to run this small test program (mqttTest.py):

from umqtt_simple import MQTTClient
c = MQTTClient(‘mqtt_client’,’raspberrypi.home’,ssl=True)
c.connect()
c.publish(b’test/test1′,b’hello’)
c.disconnect()

Fix What Didn’t Work

UPDATE (11/5/2017):  uncommented…instead, access umqtt_simple with:

c = MQTTClient(‘mqtt_client’,’raspberrypi.home’,1883)

(Include the port number in the call.  I don’t understand if the Raspberry Pi is using 1883 and NOT assuming SSL?…)

I also had to comment out these lines from umqtt_simple.py:

# if self.ssl:
#      import ussl
#      self.sock = ussl.wrap_socket(self.sock, **self.ssl_params)

Retest

Results from mosquitto_sub:

Client mosqsub/1328-raspberryp received PUBLISH (d0, q0, r0, m0, ‘test/test1’, … (5 bytes))

hello


Log lines from mosquitto -v:

 

1509818043: Received PUBLISH from mqtt_client (d0, q0, r0, m0, ‘test/test1’, … (5 bytes))

 

1509818043: Sending PUBLISH to mosqsub/1328-raspberryp (d0, q0, r0, m0, ‘test/test1’, … (5 bytes))


Yippee

ChallengesBringOpportunites  Seems to be working.  Thanks for reading this far….please find many things to smile about.


 

Porting energyic_spi.cpp (Arduino Lib) to Circuit Python

I’m having the best time with Python.  It is a very easy language to use and it’s interpretive nature makes it a dream to prototype with.

The only thing preventing me from using it on the Feather Huzzah to talk to the ATM90 FeatherWings is a SPI library for the ATM90.  But wait, Tisham wrote a SPI library for the Arduino IDE.  How hard can it be to port the library to Circuit Python?

The Goal

The goal of this post is to document the steps I took to convert the ATM90E26 SPI Arduino library to Circuit Python.

Thanks To Those That Went Before

  • Tisham Dhar (company is whatnick) for the ATM90E26 FeatherWing, supporting Arduino Library, his kindness, and his willingness to share knowledge.  This post is all about porting Tisham’s Arduino library to Circuit Python.  Porting was MUCH easier thanks to having the Arduino Library available.  If not the least to see how funky SPI seems to be on the ESP8266.  For example, the Arduino library sets SPI to MODE2 for the Huzzah…when in reality the ATM90e26 speaketh MODE 3.  This is still weird to me…grrrr…but once I figured out a potential “kludge”, I was able to fix my challenges with SPI…although I also got help from my next person I am incredibly thankful to:
  • Scott Shawcroft (aka tannewt)…I’m bumbling around trying to get the SPI gunk to work with Circuit Python.  I go to Adafruit’s Circuit Python Discord Chat…I ask a question that is stumping me.  Scott takes several (several) minutes to help me figure stuff out…and….SHAZAM… the SPI stuff falls into place.  During the debugging process, Scott was incredibly sharp, helpful, and kind.  Which gets me to my next round of thanks..
  • Tony Dicola I learn A LOT from Tony’s tutorials and videos.  The particularly useful tutorial for this post: Porting an Arduino library to CircuitPython: VL6180X Distance Sensor.
  • Adafruit – THANK YOU for the community you have built, the hardware and software that I seem to constantly crave.  I can only hope more companies see you as a role model for how to run a successful, community driven company that focuses on the positive.
THANK YOU THANK YOU THANK YOU THANK YOU THANK YOU THANK YOU…..THANK YOU…..AND MORE THANKS TO YOU…..

Open Source

    • HappyDay_energyic_spi.py : Circuit Python library for the ATM90e26…the protagonist of this post.
    • PyspiTest.py : Lil’ test app.  It imports HappyDay_energyic_spi.py and calls into the functions … like getSysStatus() ….

Stuff I Did

Install Circuit Python on a Feather Huzzah

Here’s what I learned from installing Circuit Python:

1. Use the esptool to erase the Huzzah’s flash and install Circuit Python.  I didn’t have this on my Mac, so from a terminal window…: $ pip install esptool

 Now that I had the tool that knows how to r/w the Huzzah’s flash, I needed to know the serial port the Huzzah was connected to.  From a terminal window….$ ls /dev/cu*

2.  PRACTICE SAFE FLASHING…START BY ERASING.  Sorry for shouting but I wandered down a gnarly pit before I practiced erasure.  By gnarly pit, I changed wifi’s and Circuit Python was NOT happy…Circuit Python would continually loop:

File “inisetup.py”, line 9, in wifi
OSError: can’t set AP config 

 

Adafruit CircuitPython 2.1.0 on 2017-10-17; ESP module with ESP8266
>>> scandone
no <SSID name> found, reconnect after 1s
reconnect

From a terminal window….

$ esptool.py -p /dev/cu.SLAB_USBtoUART erase_flash

3. Get the copy of Circuit Python from  Adafruit’s GitHub location that holds Circuit Python releases.  I renamed the binary to a simpler name (huzzah.bin)

4. Jumper GPIO0 and GND so the ESP826 is in boot mode.

5. From a terminal window…  $ esptool.py -p /dev/cu.SLAB_USBtoUART –baud 460800 write_flash –flash_size=detect 0 huzzah.bin

Check Install

Easiest way I found to check install was to go into REPL: 

$ screen /dev/tty.SLAB_USBtoUART 115200

OKIDOK.  Got the >>>

All is well.

Get ampy Going

Unfortunately, I can’t do a USB mount on the Huzzah.  I ended up using the  ampy tool.  I used this convenient tool to copy my Python library code to the Huzzah as well as run my test app.

Note: I didn’t do this but:

If you don’t want to constantly specify the –port option you can set the AMPY_PORT environment variable in your terminal session and ampy will use it as the board’s serial port.

$ export AMPY_PORT=/dev/tty.SLAB_USBtoUART

Write Python Library

The ATM80e26 Circuit Python library is called HappyDay_energyic_spi.py.  I’m not going to explain the code because not only is it similar to the Arduino library, Python code seems easy to understand.  Beside, you write better Python code than I do.

Resources

$ ampy -p /dev/tty.SLAB_USBtoUART put spi_device.mpy  /lib/spi_device.mpy

(after creating the /lib directory using amp’s mkdir command) 

Adjust the Voltage Gain

THIS IS IMPORTANT.  I discuss adjusting the ATM90c26 voltage gain to a specific setup in this post.

 

Figure out SPI Gobly Gook

By “Gobly Gook” I mean the SPI phase and polarity.  Now I got (ok, perhaps totally) confused on this.  The Arduino library sets the SPI Mode for the Huzzah to Mode 2.  But, the ATM90e26 speaketh in Mode 3.  Grr…

NewImage

From the data sheet (4.1): Data on SDI is shifted into the chip on the rising edge of SCLK while data on SDO is shifted out of the chip on the falling edge ofSCLK.

Bytes from the ATM90e26 Meter Status Register

The Meter Status Register is:

#define EnStatus 0x46 //Metering Status

The SPI section of the data sheet notes (section 4.1.1/Read Sequence) notes the high bit of the MOSI request is set to 1 when asking to read a register.

0x46 | 1 << 7 = 0XC6

Read Meter Status Register Request 

MOSI == SDO (Master Output —> Slave Input) -> Falling Edge: E.g. MOSI byte: 1100 0110 = 0XC6 

ArdspiTestMeterStatusMOSI

Image on Imgur 

Data Returned From ATM90e26 

MISO == SDI (Master Input <— Slave Output)  -> Rising Edge E.g. First MISO byte: 1100 1000 = 0XC8

ArdspiTestMeterStatusMISO

Image on Imgur 

Set up The Init Function

This led me to:

def __init__(self,spi_bus,cs):self._device = SPIDevice(spi_bus,cs,baudrate=200000,polarity=1,phase=1)

I set the baud rate to what the Arduino library had… then there is the polarity=1, phase=1 (i.e.: SPI mode 3).

Figure Out the GPIO Naming

As you can see in the PyspiTest.py test app,  board pins are used to set the CS:

cs = digitalio.DigitalInOut(board.GPIO0)

before creating an energyic_SPI instance.  i.e.:

>>> import board

>>> dir(board)

[‘ADC’, ‘GPIO16’, ‘GPIO14’, ‘SCK’, ‘GPIO12’, ‘MISO’, ‘GPIO13’, ‘MOSI’, ‘GPIO15’, ‘GPIO2’, ‘GPIO0’, ‘GPIO4’, ‘SDA’, ‘RX’, ‘TX’, ‘GPIO5’, ‘SCL’]

Run The Test App

s margaret$ ampy -p /dev/tty.SLAB_USBtoUART run PyspiTest.py

Here are results that I got without using a CT or power calibration.  It’s the same results I received when I ran the Arduino library:

Checksum 1:4A34
Checksum 2:D294
Sys Status:0
Meter Status:C801
Voltage:0.03
Current:0.01
Active power:0.00
p.f.:0.00

That’s It

ChallengesBringOpportunites  YIPPEE!!! IT WORKS.  Thanks for reading this far.  Please find many things to smile about.


It Works! Monitor Energy Use

ChallengesBringOpportunites  Yippee!! I clamped the CTs on the incoming power cables:

Pasted Image 10 26 17 4 27 PM

Attached the CTs to the Energy Monitor:

IMG 6330

Plug in the 9V AC Transformers:

IMG 6331

Get the Raspberry Pi to collect the data and create a visualization:

Pasted Image 10 26 17 4 35 PM

I compared results to the readings I get from the Sense monitor.  The Sense monitor’s readings are typically about 70 watts more.  A 70 watt difference seems “good enough” for me.

As I mentioned in a previous post, I used Dash (plotly) to visualize.  While the current visualization is ugly/course, it is a great start with very little code required.  Just by using Dash/plotly, I’m able to zoom into sections of the plot and get the value of individual data points:

NewImage

I zoomed into the area of the plot which monitored energy use while I was making coffee.

It is exciting – and a relief – to get to plotting power readings.  There’s some stuff I need to do before I install into my neighbor’s breaker box…but this is a nice milestone.

Setting up the Raspberry Pi for WiFi

This is a very short post to document how to connect a Raspberry Pi to an iPhone’s “Personal Hotspot.”

I’m about to go on a short trip.  I want to take my Raspberry Pi with me.  Currently, the Raspberry Pi is connected to our network using Ethernet.  I want to access using WiFi.  The challenge with that is I need WiFi access when I am not around a wireless access point.

I decide to use my iPhone’s “Personal Hotspot” feature.

 

I tried two different methods and found Adafruit’s steps for command line configuration of WiFi to work.  The directions walked me through modifying the interfaces.d file:

sudo nano /etc/network/interfaces

~ $ ifconfig wlan0

wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 172.20.10.5  netmask 255.255.255.240  broadcast 172.20.10.15

        inet6 fe80::2a5f:78b0:ba56:b15f  prefixlen 64  scopeid 0x20<link>

        ether b8:27:eb:8f:08:40  txqueuelen 1000  (Ethernet)

        RX packets 4  bytes 910 (910.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 35  bytes 6213 (6.0 KiB)



        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


I change to my iPhone’s personal hotspot SSID, and YIPPEE! I connect with my Raspberry Pi.

BUT….the Ethernet was gone on reboot.


# interfaces(5) file used by ifup(8) and ifdown(8)

 

# Please note that this file is written to be used with dhcpcd

# For static IP, consult /etc/dhcpcd.conf and ‘man dhcpcd.conf’

 

# Include files from /etc/network/interfaces.d:

source-directory /etc/network/interfaces.d

 

auto lo

auto eth0

 

iface lo inet loopback

iface eth0 inet dhcp

 

allow-hotplug wlan0

auto wlan0

 

iface wlan0 inet dhcp

        wpa-ssid “Mom’s 6”

it turned out I was missing auto eth0 in the interface.d file.

Other useful commands: 

ifdown <config>

ifup <config>

Where config = wlan0 or eth0

Challenge – Monitoring Three-Wire Single Phased System

Three-Wire Single Phased System

I noted in a previous post,  North American houses are wired with a three-wire single phased system

Pasted Image 9 20 17 3 50 AM

Learning Resources:

Measuring Power

The OpenEnergyMonitor reference does an excellent job describing the setup of CTs for their products:

 As well as details on how to calculate power with this configuration:

NewImage

The OpenEnergyMonitor reference notes: “Because there are three wires (discounting the protective earth conductor), classical theory dictates that two wattmeters are needed, and for ‘wattmeter’ read a pair of voltage and current measurements. Unfortunately, the emonTx has only one voltage input, therefore a compromise is required. Fortunately, because the voltage balance between the two legs is good, little error is introduced by assuming the voltages are equal in magnitude. However, two current measurements are always necessary.” 

The ATM90 FeatherWing measure both CT and Voltage for a circuit.  This means, besides two CTs, two voltage transformers are needed.

Testing 3-Wire Single Phase Power Measurements with ATM90 FeatherWings

Hardware

I am using:

  • 2 ATM90 FeatherWings.  The Arduino Sketch talks to each through SPI.  One FeatherWing is set to CS =0 and the other to CS = 15.
  • 2 SCT-013-000 CTs  (data sheet).
  • 2 9V AC-AC Transformers.

Firmware

I wrote WholeHousePowerReading.ino to read the two power measurements and send the sum of the two over mqtt.

I decided to send only the power readings:

void myEnergyMonitor_getReadings() {
  // Check to see if the ATM90 is working...if the system status returns 2, it is...
  bool atm90needsReset = false;
  if ( (eic0.GetSysStatus() != 2) || (eic1.GetSysStatus() != 2) ) {
    atm90needsReset = true;
  }
  if (atm90needsReset) {
    myEnergyMonitor_init();
    strcpy_P(energyReadingJSON, msgRESET);
    return;
  }
  int wholePower = eic0.GetActivePower() + eic1.GetActivePower();
  // Make JSON string
  strcpy(energyReadingJSON, "{\"p\":");
  char buf[10];
  itoa(wholePower, buf, 10);
  strcat(energyReadingJSON, buf);
  strcat(energyReadingJSON, "}");
}

I’ve gone through several iterations on what values I send to the server/cloud for visualization.  On one hand, the more info, the better.  So bring everything that is read back and store.  On the other hand, the focus is on understanding power usage.  Over time, I’d love to evolve energy monitoring to include device detection and perhaps other things that evolved understanding of a technology brings.  However, for now, I’m going to focus on whole house power.  This will be an Int sent as a mqtt message to em/# topic:

// Topic = em. Specific node = 1.  1 will map to an entry in the Address table to the house address for node 1.
const char* mqttTopic = "em/1";

The firmware sends two types of mqtt messages:

  • Info
// MQTT info messages
const char msgBOOT[] PROGMEM = "{\"m\": \"Boot\"}";
const char msgRESET[]  PROGMEM = "{\"m\": \"Reset\"}";
  • Power Readings (see myEnergyMonitor_getReadings() ).

Database

I’ve evolved the SQLite database definition on the Raspberry Pi to three tables: 

class Reading(peewee.Model):
    addressID=peewee.IntegerField()
    power = peewee.IntegerField()

    class Meta:
        database = database

class Info(peewee.Model):
    addressID=peewee.IntegerField()
    message=peewee.TextField()
    time = peewee.TimestampField()

    class Meta:
        database = database

class Address(peewee.Model):
    addressID=peewee.IntegerField()
    firstName=peewee.TextField()
    lastName=peewee.TextField()
    streetAddress=peewee.TextField()
    city=peewee.TextField()
    state=peewee.TextField()
    zipCode=peewee.TextField()

    class Meta:
        database = database
  • Address Table: An entry into the Address table is inserted by a person.  Here’s mine:
import peewee
from EnergyReadingModel import Address
addressID = 1
firstName = 'Margaret'
lastName = 'Johnson'
streetAddress = '555 Main Street'
city = 'Kirkland'
state = 'WA'
zipCode = '98033'

try:
    Address.get(Address.firstName==firstName,Address.lastName==lastName,
        Address.streetAddress==streetAddress,Address.city==city,
        Address.state==state,Address.zipCode==zipCode)
    print("Address record already exists.")
except:
    Address.create(addressID=addressID,firstName=firstName,
    lastName=lastName,streetAddress=streetAddress,city=city,
    state=state,zipCode=zipCode)
    print("Address record created.")

The addressID is part of the mqtt topic the Energy Monitor sends.  For my Energy Monitor, the mqtt topic is em/1.

  • Info Table: an entry is added to the Info Table when an Energy Monitor sends an mqtt message like:
// MQTT info messages
const char msgBOOT[] PROGMEM = "{\"m\": \"Boot\"}";
const char msgRESET[]  PROGMEM = "{\"m\": \"Reset\"}";
  • Reading Table: The power readings that are picked up from Energy Monitors.

Python App on Raspberry Pi  

The Python app on the Raspberry Pi – addMessageToEnergyDB.py – picks up the energy monitor’s mqtt message and inserts the content of the payload into the right table.

def on_message(client, userdata, msg):
    # Parse out the Energy Monitor ID
    topicHead,addressID = msg.topic.split("/")
    # handle the mqtt payload
    json_data = str(msg.payload)
    data = json.loads(json_data)
    # The mqtt message could be for either the Reading or Info table.
    try: # Try treating message as an Info message
        messageValue =  data[u'm']
        timeValue = time.time()
        try:
            Info.create(addressID=addressID,message=messageValue,time=timeValue)
        except:
            print("could not create a info record")
    except:
        # The message could be an energy reading or something I don't know how
        # to handle.
            try:
                powerValue = data[u'p']
                timeValue = time.time()
                Reading.create(addressID=addressID,power=powerValue,time=timeValue)
            except:
                print("could not create a reading record")

It Works

I can read both wires that are required in the 3-wire single phase system that is used to “pump” electricity into our house.  My next concern is the strength of the wiFi radio on the Feather Huzzah….

 

 

Until next time.  Thank you so much for reading this far.  Please find many things to smile about.

Challenge – Getting Readings of 0 from the ATM90E26

I noticed at some point the ATM90 started returning 0’s for the results.  The datasheet notes – in 4.1.3 TIMEOUT AND PROTECTION – : A read access to an invalid address returns all zero.  Hmmm…

Thanks To Those That Went Before

 Tisham Dhar quickly answered a question I had after debugging that pointed directly to what the problem was.  I want Tisham to be successful and believe he does quality work.  I highly recommend the ATM89e26 energy monitor kits.

Debug

Things I did to debug/fix:

  • Change the Energy DB model to have two tables, one for the readings, one for log information.  The log information will give a way for the firmware to let me know about events going on in the code that are worth looking at.  I added two events to the Arduino firmware: 1) Reset. 2) Boot.  I like having a way of informing me about events in the firmware.  I think sending simple mqtt messages is an easy-peezy way to do this.
  • Modified the Arduino code in EnergyMonitorStuff.h to check a reading and reset if all the values are 0.  If they are, reinitialize the ATM89.  I like having this in the code regardless.  
  if (sysStatus == 0 && voltage == 0.0 && current == 0.0 && pf == 0.0 && power == 0) {
    myEnergyMonitor_init();
    strcpy(energyReadingJSON,"{\"m\": \"Reset.\"}");
    return (energyReadingJSON);
  }

it also sets up a “Reset.” mqtt message for insertion into the EnergyMonitor.db database.  The other Info I currently collect is when the firmware boots:

void setup() {

  BlinkInSetupStart
  initStuff();
  BlinkInSetupEnd
  myMQTT_publish("{\"m\": \"Boot.\"}");
}
/********************************************************
   LOOP
 ********************************************************/
void loop() {
  BlinkInLoop
  char *json = myEnergyMonitor_getReadings();
  DEBUG_PRINT(F("JSON: ")); DEBUG_PRINTLN(json);
  DEBUG_PRINTLN(strlen(json));
  myMQTT_publish(json);
  delay(10000);  // take a reading every 10 seconds.
}

Restarts

The image is rather small.  It is a plot showing power readings (measuring our microwave).  The dots represent resets (upper) and reboots (lower).  For an ~ 15 hour time period, there were 56 resets and 6 reboots.

pi@raspberrypi:~/EnergyDB $ python infoquery.py

Readings: 2724 Resets: 0, Reboots: 0

 Shoot…I did not take a reading prior to “fixing” the challenge so there are no resets or reboots in the infoquery results.

Fix

The cause of the 56 resets and 6 reboots was noise from the power supply.

NewImage       NewImage

Instead of using a separate power source for the FeatherWing, I was using the 9V AC-AC transformer.  No resets/reboots happened when the FeatherWing was powered through the USB port.

Thoughts

Ideally, only one power supply is needed to handle both the FeatherWing as well as the 9V AC-AC power measurement requirements.  I want to explore the characteristics of the DC power being provided to the FeatherWing.  Is there something in the power circuit that could be fixed in order for the 9VAC in to 3.5VDC out to be smoother?  My first step here would be to check this out using a scope.

 

Well, there it is.  Challenge: a lot of reboots and resets.  Originally, I thought this might be because of that darn ATM90.  Nope.  Power supply.  Of course, after the debugging this makes TONS of sense.  I have enjoyed the results of testing for myself with convincing results.

HappyDay Neighbors Web Service Side – Let’s Give Raspberry Pi a Twirl!

I’m getting my head around the collecting and visualizing of power readings.  As in, it’s not hard to collect a lot of power readings.  But what should be displayed and how should it be displayed?  This post is about coming up with a self contained, easy to use data collecting and visualization software that moves my thoughts farther in the direction of what concerns I need to think about before I figure out the best way to build an awesome experience.  I find it is first best to try stuff out.  This way I gain knowledge on what’s the best way to build what I want to deploy to others.  In my previous attempt at building a prototype back end for collecting and visualizing energy data, I used AWS Services. 

Frustrated

Bumbling through the AWS services to do what I want felt like I was wading through something that is gooey, muddy, and slimy.  In fact, I bumbled around so successfully that apparently in just one day I had blown through my free level and started to get charged.  Amazon’s charging capability – seemingly down to the micro transaction – is a true testimonial in their ability to suck up every penny on the table.  Here’s part of a mail I got from AWS billing when I asked about charges: Although your account is eligible for the Free Tier, your usage charges were based on activity outside the specifications of the offer. Please note that the AWS Free Tier provides up to 750 hours per month of a single-AZ t2.micro.elasticsearch or t2.small.elasticsearch instance and 10GB per month of optional Amazon EBS storage (Magnetic or General Purpose), while you were exceeding the limitation of hours.  Hmmm…..I just want to collect energy data and draw a line chart here, a bar chart there….

 

 

So I changed direction on technologies to use for collecting and visualizing energy data:

In this prototype exploration, I  use a Raspberry Pi 3 to collect readings coming in from the energy monitor using mqtt.  The Raspberry Pi will also act as a web server to display visualizations of the energy data.  I’ll use Python as the programming language and hook together:

  • mosquitto – An mqtt Broker.
  • peewee – Makes database inserts/queries easy.
  • Dash – Puts together “the best of” Flask and plotly to make web pages with data visualization.

The Goal

The goal of this post is to build what I call a “sacrificial draft” of the Collect and Visualize component of the HappyDay Neighborhood project.  I will capture mqtt messages from a Monitor reading the current used by a lamp as I turn the lamp off and on.  This is a simple prototype that will give me an idea how all the “goo” fits together.

Step1:Install the Raspbian OS

Ooh – Raspberry Pi…I feel like I get to play with the cool kids.  I haven’t played around with one.I’ll use a Raspberry Pi as the “command device.”  It will receive mqtt messages from energy monitors and store them into a database.  My browser will access a web page on the Raspberry Pi to look at the power readings.  I got a Raspberry Pi in the mail.  Next I need to install an OS on it.  

NewImage

I have a natural instinct at this point to start installs at the Adafruit Learning Center.  I wasn’t disappointed.  I found and followed along.  I decided to give Raspbian Stretch Lite a twirl.  I made an image…and then…hmmm…how do I work on a headless Raspberry Pi? I don’t want to use a separate monitor and keyboard…but still the Raspberry Pi wouldn’t boot.  The challenge was the original format of the SD card.  I bought the cheapest SD micro I could find on Amazon.  I didn’t format prior to using Etcher to burning the Raspbian image.  What worked for me was first formatting the card with SDFormatter (overwrite).

Step 2: Read/Write Files to the Raspberry Pi

I am not fluent with “Unixy” copying and saving files between a Raspberry Pi and a Mac.  I wanted to use my Mac editors on files stored on the Raspberry Pi.  The way to do this is by extending the Mac file system “goo” to include the Raspberry Pi’s drive.

To be able to drag/drop/open files on the Raspberry Pi from within my Mac’s Finder, I:

    • Installed SSHFS from here.
    • Created a directory named ‘mount’ within my home directory.  I.e.: in my case this would be /Users/margaret/mount .
    • Opened a Terminal window and ran the command:
sshfs pi@raspberrypi.home: /Users/margaret/mount

Where the IP address is the unique IP address of the Raspberry Pi within my internal network.

Note: The ssh connection will break when the connection is lost – say because the Raspberry Pi or the Mac was rebooted.  So the sshfs command will be run quite often.

Now the Raspberry Pi’s storage appears as a disk within the Mac’s Finder.

Pasted Image 10 9 17 3 32 AM

pi@raspberrypi:~ $ pwd

/home/pi

Looks like the directory on the Raspberry Pi associated with the mount folder on my mac is /home/pi.

Step 3: Install Mosquitto

Moving right along, I need an mqtt broker on the Raspberry Pi to grab the energy monitor mqtt messages.  For this I’ll use mosquitto.  Mosquitto seems to be THE mqtt software to use on the Raspberry Pi.

Run the following on the Raspberry Pi’s command line:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo apt-get install mosquitto mosquitto-clients
pip install paho-mqtt 

Step 4: Test mqtt

OK.  Mosquitto is installed…but does it work?  I tried a few different scenarios to check this out.

Test 1 – command line

There are two command line utilities, mosquitto_pub and and mosquitto_sub.  As with all the other Linux commands I’ve used – get help on each by typing

~ $ mosquitto_pub –help

  • Open a terminal window, logon to the Raspberry PI and type the command:

 $ mosquitto_sub -t test/test1  -d

I.e.: subscribe to mqtt messages brokered on localhost sent to the test/test1 topic.  Debugging info (-d):

Client mosqsub/17589-raspberry sending CONNECT

Client mosqsub/17589-raspberry received CONNACK

Client mosqsub/17589-raspberry sending SUBSCRIBE (Mid: 1, Topic: test/test1, QoS: 0)

Client mosqsub/17589-raspberry received SUBACK

Subscribed (mid: 1): 0

  • Open a second terminal window, logon to the Raspberry Pi and type the command:

$ mosquitto_pub -t test/test1 -m “test message” -d

Client mosqpub/17627-raspberry sending CONNECT

Client mosqpub/17627-raspberry received CONNACK

Client mosqpub/17627-raspberry sending PUBLISH (d0, q0, r0, m1, ‘test/test1’, … (12 bytes))

Client mosqpub/17627-raspberry sending DISCONNECT

The terminal that was running mosquitto_sub received an mqtt message:

Client mosqsub/17589-raspberry received PUBLISH (d0, q0, r0, m0, ‘test/test1’, … (12 bytes))

 

test message

Test 2 – python

———————————-

As an aside, I just checked /usr/bin on my Raspberry Pi and did ls -l python* :

lrwxrwxrwx 1 root root 9 Jan 24 2017 /usr/bin/python -> python2.7
lrwxrwxrwx 1 root root 9 Jan 24 2017 /usr/bin/python2 -> python2.7
-rwxr-xr-x 1 root root 3162224 Jan 19 2017 /usr/bin/python2.7
lrwxrwxrwx 1 root root 36 Jan 19 2017 /usr/bin/python2.7-config -> arm-linux-gnueabihf-python2.7-config
lrwxrwxrwx 1 root root 16 Jan 24 2017 /usr/bin/python2-config -> python2.7-config
lrwxrwxrwx 1 root root 9 Jan 20 2017 /usr/bin/python3 -> python3.5
-rwxr-xr-x 2 root root 3976264 Jan 19 2017 /usr/bin/python3.5
-rwxr-xr-x 2 root root 3976264 Jan 19 2017 /usr/bin/python3.5m
lrwxrwxrwx 1 root root 10 Jan 20 2017 /usr/bin/python3m -> python3.5m
lrwxrwxrwx 1 root root 16 Jan 24 2017 /usr/bin/python-config -> python2.7-config

Jeepers! I had no idea so many versions of Python were installed. For now, I defaulted to using python2.7

———————————-

I tested out running mosquitto commands in python using the script on this page.  Here are some of the results I was getting:

$SYS/broker/connection/Usuario-PC.bridgeConnection/state 1

$SYS/broker/connection/Lucas-PC.bridgeConnection/state 1

$SYS/broker/connection/Servidor-PC.bridgeConnection/state 1

$SYS/broker/connection/caixa02.bridgeConnection/state 1

after awhile I got this error:

  File “mqtt_test.py”, line 25, in <module>

 

    client.loop_forever()

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 1481, in loop_forever

 

    rc = self.loop(timeout, max_packets)

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 1003, in loop

 

    rc = self.loop_read(max_packets)

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 1284, in loop_read

 

    rc = self._packet_read()

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 1849, in _packet_read

 

    rc = self._packet_handle()

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 2305, in _packet_handle

 

    return self._handle_publish()

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 2500, in _handle_publish

 

    self._handle_on_message(message)

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 2647, in _handle_on_message

 

    self.on_message(self, self._userdata, message)

 

  File “mqtt_test.py”, line 13, in on_message

 

    print(msg.topic+” “+str(msg.payload))

 

  File “/home/pi/.local/lib/python2.7/site-packages/paho/mqtt/client.py”, line 360, in topic

 

    return self._topic.decode(‘utf-8’)

 

  File “/usr/lib/python2.7/encodings/utf_8.py”, line 16, in decode

 

    return codecs.utf_8_decode(input, errors, True)

 

UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0xe7 in position 31: invalid continuation byte

I’m thinking there are some characters in the topic name that can’t be parsed.  For now I’ll ignore and consider the test successful.

Test 3: Mac to Raspberry Pi Test

Can I publish an mqtt message on my Mac and have the Raspberry Pi get it? Easy-Peasy!

import paho.mqtt.publish as publish
# Based on this example: https://github.com/eclipse/paho.mqtt.python/blob/master/examples/publish_single.py
publish.single(“test/topic”, “Hello Pi”, hostname=”192.168.1.17″)

Whoa.  That was too easy!

Step 5: Store Incoming Energy Messages

Store mqtt messages into a database on the Raspberry Pi.  FINALLY some fun, yah?  According to this post which I’ll take as fact until I learn better: SQLite is built into Python but is only built for access by a single connection at a time.  For now, I’ll use the built in SQLite.  When I scale to include other houses, I’ll have to change to a different database.  I should have a better feel with what data to store, how to store it, and what visualizations work the best.

The other piece of software I’ll  use for database access is the peewee ORM.  You might want to take a read at the material at this link –  if you are new to using the peewee ORM.  I chose to use the peewee ORM because of it’s simplicity.  

Install peewee

$ pip install peewee

Create the EnergyData database

For this step, I wrote EnergyReadingModel.py.  It contains the definition of the power and time variables and creates the EnergyReading.db file. I used Atom as my editor and saved the file to the EnergyDB directory I created (within the mounted directory).

~/EnergyDB $ python EnergyReadingModel.py

Created the EnergyReading.db file. This will be the database where the Energy readings will be stored.

Send Power Readings

The Feather Huzzah with the ATM90 FeatherWing is running the SendEnergyReadingsUsingMQTT.ino Arduino code.  For this part, I needed an mqtt library for the Arduino.  I am using what seems to be the most popular –  pubsubclient.

SendEnergyReadingsUsingMQTT.ino – 

  • Reads the power, current, power factor, and voltage.
  • Sends the energy Readings as an mqtt message formatted in JSON.  I ended up using a very simple name/value format.
    • {“sysStatus”:2,”voltage”:124.5,”current”:10.5,”power”:1278,”pf”:0.75}
    •  I used an online JSON parser to understand how JSON strings are parsed…
NewImage

I use a 100 byte buffer to store the message.  This is lazy of me – what if the message is longer than 100 bytes?  I decided on 100 bytes after calculating the number of bytes in the above JSON string using this letter counting calculator:

NewImage

Since this sample was 67 characters, I assume the JSON string will never be longer than 100 bytes.

Add Power Readings to the Energy DB 

I run the addPowerReadingToEnergyDB.py python app on the Raspberry Pi.  addPowerReadingToEnergyDB.py subscribes to the EnergyReading/# topic.  When it receives a message, it puts it into the energy db.

Note: while writing the python app, I learned about timestamps…which made sense to me to use as the time the energy reading was created.  I did it this way because the Raspberry Pi syncs its clock with ntp.  This means I can use the clock on the Raspberry Pi and not add an additional complexity of a RTC to the Energy monitor. 

>>> import time

>>> ts = time.time()

>>> print(ts)

1507718863.44

>>> import datetime

>>> st = datetime.datetime.fromtimestamp(ts).strftime(‘%Y-%m-%d %H:%M:%S’)

Run As a Service

I want to capture readings on the Raspberry Pi 7 x 24.  In order to do this, I need to set up the Python script to run as a background service.  I found Matt’s post – How To Autorun A Python Script On Boot Using systemd to be very helpful.

Here’s the content of the systemd (addMessageToEnergyDB.service) file I’m using:

[Unit]

Description=Add Energy Record to EnergyDB.

After=multi-user.target

 

[Service]

User=pi

Type=idle

ExecStart=/usr/bin/python /home/pi/EnergyDB/addMessageToEnergyDB.py

 

[Install]

WantedBy=multi-user.target

Executing the systemctl status command on the service gives me an indication readings are being received and entered into the EnergyDB:

 $ sudo systemctl status  addMessageToEnergyDB.service

addMessageToEnergyDB.service – Write Energy Record to the CSV file.

   Loaded: loaded (/lib/systemd/system/addMessageToEnergyDB.service; enabled; vendor preset: enabled)

   Active: active (running) since Sun 2017-11-12 09:20:01 PST; 9min ago

 Main PID: 634 (python)

   CGroup: /system.slice/addMessageToEnergyDB.service

           └─634 /usr/bin/python /home/pi/EnergyDB/addMessageToEnergyDB.py

 

Nov 12 09:26:15 raspberrypi python[634]: (‘INSERT INTO “reading” (“addressID”, “v1”, “i1”, “p1”, “pf1”, “v2”, “i2”, “p2”, “pf2”, “time”) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)’, [1, 123.92, 2.508, 284

Nov 12 09:26:36 raspberrypi python[634]: (‘INSERT INTO “reading” (“addressID”, “v1”, “i1”, “p1”, “pf1”, “v2”, “i2”, “p2”, “pf2”, “time”) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)’, [1, 123.89, 2.593, 294

Step 6: Make an HTML5 Web Page

I originally started down a path of Flask + plotly.

Then I figured out Dash was a better choice for visualizing energy readings on a web page.  I found this video gave me a good feel why I decided on Dash.  Dash builds on top of Flask and plotly so I get all the great features of these frameworks with the added benefit of doing everything in Python.  I  installed Dash using the directions in Getting Started.  Here is an image of the plot of energy readings I got after taking energy readings for our microwave over time:

NewImage
I wrote and ran the visualizeEnergy.py python code.  Whoa.  Pretty darn simple.  At least getting started is not hard.  I’ll find out how easy it is to do more complex visualizations soon enough! 🙂
 
Note:
if __name__ == __main__:
app.run_server(debug=True,port=9999,host=0.0.0.0)
 

Since I’m running this on the Raspberry Pi, I set the host to 0.0.0.0 instead of the default 127.0.0.1 (localhost). 

That’s it for now 

My-oh-my it took me a bit of bumbling to get to this point.  The learning part was thrilling.  I’m sure I’ll have many changes to my plans as I continue.  Next, I want to run the python script that adds mqtt messages to the Energy DB as a background service that starts up on boot.  Also, for some reason after “some amount” of time, the Feather starts getting 0 for all the energy readings…hmmm…. I never expect stuff to “just work.” 

 

 

Thanks for reading this far.  Please find many things to smile about.

 

 

 

Trying out the ATM90E26 FeatherWing

I’m excited! I just received Whatnick’s ATM90E26 FeatherWing that I ordered on Tindie.  I got this while I wait for the ones I ordered from Crowd Supply.  I guess I got a bit order happy.  I wanted to support Tisham’s work.

By the Way – I ran across Tisham’s discussion of his work at the Adalaide Maker Faire.  It helped give me perspective.

Spoiler alert – It works great.

The Goal

The goal of this post is to run the Arduino sketch provided by WhatNick on a Feather Huzzah with the ATM90E26 FeatherWing and get expected results for the readings.  I will use the UART option to communicate between the Huzzah and the ATM90 based on this comment made on WhatNick’s Tindie pageThe ATM90E26 has the flexibility to be accessed both over SPI and UART. However the SPI mode it supports is only Mode 3, which is an unsupported mode of the ESP8266 Arduino stack. So I ported my ATM90E26 Arduino interface code over to UART mode with CRC check and everything after a few days of head scratching.

The Arduino sketch I will run is ATM90E26_UART.ino found at this GitHub location.  

Thanks To Those That Went Before

  • To Whatnick for building the ATM90 FeatherWing and writing a very simple to use library for reading/writing to it’s registers using the Arduino IDE.  So far I am very impressed with the quality of work.  From the choice of components, PCB material, soldering, and design.  What a delight.  A HUGE THANK YOU to Tisham.  Without Tisham’s help, I would not have been successful.  THANK YOU.  Oh – and THANK YOU.
  • To Home Depot for carrying just what I need for my DIY projects (except the stuff I need from Adafruit…By the Way – can the two of you merge?) – ESPECIALLY FOR KEN. Ken used to head an electronics shop with 1,000’s of employees.  He works in the electronics section of Home Depot several mornings.  Ken is EXTREMELY knowledgable, helpful, and has stories/no stuff that is just interesting to listen to.  So much so that Home Depot is my favorite place to go. 

Materials

Test Tools

  • Kill A Watt – for comparison current and voltage measurements.  I’ve used this many times.
  • Current Clamp – another tool to compare readings.  This works in the same way the CT works.  

In order to test the ATM90 with the clamp, I made a line splitter.  With this little beauty of a tool, I can check the current use of any device.  I don’t have to go to the breaker box and muck about and mess up the electricity in a way that bothers other family members.

NewImage

 

Making the line splitter was fun/simple.   

  • 1’ green, white, black 14AWG wire $.23 * 3 = $.69
  • 3 wire female plug $2.79
  • 3 wire male plug $3.29
Total cost: $6.77.
 
Just follow the color coding.  Reminds me of coloring pictures that contained info on what color I should use (and by the way – I did not appreciate being told what colors to choose.  I would choose what I wanted.  Did a picture of the sky HAVE to be blue?).
 
Coloring info from this web site:
 
NewImage

Adjusting the Gain

Before I can start accurately measuring voltage and current, I need to calibrate by setting the UGain and IgainL.  This is discussed in section 3.2.4.5 (p. 14) of Atmel’s M90E26 app note.  Since I don’t know what I am doing, I asked Tisham about this.  His reply:

  • There are some videos that may be helpful (yes they are).  If I remember correctly, the first video was great at showing what values Tisham got when running test code.  He also mentions Ugain and Igain.
  • Do not use the calculation for Ugain that is in the app note.  Rather: new_Ugain = existing_Ugain * (expected_Voltage/current_Voltage)
  • calculate new_gain, update register, then make sure to update checksum 2.  If checksum 2 is not updated, the results are bad.
  • After the voltage is calibrated you can use a purely resistive load with known power usage or another meter to calibrate the current gain. The value already in the script should be pretty close.  
    • new_Igain = existing_Igain * (expected_power / current_Power)

Calculate Ugain

new_Ugain = existing_Ugain * (expected_Voltage/current_Voltage)

  • existing_Ugain is set in energyic_UART.cpp CommEnergyIC(0,Ugain,0xD464); //Voltage rms gain
  • expected_Voltage was measured at 124.5V
  • current_Voltage = 192.93.  I got this value from running ATM90E26_UART.ino:
Sys Status:2
Meter Status:801
Voltage:192.93
Current:0.09
Active power:11.00
p.f.:0.65

new_Ugain = 0xD464 * (124.5/192.93) = 54372 * (124.5/192.93) = 35086.891618722 = 35087 = 0x890F

Update Checksum 2

After changing the Ugain setting in energyic_UART.cpp to 0x890F, I reran the Arduino sketch.  Checksum 2 will be off, but as the data sheet points out: The readout value of the CS2 register is the calculation by the M90E26…

The library already writes out the values for the checksums and lets me know if there is an error.  There is indeed an error for Checksum 2.  The new value for Checksum 2 that I read: 0x428C.  I modified this line in energyic_UART.cpp from 

CommEnergyIC(0,CSTwo,0xD294); //Write CSTwo, as self calculated

CommEnergyIC(0,CSTwo,0x428C); //Write CSTwo, as self calculated 

Check Voltage Reading

Here are the results I got after running the Arduino sketch;

Sys Status:2
Meter Status:801
Voltage:123.60
Current:0.10
Active power:7.00
p.f.:0.66

The voltage I measured was 124.5V.  The voltage I’m getting back from the ATM90 is 123.6V.  My thoughts:

  • The reading I am getting back from the ATM90 is “close enough” to the reading I get from the Kill-A-Watt and multimeter.
  • The calculations will not be precise.
  • At this point, I’m not spending any time evolving my knowledge of the amount of error.

IgainL

I compared current readings from several devices.  The current measurements – for the ATM90 compared with the clamp were the same, if not extremely close.  For this reason, I am not adjusting the IgainL.

And…

 

ChallengesBringOpportunites YIPPEE………..IT WORKS!!!!!

But…

WhyCat hmmm… this works for one cable…but I have two:

IMG 6215

hmmm…..

Prototyping the HappyDay Neighbors Energy Monitoring Project

Tags

I’m thinking through the client side of my neighborhood energy monitoring project.  I’m starting to call this project HappyDay Neighbors.  Don’t know if that will be the final name…Naming is hard.  

The energy monitoring experience includes collecting energy readings from our breaker boxes, sending the readings using mqtt to a data store in DAH CLOUD.  The home owner then views the readings.  Through viewing the readings, the home owner will understand how much energy is being consumed and think of ways to decrease energy use.  I also want to explore social engagement using gaming/facebook/twitter…The social part is TBD.  My current focus is on reading and visualizing the data.

The Goal

The goal of this post is to explore a path to visualizing the energy readings.  I’ll do this by building a prototype that:

  • grabs 8,000 energy readings from Tisham’s data on Thingspeak and sends it to a cloud data store using mqtt.
  • shows a histogram of the energy readings in a web browser.

Thanks To Those That Went Before

  • The folks behind the abundant resources and excellent community of the OpenEnergyMonitor project.  The folks in the community as well as the resources are very accessible.  There are many of us who benefit greatly from their ongoing efforts.  I’m excited!  Just today I got an emonPi in the mail.  Hopefully the CTs I have on order will arrive soon.  Then I’ll be able to give the OEM experience a twirl.
  • A HUGE shout out to Tisham Dhar.  Tisham is the creator of the ATM90E26 Single-Phase Energy Monitor Dev Kits.   I have been impressed with the intelligence and care Tisham has taken to Open Source his work.  Tisham also kindly answered all the questions I peppered him with.  His work looks super sharp to me.  

Background Info

My exploration took me on a path that relied on AWS services.  I went down this path because:

  • eventually I want to scale collecting and analyzing energy readings throughout our neighborhood and beyond.
  • I wanted to build as quick and dirty as possible.  This meant relying on a system of services – mqtt broker, data base, graphing – that were loosely coupled together, had a lot of folks using it so that there were a lot of videos and docs on use cases.
  • I am using the prototype to better define the final experience.
When I started this post, I had no idea what AWS services to use.  My learning process was to jump right on in.  I’ll write a bit about what I’ve learned at the end of this post.

There was a lot of goo…um…technology…that I was unfamiliar with. I bumbled my way through relying on mostly these resources:

Open Source

These files are used in the post:

Part 1: Sending mqtt Messages

I wanted to divide and conquer taking energy readings using sensors/hardware from visualization.  This way, I can start  collecting energy readings prior to having the hardware reading.  It will also allow me to focus on the why’s and how’s of the data visualization step.  I’ll read power readings from Tisham’s Thingspeak feed. I started discussing Tisham’s excellent work in this post.

Step 1: Build a File of Power Readings

I did this in my first Jupyter notebook.  As I started putting together the workflow of this prototype, it became apparent that date conversions are hard to get right (…yah, no kidding 🙂 ).  The updated notebook – ReadTishamDataIntoCSV.ipynb.

Sending date/time strings are tricky. Computing loves numbers, humans love strings.  I convert the date/time string to it’s epoch value (a long datatype).

The format TIsham uses to store the date/time format:

%Y-%m-%dT%H:%M:%SZ

Sending date/time strings are tricky. Computing loves numbers, humans love strings.  I convert the date/time string to it’s epoch value (a long datatype).  Here’s the Python script I ended up using to take in the dates in the ThingSpeak feed and convert to an epoch format:

dt = datetime.datetime.strptime(entry[u’created_at’],’%Y-%m-%dT%H:%M:%SZ’)
epoch = datetime.datetime(1970, 1, 1)
epochTime = int((dt – epoch).total_seconds())

NewImage

Entries in the CSV file look like this:

176,1506532776
177,1506532795

i.e.: Energy reading in watts, epoch time when measurement was taken.

E.g.: converting an epoch time format to a human readable format using Python:

>>> import time

>>> time.strftime(‘%Y-%m-%d %H:%M:%S’, time.localtime(1506532776))

‘2017-09-27 10:19:36’

Step 2: Send the Power Readings as mqtt messages 

In order to send mqtt messages, I needed to figure out what mqtt broker to use.  My requirements for a broker included robust availability, storage and visualization of the energy messages.  I ended up choosing AWS services.

I put the code I used into the Jupyter notebook SendMqttToTishamDataTopic.ipynb (located at this GitHub location).  It sends the mqtt energymqtt messages to the topic /clientPrototype/data/Tisham.

To do this I needed boto3: and AWS_CLI.

  • Install the AWS CLI:  From the web page – The AWS CLI is an open source tool built on top of the AWS SDK for Python (Boto) that provides commands for interacting with AWS services. hmmm…I have the Anaconda variant of Python 3.6 installed.  So this seems to mean I’ll be need to use the conda package manager (instead of following Amazon’s instructions, I followed instructions from stack overflow):

$ conda install -c conda-forge awscli

$ conda install -c anaconda boto3

 
NewImage
I needed an access key and secret access key.  I followed the direction to get the access key and secret access key from this Quick Configuration Link.  Next I ran aws configure and copy / pasted the keys.  Here’s what the mqtt messages look like:
{'power': '176', 'time': '1506532776'}
{'power': '177', 'time': '1506532795'}

Part 2: Data Store and Visualization

For data store and visualization I used:

  • the AWS Elastic Search Service as the message store.
  • an AWS IoT rule that triggered when an mqtt message with the /clientPrototype/data/# topic come into the broker.
  • AWS Kibana for visualization.
It is not my goal to give tutorials on the AWS services I used.  While overwhelming at times, once I focused on each one I was able to bumble through using the available videos and tutorials.  I’ll just go through the steps I used after many starts and stops.

AWS Elastic Search Service

  • I created a test domain.
  • If the clientPrototype index (database) exists (from prior testing) delete:
curl -XDELETE search-test-XXXXXXXXXXXXXXXXXXXXXXX.us-west-2.es.amazonaws.com/clientprototype
  • Create the clientprototype index and powerReading type:
 
curl -i -X PUT
   -d
'{
  "mappings": {
    “powerReading": {
      "properties": {
        "time": {
          "type": “date",
          “format": “epoch_second"
        },
        “power": {
          "type": “long"
        }
      }
    }
  }
}
' 'https://search-test-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.es.amazonaws.com/clientPrototype'

NewImage

I followed the example in the tutorial:

  • As I mentioned above, I’m using the epoch date format.   Stuff that took me time to figure out:
    • Removing line breaks from the curl command:
      • Copy/pasted the curl command to create the powerReadingtype within the clientPrototype index of the test domain into TextWrangler to edit.  
      • Remove line breaks using the Remove Line Break option on TextWrangler under the Text menu.  
    • Figuring out the epoch date formatting – both on the Python side as well as the format to use in Kibana. 

 AWS IoT Rule

Here’s the rule I created in the AWS IoT Rule UI:

Filter

NewImage

when a message comes into the /clientPrototype/data/<anything> topic, send it on to Elastic Search.  The topic I set up in the SendMqttToTishamDataTopic.ipynb notebook, /clientPrototype/data/Tisham , will trigger this rule when the mqtt message comes into the AWS broker.  

Action

The action taken:

Pasted Image 9 29 17 10 39 AM

Role

Rules run in the system using a role set up within the IAM UI. 

NewImage

I gave the following permissions:

NewImage

Access to the CloudWatch logs is important for debugging.  I’ll get to that.  The other policy allows the rule to publish into ElectricSearch:

NewImage

 

Sending mqtt messages and publishing into ElasticSearch can now be testing.

Test

The test will:

  • use the AWS IoT test UI to see if the AWS broker is receiving the mqtt messages.
  • evaluate logs in the log file to verify power readings are published into Elastic Search.

AWS IoT Test

NewImage

Evaluate Logs

The AWS services – IoT (message broker) and ElasticSearch can be configured to log debug information.  Monitoring is done through the AWS CloudWatch service.

I couldn’t get it to work until I went through the AWS_CLI command:

aws iot set-logging-options –logging-options-payload roleArn=”arn:aws:iam::XXXXXXXXXXXX:role/service-role/pub_to_es”,logLevel=”DEBUG”

Here are the log entries that are created when the mqtt message is sent:

2017-09-30 17:17:13.659 TRACEID:f872acf1-ccaf-b6f0-6df4-f94a7e91c3c5 PRINCIPALID:XXXXXXXXXX/XXXXXXXXX/AIDAIHBTGHRY27IMGPW2M [INFO] EVENT:PublishEvent TOPICNAME:/clientPrototype/data/Tisham MESSAGE:PublishIn Status: SUCCESS


2017-09-30 17:17:13.659 TRACEID:f872acf1-ccaf-b6f0-6df4-f94a7e91c3c5 PRINCIPALID:XXXXXXXXXX/XXXXXXXXX/AIDAIHBTGHRY27IMGPW2M [INFO] EVENT:PublishEvent MESSAGE: IpAddress: 50.46.122.171 SourcePort: 51507


2017-09-30 17:17:13.712 TRACEID:f872acf1-ccaf-b6f0-6df4-f94a7e91c3c5 PRINCIPALID:393111621300/AIDAIHBTGHRY27IMGPW2M/AIDAIHBTGHRY27IMGPW2M [INFO] EVENT:MatchingRuleFound TOPICNAME:/clientPrototype/data/Tisham CLIENTID:N/A MESSAGE:Matching rule found: ClientPrototypeRule

2017-09-30 17:17:13.712 TRACEID:f872acf1-ccaf-b6f0-6df4-f94a7e91c3c5 PRINCIPALID:393111621300/AIDAIHBTGHRY27IMGPW2M/AIDAIHBTGHRY27IMGPW2M [DEBUG] EVENT:ElasticsearchActionStart TOPICNAME:/clientPrototype/data/Tisham CLIENTID:N/A MESSAGE:Starting execution of ElasticsearchAction on topic /clientPrototype/data/Tisham

2017-09-30 17:17:13.843 TRACEID:f872acf1-ccaf-b6f0-6df4-f94a7e91c3c5 PRINCIPALID:393111621300/AIDAIHBTGHRY27IMGPW2M/AIDAIHBTGHRY27IMGPW2M [INFO] EVENT:ElasticsearchActionSuccess TOPICNAME:/clientPrototype/data/Tisham CLIENTID:N/A MESSAGE:Successfully indexed document in ES. Message arrived on: /clientPrototype/data/Tisham, Action: elasticsearch, Endpoint: https://search-test-tkpccxbid4y7vlelgo6jq55kmi.us-west-2.es.amazonaws.com, Index: clientprototype, type: powerReading, id: 008edff0-11f1-46c3-9e84-76500a5a4771

The log entries are a HUGE help during debugging.  It took me many (many) attempts to get through the workflow.  The log files made it obvious what the problem was.  The problems typically were 1) ill-formed JSON 2) lack of security clearance.

Kibana Visualization

A link to Kibana is available in ElasticSearch once I created the test domain:

Pasted Image 9 29 17 1 02 PM

Once I wrapped my head around Kibana’s main use case – visualizing a gazillion data points..like millions upon millions (e.g.: all the flight data from the past few years…).  I was able to wrap my head around making this bar chart:

NewImage

For the Y-Axis, I ended up using the MAX Aggregation figuring when the data is zoomed in or out this becomes the most interesting for a bar chart given the constraints on setting the Y-Axis.  I.e.: unlike Excel – data stores and visualizers that …well um…excel with data sets in the 10,000’s but are not focused on BIG BIG DATA – that plot individual points, Kibana uses aggregates (hence one of the suggested videos is about Kibana aggregation).

Lessons Learned 

Whew.  Well, it works.  I learned:

  • AWS services are both powerful and overwhelming.  It gets confusing what is free and what ends up being charged. For example, one path I took ended up costing about $14.  This alarmed me because I had no idea I was getting charged…things “just worked.”  I need to get a better feel of free versus costs.
  • Using AWS services is an easy way to scale to many, many energy monitors.
  • ElasticSearch and Kibana are optimized for finding a needle in a haystack. In this case, the haystack is gazillions of data points.  The needle being a behavior that is in the data but needs to be searched out.  Their scenario is one of BIG DATA.  While this is a scenario I am interested in eventually, the primary interest is for evaluating energy readings within a fairly small time frame – a day, a week, a month and/or real time.  Other AWS services for the data store (e.g.: S3) might be a better solution.  This is dependent on the eventual scenarios.  Kibana is not the best visualization because it assumes the scenario is one of a gazillion data points.  Kibana’s context is one of putting data points into buckets to display.  
  • AWS IoT rules are awesome.  It is so easy to trigger an AWS service (like storage of the payload) and set up roles.
  • mqtt works well for sending energy readings from the energy monitor that will be located in the breaker box.
 
Thanks for reading this far…..    Please find many things to smile about.

 

 

For the prototype – where my goal is to tweak out all aspects – hardware, mqtt message passing, storage, visualization, scaling to our neighborhood (and beyond) – I feel the path I took works out.  There are many, many choices from different vendors…and many choices within the AWS family to get to the finish line.  For now, I’m happy with this workflow.