Using rrdtool to chart Indigo data

Indigo currently shipping version 7 is a leading Mac home automation software package. One of it’s mostly widely-used features is its ability to execute user-provided Python scripts of AppleScripts. In my previous introduction to scripting Indigo with Python I showed how to use the Indigo plugin host to execute Python scripts. In this post, I’ll describe how I use a third-party charting package rrdtool to graph data from Indigo by taking advantage of Indigo’s ability to execute arbitrary Python scripts. This tutorial is focused on using Python as a bridge between Indigo 7 and rrdtool. If you are interested in a solution that takes advantage of AppleScript and bash scripts to do the samee thing, see this thread on the Indigo forums.

Installing rrdtool

I used Homebrew to install rrdtool. Homebrew is a package manager for macOS. If you already have Homebrew, then installing rrdtool is just: brew install rrdtool. Otherwise, you can easily install Homebrew. In the terminal, execute the following:

1
2
/usr/bin/ruby -e "$(curl -fsSL \
https://raw.githubusercontent.com/Homebrew/install/master/install)"

After installing Homebrew, then you can easily install rrdtool as described above.

Installing Python bindings for rrdtool

While it’s possible to run rrdtool from ApplesScripts that are executed inside of Indigo, it may be preferrable to use Python. I’ve found Python much easier to work with and more reliable than AppleScript. If you use Python, then the bindings to rrdtool make the work much easier than setting up shell scripts to execute. To install the Python bindings, I used sudo easy_install rrdtool. That should take care of all the prerequisites and install the rrdtool Python module.

Briefly introducing rrdtool

This tutorial isn’t meant to be an exhaustive introduction to rrdtool, but I’ll briefly describe the principles of operation. As its name implies, rrdtool uses a round robin database format which is a time series storage format implemented as a circular list. This format allows us to look closely at short-term trends while losing precision over the longer term. It’s ideal and compact for examining short-term data trends. rrdtool then is a set of tools that manage and graph data from a round robin database.

Creating a round robin database

To get started using rrdtool you need to create a database that will hold the data that Indigo 7 provides from your sensors. In my first use case, I was interested in charting temperature and humidity levels in a part of my basement. I set up the database as follows:

1
2
3
4
5
6
7
8
rrdtool create /usr/local/share/obasement.rrd
--step 90 \
DS:temperature:GAUGE:2000:32:80 \
DS:humidity:GAUGE:2000:20:80 \
RRA:AVERAGE:0.5:1:350400 \
RRA:AVERAGE:0.5:96:3650 \
RRA:MIN:0.5:96:3650 \
RRA:MAX:0.5:96:3650

This asks rrdtool to create a new database as the specified location with the name obasement.rrd. The parameter descriptions are beyond the scope of this tutorial but briefly, the DS parameter has the following format: DS:variable_name:DST:heartbeat:min:max. The DST is the Data Source Type which can be any of: COUNTER, DERIVE, ABSOLUTE, GAUGE. Here, the GAUGE value for DST just means that no rate of change is saved - only the primary data point. The next parameter of DS is the heartbeat. To simplify, you can look at the heartbeat as the interval seconds between expected data points. Finally we have the expected minimum and maximum values for the data.

The RRA parameters specify a round robin archive and describes the behaviour of the consolidation algorithm. A round robin database has to compress older data to maintain its circular compact list and the RRA parameters describe how that should work. Here we’re using an AVERAGE function to consolidate old data.

I’d recommend reading through this thorough introduction to the rrd format parameters before creating your own databases.

Obtaining data from sensors via Indigo

In this example case, I want to chart temperature and humidity levels from a Aeotec MultiSensor 6. Grabbing the values from this device via Python is easy:

1
2
3
4
5
6
7
8
9
import rrdtool

# "Original basement temperature sensor"
temp = indigo.devices[401767099].sensorValue
# "Original basement humidity"
hum = indigo.devices[1437240536].sensorValue

# update our round robin database
rrdtool.update('/usr/local/share/obasement.rrd','N:{0}:{1}'.format(temp,hum))

In line 9, we update the round robin database with the incoming data. This script can be launched from and Indigo schedule that corresponds to the heartbeat we specified.

Graphing our data

Generating graphs of the data is straightforward now that we have the database functioning. Here’s the code that generates my graphs:

1
2
3
4
5
6
7
ret = rrdtool.graph("/Users/alan/Desktop/obasement.png",
"--start", "-2days", "-w 600",
'DEF:Humidity=/usr/local/share/obasement.rrd:humidity:AVERAGE',
'DEF:Temperature=/usr/local/share/obasement.rrd:temperature:AVERAGE',
'LINE1:Temperature#ff0000:Temperature (°F)',
'LINE1:Humidity#0000ff:Humidity (%)'
)

The full update-and-graph script is here:

Here you have a few choices, you can schedule the update and graph generation as a scheduled task in Indigo, or you can schedule it in the OS. I chose the latter and used the application LaunchControl to launch the job as a User Agent at the interval specified by the heartbeat parameter. The program to run in LaunchControl is: "/Library/Application Support/Perceptive Automation/Indigo 7/IndigoPluginHost.app/Contents/MacOS/IndigoPluginHost" -x /Users/alan/Documents/dev/scripts+tools/indigo_graph_obasement.py or "path-to-indigo-plugin-host -x path-to-my-script"

The result

The result of all of this is a graph of the temperature and humidity in my basement which I can incorporate in control pages or display anywhere else of my choosing.

graph.png

Of course, to use the graph in your Indigo control pages, you’ll need to change the path in the above code to “/Library/Application Support/Perceptive Automation/Indigo 7/IndigoWebServer/images”.

Acknowledgements

The inspiration to write this implementation came from users cullenfluffyjennings and webdeck in this thread on the Indigo forums.

See also

“The Italian experience provides a blueprint for how to defeat Mr. Trump. Only two men in Italy have won an electoral competition against Mr. Berlusconi: Romano Prodi and the current prime minister, Matteo Renzi (albeit only in a 2014 European election). Both of them treated Mr. Berlusconi as an ordinary opponent. They focused on the issues, not on his character. In different ways, both of them are seen as outsiders, not as members of what in Italy is defined as the political caste.”

“And an opposition focused on personality would crown Mr. Trump as the people’s leader of the fight against the Washington caste. It would also weaken the opposition voice on the issues, where it is important to conduct a battle of principles.”

Focus on issues and principles, not personality. Don’t feed the troll

Don't feed the troll

In internet speak, “to feed the troll” means to try to engage people online who are just trying to stir up discord for no other reason than to provoke people. Trolls are almost always insecure, psychologically-damaged people, if not full-blown psychopaths who lack the usual social barriers that most of us possess. Thus, a common piece of advice tossed about on the Internet is: “don’t feed the troll.” This is sound advice.

A useful corollary might be: “Don’t elect a troll.” But that’s already done, so we’re left to deal with a Troll-in-Chief. Trump can be engaged on two levels: 1) By reference to his policy decisions (which in less than two weeks have thrown the world into chaos), or 2) By reference to his childish, impulsive, ill-informed tantrums on Twitter. I’ll explain why constraining our responses to his policies is the best best.

Trump is not a politician. Judging by the fact that his personal ventures have failed to beat the S&P 500, he’s not much of a businessman either. So what exactly is Mr. Trump? He’s a showman of course. He has a pathological need for attention. Attention for Mr. Trump is an addiction like mainlining heroin. He is an arch-narcicisssist who stirs up shit on Twitter not only because he’s an angry man but also because he likes the dopamine spike when people respond to his countless micro-tantrums. By responding to his vacuous mean-spirited tirades, we the people along with the media are simply enabling an addicted, very impaired man. We must stop feeding the troll. When he publicly mocks an esteemed U.S. Senator who lost relatives in the Holocaust as he did with Senator Schumer, we need to restrain the impulse to report it or fire back. Only by interrupting the positive feedback loop do we have any hope of staying focused on priorities. After all, sticks and stones may break my bones but words will never hurt me. (Well, at least not directly.) Trump-the-Troll is going to say whatever he needs to say in order to get attention. Where does it end? Trump calling in the nuclear codes because he’s tired of being compared to a Cheeto? It’s time to starve the troll of the attention that fuels his impulsive blathering.

But more importantly, we must not feed the Troll-in-Chief because it’s a side-show. While we and the media are attending the carnival of trolls, the real architect, the real Goebbels, Bannon is working up some serious white-supremacist, Nazi-style mischief. The main attraction isn’t the Troll himself. The Troll is simply a distraction. While we’ve got our eyes on the Troll, Bannon is busy dismantling the foundations of the Republic.

Progressives have to become laser-focused and disciplined. We can start becoming more disciplined by starving the Troll of the attention he desperately craves.

AppleScript and iTerm2

Among the many reasons I use iTerm2 in lieu of the macOS Terminal is its AppleScript support.

I recently had the need to automate some tasks on my Amazon Web Services EC2 server in a way that takes advantage of iTerm2 AppleScript functionality.

Use case

I’ve found recently, that my screen sessions were disappearing. Although I haven’t completely excluded other causes, some have suggested that infrequently-reconnected sessions can be cleaned up. Since I’m not a Unix sysadmin, I’m not sure about this. However, I decided to test the hypothesis by writing an AppleScript that logs into my EC2 server, attaches to each screen session, detaches and closes the connection.

screen escape sequence implementation

The trickiest bit to solve was the ^A escape sequence that screen uses. Here’s how I solved that part:

1
2
3
key code 0 using {control down}
delay 1
keystroke "d"

By wrapping that in a subroutine, I was able to automate the detachment from the current screen.

Full implementation

The complete implementation just loops over the screen session ID’s, attaches and detaches then finally logs out and closes the window. I use LaunchControl to have the AppleScript run every 2 hours.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
--
-- Created by: Alan Duncan
-- Created on: 2017-02-02
--
-- Copyright (c) 2017 OjisanSeiuchi
-- All Rights Reserved
--

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

set screenIDs to {5546, 5208, 5129, 5580}

tell application "iTerm"
set newWindow to (create window with profile "OjisanSeiuchi EC2")
tell current session of current window
delay 3
repeat with screenID in screenIDs
-- attach to this screen session
set screenCommand to "screen -r " & (screenID as string)
write text screenCommand
delay 2
-- detach from the session
my detach()
delay 1
end repeat
write text "logout"
delay 1
end tell
close newWindow
end tell

-- detaches from the current screen session
on detach()
tell application "iTerm" to activate
tell application "System Events"
tell process "iTerm2"
key code 0 using {control down}
delay 1
keystroke "d"
end tell
end tell
end detach

We’ll see if this solves the problem of screen sessions disappearing.

NYT: Women who voted for Trump

Some insight into women who voted for Trump.

“I think he’s a really good man, deep down. This guy has such potential, and I truly believe he cares about our country and wants to help everyone.”

Well, by everyone, you mean “those exactly like me.” Actually, how about “just me”.


“But I had an 8-year-old who was totally on the Trump train. He talked me into taking him to a Trump rally.”

It’s never too early to think about getting the youngsters into the Hitler Youth.


“Trump’s a successful businessman, and I feel like that’s what America needs to bring our economy back.”

Well, let’s just forget about the multiple bankruptcies and stiffing contractors.


“Benghazi. The emails. The I.R.S. She’s a liar.”

Thank goodness we elected a truth-teller. Right.


“Look at how much Trump hires women, how much he does rely on women, how much he relies on his own daughter. I’m sort of amazed by her. She may pull him more into the middle. She’ll be a good voice for women.”

A good voice for wealthy women of privilege.


“Driving to work yesterday, I saw three homeless people. They need our help.”

So taking away any hope of medical coverage is the best way to help people?

Scripting Indigo with Python

I’ve used Indigo home automation software for a few years. It’s a integrated home automation software environment for the Mac and its a solid stable and well-supported platform.

Within Indigo, it’s possible to script triggers and actions either AppleScript or Python. It’s funny - AppleScript often looks like the easier route to take. It looks more like plain English than Python. But as they say, looks are deceiving. Two bits of bad news put the nail in AppleScript’s coffin for me - as least with Indigo.

First, I kept encountering a nasty bug when trying to talk to the Indigo Server via AppleScript run outside of the Indigo environment. You can read all about it on the Indigo forums, but basically AppleScript complained that Indigo Server wasn’t running when it plainly was running. I’m not usually one for complex workarounds such as thus that were being discussed on the forums.[1] So, I began to give Python a consideration.

Then there’s the iffy status of AppleScript coming out of Cupertino. The recent departure of Apple’s head of Mac automation technologies and the fact that Apple dissolved the whole Mac automation team doesn’t bode well for AppleScript.

Setting up to talk to Indigo via the Indigo Plugin Host (IPH) and Python

To make it easier to address the indigohost which is buried ddep in the Application Support directory, you should just create an alias.

1
2
~|⇒ echo "alias indigohost='/Library/Application\ Support/Perceptive\ Automation/Indigo\ 7/IndigoPluginHost.app/Contents/MacOS/IndigoPluginHost'" >> ~/.zshrc
~|⇒ source ~/.zshrc

I happen to use zsh so your terminal environment may be different.

Now to try out the indigohost:

1
indigohost -e 'return indigo.devices.len()'

This should return the number of devices you have. Now you can script Indigo in Python both from within the Indigo client context and as an external application.

To run a Python application as a separate file, it’s just:

1
indigohost -x iphtest.py

Resources

Here are a few resources to get you started:

Good luck and have fun!


  1. For example, some users have reported that restarting the AppleEvents daemon with sudo killall -9 appleeventsd can somehow allow external AppleScript applications to address the Indigo Server. It didn't work for me.