An approach to dealing with spurious sensor data in Indigo

Spurious sensor data can wreak havoc in an otherwise finely-tuned home automation system. I use temperature data from an Aeotech Multisensor 6 to monitor the environment in our greenhouse. Living in Canada, I cannot rely solely on passive systems to maintain the temperature, particularly at night. So, using the temperature and humidity measurements transmitted back to the controller over Z-wave, I control devices inside the greenhouse that heat and humidify the environment.

But spurious temperature and humidity data mean that I often falsely trigger the heating and humidification devices. After dealing with this for several weeks, I came up with a workable solution that can be applied to other sensor data. It’s important to note that the solution I developed uses time-averaging of the data. If it’s important to react to the data quickly, then the averaging window needs to be shortened or you may need to look for a different solution.

I started by trying to ascertain exactly what the spurious temperature data were. It turns out that usually the spurious data points were 0’s. But occasionally odd non-zero data would crop up. In all cases the values were lower than the actual value and always by a lot (i.e. 40 or more degrees F difference.)

In most cases with Indigo, for simplicity, we simply trigger events based on absolute values. When spurious data are present, for whatever reason, false triggers will result. My approach takes advantage of the fact that Indigo keeps a database of sensor data. By default it logs these data points to a SQLite database. This database is at /Library/Application Support/Perceptive Automation/Indigo 7/Logs/indigo_history.sqlite. I used the application Base a GUI SQLite client on macOS to explore the structure a bit. Each device has a table named device_history_xxxxxxxx. You simply need to know the device identifier which you can easily find in the Indigo application. Exploring the table, you can see how the data are stored.

To employ a strategy of time-averaging and filtering the data, I decided to pull the last 10 values from the SQLite database. As I get data about every 30 seconds from the sensor, my averaging window is about 5 minutes. It turns out this is quite easy:

import sqlite3

SQLITE_PATH = '/Library/Application Support/Perceptive Automation/ \
Indigo 7/Logs/indigo_history.sqlite'
SQLITE_TN = 'device_history_114161618'

conn = sqlite3.connect(SQLITE_PATH)
c = conn.cursor()
SQL = "SELECT gh.sensorvalue from {tn} as {alias} \
all_rows = c.fetchall()

Now all_rows contains a list of single-item tuples that we need to compact into a list. In the next step, I filter obviously spurious values and compact the list of tuples into a list of values:

tempsF = filter(lambda a: a > 1, [i[0] for i in all_rows])

But some spurious data remains. Remember that many of the errant values are 0.0 but some are just lower than the actual values. To do this, I create a list of the differences from one value to the next and search for significant deviations (5°F in this case.) Having found which value creates the large difference, I exclude it from the list.[1]

diffs = [abs(x[1]-x[0]) for x in zip(tempsF[1:],tempsF[:-1])]
idx = 0
for diff in diffs:
if diff > 5:
idx = idx+1
filtTempsF = tempsF[:idx+1] + tempsF[idx+2:]

Finally, since it’s a moving average I need to actually average the data.

avgTempsF = reduce(lambda x,y : x + y, filtTempsF) / len(filtTempsF)

In summary, this gives me a filtered, time-averaged dataset that excludes spurious data. For applications that are very time-sensitive, this approach won’t work as is. But for most environmental controls, it’s a workable solution to identifying and filtering wonky sensor data.

For reference, the entire script follows:

#	Update the greenhouse temperature in degrees C
# The sensor reports values in F, so we will update
# the value to see whenever the primary data has any change.

import sqlite3

# device and variable definitions
IDX_CURRENT_TEMP = 1822850463
IDX_FORMATTED = 1778207310
DEV_GH_TEMP = 114161618
SQLITE_PATH = '/Library/Application Support/Perceptive Automation/Indigo 7/Logs/indigo_history.sqlite'
SQLITE_TN = 'device_history_114161618'


def F2C(ctemp):
return round((ctemp - 32) / 1.8,1)

def CDeviceTemp(deviceID):
device = indigo.devices[deviceID]
tempF = device.sensorValue
return F2C(tempF)

def movingAverageF():
conn = sqlite3.connect(SQLITE_PATH)
c = conn.cursor()
SQL = "SELECT gh.sensorvalue from {tn} as {alias} ORDER BY ts DESC LIMIT 10".format(tn=SQLITE_TN,alias=SQLITE_TN_ALIAS)
all_rows = c.fetchall()
tempsF = filter(lambda a: a > 1, [i[0] for i in all_rows])
diffs = [abs(x[1]-x[0]) for x in zip(tempsF[1:],tempsF[:-1])]
idx = 0
for diff in diffs:
if diff > 5:
idx = idx+1
filtTempsF = tempsF[:idx+1] + tempsF[idx+2:]
avgTempsF = reduce(lambda x,y : x + y, filtTempsF) / len(filtTempsF)
return avgTempsF

def movingAverageC():
return F2C(movingAverageF())

# compute moving average
avgC = F2C(movingAverageF())

# current greenhouse temperature in degrees C
ghTempC = F2C(indigo.devices[DEV_GH_TEMP].sensorValue)
indigo.server.log("GH temp: raw={0}F, filtered moving avg={1}C".format(ghTempC,avgC))

# update the server variables (°C temp and formatted string)
indigo.variable.updateValue(IDX_FORMATTED, value="{0}°C".format(avgC))

  1. As I was preparing this post, I realized that it this approaches misses the possibility of a dataset having more than one spurious data point. Empirically, I did not notice any occurrence of that, but it's possible. I have to account for that in the future.

Follow the intent.

With Trump the usual advice of “Follow the money.” doesn’t work because Congress refuses to force him to disclose his conflicts of interest. As enormous and material as those conflicts must be, I’m just going to focus on what I can see with my own eyes, the man’s apparent intent.

In his public life, Donald Trump has never done anything that did not personally and directly benefit him. Most of us, as we go through life, assemble a collection of acts that are variously self-serving and other-serving. This is the way of life. Normal life. With Trump, not so. Even his meager philanthropic acts are tainted with controversy. The man simply cannot act in sacrificial way. He is incurable.[1]

As a corollary, when considering his dismissal of FBI Director Comey yesterday, until a special prosecutor is appointed, I plan to apply that principle. Since Trump acts only in his own personal best interest, I’m going to assume that in firing Mr. Comey, he is personally benefitting from it.

Since the evidence that Trump’s concern was over the Russia investigation, it’s safest to presume the firing was about the Russia investigation notwithstanding the feeble excuses of his staff who were caught off-guard by the event.

We would all do well to re-read Masha Gessen’s piece in the New York Review of Books, “Autocracy: Rules for Survival.” Her Rule #1: “Believe the autocrat. He means what he says.” remains applicable. If Trump is fuming about the Russia investigation, he probably fired the very man investigating his Administration’s ties to Russia because of it.

  1. In a campaign event in Fort Dodge, Iowa on November 12, 2015, Trump claimed that rival Ben Carson was "pathological" and that "...if you're pathological, there's no cure for that, folks, okay? There's no cure for that." Since Trump's own psychopathology is widely questioned, one wonders if he, too, is incurable. Given that narcissistic personality disorder is almost certainly among the potential diagnoses, he probably is incurable.

They're just paid protesters

You're wrong, Donnie

In an effort to strip protesters of their legitimacy, Trump and Fox News claim that protesters are simply there because they’re paid by powerful oppositional interests. Never mind that Trump has no evidence for his claim; he has no evidence for practically anything that emerges from his loud mouth. What is more interesting to me is that if money delegitimizes authenticity then presumably we can use this effect to come to additional conclusions.

If, as Trump claims, money discredits and taints the political process, then:

  1. Trump’s own motives, as an extremely wealthy man, should be suspect.
  2. For that matter, wealthy family members, such as Ivanka Trump™ and Jared Kushner who have been appointed to high positions in the Administration but who have no political experience should be viewed with skepticism.
  3. The Republicans pushed the Citizens United decision allowing unlimited amounts of money in politics. If money is good for the political process, as they presumably believe, then the mythical existence of paid protest should give them no pause for concern. It’s just free market capitalism in action, right?
  4. Considering the degree to which the NRA, the brothers Koch and other deep-pocket interests fund the campaigns of GOP politicians, maybe we should just consider all Republican politicians paid protesters.

As Simon Maloy, points out in Salon:

The irony here is that there’s one person we know for sure has paid people to show up and voice a prefabricated political message: Donald Trump. In the summer of 2015, Trump arranged for actors to show up at Trump Tower and cheer and wave signs as he announced his candidacy for president, offering them $50 for their services. And, in typical Trump fashion, he tried to stiff the agency that set up his Potemkin campaign launch.


And, no; no one paid me to write this. All protest is my own.

marking time

marking time,
eyes glazed, pupils constricted
to the head of a pin
from facing the blue white sterile light
for too long
a zombie tribe
numbering in the millions
if not more

this throng, agitated
in a subdued anesthetized
crowns one of its own
a clown of sorts
knowing little of the past
less of the present
and practically nothing
of the future.
“why not? it could be worse.”

in a strange unreality
a vaudeville show becomes
its own rehearsal,
a dreamish state from which
only an atomic flash
can awaken a person.

13 Random thoughts about Canada after living here for a year.

On January 1, 2016 we packed up all our earthly goods and headed south to Canada. (Yes, it’s true. When you live in Minnesota, it’s possible to move south to Canada. Look at the map!) Having lived here for a little over a year, here are some thoughts about living here, in no particular order:

  1. “Sorry” is more of a greeting than just an apology.
  2. Canadians really are polite; but put them behind the wheel of a car and all bets are off.
  3. Universal healthcare works. Americans love to go on and on about socialized medicine; but I’m here to tell you: it works.
  4. Bumper stickers are rare here.
  5. People don’t really talk politics. Well, they talk about U.S. politics.
  6. Left turn arrows on traffic lights are rare. It makes for interesting moments when the light changes.
  7. The electric utility is called “hydro”, which given the Greek origin of the word makes little sense until you realize that it stands for “hydroelectric.”
  8. Youth music is well-supported - both through private and public funding.
  9. State-church separation is fuzzier. For example, the Catholic school system is tax-payer funded. But only the Catholic schools. It has something to do with the Canadian Charter (a.k.a Constitution.) It was apparently some sort of historical compromise in the 1800’s.
  10. Don’t order iced tea in Canada. It’s way too sweet.
  11. As a practical matter, you can’t be elected Prime Minister unless you speak both English and French fluently. This is a really good thing.[1]
  12. Speaking of politics, campaigns are time-limited to 6 weeks before an election. How cool is that?
  13. Poutine sounds horrible, but it’s actually pretty good.

  1. How many languages does Donald Trump speak fluently, for example?

Serious audio processing on the command line

I’ve written previously about extracting and processing mp3 files from web pages. The use case that I described, obtaining Russian word pronunciations for Anki cards is basically the same although I’m now obtaining many of my words from Forvo. However, Forvo doesn’t seem to apply any audio dynamic range processing or normalization to the audio files. While many of the pronunciation mp3’s are excellent as-is, some need post-processing chiefly because the amplitude is too low. However, being lazy by nature, I set out to find a way of improving the audio quality automatically before I insert the mp3 file into my new vocabulary cards.

As before, the workflow depends heavily on Hazel to identify and process files coming out of Forvo. The download button on their website, sends the mp3 files to the Downloads directory. The first rule in the workflow just grabs downloaded mp3 files and moves them to ~/Documents/mp3 so that I can work on them directly there.

Another Hazel rule renames the verbosely-titled files to just the single Russian word being pronounced. It’s just neater that way.

rename 's/(pronunciation_ru_)(.*)/$2/' *.mp3

This uses the convenient rename command that you can obtain via Homebrew.

The final rule, grabs the newly-renamed mp3 file and performs a series of audio processing steps:

ffmpeg -i "$1" tmp.wav;
sox tmp.wav temp_out.wav norm gain compand 0.02,0.20 5:-60,-40,-10 -5 -90 0.1;
ffmpeg -i temp_out.wav -codec:a libmp3lame -qscale:a 2 tmp.mp3;
lame --mp3input -b 64 --resample 22.50 tmp.mp3 tmp;
mv tmp "$1";
rm tmp.mp3;
rm tmp.wav;
rm temp_out.wav

The first line ffmpeg -i "$1" tmp.wav; simply writes a temporary .wav file that we can process using sox. The second line invokes sox with a number of options that normalize and improve the dynamic range of the audio. Finally we use ffmpeg to convert the .wav file back to .mp3, compress the .mp3 file and then clean up.

Now I have excellent normalized audio for my cards, with no work for me!

See also:

Foreign language education in the U.S.: a neglected cure for xenophobia

Xenophobia has deep roots in the U.S. For all of its “global melting pot” rhetoric, the reality has been much more complex. It begins with the maltreatment of the native peoples of America with its attendant extinction of their language and culture. But even as recently as 1923, the U.S. Supreme Court ruled that persons from India were not eligible for U.S. citizenship because only “free white persons” were permitted. This case was one of the most egregious in the history of the United States and is a poignant example of how the Supreme Court reflects, rather than transcends, prevailing cultural values. In the United States v. Bhagat Singh Thind, 261 U.S. 204, the Supreme Court unanimously decided that an Indian Sikh man could not be granted citizenship because he was not white. Even today, the suspicion with which many English-speaking Americans view Spanish speakers undoubtedly contributes to their xenophobic calls to build a wall along the border with Mexico.

Language and culture are inextricably linked. We use spoken and written language to pass cultural values among one another and from one generation to another. Small wonder that imperialists throughout time have first sought to rid colonies of their native language. But perhaps it’s also true that by purposefully teaching languages, the U.S. will begin to see itself as a global participant rather than a global boss. The Latin origins of the word “educate” imply a process of leading out, drawing out, or leading away. Seen in this way, education isn’t a preparation solely to become part of an economic engine. It is to lead away from entrenched provincial ideas to a broader world view. Language education is a key component of this process.

Language education in the U.S.

Foreign language education in the U.S. is, to be blunt, deplorable. Owing to its fragmentary leadership structure, there are no national standards for education in the U.S. Local school districts, large and small are left to their own devices in writing curriculum standards. In many cases, individual states have specific standards to which local districts must adhere. But even those standards often derided by local politicians.

The effectiveness of a process can be judged by its outcomes. The desired outcome of foreign language education is fluency in the target language. In the U.S., less than 1% of Americans are fluent in a language that they studied in school.[1] According to Stephen Dubner of “Freakonomics” fame, the average U.S. student studies a foreign language for 2-3 years.[2] It’s hard to imagine anything more wasteful for children’s developing brains to spend hundreds of hours studying something they will never actually be able to apply in a meaningful way. Education budget cuts, lack of qualified teachers, and disjointed policy all contribute to poor foreign language acquisition by U.S. students. And that, in turn, heightens the perception that knowing English is sufficient to participate in the affairs of the world.

Language education worldwide

In contrast to the U.S. public education, the competent acquisition of a foreign language is required. Most European countries require the study of two foreign languages from an early age.[3]

In this study from the Pew Research Center, nearly three-quarters of European countries require the study of not just one foreign language, but two languages. Instead, the U.S. is a crazy-quilt of state and local foreign language requirements. Some states are even allowing students to substitute computer languages for foreign languages to meet this requirement. I’m sure this will help them enormously during their vacations in Computerland!

Another striking observation about language education in Europe is that it begins early, far earlier than the U.S. where most foreign language education takes place in high school. The delayed exposure to foreign languages for most U.S. students has two devastating effects. Drawing on what is known about the acquisition of musical ability, another aural skill, students who first encounter a foreign language late in their education are less likely to become fluent because certain features of brain plasticity decline. A more insidious effect of delayed foreign language study is that students become more and more rooted in their own cultures. Their world view becomes more and more fossilized. This, of course, is the start of a vicious cycle. Their poor foreign language experience means that they won’t support improvements for the next generation.


Foreign language acquisition is a must. Political rallies, calls to Make America Great Again™, walls, immigration bans and disparaging foreign nations won’t solve looming worldwide crises like climate change. But being able to understand and communicate with one another might.

  1. First, the U.S. needs to “prime the pump” by recruiting and developing qualified teachers who speak foreign languages competently. The U.S. already has an agency, the National Security Education Program (NSEP) a Federal initiative whose aim is to build the pool of U.S. citizens who fluently speak languages that are vital to its security interests.
  2. The U.S. Federal government should develop and promote a plan for compulsory foreign language education beginning in the early elementary years.
  3. The U.S. foreign language educational plan should develop a language prioritization based on the languages spoken by its own citizens and residents and on the languages spoken by both its adversaries and allies. The NSEP has developed just such a list of languages of strategic security importance.
  4. The U.S. should develop a plan for evaluating the effectiveness of foreign language education, holding school systems accountable for ensuring that each student is developing language fluency.

On a personal note, I was educated in the U.S. My first exposure to a language other than English was Latin which I studied for two years beginning in Grade 9. I went on to study French and Russian in high school with an amazing teacher Mme Collette Males to whom I’ve indescribably indebted. Had I not had this experience, I’m not quite sure where I would be.

  1. America's Lacking Language Skills, The Atlantic, May 10, 2015. Friedman, A. Link

  2. Is Learning a Foreign Language Really Worth It?, Freakonomics blog, March 5, 2014. Link

  3. Learning a foreign language a 'must' in Europe, not so in America, Devlin, K.,Pew Research Center, July 13, 2015, Link

Trump and the arts

I’ve long suspected that Trump regards the arts as an unnecessary nuisance for losers and suckers. High art for this hollow man is the vacuous reality television that made him famous. Now in his Federal budget, Trump offers proof. He proposes the elimination of the National Endowment for the Arts and the National Endowment for the Humanities.

This is a man with no soul.

But in the closing lines of her lyric poem “Renascence”, Edna St. Vincent Millay has a warning:

The world stands out on either side
No wider than the heart is wide;
Above the world is stretched the sky,—
No higher than the soul is high.
The heart can push the sea and land
Farther away on either hand;
The soul can split the sky in two,
And let the face of God shine through.
But East and West will pinch the heart
That can not keep them pushed apart;
And he whose soul is flat—the sky
Will cave in on him by and by.
- Edna St. Vincent Millay Renascence

If it’s not garish, gold, or glittering, it’s not worth our investment. Time for an artist’s march.