UQBAR clock surgery

I wrote about the clock I built for UQBAR café, and about its design and construction. Café visitors like it! It gets a lot of attention.

When I built it, one of the main things I wanted to be able to do was to make repairs to it without taking it down off the wall. That meant setting up good systems to debug software problems while the clock is running, and making sure that all the components were easy to swap out and replace.

The Arduino board I used for this project includes a real-time clock component. That lets the board keep time accurately. I’ve calibrated and configured it so that it’s accurate to one part per million — it will drift as much as one second every twelve days. Despite that accuracy, the clock tries to connect to the café’s wifi network at 3am every night, so it can contact the NIST time servers to sync the real-time clock and make sure it’s displaying the right time.

Over the past month or so, the clock has exhibited a sporadic, but frequent, bug: On some nights, usually several in a row, the attempt to connect to wifi fails. Then, eventually, one night, it succeeds.

When the clock fails to connect, it turns on a trouble light on the status bar and logs the error in non-volatile storage so I can see the message when I connect it to my laptop. As soon as it connects successfully, it clears the trouble light.

Of course, this all happens at 3am, so I’m not hanging around the café to debug the problem live.

I have a few theories about what’s wrong. The simplest explanation is that the wifi radio on the board is flaky. I have managed to reproduce the clock unable to connect several times in a row during the daytime, then connecting successfully.

I had a spare Arduino board laying around, so decided to replace the existing board with the new one and see if the problem recurs.

I went over to the café this afternoon (they’re closed on Mondays, so it’s a good time to do some electronics work). I’m super pleased that I was able to pull the old board and replace it with a new one in less than a half hour. My design-for-serviceability strategy worked great!

I spent about an hour total, doing some debugging and dumping the configuration and logs on the old board, then installing the new one, getting the config settings right and verifying that it works properly. I’m gonna let it run for a couple of weeks and see if the wifi problem persists.

I hope it doesn’t, but if it does, then I think the problem is not in the clock but in the network. Maybe there’s some kind of industrial activity going on in the neighborhood at three o’clock in the morning that generates a bunch of radio interference (it’s mixed use and light industrial, so that’s not a completely crazy possibility). Or, maybe the café wifi access point or cable modem are behaving funkily (though the rest of the gear in the place seems to function just fine on the network).

If the answer turns out to be interesting, I’ll write it up here on my Medium blog later.