First thing to mention, which doesn’t really have a category: I walked
from my home in New Westminster to UBC in one day; it was about 32km,
which is the longest walk I’ve done in one day. I am mulling the
possibility of walking across the US when I’m 60, and this is the kind
of daily distance I’d want to maintain. I got some good blisters and
was sore the next day, but not crippled; I think I could have done
that again. It’s a good sign.
Webby
Hardware hacking
-
More work on the weather vane; got it mounted on a peanut butter jar
lid. If that sounds silly, then in my defense it turns out to be
very handy to have a standalone mount for a project.
-
Made an HTML page to display readings from the weather vane, using
javascript to rotate an arrow graphic to reflect the direction it
was measuring. Surprisingly handy.
-
Bought an Ikea Vindriktning, aiming to read its measurements
directly with an ESP32. Took a while to figure out how to get
it working – turns out that a common ground between the ESP32 and
the sensor board was necessary to get the UART working – but I
think it’s coming along.
-
Took apart a coffee maker that died on us to figure out what was
wrong, and it turns out to be a thermal fuse that blew – apparently
this is quite common. Will be picking up a replacement and seeing
if I can get it going again.
ML/AI/Earth Observation
Space
- After nearly 5 years of searching, I have finally got a job in the
space industry: beginning January 9th 2023, I’ll be working for
Wyvern Space. They are building satellites to do
high-resolution hyperspectral imaging; my position is senior devops
software developer, helping to build and operate their image
processing pipeline. I couldn’t be more thrilled. 😁
Trying to get back to doing these things on a regular basis.
Hardware hacking
-
More work on an electronic weather vane, following these
instructions. Lots of figuring out what size of bearings I
should order.
-
Some soldering to make a battery holder for some ESP32 camera
modules I’ve got.
-
Weather station:
-
Try to get the tipping bucket rain meter working; there’s a loose
connection somewhere, and periodically I see that Burnaby had 5
metres of rain in the last 24 hours. I never realized just how
much you have to pay attention to loose wires.
-
Sketch out a new rain meter based on inexpensive flow meters, then
order some. We’ll see how this works.
Webby
-
Going through a number of online courses/resources:
-
Coursera UMich Web Design for Everybody course: excellent,
though aimed at people quite new to development of any sort. One
thing: I’m lucky enough to have my employer pay for this, but the
lecturer, Colleen van Lent, writes:
My motivation for creating this course content was to spread the
mission of free education to everyone. Unfortunately, many of
the platform changes has put the material behind paywells. I
highly encourage students to take the courses individually
(rather than as a specialization) to access them for free. Even
then, some of the assignments may be hidden. I am hoping to
launch a new more open version in Fall 2018.
-
Shay Howe’s HTML & CSS course; also excellent
-
Javascript.info: awesome walkthrough of JavaScript
-
Trying to get the basics down, then look into React or some other
front-end framework.
-
Gotta say, I’m really fascinated by the tie-in between JavaScript
and DOM manipulation, which I had not really grokked before.
-
Project-in-progress is a refactoring (not a redesign, as I want the
look to remain about the same) of The Floating Head of Ayn
Rand, which has been more or less untouched for HOLY CRAP
twenty-one years. (State of the art at the time was
table-based layout, which I adopted enthusiastically 😬).
-
Changed the CSS for this site to have the post titles be a bit more
prominent:
.posts-list-item-title {
font-size: xx-large;
}
Data
- But also web: begin taking up work on the New West Trees page
again.
- Newest feature: adding links to the Wikipedia page for a
tree species!
- Coming soon: adding common names for species (eg: English Oak
instead of Quercus robur)…which turns out to be surprisingly
tricky.
- Tried pytaxize, which was a yakshave to get an NCBI API
token, then gave me problems re: rate limiting
- Tried pygbif; better results, but still not great for
trees. Example: Quercus palustris is resolved to just “Oak”,
but Wikipedia clearly resolves it to “Pin Oak”.
- But this gave me the idea of trying wikidata or
wikispecies; this is up next.
Web development:
- I’ve become interested in web development recently, and have begun
working on a habit-tracking project called, unoriginally, Habit.
Currently it’s a good exercise for becoming familiar with Javascript,
Jquery, Bootstrap, Flask and SQLAlchemy.
Hardware hacking:
- I ordered a bunch of AI-Thinker ESP32 camera modules from
Universal Solder (Canadian vendor of Arduino, ESP32, electronic
components, etc; I’m a happy customer & recommend them thoroughly).
Started digging into how to make it into a timelapse camera.
Random:
- I signed up for a free account with [The SDF Public Access UNIX
System][3]. I’ve got a totes-real homepage at
[http://saintaardvark.unixcab.org][4], just like the old days.
[2]: The SDF Public Access UNIX System
[3]: http://sdf.org
[4]: http://saintaardvark.unixcab.org
Road trip to Ontario in an EV with my family to visit my parents.
Wonderful time.
Hardware hacking:
- More playing with ESP32. Try making an open-window detector with
the built-in Hall effect sensor, and sending a Grafana annotation
when it’s open.
Programming:
- Refactor my .emacs files to use a
lisp directory, and switch to
use-package rather than Cask. This is easily the longest-running
project I’ve been working on:
commit 85b1d148afdc135d725498c0384d58e7baa0866d
Author: Hugh Brown <hugh@chibi-laptop-01.(none)>
Date: Tue Mar 3 21:13:57 2009 -0800
New repo.
…and that commit came after declaring bankruptcy in the last one.
Data science:
No ML/DS work this month. But I am beginning to get interested in
microscopy, so…
Hardware hacking
-
Does it count as hardware hacking if it’s all software? A question
for the ages. Anyhow: set up motion on the Raspberry Pi running
the birdhouse camera. Set up a cron job on the Pi to copy the
captured movies back to my home machine. Set up a cron job on my
home machine to make a gallery out of it using PiGallery 2,
which is just what the doctor ordered.
-
The weather station had been saying for a while that my in-laws'
place was getting 14m of rain per day, which seemed excessive. Took
a look at that, and broke readings from the anemometer as
well. (╯°□°)╯︵ ┻━┻ Dug more and discovered that the connector
between those two pieces of equipment and the Cat5 cable (ask your
parents, kids) had rusted. Soldered up a replacement and we were
back in business.
-
A friend of mine (hi Matt!) gave me an OBD bluetooth dongle to try
on the car. I spent a truly stupid amount of time trying to query
it with Python, which led me into Bluetooth Hell. I love Linux but
OMG sometimes it’s the worst.
-
Set up an MQ135 to try and read CO2 levels at home. Getting mixed
results, which seems to be par for the course. A collection of
links in no particular order:
I turned 50 in February. Two months later, I’m still confused by
this. Was still recovering from COVID.
Hardware hacking
- Started work on a birdhouse camera with my father-in-law. He built
the birdhouse in December; this month we finally started working on
putting together the camera part. There are two in here: an
infrared camera and an endoscope for visual light. I put together a
little circuit board with 6 500mW IR LEDs to act as illumination.
Home sysadmin
- My little home machine is a Zotac ZBOX CI320 nano purchased
in 2015. It’s great – small, unobtrusive, passively cooled and
enough for everything I need…except that the 4GB of memory it has,
which felt so decadent, is starting to be a constraint. I blame
InfluxDB. Anyhow, ordered some more RAM only to realize I’d ordered
the wrong size. Made up for it by adding a swap file. Ask your
parents, kids.
Climate emergency
- Met with my MLA, Judy Darcy, again about climate change; she got us
some time with MLA George Heyman, BC’s Minister of the Environment
as well. I’m grateful to both for their time.
Welp…this took a long time to write up. In my defense, I got COVID
in January and that sort of threw me off for a couple months. But
it’s also just taken me a while to get back to it. Anyhow, onward!
Machine learning/data science
-
I submitted my entry to the Data Driven Cloud Cover
Competition! Aaaaand…my score was terrible. However, I got a
lot of practice out of this, and it was valuable for that. I
intended to go back and figure out exactly why my scores were so
abysmal, but got derailed (see first paragraph). But I think that
for whatever reason, my GAN was just not working at all. I need
to get more practice with this technique.
-
Some volunteer work for a local environmental society to demonstrate
how to use Pandas for graphing.
Climate emergency
- More letters to politicians. Did not make every week, but I’m
cutting myself some slack here.
What happened in 2021? Time to look back.
-
We got an EV! It’s a Kia Soul, and I love it.
-
Started writing in here semi-regularly. 💪
-
Expanded the weather station: precipitation meter, anemometer, soil
temp, particulate matter sensors.
-
Recorded a talk for PyCascades!
-
Much work on machine learning and data science: the dishwasher
loading critic, some Kaggle courses, mapping New Westminster trees.
-
A lot of work on Polaris: telemetry analyses, supervising our
third co-op student, and a proposal to run code on an ESA satellite
(sadly, denied).
-
Radio took a bit of a backseat by the end of the year.
-
I was asked to be an advisor for ALEASAT, and that was
wonderful.
-
The heat wave scared me, and I turned that into a focus on
climate activism. Small steps, but I’m taking them. Met with my
MLA in November to discuss climate change. I think of this as being
the start of about 30 years of work.
-
Total distance walked since getting my latest phone in January 2018:
8,664 km. Daily average distance in 2021 was 7.2 km, up from last
year (6.4 km).
-
Got more into birdwatching, phenology and natural history. Lots of
data taking, which I enjoy.
-
Bird feeder camera with ML to recognize the birds. Does a fairly
crappy job of picking out species but a good job of detecting birds.
-
Entered a couple of ML contests – no wins, but that’s expected;
it’s the practice I’m after.
-
At work: joined a new team which has a definite data science focus.
Learning a lot.
I’m leaving out all the incredibly important time with my family; this
isn’t the venue I choose to record that in.
A bit sidelined by Xmas, but still:
Machine learning
-
Much work on the DrivenData Cloud Cover Competition. I got the
benchmark solution implemented, then looked around for a better
approach. I decided to try implementing Weakly-Supervised Cloud
Detection with Fixed-Point GANs by Joachim Nyborg and Ira
Assent, which is a paper with code
(https://github.com/jnyborg/fcd). I’ve been slowly plugging away at
it, and recently finished a first full training run on a very small
subset of training data, though I still need to get a working
submission going.
There were a lot of things I had to change (because the code was
designed for the author’s problem, not mine) and fix (because the
code did not always work as described); I plan on submitting the
fixes back to the author, or at least letting them know about them.
This was done on two different machines. The first is a Fedora
server I have at home, an Intel NUC with 16 GB of memory. It
doesn’t have a GPU or a GUI (all text Emacs on this one), but it
works pretty well. It’s got more memory free than my laptop (which
also has 16GB but also runs browsers), and enough disk space that I
don’t need to think about it too much. This worked well for…let’s
call it minute-to-minute development: working in my editor, banging
on bugs and adding the features I wanted, committing to git and
pushing to the remote repo.
The training was all done on Microsoft’s Planetary Computer
platform, which gives me access to JupyterLab and a GPU. It has a
lot less disk space (~40GB or so, compared to ~500GB at home), no
Emacs 🤯 and no make, but it does have git and vim. My
workflow was usually to commit at home, push to git, then pull in
Jupyterlab. This was almost all in the shell, btw; the code I’m
working with was all regular Python modules, not Jupyter notebooks.
There were some times when a notebook would have been handy, but so
far I’m finding it hard to switch easily between the two – figuring
out easy ways to duplicate the parts of main() I want seems
non-trivial. Overall, this workflow worked pretty well for me.
As for Planetary Computer: it’s had some hiccups, but overall I’ve
been quite satisfied. This is a free trial, given because they’re
sponsoring this competition. Disk space is a constraint, but for a
free account I can’t really complain. And it continues to be
wonderful – and shocking – to see Microsoft embrace Free Software
in this way. No, they haven’t open-sourced Windows, but this still
feels like pigs flying.
With Xmas and family being the focus of the month, this took up most
of my hobby time.
Home sysadmin
- Upgrade servers to Debian 12. A bit tedious, but painless. Bless
Debian and all who work on it.
Hardware hacking
- The endoscopes I ordered last month came in, so I played with
those a bit. I think they should do for the birdhouse project,
though low light may be a problem. Set up a burner Android phone to
get the streaming application working, which was fine if still
sketchy-looking AF. It also works well enough over USB, though with
an interesting lag that kicks in if there’s too much change in the
picture – for example, if you wave it around a lot, rather than
just keeping it focused on a smaller area.
Climate emergency
Climate emergency:
-
This month, I actually got to meet with Jennifer Whiteside, my MLA
and BC’s Minister of Education, about natural gas
fracking in BC. I was there with two other folks from New
Westminster, and I think it went fairly well. She agreed to meet
with us again in a few months.
-
Lots of letters.
Hardware hacking:
- First graph for the birbcam! We’re posting data to InfluxDB, so I
get to play with it in Grafana (see below). Lots more to do, but this is a good start.
-
Ordered a couple wifi endoscopes for the next project: a birdhouse camera.
-
The tipping bucket precipitation meter that’s part of the weather
station will sometimes record a lot of tips in a very short
time…like, a ridiculous number. I’ve tried various things to
filter out spurious signals, but we still see them from time to
time. This time, I tried setting a threshold in the graph –
exclude measurements with more than 5 tips in a 30 second period.
This brings things down to a sane level. Thing is, after
experimentation it is possible to have about a tip per second or
so – but that requires literally pouring water into the funnel
constantly to keep it full, and even in the midst of this I
don’t think we were getting that much rain.
Machine learning/data science:
-
Worked on the Kaggle Time Series course. Left some
feedback. Still not done.
-
Worked on visualizing data for Russet, a project I’ve let languish
for a while. The goal was to take pictures out my office window,
which faces a lot of trees (mix of deciduous & conifers), and try to
see if I could track changes in the average colour over time. The
result – just a first pass – is a half-hour animated bar
chart. It’s definitely interesting to see the change over the
course of hours, days and months. But it is also a half hour bar
chart. Lots of room for improvement.
Natural History
- Observations & IDs for iNaturalist, observations for Nature’s
Notebook. I’m still managing to find new things to look at.
That was October. Of 2021. Time is weird, yo.
Climate Emergency
-
More letters each week, but now with a reply from the BC environment
minister now that the update to BC’s climate plan has come out.
Lots of reading through that and figuring out how I feel about it.
-
More asking my MLA (who’s the education minister) for a meeting to
talk about fracking. No response.
-
Attended a protest for the first time since university. Felt
awkward…but if that’s the worst, I’ll keep doing this.
Hardware hacking
-
Playing with a TM1638 module I ordered.
-
BIRBCAM! Set up cheap binoculars so that they’re focused on a bird
feeder at my in-laws’ house; set up a webcam behind them; get the
Coral dev board connected to their wifi; get Motion running on it;
and get Motion taking pictures, then getting an example script to
analyze the pictures post-hoc. The identification is a little all
over the map, but as a bird detector it works great. Pic can be
found here.
-
I ordered a handheld anemometer a while back; it arrived, and I
realized it did Bluetooth. That led me down the path of trying to
decode the packets, rather than install the dodgy-looking app that
I’m sure is totally fine, not even a problem Still not figured out
Data science/ML/GIS
My wife and I were out at the Coquitlam River Park today, walking
along the trails. We hadn’t been there before, and it was amazing to
see the river flow. It was fast, and it was easy to imagine the bad
things that would happen if it flooded.
On our way back, we crossed Patricia Bridge, a small footbridge
that took us back to where we’d parked. For fun, I decided to try
guestimating the river flow, and then see if I could find actual data
on it once we got home. I figured I probably wouldn’t be terribly
accurate, but it would be interesting to see how wrong I was – and
maybe why.
By dropping a stick in the river & watching it go, I estimated that
the water speed was something like 2 metres/second (at least, near the
bank). By pacing out the footbridge, I estimated the width to be 30
metres. I had no clue about the depth of the water, so I decided to
call it 5 metres; similarly, to simplify things I decided to assume an
oblique triangular profile for the river bottom. That gave me a
surface area of 75 m^2 (that is, of the face of water from the bottom of
the river to the top); assuming 2m/s, that gave me a volume of 150
m^3/s.
Is there real data on this? You damn betcha! Station 08MH002 is
maybe a kilometer from Patricia Bridge, and even has a graph:
The damn tooltip doesn’t show up in the screenshot, but it read 62.3
m^3/s at 2:15 PM PDT – right about when I was pacing off the breadth of
the river. That puts me off by about 2.5X. I was hoping to be closer
than that!
So what did I have wrong? Well, the depth – which the same graph
shows as 8.871 metres at the same time. That doesn’t help my model
any…that would make my volume about 266 m^3/s.
What about flow? My model assumes that water flow is the same from
top to bottom; is that correct? Probably not! (Incidentally, I
didn’t know that ResearchGate had its own StackExchange-like Q&A
feature…) The linked paper has this graph:
which came from this USGS publication.
At this point, I fell down another rabbit hole around modelling
velocity distribution in rivers, other papers giving empirical
results on velocity distribution, and so on. This
article gave Manning’s Equation, which I was excited about
until I realized this doesn’t really apply here. But it was
interesting reading about Manning himself.
I was hoping to get a rough-and-ready formula to figure out the
average velocity distribution, but that didn’t come up in my very
quick, not terribly attentive reading. I wonder if I could just use a
multiplier of 0.4 (my estimate vs what the readings were) as an
empirical heuristic?
Also…as far as the big surge in data goes, my assumption is that the
dam that’s upstream released a bunch of water, given the very sharp
rise that occurred right at midnight. Still digging into that.
