I’ve added repos for both parts of the Techshed server to Github in case anyone’s interested.
The first is about setting up Hypriot OS, to run Docker on our Raspberry Pi server, allowing it to pull in the second repo and spin up some services:
The second one defines Telegraf, Influx & Grafana services (in docker-compose), with custom dashboards for system stats + Luftdaten, so we can see live stats from the sensors in-session:
(outstanding problems are: power supply + mDNS through the router)
We did get a sensor running and trying to post samples to the pi server. (The big white temperature/humidity sensor does the right thing, the smaller blue one does not.)
We’re not sure what we’re seeing- we have the pi running and (sometimes) it has all three containers running. I think the database is not created, or the URL used to connect to it isn’t resolving - the URL seems to be http://influxdb:8086 but it’s not clear how that hostname is meant to be resolved. We’re not sure where to look for logs, or how to tweak configs. (I thought we might get away with switching to localhost:8086 or with putting an entry into the pi’s /etc/hosts, but neither idea seems to have legs.)
It’s possible that we need to rebuild the image that the pi is running back to a clean state, and re-explore knowing what we now know.
Certain container ports have been mapped to/exposed on the host’s ports (defined in the docker-compose file) - imagine mDNS was working for a second
grafana is on techshed.local:3000
telegraf on techshed.local:8064
The service-level hostnames are internal to the virtual network Docker creates for the containers, so the services can talk to each other.
Useful docker command for reference:
# list running containers (look for the hashes
docker ps
#show log tail of stdout & stderr from a given container hash (can be a very short first part of hash, so long as it's unique)
docker logs <container hash goes here>
Note: I believe the telegraf service starts sending data to influx before it’s finished booting, so there may be an error or 2 in the logs for telegraf on boot - was minor enough/possibly complex to resolve so I’ve left that for now.
There is a line in the Luftdaten firmware to create the DB, but tbh, I can’t remember if that worked or I had to create the initial database manually via the API (can be done with a GET request in a browser). If there is any issue I should be able to bake it into the setup of the Influx service.
As for testing it - the magic is, it’s built to set itself up, so to get back to baseline, we just reflash the SD card and boot the Pi.
(also applies if anyone wants to fire up/play with the thing on their own Pi - first draft of a guide is on the hypriot-server repo linked above)
Ah, so maybe mDNS isn’t just a nicety, it’s something which has to be running on the network in order that this cluster of docker instances will interact?
Hm, on further thinking, possible the ESP on the sensor needs to use mDNS to post results? And so if the WiFi Access Point refuses to cooperate, that will scupper things. It’s not that the AP needs to help mDNS to work, but it needs not to break it. And it’s not about the cluster of Docker instances needing to communicate. Perhaps.
(off work Sick today - hence weird daytime posting!)
To clarify: the addressing of services by name like influxdb:8086 is internal to docker (visible to other containers, but not external to host) - is a different IP range to the external network - not exposed outside the host by default.
Ports of services are mapped out to ports on the host to expose them to the outside world.
In this case those are defined in the docker-compose file - excerpt (where ‘8086:8086’ means map ‘post-port:contanier-port’)
Once that’s defined, the outside world just sees the host presenting an influxdb endpoint on port 8086 - the host can be addressed by either IP or mDNS.
mDNS just makes things easier/more friendly than looking up the host IP and typing it in.
[Even more friendly would be some reverse proxy stuff in the host to hide the port away (e.g. influx.techshed.local or techshed.local/influx would be nicer than :ports]
In the meetup, I suspect there was just an issue with the initial DB creation as you thought.
In the meetup, I suspect there was just an issue with the initial DB creation as you thought.
Yeah I’m pretty sure this was it. I thought I saw telegraf communicating with the database as I saw it send some POST requests with SQL to the database. I just wasn’t sure whether or not the database was accepting the requests, or if the data from the NodeMCU was successfully reaching the server in the first place.
Just thinking around what were covering - it’s essentially making invisible parts of our everyday world visible.
That thread goes across water, air, even awareness around data contained in bluetooth & wifi signals… … If we get a few projects moving, maybe it’s an angle we can use to weave a story.
I just found this orientation diagram. Don’t know how I missed it before, but it seems the sniffer can go three ways up, but there is not indication which is preferred. To give us the most options for casing and mounting we could run three sensors in the same location, one in each orientation, and see if there is a significant difference. It would also be interesting to test the ‘not preferred’ orientations just for the craic…
Interesting readings from (what smelled like) a bonfire burning wood yesterday, highlighting indoor vs outdoor pollutants. I took the sensor outdoors for a short time and got PM2.5 well over 200 parts per million - during that time the indoor pollutants dropped, then reversed when the sensor came back indoors:
CO2 and Formaldehyde (HCHO) both higher indoors, even when there’s notable pollution from woodsmoke outdoors
PM2.5 particles & total volatile organic compounds (TVOCs) both higher outdoors
Looks like https://opensensemap.org/ has more nodes still showing (Luftdaten Firmware can send to OpenSenseMap too), so it could be a problem with Luftdaten recieving data…?
Hey guys,
I just had a call from Anna F at the council. A few updates…
Air quality workshop in April is cancelled
She does still want us to look at creating a sensor outside a school. She’s interested in creating one that could be there for a longer period/permanent. and possibly rolled out to all schools.
She had to request funding for it today, so she is putting a request in. I said we would still have to scope it out in more detail, but in principle, we’re happy to take it on…
(i.e. is there power, wifi, some area to attach and host the unit). We’d also have to consider other factors too in terms of risk assessment etc. )
She’s thinking this could be for September.
She also asked if we knew about existing air quality and was forwarded this by another council. The large box that used to be in the centre of the town hasn’t been in place for quite some time now (at least 9 years, possibly longer). We do however monitor nitrogen dioxide across Frome.
This is done by diffusion tube, so it’s not instantaneous data. The tubes collect monthly averages from which the final annual average is obtained. This data gives a long-term analysis of one of the main pollutants in the area and provides data to inform planning, traffic decisions etc.
Ok update on this. FTC has asked us to setup x6 sensor sites with supporting website. The team met last night and a few actions came out of it;
Danny > Email FTC to clarify points
Danny > Create draft brief and share with team for feedback
Will > Explore hosting options with AWS
Danny > to buy x6 more boxes (pending agreement with FTC)
In order for us to draw up a proposal we have a few questions. Happy to discuss on phone if you prefer
Budget and timelines
What is your budget and timelines for this? As this impacts the level of experience we can create. See below points/questions that allude to this…
TC: I have £3k initially but may be able to access more if needed (although then I’ll have less for other transport projects – this is definitely a priority though). I would really like to launch this in the next month or so if at all possible so people can see and appreciate the low-ish levels now…
Experience and website
Do you envision a custom website for this? Or we could just send readings to german citizen data site - Map Sensor.Community
If on custom website do we visualise the data in easy to understand format i.e. air quality today is ‘bad’ according to WHO OR just display the readings - e.g, air quality today is 33ppm
Would the website host any other content other than sensor readings?
**Below is the info I got from Mendip re air quality monitoring at the moment – it’s certainly not very instant!
Locations & installation
Do you have any sites in Frome for the air sensors in mind?
TC: Mixture would be good including town centre, Butts and a geographic spread across from and near schools (ten in total? Or less depending on budget)
The sensors need to be within wifi range and be connected to a power source.
TC: – that’s fine – if we make a map of where they could be I’m sure I can find contacts in each area who would be happy to host
Therefor sites would need to be identified and assed for suitability (private property may be better for this reason i.e. above shops etc).
The air sensors would need to be ‘fitted’ and setup to run on wifi, so assuming you would need us to do this.
Service level support
Assume you would expect some service level. So if sensor was tampered with or website went down you’d need us to go out and fix it within a time period?
TC: Yes please
Air sensors 1.0 and beyond
The 1.0 models need a power source and a wifi connection.
However, we can develop more sophisticated models, that run off solar and don’t need a wifi source (using either cellular or low band wifi network).
This would allow for more flexible location sites. This may take more time.
Let us know if this is something you would prefer for us to explore and cost.
TC: Initial programme is fine to use powered sources I reckon
Well done Danny. Looks like an imminent order for 10 of the Frome Tech Shed Vanilla Mk1 sniffers. Shall I start printing mounting kits for the boxes which will take a while? Or do you want to wait?
On the “are we duplicating the Mendip Distric Council measurements?” question:
Our approach is aiming for near-realtime (minutes not days/weeks) with open data
May form a basis for wider data/IoT projects for the area
Our readings (v1) are of particulate matter rather than the NO² measured by Mendip
There should be some correlation between the two that may make interesting data (possibly varying by emmission source, time of year and weather). Studies on correlation:
Our methods are lower cost, faster/more accessible results, but not as accurate - consistent enough to show ranges & trends, but not aimed at legal enforcement
