I’ve added repos for both parts of the Techshed server to Github in case anyone’s interested.
The first is about setting up Hypriot OS, to run Docker on our Raspberry Pi server, allowing it to pull in the second repo and spin up some services:
The second one defines Telegraf, Influx & Grafana services (in
docker-compose), with custom dashboards for system stats + Luftdaten, so we can see live stats from the sensors in-session:
(outstanding problems are: power supply + mDNS through the router)
We did get a sensor running and trying to post samples to the pi server. (The big white temperature/humidity sensor does the right thing, the smaller blue one does not.)
We’re not sure what we’re seeing- we have the pi running and (sometimes) it has all three containers running. I think the database is not created, or the URL used to connect to it isn’t resolving - the URL seems to be http://influxdb:8086 but it’s not clear how that hostname is meant to be resolved. We’re not sure where to look for logs, or how to tweak configs. (I thought we might get away with switching to localhost:8086 or with putting an entry into the pi’s /etc/hosts, but neither idea seems to have legs.)
It’s possible that we need to rebuild the image that the pi is running back to a clean state, and re-explore knowing what we now know.
Certain container ports have been mapped to/exposed on the host’s ports (defined in the
docker-compose file) - imagine mDNS was working for a second
- grafana is on
- telegraf on
The service-level hostnames are internal to the virtual network Docker creates for the containers, so the services can talk to each other.
Useful docker command for reference:
# list running containers (look for the hashes
#show log tail of stdout & stderr from a given container hash (can be a very short first part of hash, so long as it's unique)
docker logs <container hash goes here>
Note: I believe the telegraf service starts sending data to influx before it’s finished booting, so there may be an error or 2 in the logs for telegraf on boot - was minor enough/possibly complex to resolve so I’ve left that for now.
There is a line in the Luftdaten firmware to create the DB, but tbh, I can’t remember if that worked or I had to create the initial database manually via the API (can be done with a GET request in a browser). If there is any issue I should be able to bake it into the setup of the Influx service.
As for testing it - the magic is, it’s built to set itself up, so to get back to baseline, we just reflash the SD card and boot the Pi.
(also applies if anyone wants to fire up/play with the thing on their own Pi - first draft of a guide is on the hypriot-server repo linked above)
Ah, so maybe mDNS isn’t just a nicety, it’s something which has to be running on the network in order that this cluster of docker instances will interact?
Hm, on further thinking, possible the ESP on the sensor needs to use mDNS to post results? And so if the WiFi Access Point refuses to cooperate, that will scupper things. It’s not that the AP needs to help mDNS to work, but it needs not to break it. And it’s not about the cluster of Docker instances needing to communicate. Perhaps.
(off work Sick today - hence weird daytime posting!)
To clarify: the addressing of services by name like
influxdb:8086 is internal to docker (visible to other containers, but not external to host) - is a different IP range to the external network - not exposed outside the host by default.
Ports of services are mapped out to ports on the host to expose them to the outside world.
In this case those are defined in the docker-compose file - excerpt (where ‘8086:8086’ means map ‘post-port:contanier-port’)
Once that’s defined, the outside world just sees the host presenting an influxdb endpoint on port 8086 - the host can be addressed by either IP or mDNS.
mDNS just makes things easier/more friendly than looking up the host IP and typing it in.
[Even more friendly would be some reverse proxy stuff in the host to hide the port away (e.g.
techshed.local/influx would be nicer than :ports]
In the meetup, I suspect there was just an issue with the initial DB creation as you thought.
Would we not also want to include more about what is in the air as well see
These other sensors then cover carbon monixoide, Ozone Ammonia methane etc
In the meetup, I suspect there was just an issue with the initial DB creation as you thought.
Yeah I’m pretty sure this was it. I thought I saw telegraf communicating with the database as I saw it send some POST requests with SQL to the database. I just wasn’t sure whether or not the database was accepting the requests, or if the data from the NodeMCU was successfully reaching the server in the first place.
Just thinking around what were covering - it’s essentially making invisible parts of our everyday world visible.
That thread goes across water, air, even awareness around data contained in bluetooth & wifi signals… … If we get a few projects moving, maybe it’s an angle we can use to weave a story.
I just found this orientation diagram. Don’t know how I missed it before, but it seems the sniffer can go three ways up, but there is not indication which is preferred. To give us the most options for casing and mounting we could run three sensors in the same location, one in each orientation, and see if there is a significant difference. It would also be interesting to test the ‘not preferred’ orientations just for the craic…
Interesting readings from (what smelled like) a bonfire burning wood yesterday, highlighting indoor vs outdoor pollutants. I took the sensor outdoors for a short time and got PM2.5 well over 200 parts per million - during that time the indoor pollutants dropped, then reversed when the sensor came back indoors:
- CO2 and Formaldehyde (HCHO) both higher indoors, even when there’s notable pollution from woodsmoke outdoors
- PM2.5 particles & total volatile organic compounds (TVOCs) both higher outdoors
- one more image to show how high particulates were getting (no visible smoke, but strong smell):
The dip in the middle was from going back indoors for a short time.
I don’t know if people remember what this map usually looks like (a lot less green), but these are the current Luftdaten maps for Europe:
(can’t make 4 posts in a row)
The number of sensor nodes on Luftdaten’s mapping https://maps.sensor.community/#5/48.613/3.543
Has dropped off massively…
The chinese mainland & most of Asia is now blank (I’m sure there used to be nodes there).
Looks like https://opensensemap.org/ has more nodes still showing (Luftdaten Firmware can send to OpenSenseMap too), so it could be a problem with Luftdaten recieving data…?
Could be (some of these don’t for with OpenMapSense showing more nodes):
- Luftdaten infrastructure issue or
- People not maintaining during this time
- Network traffic load/prioritisation or WiFi interference with everyone at home?
- Firmware bug
- State intervention or malicious activity
The top few seem simpler/more likely - the last one is probably just paranoia on my part.
I just had a call from Anna F at the council. A few updates…
- Air quality workshop in April is cancelled
- She does still want us to look at creating a sensor outside a school. She’s interested in creating one that could be there for a longer period/permanent. and possibly rolled out to all schools.
She had to request funding for it today, so she is putting a request in. I said we would still have to scope it out in more detail, but in principle, we’re happy to take it on…
(i.e. is there power, wifi, some area to attach and host the unit). We’d also have to consider other factors too in terms of risk assessment etc. )
She’s thinking this could be for September.
She also asked if we knew about existing air quality and was forwarded this by another council.
The large box that used to be in the centre of the town hasn’t been in place for quite some time now (at least 9 years, possibly longer). We do however monitor nitrogen dioxide across Frome.
This is done by diffusion tube, so it’s not instantaneous data. The tubes collect monthly averages from which the final annual average is obtained. This data gives a long-term analysis of one of the main pollutants in the area and provides data to inform planning, traffic decisions etc.
The met office produce a pollution forecast for DEFRA https://uk-air.defra.gov.uk/forecasting/locations?q=frome, which may be of interest to you.
"She does still want us to look at creating a sensor outside a school. "
Could it be changed to a Nursery (Rainbow on Portway)?
Ok update on this. FTC has asked us to setup x6 sensor sites with supporting website. The team met last night and a few actions came out of it;
Danny > Email FTC to clarify points
Danny > Create draft brief and share with team for feedback
Will > Explore hosting options with AWS
Danny > to buy x6 more boxes (pending agreement with FTC)
Will update once I know more…
Response from town council
In order for us to draw up a proposal we have a few questions. Happy to discuss on phone if you prefer
Budget and timelines
What is your budget and timelines for this? As this impacts the level of experience we can create. See below points/questions that allude to this…
TC: I have £3k initially but may be able to access more if needed (although then I’ll have less for other transport projects – this is definitely a priority though). I would really like to launch this in the next month or so if at all possible so people can see and appreciate the low-ish levels now…
Experience and website
- Do you envision a custom website for this? Or we could just send readings to german citizen data site - https://deutschland.maps.sensor.community/#6/51.165/10.455
- If on custom website do we visualise the data in easy to understand format i.e. air quality today is ‘bad’ according to WHO OR just display the readings - e.g, air quality today is 33ppm
- Would the website host any other content other than sensor readings?
TC: I’m thinking of a Clean Air for Frome website so the sensors would be a key part of this. I think a bespoke version that makes it really clear and obvious what is ‘good’ or ‘bad’ would be really useful, but I suppose this depends how much it costs. Also have you looked at the current data for Frome (that Mendip organise)? Would be good to ensure we are adding to rather than duplicating this.** https://www.mendip.gov.uk/media/20828/2018-Air-Quality-Annual-Status-Report/pdf/2018_Air_Quality_Annual_Status_Report.pdf?m=636854941860330000
**Below is the info I got from Mendip re air quality monitoring at the moment – it’s certainly not very instant!
Locations & installation
- Do you have any sites in Frome for the air sensors in mind?
TC: Mixture would be good including town centre, Butts and a geographic spread across from and near schools (ten in total? Or less depending on budget)
- The sensors need to be within wifi range and be connected to a power source.
TC: – that’s fine – if we make a map of where they could be I’m sure I can find contacts in each area who would be happy to host
- Therefor sites would need to be identified and assed for suitability (private property may be better for this reason i.e. above shops etc).
- The air sensors would need to be ‘fitted’ and setup to run on wifi, so assuming you would need us to do this.
Service level support
Assume you would expect some service level. So if sensor was tampered with or website went down you’d need us to go out and fix it within a time period?
TC: Yes please
Air sensors 1.0 and beyond
- The 1.0 models need a power source and a wifi connection.
- However, we can develop more sophisticated models, that run off solar and don’t need a wifi source (using either cellular or low band wifi network).
- This would allow for more flexible location sites. This may take more time.
- Let us know if this is something you would prefer for us to explore and cost.
TC: Initial programme is fine to use powered sources I reckon
@p.j @Will @Ed_S @Al_B @Snowy
Well done Danny. Looks like an imminent order for 10 of the Frome Tech Shed Vanilla Mk1 sniffers. Shall I start printing mounting kits for the boxes which will take a while? Or do you want to wait?
couple of questions…
All looks positive.
On the “are we duplicating the Mendip Distric Council measurements?” question:
- Our approach is aiming for near-realtime (minutes not days/weeks) with open data
May form a basis for wider data/IoT projects for the area
- Our readings (v1) are of particulate matter rather than the NO² measured by Mendip
- There should be some correlation between the two that may make interesting data (possibly varying by emmission source, time of year and weather). Studies on correlation:
- Our methods are lower cost, faster/more accessible results, but not as accurate - consistent enough to show ranges & trends, but not aimed at legal enforcement