Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revision Both sides next revision
projects:telegraph [2018/03/20 22:12]
darron [Telegraph (cpu.xml)]
projects:telegraph [2018/11/16 09:48]
darron
Line 1: Line 1:
-==== Telegraph (SNMP/RRD/SQL) ====+==== Telegraph ====
 <wrap right> <wrap right>
 +{{:projects:light.jpg?200}} \\ \\
 +{{:projects:bme280.jpg?200}} \\ \\
 {{:projects:telegraph.png?200}} {{:projects:telegraph.png?200}}
 </wrap> </wrap>
  
-Telegraph is a project which will poll an SNMP +Telegraph is a multi-purpose tool which provides sensor processing, data gathering, web graphing and status reporting. It is designed specfically for the Raspberry Pi and uses cheap and easily obtained sensor devices. An example device is the BME280 which can provide readings of ambient temperature, barometric pressure and relative humidity. BME280 boards can cost as little as 99p.
-server and update an RRD or SQL (SQLite3) database.+
  
-A number of external SNMP daemon commands are supplied +Telegraph sensor processes are Simple Network Management Protocol (SNMPdaemon extensions which read data from various devices on a Raspberry or similar personal computer. The values returned can be used to populate a round robin (RRD) or SQL database either with Telegraph or a tool of your own choice.
-to capture values from various probes on a Raspberry or Banana Pi.+
  
-An example PHP script demonstrates how to generate RRD +Telegraph also supports networking and can use ICMP to measure network latency (PING) and with SNMP Telegraph can measure network utilisation. Network interface names are supported which are mapped to interface indexes when querying. 
-graphs like this one shown here.+ 
 +Telegraph is run from CRON at 5 minute intervals for data gathering and the sensor processes are configured within the SNMP (net-snmp) config. Sensors generally depend on a computer like the Raspberry Pi for interfacing, but data gathering can run on any networked Linux host. 
 + 
 +At each CRON iteration Telegraph can fork a process and supply it with gathered data. This allows for a user defined function and custom processing. 
 + 
 +PHP scripts demonstrate how you can access the RRD files containing the data to generate graphs for the web like the one shown above. 
 + 
 +Telegraph can send reports via E-mail, XMPP, or with HTTP POST when either a target is offline or returns back online or if a retrieved value falls outside a specified range.
  
 === Install === === Install ===
 +
 +First install mercurial to download telegraph.
  
 <code> <code>
 +sudo apt-get install mercurial
 hg clone http://hg.kewl.org/pub/telegraph hg clone http://hg.kewl.org/pub/telegraph
 cd telegraph cd telegraph
 </code> </code>
  
-At this time you must read the file named README in the telegraph directory+Now view the instructions found in the README file in the telegraph directory which explains the installation process. 
-This file explains the installation requirements for this software.+ 
 +=== Sensor processing === 
 + 
 +For the SNMP (simple network management protocol) daemon, Telegraph supplies the following sensor processes. 
 + 
 +^ Process ^ Use ^ 
 +| ads1115   | Analog to digital conversion | 
 +| bh1750fvi | Illuminance | 
 +| bme280    | Pressure + temperature + humidity | 
 +| bmp180    | Pressure + temperature | 
 +| bmp280    | Pressure + temperature | 
 +| dht11     | Humidity + temperature | 
 +| ds18b20   | Temperature | 
 +| ds3231    | Temperature | 
 +| lm75      | Temperature | 
 +| mcp3008   | Analog to digital conversion | 
 +| pcf8591   | Analog to digital conversion | 
 +| si7021    | Humidity    | 
 +| soc       | Temperature | 
 +| tc74      | Temperature | 
 +| tm7705    | Analog to digital conversion | 
 +| tsl2561   | Illuminance | 
 + 
 +Analog to digital conversion devices all provide the following features. 
 + 
 +^ Name ^ Use ^ Schematic ^ 
 +| RAW DATA   | Device specific value   | | 
 +| LEVEL      | Value scaled 0..1       | | 
 +| SCALE LEVEL| value scaled 0..N       | | 
 +| RESISTANCE | Resistance calculation  |VCC - R - VINPUT - R? - GND| 
 +| VOLTAGE    | Voltage calculation     | | 
 +| LM19       | Temperature             | | 
 +| LM35       | Temperature             | | 
 +| LM335      | Temperature             | | 
 +| MCP9700    | Temperature             | | 
 +| MCP9701    | Temperature             | | 
 +| TMP36      | Temperature             | | 
 +| LDR        | Light  calculation      |VCC - R - VINPUT - LDR - GND| 
 +| NTC        | Temperature calculation |VCC - R - VINPUT - NTC - GND| 
 + 
 +Each process has a set of command line arguments specific to that probe type.
  
-Telegraph may be installed once all requirements are met, with 
 <code> <code>
-make +$ bmp180 
-make install+USAGE: bmp180 I2C-DEV I2C-ADDR MODE 
 +Program to query the BMP180. 
 + 
 +Error: missing arg(s). 
 + 
 +MODE: 
 + 0 TEMPERATURE 
 + 2 PRESSURE 
 + 
 +VERSION: 
 + 1.0 
 +U
 </code> </code>
  
-=== SNMPD (snmpd.conf) ===+I2C-DEV represents the device node of the i2c bus the device is on. The I2C-ADDR is the bus address. MODE determines what type of reading is required.
  
-SNMPD is set up to return a value when probed for the CPU temperature on a Raspberry Pi.+In operation it can look like this:
  
 <code> <code>
-rocommunity public X.X.X.X/24+$ bmp280 /dev/i2c-1 0x76 0 
 +20.741 
 + 
 +$ bmp280 /dev/i2c-1 0x76 2 
 +1003.037 
 + 
 +$ bmp280 /dev/i2c-1 0x76 1 
 +
 +</code> 
 + 
 +In the first two instances, temperature and pressure are retrieved, in the last case U is returned which represents UNKNOWN since this device doesn't support that MODE. 
 + 
 +For analog to digital conversion, all levels and calculations are referenced to the 3V3 supply voltage. For example, the VOLTAGE calculation simply takes the LEVEL value and multiplies it by 3.3. 
 + 
 +For the light calculation (lux) the following formula is used. 
 +<code> 
 +log(lux) = (log(ldr) - intercept) / slope 
 +</code> 
 + 
 +The slope and intercept are device dependant and this may be calculated using the Telegraph 'meter' command line tool using a reference device like the tsl2561. The results can produce a reasonable approximation for lux with very cheap components. 
 + 
 +The NTC calculation uses this formula. 
 +<code> 
 +1/T = 1/To + 1/B . ln(R/Ro) 
 +</code> 
 + 
 +B is known as the NTC B25 value and Ro is R25 on the datasheet for the device. R is the known resistance of the device at the current temperature and the B25/R25 values are representative of room temperature at 25'C. 
 + 
 +=== Simple Network Management Protocol === 
 + 
 +The SNMP server configuration uses the net-snmp extend feature which allows a process to return a text string to the SNMP client. For RRD and Telegraph, this value is text string representing a numerical value, else U which means UKNOWN. 
 + 
 +/etc/snmp/snmpd.conf 
 +<code> 
 +rocommunity public 192.168.0.0/24
 extend cpu /usr/local/bin/soc /sys/class/thermal/thermal_zone0/temp extend cpu /usr/local/bin/soc /sys/class/thermal/thermal_zone0/temp
 </code> </code>
  
-=== Telegraph (cpu.xml) ===+In this simple example, a net-snmp extend is set to return the CPU temperature when queried. The first line is access control with rocommunity describing a read-only group and public being that groups authentication. The network range limits any queries for this group to that range.
  
-This configuration file sets up query for the above cpu temperature in snmpd.conf.+Any number of extend entries can be made with each having unique name and purpose.
  
 +Whenever a change to the SNMP server configuration is made the server must be restarted for any new settings to take effect.
 +<code>
 +/etc/init.d/snmpd restart
 +</code>
 +
 +=== Data gathering ===
 +
 +At periodic intervals Telegraph is run from CRON to gather data. The data can represent gauges which are ranges of values, like temperature, or counters which are used to measure packet throughput on network interfaces.
 +
 +The type of data is configured as a gauge by default, but this can be overriden to any type supported by RRDTOOL when using an RRD. For SQL, this is user defined, and not documented here, please refer to the meter demo for assistance.
 +
 +This configuration file sets up a query for cpu temperature as configured in snmpd.conf in the section above.
 +
 +/etc/telegraph/cpu.xml
 <code> <code>
 <?xml version="1.0"?> <?xml version="1.0"?>
-<telegraph+<telegraph title="CPU"
- <graph title="CPU Temp" label="C" /> + <graph label="C" /> 
- <database type="rrd" filename="/var/lib/rrd/cpu.rrd" /> + <database type="rrd" filename="cpu.rrd" /> 
- <query source="0" name="cpu" host="X.X.X.X" community="public" miboid="cpu" />+ <query source="0" name="rpi" host="192.168.0.100" community="public" extend="cpu" />
 </telegraph> </telegraph>
 </code> </code>
 +
 +The title is used by both graphing and reporting. For graphing it will appear at
 +the head of a graph and with reporting it will be the subject of the report.
 +
 +The RRD file is where data is stored, by default this will be in /var/lib/telegraph/
 +but a path may be specified if required.
 +
 +Multiple queries may be entered, starting at source 0 and incrementing by one for
 +each new query.
 +
 +Only the same data types can exist in any one configuration file, for example, only
 +temperature or only network activity. Data types cannot be mixed and demand
 +separate configuration files and RRD files.
 +
 +If an RRD file exists and a new query is added, it will not work and a new column
 +must be added to the RRD file beforehand.
 +
 +Telegraph supplies a command line tool 'rrdadd' which will add 1 or more blank columns
 +to allow new sources to be added easily to an already existing RRD file.
  
 === CRON === === CRON ===
  
-CRON job will run every 5 minutes to update the RRD.+CRON is a daemon in Unix which invokes processes at specific times and intervals. 
 + 
 +CRON should run Telegraph every 5 minutes to gather data like this.
 <code> <code>
-*/5 * * * * /usr/local/bin/telegraph /usr/local/etc/cpu.xml+*/5 * * * * /usr/local/bin/telegraph -d 0 cpu.xml
 </code> </code>
  
-=== World wide web ===+CRON configuration is in TABLES, or CRONTABS, and is edited like this for Telegraph. It should be noted that Telegraph only uses root to set up ICMP and runs as an unprivileged user otherwise. 
 +<code> 
 +sudo crontab -e 
 +</code> 
 + 
 +This CRON job will fetch values and store them in the database as configured by cpu.xml 
 + 
 +The -d option is a delay value, it can be used to delay the process from CRON 
 +when multiple telegraph processes are executed in parallel to stagger their 
 +processing and potential load on the system or target systems. 
 + 
 +<code> 
 +*/5 * * * * /usr/local/bin/telegraph cpu.xml 
 +*/5 * * * * /usr/local/bin/telegraph -d 30 network.xml 
 +</code> 
 + 
 +In these CRON entries every 5 minutes a Telegraph process gathers cpu data whilst another waits 30 seconds before gathering network statistics. 
 + 
 +=== User defined function === 
 + 
 +A user defined function may be forked every time Telegraph fetches new data. Forking allows a new process to be created and run in parrallel by the process creating the fork. 
 + 
 +The configuration file has option added to achieve this. 
 + 
 +<code> 
 +<user fork="/usr/local/bin/myprocess" /> 
 +</code> 
 + 
 +With this line, the executable /usr/local/bin/myprocess is forked after Telegraph has finished gathering data. 
 + 
 +The command line arguments for the process are the process name, the telegraph title and then every result for each source in turn. This means that argv[2] would represent source 0 and argv[3] source 1, etc. 
 + 
 +=== Web graphing === 
 + 
 +Example configuration exists in the demo directory of the project which is used to demonstrate its operation with PHP scripts that are installed in /var/www/telegraph/ 
 + 
 +The london demo can be seen [[http://london.c2n.uk/|here]]. This demo uses a proxy cache so the resultant graphs may be slightly dated. 
 + 
 +The demo has been designed to work with apache2 but nginx will also work well. 
 + 
 +=== Status reporting === 
 + 
 +Telegraph can report problems it finds when gathering data or when a returned value is out of range. 
 + 
 +Problem reports are of two types, DOWN and UP, which signify if a source was not available or available to be queried. This could mean that the SNMP service was offline, or that ICMP PING could not reach a host. 
 + 
 +A DOWN report will occur when a problem is found, and an UP when the problem is resolved. Whilst a service continues to be DOWN, or UP, no reports are made. 
 + 
 +Range reports occur when a value goes below or equal to a preset LOW value or becomes equal to or rises above an preset HIGH value in the configuration. This could be used to send reports when temperature goes too low or rises too high, for example. Range reports only work with both LOW and HIGH values given and no reports are made unless the value returns to one state or the other. 
 + 
 +Three types of report delivery exist, e-mail, XMPP and HTTP POST. 
 + 
 +E-mail reporting requires sendmail or equivalent to be installed and XMPP demands a valid jabber/xmpp account to use as a sender. HTTP POST is set up to work with the RUN/STOP watchdog service at [[https://runstop.uk|runstop.uk]] 
 + 
 +The Telegraph XML configuration file is updated for reporting. 
 + 
 +<code> 
 +<monitor filename="cpu.db" /> 
 +<report type="email"    to="me@example.com" /> 
 +<report type="xmpp"     to="me@xmpp.example.com" auth="xmpp" /> 
 +<report type="watchdog" to="https://cgi.runstop.uk/watchdog/" auth="watchdog" /> 
 +</code> 
 + 
 +The monitor line sets up an SQL database file used to monitor query state (UP, DOWM, LOW and HIGH). 
 + 
 +The three report lines demonstrate sending reports. The first is e-mail, the second sends a jabber message to the destination using credentials found in 
 +the file /etc/telegraph/xmpp. This file is two lines, one being jabber identity and the second password. This file cannot be world readable. The last line post an HTTP message to the runstop.uk server using the credentials in NETRC format found in /etc/telegraph/watchdog. This file may not be world readable either. 
 + 
 +The content of a report contains 1 or more lines in CSV format. 
 + 
 +Here is an ICMP report of hosts which do not respond to ICMP PING. 
 +<code> 
 +"Pi2 AR9271", 192.168.0.162, ICMP, DOWN, 0 
 +"Pi4 BCM43143", 192.168.0.164, ICMP, DOWN, 0 
 +"PiC AR9287", 192.168.0.172, ICMP, DOWN, 0 
 +"PiE RTL8191SU", 192.168.0.174, ICMP, DOWN, 0 
 +</code> 
 + 
 +The columns are name, host, query type, status, value. For ICMP value will be 0 or 1. 
 + 
 +This is a range report. 
 +<code> 
 +"Pi7 DS18B20 room", 192.168.0.167, SNMP, HIGH, 20 
 +</code> 
 + 
 +This report show that a HIGH level of 20 has been reached by a temperature probe. 
 + 
 +=== Other tools === 
 + 
 +If you want to use the telegraph probes with SNMP but not use Telegraph to gather data then you can use the Telegraph 'extend' command line tool to generate the required SNMP MIBs for queries by other tools. 
 + 
 +For example, for the following SNMP configuration. 
 + 
 +<code> 
 +extend cpu /usr/local/bin/soc /sys/class/thermal/thermal_zone0/temp 
 +</code> 
 + 
 +The MIB can be determined with extend. 
 + 
 +<code> 
 +$ extend cpu 
 +1.3.6.1.4.1.8072.1.3.2.3.1.1.3.99.112.117 
 +</code> 
 + 
 +Test it somewhat like this. 
 + 
 +<code> 
 +$ snmpget -v2c -c public 192.168.0.100 1.3.6.1.4.1.8072.1.3.2.3.1.1.3.99.112.117 
 +NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."cpu" = STRING: 34.166 
 +</code>
  
-Example scripts exist in the demo directory of the project which +How you use the MIB in other tools is of course, tool specific.
-demonstrate how the RRD is queried to produce a graph of the values +
-stored within it.+
  
-The demo can be seen [[http://london.omweb.org/|here]].