Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
projects:telegraph [2018/03/20 22:12]
darron [Telegraph (cpu.xml)]
projects:telegraph [2018/11/16 04:24] (current)
Line 1: Line 1:
-==== Telegraph ​(SNMP/​RRD/​SQL) ​====+==== Telegraph ====
 <wrap right> <wrap right>
 {{:​projects:​telegraph.png?​200}} {{:​projects:​telegraph.png?​200}}
 </​wrap>​ </​wrap>​
  
-Telegraph is a project ​which will poll an SNMP +Telegraph is a multi-purpose tool which provides sensor processing, data gathering, web graphing ​and status reporting. It is designed specfically for the Raspberry Pi and uses cheap and easily obtained sensor devices. An example device is the BME280 which can provide readings of ambient temperature,​ barometric pressure and relative humidity. BME280 boards can cost as little as 99p.
-server ​and update an RRD or SQL (SQLite3) database.+
  
-A number of external ​SNMP daemon ​commands are supplied +Telegraph sensor processes are Simple Network Management Protocol (SNMPdaemon ​extensions which read data from various ​devices ​on a Raspberry or similar personal computer. The values returned can be used to populate a round robin (RRD) or SQL database either with Telegraph or a tool of your own choice.
-to capture values ​from various ​probes ​on a Raspberry or Banana Pi.+
  
-An example ​PHP script demonstrates ​how to generate ​RRD +Telegraph also supports networking and can use ICMP to measure network latency (PING) and with SNMP Telegraph can measure network utilisation. Network interface names are supported which are mapped to interface indexes when querying. 
-graphs like this one shown here.+ 
 +Telegraph is run from CRON at 5 minute intervals for data gathering and the sensor processes are configured within the SNMP (net-snmp) config. Sensors generally depend on a computer like the Raspberry Pi for interfacing,​ but data gathering can run on any networked Linux host. 
 + 
 +In each CRON job Telegraph can spawn a process and supply it with gathered data. This allows for a user defined application and custom processing. 
 + 
 +PHP scripts demonstrate ​how you can access the RRD files containing the gathered data to generate graphs ​for the web like the one shown above. 
 + 
 +Telegraph can send reports via E-mail, XMPP, or with HTTP POST when either a target is offline or returns back online or if a retrieved value falls outside a specified range.
  
 === Install === === Install ===
 +
 +First install mercurial to download telegraph.
  
 <​code>​ <​code>​
 +sudo apt-get install mercurial
 hg clone http://​hg.kewl.org/​pub/​telegraph hg clone http://​hg.kewl.org/​pub/​telegraph
 cd telegraph cd telegraph
 </​code>​ </​code>​
  
-At this time you must read the file named README in the telegraph directory+Now view the instructions found in the README ​file in the telegraph directory ​which explains the installation ​process. 
-This file explains the installation ​requirements for this software.+ 
 +=== Sensor processing === 
 + 
 +For the SNMP (simple network management protocol) daemon, Telegraph supplies the following sensor processes. 
 + 
 +^ Process ^ Use ^ 
 +| ads1115 ​  | Analog to digital conversion | 
 +| bh1750fvi | Illuminance | 
 +| bme280 ​   | Pressure + temperature + humidity | 
 +| bmp180 ​   | Pressure + temperature | 
 +| bmp280 ​   | Pressure + temperature | 
 +| dht11     | Humidity + temperature | 
 +| ds18b20 ​  | Temperature | 
 +| ds3231 ​   | Temperature | 
 +| lm75      | Temperature | 
 +| mcp3008 ​  | Analog to digital conversion | 
 +| pcf8591 ​  | Analog to digital conversion | 
 +| si7021 ​   | Humidity ​   | 
 +| soc       | Temperature | 
 +| tc74      | Temperature | 
 +| tm7705 ​   | Analog to digital conversion | 
 +| tsl2561 ​  | Illuminance | 
 + 
 +Analog to digital conversion devices all provide the following features. 
 + 
 +^ Name ^ Use ^ Schematic ^ 
 +| RAW DATA   | Device specific value   | | 
 +| LEVEL      | Value scaled 0..1       | | 
 +| SCALE LEVEL| value scaled 0..N       | | 
 +| RESISTANCE | Resistance calculation ​ |VCC - R - VINPUT - R? - GND| 
 +| VOLTAGE ​   | Voltage calculation ​    | | 
 +| LM19       | Temperature ​            | | 
 +| LM35       | Temperature ​            | | 
 +| LM335      | Temperature ​            | | 
 +| MCP9700 ​   | Temperature ​            | | 
 +| MCP9701 ​   | Temperature ​            | | 
 +| TMP36      | Temperature ​            | | 
 +| LDR        | Light  calculation ​     |VCC - R - VINPUT - LDR - GND| 
 +| NTC        | Temperature calculation |VCC - R - VINPUT - NTC - GND| 
 + 
 +Each process has a set of command line arguments specific to that probe type.
  
-Telegraph may be installed once all requirements are met, with 
 <​code>​ <​code>​
-make +$ bmp180 
-make install+USAGE: bmp180 I2C-DEV I2C-ADDR MODE 
 +Program to query the BMP180. 
 + 
 +Error: missing arg(s). 
 + 
 +MODE: 
 + 0 TEMPERATURE 
 + 2 PRESSURE 
 + 
 +VERSION: 
 + 1.0 
 +U
 </​code>​ </​code>​
  
-=== SNMPD (snmpd.conf) ===+I2C-DEV represents the device node of the i2c bus the device is on. The I2C-ADDR is the bus address. MODE determines what type of reading is required.
  
-SNMPD is set up to return a value when probed for the CPU temperature on a Raspberry Pi.+In operation it can look like this:
  
 +<​code>​
 +$ bmp280 /dev/i2c-1 0x76 0
 +20.741
 +
 +$ bmp280 /dev/i2c-1 0x76 2
 +1003.037
 +
 +$ bmp280 /dev/i2c-1 0x76 1
 +U
 +</​code>​
 +
 +In the first two instances, temperature and pressure are retrieved, in the last case U is returned which represents UNKNOWN since this device doesn'​t support that MODE.
 +
 +For analog to digital conversion, all levels and calculations are referenced to the 3V3 supply voltage. For example, the VOLTAGE calculation simply takes the LEVEL value and multiplies it by 3.3.
 +
 +For the light calculation (lux) the following formula is used.
 +<​code>​
 +log(lux) = (log(ldr) - intercept) / slope
 +</​code>​
 +
 +The slope and intercept are device dependant and this may be calculated using the Telegraph '​meter'​ command line tool using a reference device like the tsl2561. The results can produce a reasonable approximation for lux with very cheap components.
 +
 +The NTC calculation uses this formula.
 +<​code>​
 +1/T = 1/To + 1/B . ln(R/Ro)
 +</​code>​
 +
 +B is known as the NTC B25 value and Ro is R25 on the datasheet for the device. R is the known resistance of the device at the current temperature and the B25/R25 values are representative of room temperature at 25'C.
 +
 +=== Simple Network Management Protocol ===
 +
 +The SNMP server configuration uses the net-snmp extend feature which allows a process to return a text string to the SNMP client. For RRD and Telegraph, this value is text string representing a numerical value, else U which means UKNOWN.
 +
 +/​etc/​snmp/​snmpd.conf
 <​code>​ <​code>​
 rocommunity public X.X.X.X/24 rocommunity public X.X.X.X/24
Line 38: Line 130:
 </​code>​ </​code>​
  
-=== Telegraph (cpu.xml) ​===+In this simple example, a net-snmp extend is set to return the CPU temperature when queried. The first line is access control with rocommunity describing a read-only group and public being that groups authentication. The network range limits any queries for this group to that range. 
 + 
 +Any number of extend entries can be made with each having a unique name and purpose. 
 + 
 +Whenever a change to the SNMP server configuration is made the server must be restarted for any new settings to take effect. 
 +<​code>​ 
 +/​etc/​init.d/​snmpd restart 
 +</​code>​ 
 + 
 +=== Data gathering ​===
  
 This configuration file sets up a query for the above cpu temperature in snmpd.conf. This configuration file sets up a query for the above cpu temperature in snmpd.conf.
  
 +/​etc/​telegraph/​cpu.xml
 <​code>​ <​code>​
 <?xml version="​1.0"?>​ <?xml version="​1.0"?>​
-<​telegraph+<​telegraph title="​CPU"​
- <​graph ​title="​CPU ​Temp" label="​C"​ /> + <​graph ​label="​C"​ /> 
- <​database type="​rrd"​ filename="​/​var/​lib/​rrd/​cpu.rrd"​ /> + <​database type="​rrd"​ filename="​cpu.rrd"​ /> 
- <​query source="​0"​ name="​cpu"​ host="​X.X.X.X"​ community="​public" ​miboid="​cpu"​ />+ <​query source="​0"​ name="​cpu"​ host="​X.X.X.X"​ community="​public" ​extend="​cpu"​ />
 </​telegraph>​ </​telegraph>​
 </​code>​ </​code>​
 +
 +The title is used by both graphing and reporting. For graphing it will appear at
 +the head of a graph and with reporting it will be the subject of the report.
 +
 +The rrd file is where data is stored, by default this will be in /​var/​lib/​telegraph
 +but a path may be specified if required.
 +
 +Multiple queries may be entered, starting at source 0 and incrementing by one for
 +each new query.
 +
 +Only similar data types can exist in one configuration file, for example, only
 +temperature or only network activity. Data types cannot be mixed and demand
 +separate configurations and databases.
 +
 +If an RRD file exists and a new query is added, it will not work and a new column
 +must be added to the RRD file beforehand.
 +
 +Telegraph supplies a command line tool '​rrdadd'​ which will add 1 or more blank columns
 +to allow new sources to be added easily to an already existing RRD file.
  
 === CRON === === CRON ===
  
-CRON job will run every 5 minutes to update the RRD.+CRON  ​should ​run Telegraph ​every 5 minutes to gather data.
 <​code>​ <​code>​
-*/5 * * * * /​usr/​local/​bin/​telegraph ​/​usr/​local/​etc/​cpu.xml+*/5 * * * * /​usr/​local/​bin/​telegraph ​-d 0 cpu.xml
 </​code>​ </​code>​
  
-=== World wide web ===+This cron job will fetch values and store them in the database as configured by cpu.xml 
 + 
 +The -d option is a delay value, it can be used to delay the process from cron 
 +when multiple telegraph processes are executed in parallel to stagger their 
 +processing and potential load on the system or target systems. 
 + 
 +<​code>​ 
 +*/5 * * * * /​usr/​local/​bin/​telegraph -d 0  cpu.xml 
 +*/5 * * * * /​usr/​local/​bin/​telegraph -d 30 network.xml 
 +</​code>​ 
 + 
 +=== Web graphing ​=== 
 + 
 +Example configuration exists in the demo directory of the project which is used to demonstrate its operation 
 +with PHP scripts that are installed in /​var/​www/​telegraph 
 + 
 +The london demo can be seen [[http://​london.c2n.uk/​|here]]. This demo uses a proxy cache so the resultant 
 +graphs may be slightly dated. 
 + 
 +The demo has been designed to work with apache2 but nginx will also work well. 
 + 
 +=== Status reporting === 
 + 
 +Telegraph can report problems it finds when gathering data or when a returned value is out of range. 
 + 
 +Problem reports are of two types, DOWN and UP, which signify if a source was not available or available to be queried. This could mean that the SNMP service was offline, or that ICMP PING could not reach a host. 
 + 
 +A DOWN report will occur when a problem is found, and an UP when the problem is resolved. Whilst a service continues to be DOWN, or UP, no reports are made. 
 + 
 +Range reports occur when a value goes below or equal to a preset LOW value or becomes equal to or rises above an preset HIGH value in the configuration. This could be used to send reports when temperature goes too low or rises too high, for example. Range reports only work with both LOW and HIGH values given and no reports are made unless the value returns to one state or the other. 
 + 
 +Three types of report delivery exist, e-mail, XMPP and HTTP POST. 
 + 
 +E-mail reporting requires sendmail or equivalent to be installed and XMPP demands a valid jabber/xmpp account to use as a sender. HTTP POST is set up to work with the RUN/STOP watchdog service at [[https://​runstop.uk|runstop.uk]] 
 + 
 +The Telegraph XML configuration file is updated for reporting. 
 + 
 +<​code>​ 
 +<monitor filename="​cpu.db"​ /> 
 +<report type="​email" ​   to="​me@example.com"​ /> 
 +<report type="​xmpp" ​    ​to="​me@xmpp.example.com"​ auth="​xmpp"​ /> 
 +<report type="​watchdog"​ to="​https://​cgi.runstop.uk/​watchdog/"​ auth="​watchdog"​ /> 
 +</​code>​ 
 + 
 +The monitor line sets up an SQL database file used to monitor query state (UP, DOWM, LOW and HIGH). 
 + 
 +The three report lines demonstrate sending reports. The first is e-mail, the second sends a jabber message to the destination using credentials found in 
 +the file /​etc/​telegraph/​xmpp. This file is two lines, one being jabber identity and the second password. This file cannot be world readable. The last line post an HTTP message to the runstop.uk server using the credentials in NETRC format found in /​etc/​telegraph/​watchdog. This file may not be world readable either. 
 + 
 +The content of a report contains 1 or more lines in CSV format. 
 + 
 +Here is an ICMP report of hosts which do not respond to ICMP PING. 
 +<​code>​ 
 +"Pi2 AR9271",​ 192.168.0.162,​ ICMP, DOWN, 0 
 +"Pi4 BCM43143",​ 192.168.0.164,​ ICMP, DOWN, 0 
 +"PiC AR9287",​ 192.168.0.172,​ ICMP, DOWN, 0 
 +"PiE RTL8191SU",​ 192.168.0.174,​ ICMP, DOWN, 0 
 +</​code>​ 
 + 
 +The columns are name, host, query type, status, value. For ICMP value will be 0 or 1. 
 + 
 +This is a range report. 
 +<​code>​ 
 +"Pi7 DS18B20 room", 192.168.0.167,​ SNMP, HIGH, 20 
 +</​code>​ 
 + 
 +This report show that a HIGH level of 20 has been reached by a temperature probe. 
 + 
 + 
 +=== Other tools === 
 + 
 +If you want to use the telegraph probes with SNMP but not use Telegraph to gather data then you can use the Telegraph '​extend'​ command line tool to generate the required SNMP MIBs for queries by other tools. 
 + 
 +For example, for the following SNMP configuration. 
 + 
 +<​code>​ 
 +extend cpu /​usr/​local/​bin/​soc /​sys/​class/​thermal/​thermal_zone0/​temp 
 +</​code>​ 
 + 
 +The MIB can be determined with extend. 
 + 
 +<​code>​ 
 +$ extend cpu 
 +1.3.6.1.4.1.8072.1.3.2.3.1.1.3.99.112.117 
 +</​code>​ 
 + 
 +Test it somewhat like this. 
 + 
 +<​code>​ 
 +$ snmpget -v2c -c public X.X.X.X 1.3.6.1.4.1.8072.1.3.2.3.1.1.3.99.112.117 
 +NET-SNMP-EXTEND-MIB::​nsExtendOutput1Line."​cpu"​ = STRING: 34.166 
 +</​code>​
  
-Example scripts exist in the demo directory of the project which +How you use the MIB in other tools is of course, tool specific.
-demonstrate how the RRD is queried to produce a graph of the values +
-stored within it.+
  
-The demo can be seen [[http://​london.omweb.org/​|here]].