FreedomBox for Communities/Monitoring Server

The monitoring client on the FreedomBox server of each of the communities will need to send the collected metrics to a central location. This central location, the monitoring server, needs to store the data sent, visualize the data for administrator and alert them when something is off. This section describes the setup on the server. It is assumed that the server is also running a FreedomBox. However, with minor changes to the instructions any server running Debian GNU/Linux Buster or above can be used as a server. Setup Graphite Carbon Graphite Carbon server is responsible collecting the data from the various FreedomBox servers and storing them in a time-series database called Whisper. Run the following commands to setup Graphite Carbon. Install Graphite Carbon. apt install graphite-carbon Open up ports in the firewall for receiving data from FreedomBox Servers. firewall-cmd --zone=external --add-port=2003/tcp firewall-cmd --zone=external --add-port=2003/tcp --permanent After this, graphite-carbon server will be running and listening on port 2003. This can be checked using the following commands: systemctl status carbon-cache ss -nlpt | grep carbon-cache Setup Graphite Web Graphite Web service is responsible for serving the data stored by Graphite Carbon to consumers that can visualize the data such as Grafana. Run the following commands to setup Graphite Web. Install Graphite Web apt install graphite-web libapache2-mode-wsgi-py3 Generate a secret key for Django framework used by Graphite Web. python3 -c 'import random; result = "".join([random.choice("abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)") for i in range(64)]); print(result)' Note the key generated from previous command and set it as secret key in the file /etc/graphite/local_settings.py. SECRET_KEY="<output from the key generation command>" Create a database for Graphite Web to store its own data. sudo --user _graphite graphite-manage migrate --run-syncdb chown _graphite:_graphite /var/lib/graphite/graphite.db chmod 0600 /var/lib/graphite/graphite.db Create Apache configuration for graphite-web and start serving. cat << EOF > /etc/apache2/sites-available/graphite-web.conf Listen 8080 <VirtualHost *:8080> WSGIDaemonProcess _graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120 user=_graphite group=_graphite WSGIProcessGroup _graphite WSGIImportScript /usr/share/graphite-web/graphite.wsgi process-group=_graphite application-group=%{GLOBAL} WSGIScriptAlias / /usr/share/graphite-web/graphite.wsgi Alias /static/ /usr/share/graphite-web/static/ <Location "/static/"> SetHandler None </Location> </VirtualHost> EOF a2ensite graphite-web systemctl restart apache2 Setup Grafana Grafana consumes data from Graphite Web and produces beautiful visualizations. It can also be configured for alerting when certain event occurs in the data. Run the following commands to setup Grafana. Install Grafana from it's project's Debian repositories. apt-get install -y apt-transport-https wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add - cat << EOF > /etc/apt/sources.list.d/grafana.list deb https://packages.grafana.com/oss/deb stable main EOF apt update apt install grafana Configure Grafana to be served from the a existing domain under a sub-path. Edit the file /etc/grafana/grafana.ini and ensure the following two directives are configured as shown. root_url = https://<domain_name>/monitor/ serve_from_sub_path = true Configure Grafana to start every time the machine is booted and start Grafana. systemctl daemon-reload systemctl enable grafana-server.service systemctl start grafana-server.service Create Apache configuration to proxy Grafana from a sub-path and start serving. cat << EOF > /etc/apache2/conf-available/grafana.conf <Location /monitor/> ProxyPass http://localhost:3000/monitor/ </Location> EOF a2enconf grafana systemctl reload apache2 Setup credentials for Grafana access. Visit https://<domain_name>/monitor/ and login with the following credentials. You will be prompted to change the password. Set a strong password. Username: admin Password: admin Create user accounts for each for users who will using the monitoring setup. In the left menu, go to Server Admin, Users. Click on the New User button and provide details such as name, email, username and password to create a new user account. Connect Grafana to the Graphite Carbon database via Graphite Web we have configured in the previous sections. Go to Configuration, Data Sources and select Carbon data source. In the configuration dialog that shows up, provide the following details: Name: Carbon (or anything you choose) URL: http://localhost:8080 Access: Server Version: 1.1 The press Save & Test. You should see a message confirming the data source is accessible. Creating a Dashboard for Each Community In Grafana interface, you need to create a dashboard for monitoring each of the communities you are the administering. The following instructions provide an overview of how this can be done: Login to Grafana interface. In the menu, select Create, Dashboard. A dashboard is created and a prompt is shown to add a new panel. Select a Graph. A new dashboard is created with a simple graph. Save dashboard by clicking the save icon. In the save dialog, provide a name for the dashboard based on the community for which the dashboard is being created. Click Save. To start showing proper data in the panel, click on Panel Title and select Edit. In the Metric tab, click on select metric drop down. You should see the name of the domain you assigned to your community as configured in collectd. Then select a metric such as cpu-0 and then cpu-user. You should start seeing the plot of the metric in the graph. Save the dashboard. Parameters to Monitor It is recommended that you create a dashboard to your taste but monitor the following parameters at least for each community: CPUs Series: <community> • * • cpu-user Functions: averageSeries() • alias(User) Series: <community> • * • cpu-nice Functions: averageSeries() • alias(Nice) Series: <community> • * • cpu-system Functions: averageSeries() • alias(System) Series: <community> • * • cpu-wait Functions: averageSeries() • alias(Wait) Memory Series: <community> • memory • percent-used Functions: alias(% Used) Load Series: <community> • load • load • shorterm Functions: alias(5min) Series: <community> • load • load • midterm Functions: alias(10min) Series: <community> • load • load • longterm Functions: alias(15min) Disk Series: <community> • <disk> • disk_ops • read Functions: alias(Read ops) Series: <community> • <disk> • disk_ops • write Functions: alias(Write ops) Series: <community> • <disk> • disk_octets • read Functions: alias(Read Bytes) Series: <community> • <disk> • disk_octets • write Functions: alias(Write Bytes) Disk I/O Time Series: <community> • <disk> • pending_operations Functions: alias(Pending Operations) Series: <community> • <disk> • disk_io_time • io_time Functions: alias(I/O Time) Series: <community> • <disk> • disk_io_time • weighted_io_time Functions: alias(Weighted I/O Time) <mountpoint> Partition (for each partition) Series: <community> • df-<mountpoint> • percent_bytes-used Functions: alias(Used) Series: <community> • df-<mountpoint> • percent_bytes-reserved Functions: alias(Reserved) <LAN/WAN> Network Bandwidth (for each interface) Series: <community> • interface-<interface> • if_octets • rx Functions: alias(Received) Series: <community> • interface-<interface> • if_octets • tx Functions: alias(Transmitted) Connections Tracked Series: <community> • conntrack • conntrack Functions: alias(Connections Tracked) Wi-Fi Access Points alias(sumSeries(isNonNull(<community>.ping.ping-10_42_0_[1-9][0-9])), 'Wi-Fi Access Points') Internet Uplink alias(isNonNull(<community>.ping.ping-10_42_0_4), 'Internet Router') alias(isNonNull(<community>.ping.ping-10_42_0_5), 'PTP Link - Router A') alias(isNonNull(<community>.ping.ping-10_42_0_6), 'PTP Link - Router B') Internet Ping Series: <community> • ping • ping-1_1_1_1 Functions: alias(Internet Ping) Processes Series: <community> • processes • ps_state-running Functions: alias(Running) Series: <community> • processes • ps_state-sleeping Functions: alias(Sleeping) Series: <community> • processes • ps_state-paging Functions: alias(Paging) Series: <community> • processes • ps_state-blocked Functions: alias(Blocked) Series: <community> • processes • ps_state-zombies Functions: alias(Zombies) Series: <community> • processes • ps_state-stopped Functions: alias(Stopped) Uptime Series: <community> • uptime • uptime Functions: alias(Time Since Reboot) Users Series: <community> • users • users Functions: alias(Logged-In Users) collectd Queue Series: <community> • collectd-cache • cache_size Functions: alias(Cache Size) Series: <community> • collectd-write_queue • derive-dropped Functions: alias(Dropped Metrics) Series: <community> • collectd-write_queue • queue_length Functions: alias(Queue Length)