September 5th

Installing StatsD on an Ubuntu server (10.04) was surprisingly easy. Here are the steps that I took. These will work from a fresh install of Ubuntu server 10.04 or after having installed and started graphite.

# INSTALL NODE.JS
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python-software-properties git-core
sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs
 
# CLONE THE STATSD PROJECT
git clone https://github.com/etsy/statsd.git
 
# CREATE A CONFIG FILE
cd statsd
cp exampleConfig.js dConfig.js
# edit config file your settings
sudo vim dConfig.js
node stats.js dConfig.js
 
# START STATSD
node stats.js dConfig.js

When you start StatsD, it will probably give you a warning like

(node) process.compile should not be used. Use require(‘vm’).runInThisContext instead.

but it is still working as expected so you can ignore it.

You can then test to make sure StatsD is working. Edit and run this simple PHP script to throw some stats at your StatsD.

<?php
 
StatsD::increment("testing.increment");
StatsD::timing("testing.timing", 2345);
 
class StatsD {
    public static function timing($stat, $time, $sampleRate=1) {
        StatsD::send(array($stat => "$time|ms"), $sampleRate);
    }
    public static function increment($stats, $sampleRate=1) {
        StatsD::updateStats($stats, 1, $sampleRate);
    }
    public static function decrement($stats, $sampleRate=1) {
        StatsD::updateStats($stats, -1, $sampleRate);
    }
    public static function updateStats($stats, $delta=1, $sampleRate=1) {
        if (!is_array($stats)) { $stats = array($stats); }
        $data = array();
        foreach($stats as $stat) {
            $data[$stat] = "$delta|c";
        }
        StatsD::send($data, $sampleRate);
    }
    public static function send($data, $sampleRate=1) {
        $sampledData = array();
        if ($sampleRate < 1) {
            foreach ($data as $stat => $value) {
                if ((mt_rand() / mt_getrandmax()) <= $sampleRate) {
                    $sampledData[$stat] = "$value|@$sampleRate";
                }
            }
        } else {
            $sampledData = $data;
        }
        if (empty($sampledData)) { return; }
        try {
            $host = [your graphite host here];
            $port = [your graphite port here 8125?];
            $fp = fsockopen("udp://$host", $port, $errno, $errstr);
            if (! $fp) { return; }
            foreach ($sampledData as $stat => $value) {
                fwrite($fp, "$stat:$value");
            }
            fclose($fp);
        } catch (Exception $e) {
        }
    }
}
 
?>

Be sure to edit the $host and $port settings towards the bottom with the appropriate graphite settings.

When you run this script, you can run a tcpdump on the receiving machine to see if the UDP packets are getting to the machine.

sudo tcpdump -tn port 8125

If your StatsD is installed on a different machine/IP than Graphite, you can also watch StatsD send of the data by running

sudo tcpdump -tn port 8125 or port 2003

This is especially helpful if you have not properly set up your Amazon EC2 security group settings. Be sure to open a UDP port for the traffic coming through.

How-to Install Graphite on Ubuntu

MikeGrace

26 Comments

September 3rd

How to, Linux, Web Design -Dev

UPDATED TO WORK WITH GRAPHITE 0.9.9!!! (10/13/2011)

The new release of Graphite 0.9.9 includes an experimental Flot JavaScript graph. Awesome!

Steps to installing Graphite on Ubuntu -> gist.github.com
Ubuntu Graphite install screencast -> youtube.com

If you are looking for a how-to on graphite one (CAD) or statsd this is not the place.

A few months ago I blogged abou how much I love stats. One of the things that I shared in that post was a blog post done by the etsy developer team about statsd and graphite to track everything.

This week I was able to get graphite setup on a dedicated Ubuntu 10.04 on my Mac using VMware Fusion. I thought I would share the steps I took so others might be able to save some time in their setup. In setting up apache I just copied over the default virtual host settings since graphite will be the only thing running on the server. If you are planning on having multiple virtual hosts on the machine then you will want to configure the virtual hosts differently.

####################################
# BASIC REQUIREMENTS
# http://graphite.wikidot.com/installation
# http://geek.michaelgrace.org/2011/09/how-to-install-graphite-on-ubuntu/
# Last tested & updated 10/13/2011
####################################
 
sudo apt-get update
sudo apt-get upgrade
 
wget http://launchpad.net/graphite/0.9/0.9.9/+download/graphite-web-0.9.9.tar.gz
wget http://launchpad.net/graphite/0.9/0.9.9/+download/carbon-0.9.9.tar.gz
wget http://launchpad.net/graphite/0.9/0.9.9/+download/whisper-0.9.9.tar.gz
tar -zxvf graphite-web-0.9.9.tar.gz
tar -zxvf carbon-0.9.9.tar.gz
tar -zxvf whisper-0.9.9.tar.gz
mv graphite-web-0.9.9 graphite
mv carbon-0.9.9 carbon
mv whisper-0.9.9 whisper
rm carbon-0.9.9.tar.gz
rm graphite-web-0.9.9.tar.gz
rm whisper-0.9.9.tar.gz
sudo apt-get install --assume-yes apache2 apache2-mpm-worker apache2-utils apache2.2-bin apache2.2-common libapr1 libaprutil1 libaprutil1-dbd-sqlite3 python3.1 libpython3.1 python3.1-minimal libapache2-mod-wsgi libaprutil1-ldap memcached python-cairo-dev python-django python-ldap python-memcache python-pysqlite2 sqlite3 erlang-os-mon erlang-snmp rabbitmq-server bzr expect ssh libapache2-mod-python python-setuptools
sudo easy_install django-tagging
 
####################################
# INSTALL WHISPER
####################################
 
cd ~/whisper
sudo python setup.py install
 
####################################
# INSTALL CARBON
####################################
 
cd ~/carbon
sudo python setup.py install
# CONFIGURE CARBON
####################
cd /opt/graphite/conf
sudo cp carbon.conf.example carbon.conf
sudo cp storage-schemas.conf.example storage-schemas.conf
sudo vim storage-schemas.conf
### edited storage-schemas.conf to be the following
[stats]
priority = 110
pattern = .*
retentions = 10:2160,60:10080,600:262974
###
 
####################################
# CONFIGURE GRAPHITE (webapp)
####################################
 
cd ~/graphite
sudo python check-dependencies.py
sudo python setup.py install
 
# CONFIGURE APACHE
###################
cd ~/graphite/examples
sudo cp example-graphite-vhost.conf /etc/apache2/sites-available/default
sudo cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
sudo vim /etc/apache2/sites-available/default
# moved 'WSGIImportScript /opt/gr..' to right before virtual host since it gave me an error saying
# WSGIImportScript cannot occur within <VirtualHost> section
# if this path does not exist make it!!!!!!
# /etc/httpd/wsgi
sudo mkdir /etc/httpd
sudo mkdir /etc/httpd/wsgi
sudo /etc/init.d/apache2 reload
 
####################################
# INITIAL DATABASE CREATION
####################################
cd /opt/graphite/webapp/graphite/
sudo python manage.py syncdb
# follow prompts to setup django admin user
sudo chown -R www-data:www-data /opt/graphite/storage/
sudo /etc/init.d/apache2 restart
cd /opt/graphite/webapp/graphite
sudo cp local_settings.py.example local_settings.py
 
####################################
# START CARBON
####################################
cd /opt/graphite/
sudo ./bin/carbon-cache.py start
 
####################################
# SEND DATA TO GRAPHITE
####################################
cd ~/graphite/examples
sudo chmod +x example-client.py
# [optional] edit example-client.py to report data faster
# sudo vim example-client.py
sudo ./example-client.py