Xen VMs performance collection
Unlike VMware Server, Xen’s HyperVisor does not allow an easy collection of performance information. The management machine, called “Domain-0” is actually a privileged virtual machine, and thus – get its own small share of CPUs and RAM. Collecting performance information on it will lead to, well, collecting performance information for a single VM, and not the whole bunch.
Local tools, such as “xentop” allows collection of information, however, combining this with Cacti, or any other SNMP-based collection tool is a bit tricky.
A great solution is provided by Ian P. Christian in his blog post about Xen monitoring. He has created a Perl script to collect information. I have taken the liberty to fix several minor things with his permission. The modified scripts are presented below. Name the script (according to your version of Xen) “/usr/local/bin/xen_stats.pl” and set it to be executable:
For Xen 3.1
#!/usr/bin/perl -w
use strict;
# declare...
sub trim($);
#xen_cloud.tar.gz
# we need to run 2 iterations because CPU stats show 0% on the first, and I'm putting .1 second betwen them to speed it up
my @result = split(/n/, `xentop -b -i 2 -d.1`);
# remove the first line
shift(@result);
shift(@result) while @result && $result[0] !~ /^xentop - /;
# the next 3 lines are headings..
shift(@result);
shift(@result);
shift(@result);
shift(@result);
foreach my $line (@result)
{
my @xenInfo = split(/[t ]+/, trim($line));
printf("name: %s, cpu_sec: %d, cpu_percent: %.2f, vbd_rd: %d, vbd_wr: %dn",
$xenInfo[0],
$xenInfo[2],
$xenInfo[3],
$xenInfo[14],
$xenInfo[15]
);
}
# trims leading and trailing whitespace
sub trim($)
{
my $string = shift;
$string =~ s/^s+//;
$string =~ s/s+$//;
return $string;
}
For Xen 3.2 and Xen 3.3
#!/usr/bin/perl -w
use strict;
# declare…
sub trim($);
# we need to run 2 iterations because CPU stats show 0% on the first, and I’m putting .1 second between them to speed it up
my @result = split(/n/, `/usr/sbin/xentop -b -i 2 -d.1`);
# remove the first line
shift(@result);
shift(@result) while @result && $result[0] !~ /^[t ]+NAME/;
shift(@result);
foreach my $line (@result)
{
my @xenInfo = split(/[t ]+/, trim($line));
printf(“name: %s, cpu_sec: %d, cpu_percent: %.2f, vbd_rd: %d, vbd_wr: %dn“,
$xenInfo[0],
$xenInfo[2],
$xenInfo[3],
$xenInfo[14],
$xenInfo[15]
);
}
# trims leading and trailing whitespace
sub trim($)
{
my $string = shift;
$string =~ s/^s+//;
$string =~ s/s+$//;
return $string;
}
Cron settings for Domain-0
Create a file “/etc/cron.d/xenstat” with the following contents:
# This will run xen_stats.pl every minute
*/1 * * * * root /usr/local/bin/xen_stats.pl > /tmp/xen-stats.new && cat /tmp/xen-stats.new > /var/run/xen-stats
SNMP settings for Domain-0
Add the line below to “/etc/snmp/snmpd.conf” and then restart the snmpd service
extend xen-stats /bin/cat /var/run/xen-stats
Cacti
I reduced Ian Cacti script to be based on a per-server setup, meaning this script gets the host (dom-0) name from Cacti, but cannot support live migrations. I will try to deal with combining live migrations with Cacti in the future.
Download and extract my modified xen_cloud.tar.gz file. Extract it, place the script and config in its relevant location, and import the template into Cacti. It should work like charm.
A note – the PHP script will work only on PHP5 and above. Works flawlessly on Centos5.2 for me.
With Xen 3.1 ‘/usr/sbin/xentop -b -i 2 -d.1’ – always returns Domain-0 CPU usage about 100%.
And it should be full path to xentop since not all crontab jobs have right environment set.
P.S. Thanks for the tool!
Thanks for your comment.
With Cacti, I tend to sample dom0 from other aspects, and not through this specific command. However, thanks for the info!
Ez
Hi,
Nice job, but sadly the link to xen_cloud.tar.gz seems broken : “Error establishing a database connection”
Cheers,
Thanks for letting me know.
There was a minor problem with the server which caused its backend DB to fail. It was fixed, and should not operate correctly. I have tested it to work.
Let me know if it doesn’t, please.
Thanks
Ez
I’m getting this error:
Error: XML: Hash version does not exist.
I think this is because you exported template with different version of cacti. Could you let me know which version you used?
I use 0.8.7c
Weird…
Ez
Hi,
im having trouble getting things to work together.
I could import the scripts into cacti and copied the 2 scripts into the cactio folders,
than i used your script on the dom0, and can snmpwalk onto that machine:
# snmpwalk -v2c -c public 172.17.4.101 NET-SNMP-EXTEND-MIB::nsExtendOutputFull.”xen-stats”
NET-SNMP-EXTEND-MIB::nsExtendOutputFull.”xen-stats” = STRING: name: Domain-0, cpu_sec: 1563511, cpu_percent: 97.10, vbd_rd: 0, vbd_wr: 0
name: monitoring-2, cpu_sec: 3833949, cpu_percent: 194.00, vbd_rd: 14746, vbd_wr: 20419
name: uat-vm2, cpu_sec: 154919, cpu_percent: 0.00, vbd_rd: 25191, vbd_wr: 88203
name: uat-vm4, cpu_sec: 94536, cpu_percent: 0.00, vbd_rd: 16, vbd_wr: 4
name: vpn-vm2, cpu_sec: 110325, cpu_percent: 0.00, vbd_rd: 21, vbd_wr: 1
But within cacti no graphs will be selectable when I try to create some,
this is the output of the verbose query:
+ Running data query [19].
+ Found type = ‘4 ‘[script query].
+ Found data query XML file at ‘/var/www/cacti/resource/script_queries/xen_cloud_stats.xml’
+ XML file parsed ok.
+ Executing script for list of indexes ‘/usr/bin/php /var/www/cacti/scripts/xen_cloud_stats.php index’
+ Executing script query ‘/usr/bin/php /var/www/cacti/scripts/xen_cloud_stats.php query xen_name’
+ Found item [xen_name=”] index:
+ Found data query XML file at ‘/var/www/cacti/resource/script_queries/xen_cloud_stats.xml’
+ Found data query XML file at ‘/var/www/cacti/resource/script_queries/xen_cloud_stats.xml’
+ Found data query XML file at ‘/var/www/cacti/resource/script_queries/xen_cloud_stats.xml’
Do i miss anything?
greetings
If you get errors like:
NET-SNMP-EXTEND-MIB::nsExtendOutputFull.”xen-stats” = STRING: /bin/cat: /var/run/xen-stats: Permission denied
it might be that SELinux is blocking access. I created this policy using audit2allow:
module snmpd 1.0;
require {
type var_run_t;
type snmpd_t;
class file { read getattr };
}
#============= snmpd_t ==============
allow snmpd_t var_run_t:file { read getattr };
compile & load it, and snmpd is able to read the xen-stats file.
Hi, i try this script on xen express 5.5 but i’ve this error
Operator or semicolon missing before & at ./xen-stat.pl line 14.
Ambiguous use of & resolved as operator & at ./xen-stat.pl line 14.
Useless use of not in void context at ./xen-stat.pl line 14.
Bareword “amp” not allowed while “strict subs” in use at ./xen-stat.pl line 14.
Execution of ./xen-stat.pl aborted due to compilation errors.
how can i fix it?
This script is meant for Xen Community, and was tested on Centos5 versions. I did not try to adjust it to run on XS just yet.
Sorry.
> With Xen 3.1 ‘/usr/sbin/xentop -b -i 2 -d.1′ – always returns Domain-0 CPU usage about 100%.
> And it should be full path to xentop since not all crontab jobs have right environment set.
Change “-d.1” to “-d 1”. Apparently, some xentop implementations only accept seconds, not tenths of seconds.
Hi,
strangely I also see this error about & when trying it on OracleVM – which is VERY close to CentOS5 (just using a current Xen version – god bless them)
[root@waxh0004 tmp]# perl xen_stats.pl
Operator or semicolon missing before & at xen_stats.pl line 14.
Ambiguous use of & resolved as operator & at xen_stats.pl line 14.
Useless use of not in void context at xen_stats.pl line 14.
Bareword “amp” not allowed while “strict subs” in use at xen_stats.pl line 14.
Execution of xen_stats.pl aborted due to compilation errors.
I’ll ask my perlspeaking friend to look into it, maybe she can fix it.
(RHEL/CentOS are just not desirable for me any more. I think they spend too much energy on hiding all features / benefits of Xen with their old kernels and versions and libvirt, so they can promote the KVM better. It says enough about XENs superioty if they have to compare 2006’s Xen with 2010’s KVM :))
Hi.
It seems the script had an error due to WordPress formatting aids, and two ampersand signs together got translated to HTML marks of &
I have fixed this issue on both scripts (you will probably use the 3.3 one) so if you download it now, you will get the correct (and working) version.
Thanks for letting me know!
Hi,
very kind of you.
Still having some other issue, I’ll post back once that is found.
Florian Heigl
hi darkfader,
check the printf line
quotation marks after ( and after n
printf(“name: %s, cpu_sec: %d, cpu_percent: %.2f, vbd_rd: %d, vbd_wr: %dn”,
georg doern
The Script does not run with SLES10SP3 with xen 3.2.3
The output from xentop differs. I modified your script to work for me
Here it is:
#!/usr/bin/perl -w
use strict;
# declare…
sub trim($);
#xen_cloud.tar.gz
# we need to run 2 iterations because CPU stats show 0% on the first, and I’m putting .1 second betwen them to speed it up
my @result = split(/n/, `/usr/sbin/xentop -b -i 2 -d.1`);
# remove the first line
shift(@result);
# remove the rest line till the Next “Name”-Header
shift(@result) while @result && $result[0] !~ /^[t ]*NAME/;
foreach my $line (@result)
{
my @xenInfo = split(/[t ]+/, trim($line));
if ( $xenInfo[0] ne “NAME” ) {
printf(“name: %s, cpu_sec: %d, cpu_percent: %.2f, vbd_rd: %d, vbd_wr: %dn”,
$xenInfo[0],
$xenInfo[2],
$xenInfo[3],
$xenInfo[14],
$xenInfo[15]
);
}
}
# trims leading and trailing whitespace
sub trim($)
{
my $string = shift;
$string =~ s/^s+//;
$string =~ s/s+$//;
return $string;
}
Php script yields this error. Any thoughts as to why?
$ /usr/bin/php /var/www/html/cacti/scripts/xen_cloud_stats.php public query xen_name
PHP Notice: Undefined offset: 1 in /var/www/html/cacti/scripts/xen_cloud_stats.php on line 90
PHP Notice: Undefined index: name in /var/www/html/cacti/scripts/xen_cloud_stats.php on line 93
PHP Notice: Undefined index: name in /var/www/html/cacti/scripts/xen_cloud_stats.php on line 93
PHP Notice: Undefined index: name in /var/www/html/cacti/scripts/xen_cloud_stats.php on line 97
PHP Notice: Undefined index: name in /var/www/html/cacti/scripts/xen_cloud_stats.php on line 60
PHP Notice: Undefined index: name in /var/www/html/cacti/scripts/xen_cloud_stats.php on line 60
#php -v
PHP 5.2.10 (cli) (built: Nov 13 2009 11:44:05)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2009 Zend Technologies
John try this :
sudo snmpwalk -v2c -c public 10.10.1.19 NET-SNMP-EXTEND-MIB::nsExtendOutputFull.”xen-stats”
For have a answer at this request, you need to have :
* SNMPD.conf of the XenServer OK (extend xen-stats /bin/cat /var/run/xen-stats)
Restart the snmp daemon after the modification
* A good xen_stats.pl, you can download the script retouched by Bjoern (be careful when you copy from the web – ” = . after a copy/paste into your ssh connection)
For test your xen_stats.pl, you must have information in the /var/run/xen-stats (result of the cron : */1 * * * * root /srv/scripts/xen_stats.pl > /tmp/xen-stats.new && cat /tmp/xen-stats.new > /var/run/xen-stats)
* Import file in the cloud.tar.gz and verbose query the data queries xen cloud and you should see information of your XenServer
It’s works on a XenServer 5.6.1 (CentOS) with a Cacti 0.8.7e on a Ubuntu Server
if anybody have this problem ” /usr/bin/perl xen_stats.pl
Unrecognized character xE2 in column 16 at xen_stats.pl line 19.”
is because of the quotes, simply change ” by ” in line 19,
are 2 different quote types, almost undetectable
If anyone has a problem, “can not load the module. SNMP-FRAMEWORK-MIB
The solution is to install the Management Information Base files
aptitude install snmp-mibs-downloader
Finally, after much fighting the plugin works perfectly =)
Went threw all of the steps however the disk access graphs do not populate. I also have get this output returned from snmpwalk
snmpwalk -v2c -c public hfcl05-03.vm NET-SNMP-EXTEND-MIB::nsExtendOutputFull..xen-stats.
NET-SNMP-EXTEND-MIB::nsExtendOutputFull..xen-stats.: Unknown Object Identifier (Sub-id not found: nsExtendToken -> xen-stats.)
Went threw all of the steps however the disk access graphs do not populate. I also have get this output returned from snmpwalk
snmpwalk -v2c -c public host.vm NET-SNMP-EXTEND-MIB::nsExtendOutputFull..xen-stats.
NET-SNMP-EXTEND-MIB::nsExtendOutputFull..xen-stats.: Unknown Object Identifier (Sub-id not found: nsExtendToken -> xen-stats.)