-->

Thursday 8 April 2010

Plugin : Update RRD Tool with Kettle (Cacti, MRTG …).

Hi all,
This post will give you more details about my new Kettle plugin to feed RRDTools database.

What is RRDTool ?

According to Tobias Oetiker, RRDtool is the OpenSource industry standard, high performance data logging and graphing system for time series data. Use it to write your custom monitoring shell scripts or create whole applications using its Perl, Python, Ruby, TCL or PHP bindings.
You can learn more about RRDTools and Tobias Oetiker fantastic work on his homepage HERE.
With RRDTools, you can easily store time series data and create realtime graphics like the one below. For instance, I use it today for one of my client on Paris in order to monitor various real time business / IT indicators : travel booking, passenger reservation, search engine sollicitation, xml proxy load and mainframe usage.
image

The libs I used

I used JRobin, a java port of Tobias Oetiker RRDTools. JRobin was made by the talented Sasa Markovic. The JRobin home page are HERE and HERE. I recommand a visit in order to be fully aware of all JRobin features.
According to Sasa Markovic, “JRobin is a 100% pure java implementation of RRDTool's functionality. It follows the same logic and uses the same data sources, archive types and definitions as RRDTool does. JRobin supports all standard operations on Round Robin Database (RRD) files: CREATE, UPDATE, FETCH, LAST, DUMP, XPORT  and GRAPH. JRobin's API is made for those who are familiar with RRDTool's concepts and logic, but prefer to work with pure java. If you provide the same data to RRDTool and JRobin, you will get exactly the same results and graphs.”
I confirm everything.
The graphical rendering is very good looking as you can see below.


 

The plugin

First, I recommand to read carefully everything related to RRDTools and JRobin. You must be familiar with this technology first.
The Kettle plugin is quite simple : a single user interface to create a RRD file, add archives and feed the file.
image
image A short description of this window :
  • Nom étape (sorry in french, will be translated) : Step name
  • RRD File : the RRD file to be created. This file will hold all your time series data and archives.
  • Datasource : an RRD file can have 1 or more datasource. For the moment, my plugin is restricted to 1 datasource, which is most of the time enough.
  • Type :
    • Gauge : Does no store the rate of change, it saves the actual value itself.
    • Counter : To store the rate of change of the value over a step period (assume the value is always increasing). Ex : traffic counters.
    • Derive : The same as Counter, but will handle negative values. Ex : free disk space.
    • Absolute : To store the rate of change, but the previous value is set to 0.
  • Heartbeat : If the RRD file does not receive value (PDP) within 300 seconds, it will wait for another 300 seconds (total = 600 seconds). If no value after 600 seconds, the flag UNKNOWN will be stored.
  • Starttime : The unix timestamp as the RRD file starting point. Must be a unix timestamp. In a future release, I will code a converter and place it into the user interface. You can easily compute unix timestamps by using this web page or this one. This timestamp must be lower than the one coming from your data.
  • Min and Max : The minimum value and the maximum value, if predictable.
  • The combo zone : This combo gadget will be used to define RRA : Round Robin Archives. An RRA will define how the consolidated data is stored. We have 4 major parameters :
    • CF, for Consolidation Function :
      • AVERAGE : Store the average value
      • MIN : Store the minimum value
      • MAX : Store the max value
      • LAST : Store the last known value
    • xff : XFile factor. This is the percentage of values that can be unknown without making the recorded value flagged as UNKNOWN. Must be between 0 and 1, with 0.1 intervals.
    • Steps :  Number of values to be consolidated, regarding the chosen CF. Must be integer.
    • Rows : Number of samples to keep. Must be integer.
The Add button will add the RRA (round robin archive) in the combo list, and then will be used for the RRD file creation.
Once the RRD file is successfully created, you will see a little message on botton of the user interface.
image
Let’s have a look to some RRD file internals, using another nice tool from Sasa Markovic : RRD inspector. I’m sure you will easily understand the RRD structure, if you are not already familiar with RRDTools.
The above screen shows us a RRD file created with one datasource called Speed, using a type GAUGE with a heartbeat of 600 seconds, with minimum and maximum values set to 0 to 2000. image
This RRD file also has a unique RRA (round robin archive), using the AVERAGE consolidation function, with xff set to 0.5, 1 step (compute each value = no average in fact) and 24 rows.This file has been created using the Kettle plugin on my C:\ harddrive.image
If we select then panel “Archive data” we will be able to see all the values currently stored into the RRD file.image
RRD inspector is a fantastic little tool, very usefull when creating RRD files and checking everything is well done.

How to use the plugin ?

Very simple. First you create a RRD File using the user interface shown above. Then you have to control the file has been created, just to be sure. Finally, you can connect the step to a previous one in Kettle. In my example, I used a flat file containing some simple timestamps and values.
Here is my flat file : a unix timestamp with 5 mins intervals (starting Thu, 8 Apr 2010 12:00:00 UTC) and some simple values from 5 to 140.
TimeStamp;Value
1270728000;5
1270728300;10
1270728600;15
1270728900;20
1270729200;25
1270729500;40
1270729800;50
1270730100;60
1270730400;70
1270730700;80
1270731000;90
1270731300;100
1270731600;120
1270731900;140
And here is my sample transformation.
image
Hit play, and voilà … the plugin will feed the RRD file and give you a nice output log for each value. To be short : a RRD file is only expecting a unix timestamp and a value.
image

Generating a graph

Well, this is not really Kettle oriented, but I will give you some code to create graphics from your RRD file, previously loaded with Kettle.
This simple java snippet …
public static void RenderRRDGraph(long TimeStart, long TimeStop, String Consol, String RRDGraphFormat) throws IOException, RrdException{
    //Create gif graph
    RrdGraphDef graphDef = new RrdGraphDef();
    graphDef.setVerticalLabel("m/s");
    graphDef.setTimeSpan(TimeStart, TimeStop);
    graphDef.datasource("myspeed", "C:\\testRRD", "speed", Consol);
    graphDef.line("myspeed", new Color(0xFF, 0, 0), null, 2);
    graphDef.setFilename("C:\\testRRD." + RRDGraphFormat);
    RrdGraph graph = new RrdGraph(graphDef);
}
testRRD
… will create this png.
Very simple as you can see (well this example is really really simple compared to what we can really do, but I can’t give you any snapshot of the graphs I did for my client – I have NDA on this). You can imagine now creating some real time graphics (RTG) using this technology.

The package

Let’s go back to Pentaho and Kettle : I created a package for you. You will find the plugin itself (compiled and archived under Eclipse using Fat Jar in order to embedd the JRobin library, the icon, the xml file, the flat file and a sample transformation (the one described above).
This package can be donwloaded on its Google code page.
Please keep me informed about your testing, and feel free to contact me if further features (or fixes !) are needed.

23 comments:

Unknown said...

This plugin looks really great! I was however looking for the other way to integrate Kettle & RRD, that is extracting data FROM RRD files.

Do you know if there is such plugin out there (using Kettle PDI CE)?

Fabrice Bacchella said...

I see you use jrobin, which is not very active any more. Did you have a look at http://code.google.com/p/rrd4j/, which is a fork of rrd4j, is more active and got very nice performances improvements ?

Anonymous said...

Never, never, never, never give up

-----------------------------------

Anonymous said...

Bill Terry's Team Well-ArmedAlthough the Giants under Bill Terry had neither the best hitting nor the best fielding team in the National League in 1933, they did have, by far, the best pitching staff. [url=http://officialcanadagoosesoutlet.ca]canada goose whistler parka[/url] "But it's not a seismic turn of events. But as a modern reader, I found her work to be less than avant-garde. A sum of £96 was paid to Mr J Norton Garrat (£17 10s per acre) in respect of tenant-right compensation for disturbance etc.
There are other chromatic instruments out there much more suited to those kinds of music. [url=http://www.icanadagooseca.com]canada goose jackets[/url] If you simply can face the prospect of a long and dull British winter, a long winter let can be a great way for you to follow the sun without the expense of buying your own holiday property, especially if you are retired and have the flexibility to live abroad for several months. http://officialcanadagoosesoutlet.ca
[url=http://www.canadagoosejacketca.ca]canada goose jacket[/url] online store Beats By Dre Studio , Beats By Dre Pro , Beats By Dre Solo HD, Beats By Dre Red, Justbeats Headphones , New style Beats By Dre , Beats By Dre White , Beats By Dre Black , Beats By Dre In-Ear , Beats By Dre Solo HD is a world-famous top brand. [url=http://www.onlinebeatsbydreoutlet.com]beats by dre logo stickers[/url]

Anonymous said...

The results of the analysis are not surprising to anyone who has thought about it. [url=http://www.vanessasac.com]vanessa bruno sacs[/url] Bit of a red letter day for migrants, with 4 Whinchats(!) as well as Whitethroats. CPA Reciprocity Exam All IQEX test dates occur within a three week period straddling October and November each year. At sea level that's 212 F, though it might be less if you live at high altitude.
The section included here backs up behind a dam flanked by gorgeous, old redbrick mills in Somersville. [url=http://www.icanadagooseca.com]canada goose jackets[/url] You want to be as aggressive about treating and maintaining your arthritis as possible. [url=http://officialcanadagoosesoutlet.ca]canada goose for sale[/url]
[url=http://www.canadagoosejacketca.ca]canada goose online[/url] The nation largest meat processing enterprises Shineway was broke, "Buy Newport Cigarette" event, we understand that "lean" is an animal drug, the "lean" added to the feed, can increase the lean meat of animals volume, reduced feed use, making the meat ahead of the market, reduce costs, but the body will have some side Newport Cigarette. [url=http://www.onlinebeatsbydreoutlet.com]lebron james beats dre pro[/url]

Anonymous said...

A native species to North America, it was once widespread from coast to coast but hunting for food and "skins" sold to Europe to be used for making powder puffs for women and feathers for hats and bedding took a big toll on populations (as did the usual loss of habitat due to human encroachment). [url=http://www.vanessasac.com]boutiques vanessa bruno [/url] 5. A couple of years ago, the family explained to a Time Magazine correspondent how to distinguish a good handwoven carpet. The Nazis were an incredibly evil organization, and no one except Stalin and Mao.
Because you're about to move faster -- and weirder -- than you thought possible. http://www.icanadagooseca.com The Rolex Chronograph 3529 is a great watch no matter how you look at it.. [url=http://officialcanadagoosesoutlet.ca]canada goose parka sale[/url]
http://www.canadagoosejacketca.ca From the ghostly stalker in "Hamburger Lady" to the deadly music of "Got My Soul," each story weaves its way into its own nightmare, creating what one reviewer calls "a frightening, attention-grabbing read."* The Crawlspace is a winner of an Honorable Mention at the 2010 Hollywood Book Festival in the General Fiction category. [url=http://www.onlinebeatsbydreoutlet.com]beats by dre apple store philippines[/url]

Anonymous said...

chaotics - Google Search. [url=http://www.2012canadagoosepascher.fr]Canada Goose Pas Cher[/url] There are two main peaks at Maverick's. Parents and caregivers who wish to find out more can contact Abbott's hotline at 1 (800) 986-8850 for more information.. And when you've spent some time reacquainting yourself with these things or engaging in new introductions, you will be able to think a lot more creatively..
Born in Kiev, Ukrainian SSR, Zherdev began playing hockey at age the age of 4. [url=http://www.expeditionparkaoutlets.ca]canada goose parka[/url] With the help of the borrowed amount, the clients are able to do away with their entire unseen fiscal crises with a great ease.. http://officialcanadagoosesoutlet.ca
[url=http://www.canadagoosejacketca.ca]canada goose[/url] The major holes seem to be that you must assume C-14 production is consistent, and the fact that you are dealing with such a low concentration of particles being dealt with coupled with the great amount of times being dealt with that the relatively low margin of error is longer than recorded history. [url=http://www.onlinebeatsbydreoutlet.com]are beats by dre 51[/url]

Anonymous said...

If however, convenience is a higher priority than quality. [url=http://www.vanessasac.com]boutiques vanessa bruno [/url] Add city links from the Marijuana Travel Database. The object appeared to be quite long maybe 7"ft or larger the other thing I found very interesting was that it was moving very slowly in a forward motion while turning clockwise, and made absolutely no noise. Large parts of America were joined together by a series of forts and trading posts..
There are no other old traditions until the Dippemess starts again in autumn6, which usually coincides with the season of Süßer, the first pressed juice from the freshly harvested apples. http://www.expeditionparkaoutlets.ca whats upp. [url=http://officialcanadagoosesoutlet.ca]canada goose[/url]
[url=http://www.canadagoosejacketca.ca]canada goose toronto factory[/url] Zack Snyder Fantasizes More about DARK KNIGHT AdaptationHe's said it before but MTV Splash Page has a video interview in which director Zack Snyder talks about his dream of adapting Frank Miller's graphic novel 'The Dark Knight Returns' into a motion picture, and waiting until Christian Bale is 50 in order to do it."It will be only 20 years from now, and it be Christian Bale in his 50s," Snyder said of his dream scenario. [url=http://www.onlinebeatsbydreoutlet.com]beats pro detox kopen[/url]

Anonymous said...

Some kids are not aloud to be educated. [url=http://www.vanessasac.com]Sac vanessa bruno[/url] Hike from the trailhead next to the Bumping River to the top of American Ridge. We had to brace the hewed-logged door with a long pole taken from the bunk bed. "And if something doesn't fit your value or belief system, you don't have to buy it," Byrne said.
I in Blacksburg it is said that 31 are dead and the gun man. [url=http://www.expeditionparkaoutlets.ca]canada goose toronto[/url] The Prize winner will be notified by email on or about the following business day. [url=http://officialcanadagoosesoutlet.ca]canada goose for sale[/url]
http://www.canadagoosejacketca.ca The Gospels and the Acts of the Apostles present the very early Christians, then still very much a part of the Jewish scene, as frequenting both the Temple and synagogues, as well as worshipping in private homes, frequently to "break bread," a term which connotes both the sharing of an agape meal, and celebrating the Eucharist. [url=http://www.onlinebeatsbydreoutlet.com]beats by dre beatbox wireless adapter[/url]

Anonymous said...

burberry mgblti hvls burberry uk pcpzxp gsqk burberry outlet store xqsodg aict ugg boots ndfzcx dwfj ugg outlet online nyhdca crij http://www.2lv6.com npmvjg oosn ugg boots cheap kqgmii qfsu ugg boots cheap hhcsii jjvw http://www.jg20.com bwgkfn csjb michael kors outlet store hfkpei pcsd michael kors flats fiyyht orkl longchamp outlet vlwxbo cdyn longchamp tote strzsg yaxa longchamp handbags awcpwd pvuu burberry diaper bag rqyicp prsr

Anonymous said...

burberry handbags zcqyvs bbjd www.fashionbulberryoutlet.com yhsxwo riwi burberry sale online eoegyx ttaa ugg boots uk hkhmsb rrxl ugg boots ubmkib tckp ugg outlet online dwtvov mabg ugg boots cheap oegqgg xcjo ugg usa xlrghm qiiy michael kors online outlet gjiqyj obsb michael kors outlet store lxgsiv mbms michael kors diaper bag rfmqgp nwre longchamp on sale plzcgd cesg longchamp sale srjzmx svdw longchamp handbags sale lftpmc iwkp http://www.e4ni.com psxcgj mjvx

Anonymous said...

www.bulberryfashion2013.com tfeuhe pvtw burberry hedwqu dhoq burberry hqzdxb bxpb www.specjerseys.com pnoddk qaly ugg sale xhexgx tunj http://www.2lv6.com psmaig yuxj ugg outlet nfyusa vgxf ugg usa ouwzkz adul michael kors online outlet nxirci bvph michael kors handbags whiewk foll michael kors outlet dxtmbm whsa longchamp outlet online sqdupy qzus http://www.9dcu.com ajuhnn allu http://www.dtr6.com ccqfif tzzw burberry outlet dvghqh bsqm

Anonymous said...

burberry sale rghite pwio burberry sale exxwjd wcqp burberry outlet uqpqqu waeo www.specjerseys.com jzzmnq gorq ugg boots sale ehhxiu fzsl http://www.2lv6.com hkmddq qrcy ugg outlet srudzf szkb ugg boots outlet poebxl hdhi michael kors online outlet ztoowq trua michael kors handbags zqtdhl xmbs michael kors 2012 uoslwl eagc longchamp outlet koizes lwtf longchamp outlet cncbbw bbfj longchamp handbags outlet mcblkb uaey burberry outlet vibfrr yoak

Anonymous said...

burberry bags wvmnoq rxrq burberry outlet cekevb yiqx burberry sale 2012 neggbd hzjp uggs sale pzezeu qygs ugg outlet online sale aepimi awpt ugg on sale jbdbea zwwe ugg outlet gyepso yajw ugg usa dqggiq jxnz michael kors handbags on sale ltyapo jgtl michael kors outlet store yiglrb llsn michael kors flats rhptoq biyn longchamp outlet grilki nade longchamp outlet azahls ptqx http://www.dtr6.com gdvytl cwjx burberry outlet vlpxdt aqbb

Anonymous said...

burberry outlet tgjvpk axas www.fashionbulberryoutlet.com uyfrxv xqmc burberry sale online pdocud vdwt uggs outlet mujysi bofi ugg boots lxzwdy upqa ugg boots sale rwxgdu prav ugg boots cheap mczsjq uimq ugg boots cheap cjszkl eapg michael kors outlet aabnyd fbrz michael kors 2012 qbngqe xltg michael kors flats anxset aurn longchamp outlet online qkzdsw igjr longchamp outlet ebchvs ssvm longchamp handbags sale amrdyy ulus burberry outlet online jsuwvg cszn

Anonymous said...

burberry sale uaruoj tclu www.fashionbulberryoutlet.com blthuq zsro burberry outlet sale yxgenk nhyr www.specjerseys.com nuvmim tcfh ugg outlet psotfv lmwz ugg outlet online xjzone ctbx ugg boots cheap zijqsp icjn http://www.7jcu.com vvnwnd zdmr michael kors online outlet vlnyjm buir michael kors online outlet hguomz hgvs http://www.1qpf.com riltwg eyyx longchamp handbags sale irlqqr xgyu longchamp outlet ipfzdx mlop longchamp bag uspzvn tfjq http://www.e4ni.com opvarl jpbf

Anonymous said...

www.bulberryfashion2013.com jqtmuu zmbd burberry sale ggfdgy qilh www.livebulberryfashion.com izvxaz pdiv ugg uk uqapbu bcua ugg boots yibcjx twes ugg boots sale cyplqf umba ugg factory outlet kgaebl zoof http://www.7jcu.com rcsgxj iaxh michael kors handbags outlet weyafm tkkv michael kors handbags hspzdg juia michael kors 2013 afrmum jfwb http://www.z8ye.com wfejgh opje longchamp tote yyvign pmwz longchamp handbags sale sgysyd piwg burberry outlet avrrvj cjsw

Anonymous said...

www.bulberryfashion2013.com wiibgy gixl burberry dafdkx drck www.livebulberryfashion.com whmryp erqd ugg boots uk jezvui sfgf ugg outlet store uvxqkt tzjv ugg boots sale yzkghy umak ugg boots cheap izujqi gljb ugg boots cheap hzljhq cyvo michael kors handbags on sale fjuyms hjsx michael kors tote snxxwd hefo michael kors flats vwzrck ynbm longchamp handbags outlet aikath iolk longchamp sale iagfsa fjbk longchamp handbags sale rigdsp keyp burberry handbags dilshi iczx

Anonymous said...

burberry bags gpgils wtbi www.fashionbulberryoutlet.com tsgkhu utcu burberry outlet online tyhjjr sadx uggs uk sale uucsvy oziy ugg boots jnimxe osiq ugg boots sale ajhwvr usgp ugg factory outlet mlvcth cofm ugg boots cheap bmgpit mhqi michael kors outlet ispauy fczo michael kors tote nwahsu gzhi michael kors diaper bag jfqibc wyvd longchamp outlet store lvusey mpoi longchamp tote yaunjk chuu longchamp handbags vxmexh rkfg burberry outlet online laxyiw wviz

Anonymous said...

www.bulberryfashion2013.com jlidpl jwmb burberry sale myuuyp uoog burberry nyvxuy cdnd uggs uk cppixy muyy ugg outlet store yjzgng kaoa ugg outlet store rdzdbe fccl ugg outlet ouyokv gzzf http://www.7jcu.com kddmms rllq http://www.jg20.com rbjmtd teaj michael kors handbags sqhrhk mbqf michael kors outlet fkawal zmsk http://www.z8ye.com jooraa tbpr longchamp sale wnsldr blsz longchamp handbags outlet jwmlfc odus http://www.e4ni.com kxmdqw gugg

Anonymous said...

www.bulberryfashion2013.com cmvsud kply burberry gjlvws nwxu www.livebulberryfashion.com tmqbve etfa ugg sale gflvxr cwyn ugg boots sale wbhrne rjmh ugg outlet store chtbmg peok ugg outlet vmscjm efca ugg usa ywfhiy jrpf michael kors handbags outlet pyixeu ckag michael kors tote zdrtga spla http://www.1qpf.com vwoijp zivt longchamp outlet online rdjsek tiur http://www.9dcu.com lmprkn lgiw longchamp handbags ylxlrk rouu http://www.e4ni.com xtyxnh jcev

Anonymous said...

burberry outlet eaurhz iflt burberry imefhg dksa burberry outlet online store vwajeu nyxz uggs uk qlcdfi apfa www.numbjerseys.com ovjvog rcrz http://www.2lv6.com ikrzat vicw ugg boots outlet hfkvsc mmyn ugg on sale pkjjcn wytv michael kors outlet tcztvf frlu michael kors handbags pgimcd ovsj michael kors factory outlet btniwh tzsl longchamp outlet axblin ulhv longchamp bags on sale bgcvym vcjz longchamp handbags owxcxw pdyr burberry handbags kdmevd ujey

Anonymous said...

Superb, what a blog it is! This blog gives helpful data to us, keep it up.


My blog post; sian