[prev in list] [next in list] [prev in thread] [next in thread]
List: opennms-discuss
Subject: [opennms-discuss] RRD, Newts or both?
From: Kouli <dev () kou ! li>
Date: 2018-02-06 15:04:17
Message-ID: CAPgN8wKwdLGej5jXvv3D=8=b8Bh4uhenx86DgJADFQ3Jo8HCsA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hello,
I am running an ONMS instance which stores collected metrics to RRD files
(JRobin). Currently, we update 150 000 RRDs every 5 minutes. To be able to
run such a load of RRD updates, we must have moved RRDs to a RAM disk. (It
is a Linux ZRAM, i.e. LZMA compressed RAM disk formatted as XFS, because
XFS supports freeze operation, which is needed to dump whole RAM disk to
HDD every hour in a consistent way). This way, it works very fast even with
150 k RRDs.
However, we would like to shorten colletion interval, our target is to
lower it down to 10 seconds. With the RAM disk trick, it is doable on a
single computer even with RRD. But, isn't it time to switch to Newts? It
does not "loose precision" of old data (hard disk space should not be a
problem with current HDD prices) and could potentially scale even more than
RRD-on-RAM-disk.
I believe a single Cassandra/Newts node will accept our collection load (it
would be 15 k values per second), *BUT how will the aggregation work?*
You guys are using Newts with 5 minute interval, so it must get approx. 10
k (100 k) values to show an aggregated graph spanning one month (one year).
Without knowing Cassandra/Newts internals, I still believe it is feasible
in "near real time" (i.e. aggregate data on-demand of visualization). But
what if the data are gathered every 10 seconds? How long will it take to
load a graph aggregating 270 k (3 M) data points?
And what are disk space requirements of Newts/Cassandra? With 150 k metrics
and 10 seconds interval, it is 480 G data points per year... With RRD's 8
bytes per data point, one year would fit into 4 TB...
Kouli
[Attachment #5 (text/html)]
<div dir="ltr">Hello,<div><br></div><div>I am running an ONMS instance which stores \
collected metrics to RRD files (JRobin). Currently, we update 150 000 RRDs every 5 \
minutes. To be able to run such a load of RRD updates, we must have moved RRDs to a \
RAM disk. (It is a Linux ZRAM, i.e. LZMA compressed RAM disk formatted as XFS, \
because XFS supports freeze operation, which is needed to dump whole RAM disk to HDD \
every hour in a consistent way). This way, it works very fast even with 150 k \
RRDs.</div><div><br></div><div>However, we would like to shorten colletion interval, \
our target is to lower it down to 10 seconds. With the RAM disk trick, it is doable \
on a single computer even with RRD. But, isn't it time to switch to Newts? It \
does not "loose precision" of old data (hard disk space should not be a \
problem with current HDD prices) and could potentially scale even more than \
RRD-on-RAM-disk.</div><div><br></div><div>I believe a single Cassandra/Newts node \
will accept our collection load (it would be 15 k values per second), <b>BUT how will \
the aggregation work?</b></div><div><br></div><div>You guys are using Newts with 5 \
minute interval, so it must get approx. 10 k (100 k) values to show an aggregated \
graph spanning one month (one year). Without knowing Cassandra/Newts internals, I \
still believe it is feasible in "near real time" (i.e. aggregate data \
on-demand of visualization). But what if the data are gathered every 10 seconds? How \
long will it take to load a graph aggregating 270 k (3 M) data \
points?</div><div><br></div><div>And what are disk space requirements of \
Newts/Cassandra? With 150 k metrics and 10 seconds interval, it is 480 G data points \
per year... With RRD's 8 bytes per data point, one year would fit into 4 \
TB...</div><div><br></div><div>Kouli</div></div>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ
opennms-discuss mailing list
To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic