[prev in list] [next in list] [prev in thread] [next in thread]
List: hyperledger-tsc
Subject: Re: [Hyperledger TSC] Test net thinking
From: "Christopher Ferris" <chris.ferris () gmail ! com>
Date: 2018-10-30 14:16:38
Message-ID: CANHbT8genzCNe=jKUWWLrC-gP+d9iZVt2PTNrVeT8tM1NGShbA () mail ! gmail ! com
[Download RAW message or body]
It strikes me, though, that not everyone is aware of what Hyperledger
already has available to the various top-level projects.
Fabric uses Jenkins to drive testing of CRs and merge commits, and we also
do nightly and weekly tests leveraging the LF cloud provider's VMs. They
aren't fast, the network is slow, but they get the job done. We spin up
networks on-demand, run the tests and tear it all down. Lather, rinse and
repeat. These aren't large networks (yet), just simple docker-compose
configurations designed to run in a single VM.
Do we want to have the ability to test across multiple VMs leveraging a k8s
environment? You bet we do. I think that for the most part, that we'd
continue for CRs and merges we would continue down the path we currently
have chosen.
For integration, performance, scale, long-running and chaos testing having
a persistent k8s platform in which to launch pre-configured networks
leveraging the latest images and binaries for testing purposes would be "a
good thing". Managing that platform is probably something that the LF is
not set up to do. Leveraging a provider such as IBM, or AWS or Azure
kubernetes service would make sense. Then there is the problem of managing
the use of that (it is still a scarce resource that needs managing) across
the various projects.
Again, this all presumes we are talking effectively about testing FOR the
projects, not OF the projects by end users.
Chris
On Tue, Oct 30, 2018 at 9:59 AM Christopher Ferris <chris.ferris@gmail.com>
wrote:
> You cannot do performance and scale testing on nominal hardware. Long
> running and chaos tests, sure.
>
> Chris
>
> On Mon, Oct 29, 2018 at 9:15 PM David Huseby <dhuseby@linuxfoundation.org>
> wrote:
>
>> Does it really take big iron to run these tests? $10k/month/project
>> seems a bit much for this kind of thing. If the goal is to just kick
>> the tires or to have a soak test, why couldn't this be done on minimal
>> virtual machines or minimal hardware?
>>
>> Dave
>> ---
>> David Huseby
>> Security Maven, Hyperledger
>> The Linux Foundation
>> +1-206-234-2392
>> dhuseby@linuxfoundation.org
>>
>> On Fri, Oct 26, 2018 at 12:30 PM Christopher Ferris
>> <chris.ferris@gmail.com> wrote:
>> >
>> > Interesting that Casey thinks of long-running tests, performance,
>> chaos, etc. I always think of a testnet as a place where devs can kick the
>> tires without having to do the heavy lifting of setting up a network
>> themselves.
>> >
>> > For the long-running tests, etc. we have nightly runs and we are
>> gearing up a weekly as well that will be long running test that has kitchen
>> sinks thrown at it and stuff. Not a large environment, but more than what
>> we do for the merge and nightly CI.
>> >
>> > IBM also runs a series of long running tests leading up to a release on
>> our platforms. We do share the results indirectly and it would be nice if
>> they were just public, frankly (this is me speaking not for IBM but as a
>> member of the community). We are working on an ability to spin up networks
>> arbitrary complexity and scale, so that tests can be run. It would be nice
>> to have that running in the community, but again, not as a test net for
>> users as much as for what Casey was advocating for.
>> >
>> > Cheers
>> >
>> > Chris
>> >
>> > On Fri, Oct 26, 2018 at 10:24 AM Middleton, Dan <
>> dan.middleton@intel.com> wrote:
>> >>
>> >> Tldr: We only scale this if we have multiple companies hosting. What
>> kind of HL support is required to facilitate that?
>> >>
>> >>
>> >>
>> >> I agree with Chris that at least for native contracts like transaction
>> families and chaincode our projects have scripts so developers can launch
>> mini environments on their laptops (in sawtooth it's basically
>> `docker-compose up`). For on-chain smart contracts like burrow, sabre, and
>> so forth developers might appreciate a running testnet to deploy to.
>> >>
>> >>
>> >>
>> >> I kind of expected Casey to come in along those lines, so it's really
>> helpful to read his priority for stress testing infrastructure.
>> >>
>> >>
>> >>
>> >> Last year, running up to our 1.0 launch, the sawtooth team found they
>> needed to automate the long running tests and associated cloud deployments.
>> We've taken to calling these our LR networks.
>> >>
>> >>
>> >>
>> >> LRs are typically 10 node networks + load generation and stats
>> collections for 1 to 7 days. These LR networks have been invaluable to
>> testing features before issuing PRs.
>> >>
>> >>
>> >>
>> >> This stress testing is not cheap though. It's one thing to pay for an
>> idle instance and it's another to pay for 10 fully utilized instances.
>> >>
>> >> And as we are normally testing nightly builds along with experimental
>> branches etc., it's really more like a multiple of 10.
>> >>
>> >>
>> >>
>> >> Based on the CNCF readme I'd bet the sawtooth project alone could
>> easily consume all their credits. CNCF, then is probably a good start but
>> not a complete solution to support all the HL projects. Similarly any
>> direct budget estimate for infrastructure would need to be on the order of
>> $10k/month/project which is probably not tenable.
>> >>
>> >>
>> >>
>> >> For sawtooth, we are looking at a crawl, walk, run with the community
>> to expand from LR nets to larger longer testnets. Start with some machines
>> provided by companies x, y, z and then add more users/instances. Actually
>> demonstrating that multiple companies can attach to a running network is a
>> highly valuable exercise. We're shooting to have the crawl version of the
>> testnet up imminently. In getting that underway we may discover some of the
>> administrative or other challenges that HL can solve in subsequent
>> iterations.
>> >>
>> >>
>> >>
>> >> Regards,
>> >> Dan
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> From: <tsc@lists.hyperledger.org> on behalf of Ry Jones <
>> rjones@linuxfoundation.org>
>> >> Date: Friday, October 26, 2018 at 12:06 AM
>> >> To: Casey Kuhlman <casey@monax.io>
>> >> Cc: Christopher Ferris <chris.ferris@gmail.com>, "
>> tsc@lists.hyperledger.org" <tsc@lists.hyperledger.org>
>> >> Subject: Re: [Hyperledger TSC] Test net thinking
>> >>
>> >>
>> >>
>> >> For people who are interested in CNCF services, the gatekeeper is
>> CNCF, not Hyperledger.
>> >>
>> >>
>> >>
>> >> Go here: https://github.com/cncf/cluster
>> >>
>> >>
>> >>
>> >> Following the template, file an issue, make your case. Note it is a
>> limited resource and priority is given to CNCF projects. I've talked to Dan
>> Kohn a couple of times and he wants to use the resource as much as possible.
>> >>
>> >>
>> >>
>> >> Ry
>> >>
>> >>
>> >>
>> >> On Thu, Oct 25, 2018 at 8:28 AM Casey Kuhlman via
>> Lists.Hyperledger.Org <casey=monax.io@lists.hyperledger.org> wrote:
>> >>
>> >> From the burrow perspective, we'd be very interested in test networks
>> to actually run testing in terms of performance testing, chaos testing,
>> soak testing, long-lived network testing which a simple CI run post merge
>> cannot really handle.
>> >>
>> >>
>> >>
>> >> While I think both Dan and Chris make valid points in terms of
>> positioning the test networks as things to help users get started faster,
>> our interest from the burrow team is less in that area and more in having
>> an HL managed infrastructure which can help us.... test.
>> >>
>> >>
>> >>
>> >> My point, and the reason that Silas brought this up a few months ago
>> to the TSC is that from a testing and performance perspective I think we
>> could do much better as a community. CCNF and the Kubernetes project have a
>> super clean and easy to approach CI system that made it dead simple for me
>> as a contributor when I started making contributions to helm. From my
>> perspective we'd be very interested to see if HL could coordinate with CCNF
>> (as Ry mentioned on RocketChat).
>> >>
>> >> --
>> >>
>> >> Ry Jones
>> >>
>> >> Community Architect, Hyperledger
>> >>
>> >> Chat: @ry
>> >
>> >
>>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1819): https://lists.hyperledger.org/g/tsc/message/1819
Mute This Topic: https://lists.hyperledger.org/mt/27623814/953895
Group Owner: tsc+owner@lists.hyperledger.org
Unsubscribe: https://lists.hyperledger.org/g/tsc/unsub [hyperledger-tsc@marc.info]
-=-=-=-=-=-=-=-=-=-=-=-
[Attachment #3 (text/html)]
<div dir="ltr"><div>It strikes me, though, that not everyone is aware of what \
Hyperledger already has available to the various top-level projects. \
<br></div><div><br></div><div>Fabric uses Jenkins to drive testing of CRs and merge \
commits, and we also do nightly and weekly tests leveraging the LF cloud \
provider's VMs. They aren't fast, the network is slow, but they get the job \
done. We spin up networks on-demand, run the tests and tear it all down. Lather, \
rinse and repeat. These aren't large networks (yet), just simple docker-compose \
configurations designed to run in a single VM.</div><div><br></div><div>Do we want to \
have the ability to test across multiple VMs leveraging a k8s environment? You bet we \
do. I think that for the most part, that we'd continue for CRs and merges we \
would continue down the path we currently have chosen. \
<br></div><div><br></div><div>For integration, performance, scale, long-running and \
chaos testing having a persistent k8s platform in which to launch pre-configured \
networks leveraging the latest images and binaries for testing purposes would be \
"a good thing". Managing that platform is probably something that the LF is \
not set up to do. Leveraging a provider such as IBM, or AWS or Azure kubernetes \
service would make sense. Then there is the problem of managing the use of that (it \
is still a scarce resource that needs managing) across the various \
projects.</div><div><br></div><div>Again, this all presumes we are talking \
effectively about testing FOR the projects, not OF the projects by end \
users.</div><div><br></div><div>Chris<br></div></div><br><div \
class="gmail_quote"><div dir="ltr">On Tue, Oct 30, 2018 at 9:59 AM Christopher Ferris \
<<a href="mailto:chris.ferris@gmail.com">chris.ferris@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>You cannot do \
performance and scale testing on nominal hardware. Long running and chaos tests, \
sure.</div><div><br></div><div>Chris<br></div></div><br><div class="gmail_quote"><div \
dir="ltr">On Mon, Oct 29, 2018 at 9:15 PM David Huseby <<a \
href="mailto:dhuseby@linuxfoundation.org" \
target="_blank">dhuseby@linuxfoundation.org</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">Does it really take big iron to run these tests? \
$10k/month/project<br> seems a bit much for this kind of thing. If the goal is to \
just kick<br> the tires or to have a soak test, why couldn't this be done on \
minimal<br> virtual machines or minimal hardware?<br>
<br>
Dave<br>
---<br>
David Huseby<br>
Security Maven, Hyperledger<br>
The Linux Foundation<br>
+1-206-234-2392<br>
<a href="mailto:dhuseby@linuxfoundation.org" \
target="_blank">dhuseby@linuxfoundation.org</a><br> <br>
On Fri, Oct 26, 2018 at 12:30 PM Christopher Ferris<br>
<<a href="mailto:chris.ferris@gmail.com" \
target="_blank">chris.ferris@gmail.com</a>> wrote:<br> ><br>
> Interesting that Casey thinks of long-running tests, performance, chaos, etc. I \
always think of a testnet as a place where devs can kick the tires without having to \
do the heavy lifting of setting up a network themselves.<br> ><br>
> For the long-running tests, etc. we have nightly runs and we are gearing up a \
weekly as well that will be long running test that has kitchen sinks thrown at it and \
stuff. Not a large environment, but more than what we do for the merge and nightly \
CI.<br> ><br>
> IBM also runs a series of long running tests leading up to a release on our \
platforms. We do share the results indirectly and it would be nice if they were just \
public, frankly (this is me speaking not for IBM but as a member of the community). \
We are working on an ability to spin up networks arbitrary complexity and scale, so \
that tests can be run. It would be nice to have that running in the community, but \
again, not as a test net for users as much as for what Casey was advocating for.<br> \
><br> > Cheers<br>
><br>
> Chris<br>
><br>
> On Fri, Oct 26, 2018 at 10:24 AM Middleton, Dan <<a \
href="mailto:dan.middleton@intel.com" target="_blank">dan.middleton@intel.com</a>> \
wrote:<br> >><br>
>> Tldr: We only scale this if we have multiple companies hosting. What kind of \
HL support is required to facilitate that?<br> >><br>
>><br>
>><br>
>> I agree with Chris that at least for native contracts like transaction \
families and chaincode our projects have scripts so developers can launch mini \
environments on their laptops (in sawtooth it's basically `docker-compose up`). For \
on-chain smart contracts like burrow, sabre, and so forth developers might appreciate \
a running testnet to deploy to.<br> >><br>
>><br>
>><br>
>> I kind of expected Casey to come in along those lines, so it's really \
helpful to read his priority for stress testing infrastructure.<br> >><br>
>><br>
>><br>
>> Last year, running up to our 1.0 launch, the sawtooth team found they needed \
to automate the long running tests and associated cloud deployments. We've taken to \
calling these our LR networks.<br> >><br>
>><br>
>><br>
>> LRs are typically 10 node networks + load generation and stats collections \
for 1 to 7 days. These LR networks have been invaluable to testing features before \
issuing PRs.<br> >><br>
>><br>
>><br>
>> This stress testing is not cheap though. It's one thing to pay for an idle \
instance and it's another to pay for 10 fully utilized instances.<br> >><br>
>> And as we are normally testing nightly builds along with experimental \
branches etc., it's really more like a multiple of 10.<br> >><br>
>><br>
>><br>
>> Based on the CNCF readme I'd bet the sawtooth project alone could easily \
consume all their credits. CNCF, then is probably a good start but not a complete \
solution to support all the HL projects. Similarly any direct budget estimate for \
infrastructure would need to be on the order of $10k/month/project which is probably \
not tenable.<br> >><br>
>><br>
>><br>
>> For sawtooth, we are looking at a crawl, walk, run with the community to \
expand from LR nets to larger longer testnets. Start with some machines provided by \
companies x, y, z and then add more users/instances. Actually demonstrating that \
multiple companies can attach to a running network is a highly valuable exercise. \
We're shooting to have the crawl version of the testnet up imminently. In getting \
that underway we may discover some of the administrative or other challenges that HL \
can solve in subsequent iterations.<br> >><br>
>><br>
>><br>
>> Regards,<br>
>> Dan<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> From: <<a href="mailto:tsc@lists.hyperledger.org" \
target="_blank">tsc@lists.hyperledger.org</a>> on behalf of Ry Jones <<a \
href="mailto:rjones@linuxfoundation.org" \
target="_blank">rjones@linuxfoundation.org</a>><br> >> Date: Friday, October \
26, 2018 at 12:06 AM<br> >> To: Casey Kuhlman <<a \
href="mailto:casey@monax.io" target="_blank">casey@monax.io</a>><br> >> Cc: \
Christopher Ferris <<a href="mailto:chris.ferris@gmail.com" \
target="_blank">chris.ferris@gmail.com</a>>, "<a \
href="mailto:tsc@lists.hyperledger.org" \
target="_blank">tsc@lists.hyperledger.org</a>" <<a \
href="mailto:tsc@lists.hyperledger.org" \
target="_blank">tsc@lists.hyperledger.org</a>><br> >> Subject: Re: \
[Hyperledger TSC] Test net thinking<br> >><br>
>><br>
>><br>
>> For people who are interested in CNCF services, the gatekeeper is CNCF, not \
Hyperledger.<br> >><br>
>><br>
>><br>
>> Go here: <a href="https://github.com/cncf/cluster" rel="noreferrer" \
target="_blank">https://github.com/cncf/cluster</a><br> >><br>
>><br>
>><br>
>> Following the template, file an issue, make your case. Note it is a limited \
resource and priority is given to CNCF projects. I've talked to Dan Kohn a couple \
of times and he wants to use the resource as much as possible.<br> >><br>
>><br>
>><br>
>> Ry<br>
>><br>
>><br>
>><br>
>> On Thu, Oct 25, 2018 at 8:28 AM Casey Kuhlman via <a \
href="http://Lists.Hyperledger.Org" rel="noreferrer" \
target="_blank">Lists.Hyperledger.Org</a> <casey=<a \
href="mailto:monax.io@lists.hyperledger.org" \
target="_blank">monax.io@lists.hyperledger.org</a>> wrote:<br> >><br>
>> From the burrow perspective, we'd be very interested in test networks to \
actually run testing in terms of performance testing, chaos testing, soak testing, \
long-lived network testing which a simple CI run post merge cannot really handle.<br> \
>><br> >><br>
>><br>
>> While I think both Dan and Chris make valid points in terms of positioning \
the test networks as things to help users get started faster, our interest from the \
burrow team is less in that area and more in having an HL managed infrastructure \
which can help us.... test.<br> >><br>
>><br>
>><br>
>> My point, and the reason that Silas brought this up a few months ago to the \
TSC is that from a testing and performance perspective I think we could do much \
better as a community. CCNF and the Kubernetes project have a super clean and easy to \
approach CI system that made it dead simple for me as a contributor when I started \
making contributions to helm. From my perspective we'd be very interested to see \
if HL could coordinate with CCNF (as Ry mentioned on RocketChat).<br> >><br>
>> --<br>
>><br>
>> Ry Jones<br>
>><br>
>> Community Architect, Hyperledger<br>
>><br>
>> Chat: @ry<br>
>
</blockquote></div>
</blockquote></div>
<div width="1" style="color:white;clear:both">_._,_._,_</div>
<hr>
Links:<p>
You receive all messages sent to this group.
<p>
<a target="_blank" href="https://lists.hyperledger.org/g/tsc/message/1819">View/Reply \
Online (#1819)</a> |
<a target="_blank" \
href="mailto:chris.ferris@gmail.com?subject=Private:%20Re:%20Re%3A%20%5BHyperledger%20TSC%5D%20Test%20net%20thinking">Reply \
To Sender</a>
| <a target="_blank" \
href="mailto:tsc@lists.hyperledger.org?subject=Re:%20Re%3A%20%5BHyperledger%20TSC%5D%20Test%20net%20thinking">Reply \
To Group</a>
>
<a target="_blank" href="https://lists.hyperledger.org/mt/27623814/953895">Mute \
This Topic</a>
> <a href="https://lists.hyperledger.org/g/tsc/post">New Topic</a><br>
<br>
<a href="https://lists.hyperledger.org/g/tsc/editsub/953895">Your Subscription</a> |
<a href="mailto:tsc+owner@lists.hyperledger.org">Contact Group Owner</a> |
<a href="https://lists.hyperledger.org/g/tsc/unsub">Unsubscribe</a>
[hyperledger-tsc@marc.info]<br>
<div width="1" style="color:white;clear:both">_._,_._,_</div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic