[prev in list] [next in list] [prev in thread] [next in thread]
List: nix-dev
Subject: Re: [Nix-dev] nixos-container networking
From: Danylo Hlynskyi <abcz2.uprola () gmail ! com>
Date: 2017-03-14 6:01:56
Message-ID: CANZg+yddUwFXzNjwB4KZAnuxZHhSRF=4zObsp=PsC35gnyk7Mg () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Strange, I have lot's of containers with "-" and experience no problems.
But maybe you've exceeded by accident limit 13 symbols per container name?
Also, last time I tried "veth" networking, I was struggling from
https://github.com/NixOS/nixpkgs/issues/16330. My container experience was
awful when I tried container renames. That's why I've already switched to
bridged networking
---
BTW, I highly recommend patch to switch-to-configuration.pl
<https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b277f78ece1bb81b79b5651897e46a2bf#diff-0a057d6ff3f6f83f68b859178484f4fe>
from
https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b277f78ece1bb81b79b5651897e46a2bf
It isn't clear from commit message, but it does the following: makes
declarative containers truly reloadable (when you change
container config, it activates new configuration for container). The
culprit is *it should be* default behavior, because of
1.
https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/containers.nix#L225-L230
2.
https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/containers.nix#L676
I'd like to PR this, but got no time to test properly other parts of Nixos.
2017-03-14 4:42 GMT+02:00 Tomasz Czyż <tomasz.czyz@gmail.com>:
> Michael, Ian, thank you for your answers.
>
> Looks like my problem was with the container name. I tried bunch of
> different setups which didn't work and I discovered that when I'm using "-"
> in container name it doesn't work (I had impression that worked one or two
> times when I started machine from scratch, but most of the time didn't).
>
> After I removed "-" from the name, looks like private network is working
> (I can access private IP of container) so I don't need NAT actually.
>
> Tom
>
> 2017-03-13 23:54 GMT+00:00 Ian-Woo Kim <ianwookim@gmail.com>:
>
> > I've recently made nixos-container port forwarding easier (both
> > imperative and declarative) and it's now merged into master.
> >
> > https://github.com/NixOS/nixpkgs/pull/20869
> >
> > Hope that this helps.
> >
> > Ian
> >
> > On Sun, Mar 12, 2017 at 7:52 PM, Michael Walker <mike@barrucadu.co.uk>
> > wrote:
> > > Tomasz,
> > >
> > > I have declarative container networking set up and working on a VPS,
> > > but I wrote most of the configuration as I was learning things, so it
> > > may not be the best way.
> > >
> > > Here's the configuration.nix for the VPS:
> > > https://github.com/barrucadu/nixfiles/blob/master/hosts/innsmouth.nix
> > > Each container has a config file here:
> > > https://github.com/barrucadu/nixfiles/tree/master/containers
> > >
> > > Containers have ports forwarded to them via NAT; each container is
> > > running a web server on port 80 with the host reverse-proxying via
> > > nginx; the host also does https and letsencrypt for all the proxied
> > > containers.
> > >
> > > At the top of the innsmouth.nix file, I have a "containerSpecs" record
> > > which has all the details for each container. The relevant bits of the
> > > config are:
> > >
> > > 1. Set up the networking and NAT:
> > >
> > > networking.nat.enable = true;
> > > networking.nat.internalInterfaces = ["ve-+"];
> > > networking.nat.externalInterface = "enp0s4";
> > >
> > > 2. Forward ports to containers:
> > >
> > > networking.nat.forwardPorts = concatMap
> > > ( {num, ports, ...}:
> > > map (p: { sourcePort = p; destination =
> > > "192.168.255.${toString num}:${toString p}"; }) ports
> > > ) containerSpecs';
> > >
> > > 3. Define all the containers:
> > >
> > > containers = mapAttrs
> > > (_: {num, config, ...}:
> > > { autoStart = true
> > > ; privateNetwork = true
> > > ; hostAddress = "192.168.254.${toString num}"
> > > ; localAddress = "192.168.255.${toString num}"
> > > ; config = config
> > > ; }
> > > ) containerSpecs;
> > >
> > > 4. Reverse-proxy HTTPS to HTTP in each container, manage letsencrypt
> > > certificates, and forward HTTP to HTTPS.
> > >
> > > This is a little complex as I have a fairly custom nginx config (see
> > > the services/nginx.nix file in the repository), but the
> > > reverse-proxying is fairly straightfoward. Here is the generated
> > > nginx.conf: https://misc.barrucadu.co.uk/nginx.txt
> > >
> > > On 13 March 2017 at 02:12, Tomasz Czyż <tomasz.czyz@gmail.com> wrote:
> > > > Hey,
> > > >
> > > > could anyone using nixos-container (declarative style) share how you
> > setup
> > > > networking?
> > > >
> > > > I'm trying to setup few containers with private network and http proxy
> > at
> > > > the front. Each container potentially could run application on port 80
> > and I
> > > > would like to expose them through proxy.
> > > >
> > > > I tried to set this up with
> > > >
> > > > privateNetwork=true;
> > > > hostAddress
> > > > localAddress
> > > >
> > > > and I tried to also run nat on the host with (just to enable outbound
> > > > traffic)
> > > > internalInterfaces = ["ve-+"];
> > > > externalInterfaces = "eth0";
> > > >
> > > > but no luck.
> > > > My next try will be creating bridge on the host and add containers to
> > that
> > > > bridge. Is that how you do stuff or are better ways of doing container
> > > > networking?
> > > >
> > > > Tom
> > > >
> > > > _______________________________________________
> > > > nix-dev mailing list
> > > > nix-dev@lists.science.uu.nl
> > > > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> > > >
> > >
> > >
> > >
> > > --
> > > Michael Walker (http://www.barrucadu.co.uk)
> > > _______________________________________________
> > > nix-dev mailing list
> > > nix-dev@lists.science.uu.nl
> > > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >
>
>
>
> --
> Tomasz Czyż
>
> _______________________________________________
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
[Attachment #5 (text/html)]
<div dir="ltr"><div><div><div><div>Strange, I have lot's of containers with \
"-" and experience no problems. But maybe you've exceeded by accident \
limit 13 symbols per container name?<br><br></div>Also, last time I tried \
"veth" networking, I was struggling from <a \
href="https://github.com/NixOS/nixpkgs/issues/16330">https://github.com/NixOS/nixpkgs/issues/16330</a>. \
My container experience was awful when I tried container renames. That's why \
I've already switched to bridged networking<br><br>---<br><br></div>BTW, I highly \
recommend patch to <a \
href="https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b277f78ece1bb81b79b5651897e46a2bf#diff-0a057d6ff3f6f83f68b859178484f4fe" \
class="gmail-link-gray-dark" \
title="nixos/modules/system/activation/switch-to-configuration.pl">switch-to-configuration.pl
</a> from <a href="https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b277f78 \
ece1bb81b79b5651897e46a2bf">https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b277f78ece1bb81b79b5651897e46a2bf</a><br><br></div>It \
isn't clear from commit message, but it does the following: makes declarative \
containers truly reloadable (when you change <br>container config, it activates new \
configuration for container). The culprit is *it should be* default behavior, because \
of<br><br>1. <a href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtu \
alisation/containers.nix#L225-L230">https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/containers.nix#L225-L230</a><br>2. \
<a href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/con \
tainers.nix#L676">https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/containers.nix#L676</a><br><br></div>I'd \
like to PR this, but got no time to test properly other parts of Nixos.<br></div><div \
class="gmail_extra"><br><div class="gmail_quote">2017-03-14 4:42 GMT+02:00 Tomasz \
Czyż <span dir="ltr"><<a href="mailto:tomasz.czyz@gmail.com" \
target="_blank">tomasz.czyz@gmail.com</a>></span>:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div><div>Michael, Ian, thank you for your \
answers.<br><br></div><div>Looks like my problem was with the container name. I tried \
bunch of different setups which didn't work and I discovered that when I'm \
using "-" in container name it doesn't work (I had impression that \
worked one or two times when I started machine from scratch, but most of the time \
didn't).<br><br></div><div>After I removed "-" from the name, looks \
like private network is working (I can access private IP of container) so I don't \
need NAT actually.<br><br></div><div>Tom<br></div></div></div><div \
class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">2017-03-13 \
23:54 GMT+00:00 Ian-Woo Kim <span dir="ltr"><<a href="mailto:ianwookim@gmail.com" \
target="_blank">ianwookim@gmail.com</a>></span>:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">I've recently made nixos-container port forwarding easier \
(both<br> imperative and declarative) and it's now merged into master.<br>
<br>
<a href="https://github.com/NixOS/nixpkgs/pull/20869" rel="noreferrer" \
target="_blank">https://github.com/NixOS/nixpk<wbr>gs/pull/20869</a><br> <br>
Hope that this helps.<br>
<span class="m_-7295093421595560673HOEnZb"><font color="#888888"><br>
Ian<br>
</font></span><div class="m_-7295093421595560673HOEnZb"><div \
class="m_-7295093421595560673h5"><br> On Sun, Mar 12, 2017 at 7:52 PM, Michael Walker \
<<a href="mailto:mike@barrucadu.co.uk" \
target="_blank">mike@barrucadu.co.uk</a>> wrote:<br> > Tomasz,<br>
><br>
> I have declarative container networking set up and working on a VPS,<br>
> but I wrote most of the configuration as I was learning things, so it<br>
> may not be the best way.<br>
><br>
> Here's the configuration.nix for the VPS:<br>
> <a href="https://github.com/barrucadu/nixfiles/blob/master/hosts/innsmouth.nix" \
rel="noreferrer" target="_blank">https://github.com/barrucadu/n<wbr>ixfiles/blob/master/hosts/inns<wbr>mouth.nix</a><br>
> Each container has a config file here:<br>
> <a href="https://github.com/barrucadu/nixfiles/tree/master/containers" \
rel="noreferrer" target="_blank">https://github.com/barrucadu/n<wbr>ixfiles/tree/master/containers</a><br>
><br>
> Containers have ports forwarded to them via NAT; each container is<br>
> running a web server on port 80 with the host reverse-proxying via<br>
> nginx; the host also does https and letsencrypt for all the proxied<br>
> containers.<br>
><br>
> At the top of the innsmouth.nix file, I have a "containerSpecs" \
record<br> > which has all the details for each container. The relevant bits of \
the<br> > config are:<br>
><br>
> 1. Set up the networking and NAT:<br>
><br>
> networking.nat.enable = true;<br>
> networking.nat.internalInterfa<wbr>ces = ["ve-+"];<br>
> networking.nat.externalInterfa<wbr>ce = "enp0s4";<br>
><br>
> 2. Forward ports to containers:<br>
><br>
> networking.nat.forwardPorts = concatMap<br>
> ( {num, ports, ...}:<br>
> map (p: { sourcePort = p; destination =<br>
> "192.168.255.${toString num}:${toString p}"; }) ports<br>
> ) containerSpecs';<br>
><br>
> 3. Define all the containers:<br>
><br>
> containers = mapAttrs<br>
> (_: {num, config, ...}:<br>
> { autoStart = true<br>
> ; privateNetwork = true<br>
> ; hostAddress = "192.168.254.${toString num}"<br>
> ; localAddress = "192.168.255.${toString num}"<br>
> ; config = config<br>
> ; }<br>
> ) containerSpecs;<br>
><br>
> 4. Reverse-proxy HTTPS to HTTP in each container, manage letsencrypt<br>
> certificates, and forward HTTP to HTTPS.<br>
><br>
> This is a little complex as I have a fairly custom nginx config (see<br>
> the services/nginx.nix file in the repository), but the<br>
> reverse-proxying is fairly straightfoward. Here is the generated<br>
> nginx.conf: <a href="https://misc.barrucadu.co.uk/nginx.txt" rel="noreferrer" \
target="_blank">https://misc.barrucadu.co.uk/n<wbr>ginx.txt</a><br> ><br>
> On 13 March 2017 at 02:12, Tomasz Czyż <<a \
href="mailto:tomasz.czyz@gmail.com" target="_blank">tomasz.czyz@gmail.com</a>> \
wrote:<br> >> Hey,<br>
>><br>
>> could anyone using nixos-container (declarative style) share how you \
setup<br> >> networking?<br>
>><br>
>> I'm trying to setup few containers with private network and http proxy \
at<br> >> the front. Each container potentially could run application on port \
80 and I<br> >> would like to expose them through proxy.<br>
>><br>
>> I tried to set this up with<br>
>><br>
>> privateNetwork=true;<br>
>> hostAddress<br>
>> localAddress<br>
>><br>
>> and I tried to also run nat on the host with (just to enable outbound<br>
>> traffic)<br>
>> internalInterfaces = ["ve-+"];<br>
>> externalInterfaces = "eth0";<br>
>><br>
>> but no luck.<br>
>> My next try will be creating bridge on the host and add containers to \
that<br> >> bridge. Is that how you do stuff or are better ways of doing \
container<br> >> networking?<br>
>><br>
>> Tom<br>
>><br>
>> ______________________________<wbr>_________________<br>
>> nix-dev mailing list<br>
>> <a href="mailto:nix-dev@lists.science.uu.nl" \
target="_blank">nix-dev@lists.science.uu.nl</a><br> >> <a \
href="http://lists.science.uu.nl/mailman/listinfo/nix-dev" rel="noreferrer" \
target="_blank">http://lists.science.uu.nl/mai<wbr>lman/listinfo/nix-dev</a><br> \
>><br> ><br>
><br>
><br>
> --<br>
> Michael Walker (<a href="http://www.barrucadu.co.uk" rel="noreferrer" \
target="_blank">http://www.barrucadu.co.uk</a>)<br> > \
______________________________<wbr>_________________<br> > nix-dev mailing \
list<br> > <a href="mailto:nix-dev@lists.science.uu.nl" \
target="_blank">nix-dev@lists.science.uu.nl</a><br> > <a \
href="http://lists.science.uu.nl/mailman/listinfo/nix-dev" rel="noreferrer" \
target="_blank">http://lists.science.uu.nl/mai<wbr>lman/listinfo/nix-dev</a><br> \
</div></div></blockquote></div><br><br clear="all"><br></div></div><span \
class="HOEnZb"><font color="#888888">-- <br><div \
class="m_-7295093421595560673gmail_signature" data-smartmail="gmail_signature">Tomasz \
Czyż</div> </font></span></div>
<br>______________________________<wbr>_________________<br>
nix-dev mailing list<br>
<a href="mailto:nix-dev@lists.science.uu.nl">nix-dev@lists.science.uu.nl</a><br>
<a href="http://lists.science.uu.nl/mailman/listinfo/nix-dev" rel="noreferrer" \
target="_blank">http://lists.science.uu.nl/<wbr>mailman/listinfo/nix-dev</a><br> \
<br></blockquote></div><br></div>
_______________________________________________
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic