[prev in list] [next in list] [prev in thread] [next in thread]
List: nix-dev
Subject: Re: [Nix-dev] nixos-container networking
From: Thomas Hunger <tehunger () gmail ! com>
Date: 2017-03-14 14:48:32
Message-ID: CAPw-HwmnNkY-ZEk1RkPT1qi8re7u2PoDQhf-S8xY1NpLtw8f_g () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Would it be possible to add an assert if there are any restrictions on the
naming? I don't know enough about this to be of much help though.
On 14 March 2017 at 06:01, Danylo Hlynskyi <abcz2.uprola@gmail.com> wrote:
> Strange, I have lot's of containers with "-" and experience no problems.
> But maybe you've exceeded by accident limit 13 symbols per container name?
>
> Also, last time I tried "veth" networking, I was struggling from
> https://github.com/NixOS/nixpkgs/issues/16330. My container experience
> was awful when I tried container renames. That's why I've already switched
> to bridged networking
>
> ---
>
> BTW, I highly recommend patch to switch-to-configuration.pl
> <https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b277f78ece1bb81b79b5651897e46a2bf#diff-0a057d6ff3f6f83f68b859178484f4fe>
> from https://github.com/NixOS/nixpkgs/pull/3021/commits/
> 6e36619b277f78ece1bb81b79b5651897e46a2bf
>
> It isn't clear from commit message, but it does the following: makes
> declarative containers truly reloadable (when you change
> container config, it activates new configuration for container). The
> culprit is *it should be* default behavior, because of
>
> 1. https://github.com/NixOS/nixpkgs/blob/master/nixos/
> modules/virtualisation/containers.nix#L225-L230
> 2. https://github.com/NixOS/nixpkgs/blob/master/nixos/
> modules/virtualisation/containers.nix#L676
>
> I'd like to PR this, but got no time to test properly other parts of Nixos.
>
> 2017-03-14 4:42 GMT+02:00 Tomasz Czyż <tomasz.czyz@gmail.com>:
>
> > Michael, Ian, thank you for your answers.
> >
> > Looks like my problem was with the container name. I tried bunch of
> > different setups which didn't work and I discovered that when I'm using "-"
> > in container name it doesn't work (I had impression that worked one or two
> > times when I started machine from scratch, but most of the time didn't).
> >
> > After I removed "-" from the name, looks like private network is working
> > (I can access private IP of container) so I don't need NAT actually.
> >
> > Tom
> >
> > 2017-03-13 23:54 GMT+00:00 Ian-Woo Kim <ianwookim@gmail.com>:
> >
> > > I've recently made nixos-container port forwarding easier (both
> > > imperative and declarative) and it's now merged into master.
> > >
> > > https://github.com/NixOS/nixpkgs/pull/20869
> > >
> > > Hope that this helps.
> > >
> > > Ian
> > >
> > > On Sun, Mar 12, 2017 at 7:52 PM, Michael Walker <mike@barrucadu.co.uk>
> > > wrote:
> > > > Tomasz,
> > > >
> > > > I have declarative container networking set up and working on a VPS,
> > > > but I wrote most of the configuration as I was learning things, so it
> > > > may not be the best way.
> > > >
> > > > Here's the configuration.nix for the VPS:
> > > > https://github.com/barrucadu/nixfiles/blob/master/hosts/innsmouth.nix
> > > > Each container has a config file here:
> > > > https://github.com/barrucadu/nixfiles/tree/master/containers
> > > >
> > > > Containers have ports forwarded to them via NAT; each container is
> > > > running a web server on port 80 with the host reverse-proxying via
> > > > nginx; the host also does https and letsencrypt for all the proxied
> > > > containers.
> > > >
> > > > At the top of the innsmouth.nix file, I have a "containerSpecs" record
> > > > which has all the details for each container. The relevant bits of the
> > > > config are:
> > > >
> > > > 1. Set up the networking and NAT:
> > > >
> > > > networking.nat.enable = true;
> > > > networking.nat.internalInterfaces = ["ve-+"];
> > > > networking.nat.externalInterface = "enp0s4";
> > > >
> > > > 2. Forward ports to containers:
> > > >
> > > > networking.nat.forwardPorts = concatMap
> > > > ( {num, ports, ...}:
> > > > map (p: { sourcePort = p; destination =
> > > > "192.168.255.${toString num}:${toString p}"; }) ports
> > > > ) containerSpecs';
> > > >
> > > > 3. Define all the containers:
> > > >
> > > > containers = mapAttrs
> > > > (_: {num, config, ...}:
> > > > { autoStart = true
> > > > ; privateNetwork = true
> > > > ; hostAddress = "192.168.254.${toString num}"
> > > > ; localAddress = "192.168.255.${toString num}"
> > > > ; config = config
> > > > ; }
> > > > ) containerSpecs;
> > > >
> > > > 4. Reverse-proxy HTTPS to HTTP in each container, manage letsencrypt
> > > > certificates, and forward HTTP to HTTPS.
> > > >
> > > > This is a little complex as I have a fairly custom nginx config (see
> > > > the services/nginx.nix file in the repository), but the
> > > > reverse-proxying is fairly straightfoward. Here is the generated
> > > > nginx.conf: https://misc.barrucadu.co.uk/nginx.txt
> > > >
> > > > On 13 March 2017 at 02:12, Tomasz Czyż <tomasz.czyz@gmail.com> wrote:
> > > > > Hey,
> > > > >
> > > > > could anyone using nixos-container (declarative style) share how you
> > > setup
> > > > > networking?
> > > > >
> > > > > I'm trying to setup few containers with private network and http
> > > proxy at
> > > > > the front. Each container potentially could run application on port
> > > 80 and I
> > > > > would like to expose them through proxy.
> > > > >
> > > > > I tried to set this up with
> > > > >
> > > > > privateNetwork=true;
> > > > > hostAddress
> > > > > localAddress
> > > > >
> > > > > and I tried to also run nat on the host with (just to enable outbound
> > > > > traffic)
> > > > > internalInterfaces = ["ve-+"];
> > > > > externalInterfaces = "eth0";
> > > > >
> > > > > but no luck.
> > > > > My next try will be creating bridge on the host and add containers to
> > > that
> > > > > bridge. Is that how you do stuff or are better ways of doing container
> > > > > networking?
> > > > >
> > > > > Tom
> > > > >
> > > > > _______________________________________________
> > > > > nix-dev mailing list
> > > > > nix-dev@lists.science.uu.nl
> > > > > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Michael Walker (http://www.barrucadu.co.uk)
> > > > _______________________________________________
> > > > nix-dev mailing list
> > > > nix-dev@lists.science.uu.nl
> > > > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> > >
> >
> >
> >
> > --
> > Tomasz Czyż
> >
> > _______________________________________________
> > nix-dev mailing list
> > nix-dev@lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >
> >
>
> _______________________________________________
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
[Attachment #5 (text/html)]
<div dir="ltr">Would it be possible to add an assert if there are any restrictions on \
the naming? I don't know enough about this to be of much help though.</div><div \
class="gmail_extra"><br><div class="gmail_quote">On 14 March 2017 at 06:01, Danylo \
Hlynskyi <span dir="ltr"><<a href="mailto:abcz2.uprola@gmail.com" \
target="_blank">abcz2.uprola@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Strange, I have lot's \
of containers with "-" and experience no problems. But maybe you've \
exceeded by accident limit 13 symbols per container name?<br><br></div>Also, last \
time I tried "veth" networking, I was struggling from <a \
href="https://github.com/NixOS/nixpkgs/issues/16330" \
target="_blank">https://github.com/NixOS/<wbr>nixpkgs/issues/16330</a>. My container \
experience was awful when I tried container renames. That's why I've already \
switched to bridged networking<br><br>---<br><br></div>BTW, I highly recommend patch \
to <a href="https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b277f78ece1bb81b79b5651897e46a2bf#diff-0a057d6ff3f6f83f68b859178484f4fe" \
class="m_2792496034223023470gmail-link-gray-dark" \
title="nixos/modules/system/activation/switch-to-configuration.pl" \
target="_blank">switch-to-configuration.pl </a> from <a \
href="https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b277f78ece1bb81b79b5651897e46a2bf" \
target="_blank">https://github.com/NixOS/<wbr>nixpkgs/pull/3021/commits/<wbr>6e36619b277f78ece1bb81b79b5651<wbr>897e46a2bf</a><br><br></div>It \
isn't clear from commit message, but it does the following: makes declarative \
containers truly reloadable (when you change <br>container config, it activates new \
configuration for container). The culprit is *it should be* default behavior, because \
of<br><br>1. <a href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/containers.nix#L225-L230" \
target="_blank">https://github.com/NixOS/<wbr>nixpkgs/blob/master/nixos/<wbr>modules/virtualisation/<wbr>containers.nix#L225-L230</a><br>2. \
<a href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/containers.nix#L676" \
target="_blank">https://github.com/NixOS/<wbr>nixpkgs/blob/master/nixos/<wbr>modules/virtualisation/<wbr>containers.nix#L676</a><br><br></div>I'd \
like to PR this, but got no time to test properly other parts of Nixos.<br></div><div \
class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div \
class="gmail_quote">2017-03-14 4:42 GMT+02:00 Tomasz Czyż <span dir="ltr"><<a \
href="mailto:tomasz.czyz@gmail.com" \
target="_blank">tomasz.czyz@gmail.com</a>></span>:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div><div>Michael, Ian, thank you for your \
answers.<br><br></div><div>Looks like my problem was with the container name. I tried \
bunch of different setups which didn't work and I discovered that when I'm \
using "-" in container name it doesn't work (I had impression that \
worked one or two times when I started machine from scratch, but most of the time \
didn't).<br><br></div><div>After I removed "-" from the name, looks \
like private network is working (I can access private IP of container) so I don't \
need NAT actually.<br><br></div><div>Tom<br></div></div></div><div \
class="gmail_extra"><div><div class="m_2792496034223023470h5"><br><div \
class="gmail_quote">2017-03-13 23:54 GMT+00:00 Ian-Woo Kim <span dir="ltr"><<a \
href="mailto:ianwookim@gmail.com" \
target="_blank">ianwookim@gmail.com</a>></span>:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">I've recently made nixos-container port forwarding easier \
(both<br> imperative and declarative) and it's now merged into master.<br>
<br>
<a href="https://github.com/NixOS/nixpkgs/pull/20869" rel="noreferrer" \
target="_blank">https://github.com/NixOS/nixpk<wbr>gs/pull/20869</a><br> <br>
Hope that this helps.<br>
<span class="m_2792496034223023470m_-7295093421595560673HOEnZb"><font \
color="#888888"><br> Ian<br>
</font></span><div class="m_2792496034223023470m_-7295093421595560673HOEnZb"><div \
class="m_2792496034223023470m_-7295093421595560673h5"><br> On Sun, Mar 12, 2017 at \
7:52 PM, Michael Walker <<a href="mailto:mike@barrucadu.co.uk" \
target="_blank">mike@barrucadu.co.uk</a>> wrote:<br> > Tomasz,<br>
><br>
> I have declarative container networking set up and working on a VPS,<br>
> but I wrote most of the configuration as I was learning things, so it<br>
> may not be the best way.<br>
><br>
> Here's the configuration.nix for the VPS:<br>
> <a href="https://github.com/barrucadu/nixfiles/blob/master/hosts/innsmouth.nix" \
rel="noreferrer" target="_blank">https://github.com/barrucadu/n<wbr>ixfiles/blob/master/hosts/inns<wbr>mouth.nix</a><br>
> Each container has a config file here:<br>
> <a href="https://github.com/barrucadu/nixfiles/tree/master/containers" \
rel="noreferrer" target="_blank">https://github.com/barrucadu/n<wbr>ixfiles/tree/master/containers</a><br>
><br>
> Containers have ports forwarded to them via NAT; each container is<br>
> running a web server on port 80 with the host reverse-proxying via<br>
> nginx; the host also does https and letsencrypt for all the proxied<br>
> containers.<br>
><br>
> At the top of the innsmouth.nix file, I have a "containerSpecs" \
record<br> > which has all the details for each container. The relevant bits of \
the<br> > config are:<br>
><br>
> 1. Set up the networking and NAT:<br>
><br>
> networking.nat.enable = true;<br>
> networking.nat.internalInterfa<wbr>ces = ["ve-+"];<br>
> networking.nat.externalInterfa<wbr>ce = "enp0s4";<br>
><br>
> 2. Forward ports to containers:<br>
><br>
> networking.nat.forwardPorts = concatMap<br>
> ( {num, ports, ...}:<br>
> map (p: { sourcePort = p; destination =<br>
> "192.168.255.${toString num}:${toString p}"; }) ports<br>
> ) containerSpecs';<br>
><br>
> 3. Define all the containers:<br>
><br>
> containers = mapAttrs<br>
> (_: {num, config, ...}:<br>
> { autoStart = true<br>
> ; privateNetwork = true<br>
> ; hostAddress = "192.168.254.${toString num}"<br>
> ; localAddress = "192.168.255.${toString num}"<br>
> ; config = config<br>
> ; }<br>
> ) containerSpecs;<br>
><br>
> 4. Reverse-proxy HTTPS to HTTP in each container, manage letsencrypt<br>
> certificates, and forward HTTP to HTTPS.<br>
><br>
> This is a little complex as I have a fairly custom nginx config (see<br>
> the services/nginx.nix file in the repository), but the<br>
> reverse-proxying is fairly straightfoward. Here is the generated<br>
> nginx.conf: <a href="https://misc.barrucadu.co.uk/nginx.txt" rel="noreferrer" \
target="_blank">https://misc.barrucadu.co.uk/n<wbr>ginx.txt</a><br> ><br>
> On 13 March 2017 at 02:12, Tomasz Czyż <<a \
href="mailto:tomasz.czyz@gmail.com" target="_blank">tomasz.czyz@gmail.com</a>> \
wrote:<br> >> Hey,<br>
>><br>
>> could anyone using nixos-container (declarative style) share how you \
setup<br> >> networking?<br>
>><br>
>> I'm trying to setup few containers with private network and http proxy \
at<br> >> the front. Each container potentially could run application on port \
80 and I<br> >> would like to expose them through proxy.<br>
>><br>
>> I tried to set this up with<br>
>><br>
>> privateNetwork=true;<br>
>> hostAddress<br>
>> localAddress<br>
>><br>
>> and I tried to also run nat on the host with (just to enable outbound<br>
>> traffic)<br>
>> internalInterfaces = ["ve-+"];<br>
>> externalInterfaces = "eth0";<br>
>><br>
>> but no luck.<br>
>> My next try will be creating bridge on the host and add containers to \
that<br> >> bridge. Is that how you do stuff or are better ways of doing \
container<br> >> networking?<br>
>><br>
>> Tom<br>
>><br>
>> ______________________________<wbr>_________________<br>
>> nix-dev mailing list<br>
>> <a href="mailto:nix-dev@lists.science.uu.nl" \
target="_blank">nix-dev@lists.science.uu.nl</a><br> >> <a \
href="http://lists.science.uu.nl/mailman/listinfo/nix-dev" rel="noreferrer" \
target="_blank">http://lists.science.uu.nl/mai<wbr>lman/listinfo/nix-dev</a><br> \
>><br> ><br>
><br>
><br>
> --<br>
> Michael Walker (<a href="http://www.barrucadu.co.uk" rel="noreferrer" \
target="_blank">http://www.barrucadu.co.uk</a>)<br> > \
______________________________<wbr>_________________<br> > nix-dev mailing \
list<br> > <a href="mailto:nix-dev@lists.science.uu.nl" \
target="_blank">nix-dev@lists.science.uu.nl</a><br> > <a \
href="http://lists.science.uu.nl/mailman/listinfo/nix-dev" rel="noreferrer" \
target="_blank">http://lists.science.uu.nl/mai<wbr>lman/listinfo/nix-dev</a><br> \
</div></div></blockquote></div><br><br clear="all"><br></div></div><span \
class="m_2792496034223023470HOEnZb"><font color="#888888">-- <br><div \
class="m_2792496034223023470m_-7295093421595560673gmail_signature" \
data-smartmail="gmail_signature">Tomasz Czyż</div> </font></span></div>
<br>______________________________<wbr>_________________<br>
nix-dev mailing list<br>
<a href="mailto:nix-dev@lists.science.uu.nl" \
target="_blank">nix-dev@lists.science.uu.nl</a><br> <a \
href="http://lists.science.uu.nl/mailman/listinfo/nix-dev" rel="noreferrer" \
target="_blank">http://lists.science.uu.nl/mai<wbr>lman/listinfo/nix-dev</a><br> \
<br></blockquote></div><br></div> \
</div></div><br>______________________________<wbr>_________________<br> nix-dev \
mailing list<br> <a href="mailto:nix-dev@lists.science.uu.nl">nix-dev@lists.science.uu.nl</a><br>
<a href="http://lists.science.uu.nl/mailman/listinfo/nix-dev" rel="noreferrer" \
target="_blank">http://lists.science.uu.nl/<wbr>mailman/listinfo/nix-dev</a><br> \
<br></blockquote></div><br></div>
_______________________________________________
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic