[prev in list] [next in list] [prev in thread] [next in thread]
List: freebsd-hackers
Subject: Re: devctl: Failed to detach pci0:2:0:0: Device busy
From: Mario Marietto <marietto2008 () gmail ! com>
Date: 2023-02-03 17:09:40
Message-ID: CA+1FSiijaRF+AwP8=9Ob+tS2pkakxnZ81h9zV-DwUwAJB=p+rQ () mail ! gmail ! com
[Download RAW message or body]
Hello, bhyve and / or FreeBSD developers and lovers.
I'm trying to pass my GTX 1060 from my host os (FreeBSD 13.1-RELEASE) to a
Linux/Ubuntu VM virtualized with bhyve,but unsuccessfully. Last year I
tried to do the same with another GPU that I have on my PC,the RTX 2080 ti
and with the help of @generix
<https://forums.developer.nvidia.com/u/generix>,after a tricky trouble
shooting we have been able to do that. Point is that the same patches which
worked (and that works right now) for the 2080 ti aren't working for the
GTX 1060. The error that I see is the same that I saw when I have
cooperated with Generix for passing through the 2080 ti. I really would
like to know which differences there are between the two gpus because for
the 1060 I'm using the same new driver that I'm using for the 2080 ti and
the latter one can be passed without problems. Anyway,I would know what I
should do to fix that error. For this matter I've created a new post on the
nvidia forum. If someone would like to give it a look,it is here :
https://forums.developer.nvidia.com/t/rminitadapter-failed-error-produced-when-i-try-t \
o-passthru-my-gtx-1060-to-a-linux-ubuntu-bhyve-vm-instead-my-rtx-2080ti-is-passed-correctly/241594
Il giorno ven 3 feb 2023 alle ore 15:21 Mario Marietto <
marietto2008@gmail.com> ha scritto:
> Very interesting behavior. I've added this new line in /etc/rc.conf :
> (until some minutes ago I didn't use it because I was using the intel
> driver,installed by the package : xf86-video-intel-2.99.917.916_2,1)
>
> kld_list="i915kms acpi_video"
>
> and this line in /boot/loader.conf :
>
> pptdevs="1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3 5/0/0"
>
> and boom. pptdevs worked again:
>
> # pciconf -l
>
> ppt0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de
> device=0x1c02 subvendor=0x19da subdevice=0x2438
> ppt1@pci0:1:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de
> device=0x10f1 subvendor=0x19da subdevice=0x2438
> ppt2@pci0:2:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de
> device=0x1e04 subvendor=0x19da subdevice=0x2503
> ppt3@pci0:2:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de
> device=0x10f7 subvendor=0x19da subdevice=0x2503
> ppt4@pci0:2:0:2: class=0x0c0330 rev=0xa1 hdr=0x00 vendor=0x10de
> device=0x1ad6 subvendor=0x19da subdevice=0x2503
> ppt5@pci0:2:0:3: class=0x0c8000 rev=0xa1 hdr=0x00 vendor=0x10de
> device=0x1ad7 subvendor=0x19da subdevice=0x2503
> ppt6@pci0:5:0:0: class=0x0c0330 rev=0x10 hdr=0x00 vendor=0x1b73
> device=0x1100 subvendor=0x1b73 subdevice=0x1100
>
> So,maybe there is some incompatibility between the intel and the pptdevs
> driver ? take also in consideration that on xorg.cong I'm still using the
> intel driver :
>
> Section "Device"
> Identifier "Card0"
> Driver "intel"
> BusID "PCI:0:2:0"
> EndSection.
>
> It's a confusing situation.
>
> Il giorno ven 3 feb 2023 alle ore 14:13 Mario Marietto <
> marietto2008@gmail.com> ha scritto:
>
> > I've sent an email to the hackers ML with this content :
> >
> > > On 30 Jan 2023, at 05:37, Mario Marietto <marietto2008@gmail.com>
> > wrote:
> > > In some FreeBSD 13.1 machines I have the problem below,in some others I
> > don't have it. I would like to know what the causes could be and how to fix
> > it.
> > >
> > > root@marietto:/usr/home/marietto # devctl detach pci0:1:0:0
> > >
> > > devctl: Failed to detach pci0:1:0:0: Device busy
> > >
> > > root@marietto:/usr/home/marietto # devctl detach pci0:2:0:0
> > >
> > > devctl: Failed to detach pci0:2:0:0: Device busy
> > >
> > > Not even it works if instead of detach them, I try to attach them
> > directly to the ppt driver :
> > >
> > > root@marietto:/usr/home/marietto/bhyve/Files # devctl set driver
> > pci0:2:0:0 ppt
> > > devctl: Failed to set pci0:2:0:0 driver to ppt: Device busy
> > >
> > > root@marietto:/usr/home/marietto/bhyve/Files # devctl detach
> > pci0:2:0:0
> > > devctl: Failed to detach pci0:2:0:0: Device busy
> > >
> > > Pci addresses 1:0:0 and 2:0:0 belong to the two GPUs that I have on my
> > PC :
> > >
> > > vgapci0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de
> > device=0x1c02 subvendor=0x19da subdevice=0x2438
> > > vgapci1@pci0:2:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de
> > device=0x1e04 subvendor=0x19da subdevice=0x2503
> > >
> > > Actually I have commented this line on /boot/loader.conf,because it
> > makes no difference if I keep it uncommented or not. It is totally ignored.
> > >
> > > #pptdevs="1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3"
> >
> > and George replied to me like this :
> >
> > Do you have 'vmm_load="YES"' in loader.conf?
> >
> > If not I am not sure it will have an effect - vmm must be loaded early
> > otherwise the card will be grabbed by the vgapci driver rather than be put
> > under ppt control.
> >
> > I think this is an interesting answer. Yes,I keep vmm_load="YES" in
> > loader.conf,BUT putting the 4 slots under ppt control using the loader.conf
> > did not work for me for months. We haven't taken this into consideration.
> > And you didn't reply when I've asked you if you have modified the bhyve
> > source code,because putting the pci addresses under ppt using the
> > loader.conf always worked for me. I would like to understand why it does
> > not work anymore. This is important to understand why I'm not able to
> > attach / detach the nVidia driver from the host to the guest os. And maybe
> > George's consideration is relevant.
> >
> > Il giorno ven 3 feb 2023 alle ore 12:11 Corvin Köhne <
> > C.Koehne@beckhoff.com> ha scritto:
> >
> > >
> > > - ---> Figure out if you can blacklist your gpu in the nvidia
> > > driver. So, the nVidia driver never gets attached to your device.
> > >
> > >
> > > - nope. I need to attach the nVidia driver to my gpu if I want to
> > > run stable diffusion + pytorch + cuda on the Linuxulator on FreeBSD. I'm
> > > thinking of installing Stable Diffusion within the Linux VM,instead of
> > > doing it on FreeBSD.
> > >
> > >
> > > - ---> Wait for drm drivers and use the nouveau driver. Nouveau
> > > isn't maintained by nVidia, so it's more likely to get fixed.
> > >
> > >
> > > - The nouveau driver is always bugged and it never worked well. I
> > > wonder if it is still supported. It has even limited features.
> > >
> > >
> > >
> > > Afaik, you don't have other options, sry. I'm no gpu driver developer,
> > > so I can't help you.
> > >
> > > Btw: Do you refer to linux nouveau or freebsd nouveau? Freebsd made much
> > > progress in porting linux drm to freebsd with 14.0. So, if latest linux
> > > nouveau fits your requirements, it's likely that freebsd 14.0 may fit them
> > > too.
> > >
> > >
> > >
> > > - Are the following commands enough to update the bhyve source code
> > > on my machine ? take in consideration that I need to grab only your new
> > > last patch.
> > >
> > >
> > > - cd /usr/src
> > >
> > >
> > > - git checkout -f origin/phab/corvink/13.1/nvidia-wip
> > >
> > >
> > > - # cd /wherever/you/put/my/Makefile
> > >
> > > # make -j 8
> > >
> > >
> > >
> > > My latest patch is a mouse driver patch for the host. So, no, you have
> > > to use:
> > >
> > > cd /usr/src
> > >
> > > git fetch
> > >
> > > git checkout -f origin/phab/corvink/13.1/nvidia-wip
> > >
> > > make kernel
> > >
> > > reboot
> > >
> > >
> > >
> > >
> > >
> > > Best regards
> > >
> > > Corvin
> > >
> > >
> > >
> > >
> > >
> > > This email contains confidential information. If you have received it in
> > > error, you must not read, use, copy or pass on this e-mail or its
> > > attachments.
> > > If you have received the e-mail in error, please inform me immediately
> > > by reply e-mail and then delete this e-mail from your system. Thank you
> > >
> > > Diese E-Mail enthaelt vertrauliche Informationen. Sollten Sie sie
> > > irrtuemlich erhalten haben, duerfen Sie diese E-Mail oder ihre Anhaenge
> > > nicht lesen, verwenden, kopieren oder weitergeben.
> > > Sollten Sie die Mail versehentlich erhalten haben, teilen Sie mir dies
> > > bitte umgehend per Antwort-E-Mail mit und loeschen Sie diese E-Mail dann
> > > aus Ihrem System. Vielen Dank
> > >
> > > Beckhoff Automation GmbH & Co. KG | Managing Director: Dipl. Phys. Hans
> > > Beckhoff
> > > Registered office: Verl, Germany | Register court: Guetersloh HRA 7075
> > >
> > >
> >
> > --
> > Mario.
> >
>
>
> --
> Mario.
>
--
Mario.
[Attachment #3 (text/html)]
<div dir="ltr"><div></div><div><p>Hello, bhyve and / or FreeBSD developers and \
lovers.<br></p><p>I'm trying to pass my GTX 1060 from my host os (FreeBSD \
13.1-RELEASE) to a Linux/Ubuntu VM virtualized with bhyve,but unsuccessfully. Last \
year I tried to do the same with another GPU that I have on my PC,the RTX 2080 ti \
and with the help of <a class="gmail-mention" \
href="https://forums.developer.nvidia.com/u/generix">@generix</a>,after a tricky \
trouble shooting we have been able to do that. Point is that the same patches which \
worked (and that works right now) for the 2080 ti aren't working for the GTX 1060. \
The error that I see is the same that I saw when I have cooperated with Generix for \
passing through the 2080 ti. I really would like to know which differences there are \
between the two gpus because for the 1060 I'm using the same new driver that I'm \
using for the 2080 ti and the latter one can be passed without problems. Anyway,I \
would know what I should do to fix that error. For this matter I've created a \
new post on the nvidia forum. If someone would like to give it a look,it is here \
:<br></p></div><div><a \
href="https://forums.developer.nvidia.com/t/rminitadapter-failed-error-produced-when-i \
-try-to-passthru-my-gtx-1060-to-a-linux-ubuntu-bhyve-vm-instead-my-rtx-2080ti-is-passed-correctly/241594" \
target="_blank">https://forums.developer.nvidia.com/t/rminitadapter-failed-error-produ \
ced-when-i-try-to-passthru-my-gtx-1060-to-a-linux-ubuntu-bhyve-vm-instead-my-rtx-2080ti-is-passed-correctly/241594</a></div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno ven 3 feb 2023 alle \
ore 15:21 Mario Marietto <<a href="mailto:marietto2008@gmail.com" \
target="_blank">marietto2008@gmail.com</a>> ha scritto:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><span \
style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)">Very interesting \
behavior. I've added this new line in /etc/rc.conf : (until some minutes ago I \
didn't use it because I was using the intel driver,installed by the package : \
<span style="font-family:monospace"><span style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)">xf86-video-intel-2.99.917.916_2,1)</span></span></span></span></span></div><div><span \
style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span \
style="font-family:monospace"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><br></span></span></span></span></div><div><span \
style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)">kld_list="i915kms \
acpi_video"</span><br></span></span></div><div><span \
style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><br></span></span></div><div><span \
style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)">and this line in \
/boot/loader.conf :</span></span></div><div><span \
style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><br></span></span></div><div><span \
style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span \
style="font-family:monospace"><span style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)">pptdevs="1/0/0 1/0/1 \
2/0/0 2/0/1 2/0/2 2/0/3 \
5/0/0"</span></span><br></span></span></span></div><div><span \
style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><br></span></span></div><div><span \
style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)">and boom. pptdevs worked \
again:</span></span></div><div><span style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><br></span></span></div><div><span \
style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)"># pciconf \
-l<br></span></span></div><div><span style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><br></span></span></div><div><span \
style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)">ppt0@pci0:1:0:0: \
class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1c02 subvendor=0x19da \
subdevice=0x2438 </span></span></div><span style="font-family:monospace"><span \
style="font-family:arial,sans-serif">ppt1@pci0:1:0:1: class=0x040300 \
rev=0xa1 hdr=0x00 vendor=0x10de device=0x10f1 subvendor=0x19da subdevice=0x2438 \
<br>ppt2@pci0:2:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de \
device=0x1e04 subvendor=0x19da subdevice=0x2503 <br>ppt3@pci0:2:0:1: \
class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x10f7 subvendor=0x19da \
subdevice=0x2503 <br>ppt4@pci0:2:0:2: class=0x0c0330 rev=0xa1 hdr=0x00 \
vendor=0x10de device=0x1ad6 subvendor=0x19da subdevice=0x2503 <br>ppt5@pci0:2:0:3: \
class=0x0c8000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad7 subvendor=0x19da \
subdevice=0x2503</span><br></span><div><span style="font-family:monospace"><span \
style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)">ppt6@pci0:5:0:0: \
class=0x0c0330 rev=0x10 hdr=0x00 vendor=0x1b73 device=0x1100 subvendor=0x1b73 \
subdevice=0x1100</span></span><br></span></div><div><span \
style="font-family:monospace"><br></span></div><div>So,maybe there is some \
incompatibility between the intel and the pptdevs driver ? take also in \
consideration that on xorg.cong I'm still using the intel driver \
:<br></div><div><br></div><div><span style="font-family:monospace"><span \
style="font-family:arial,sans-serif"><span \
style="color:rgb(0,0,0);background-color:rgb(255,255,255)">Section "Device" \
</span><br> Identifier "Card0" <br> Driver \
"intel" <br> BusID "PCI:0:2:0"
<br>EndSection. <br></span></span></div><div><span \
style="font-family:monospace"><span \
style="font-family:arial,sans-serif"><br></span></span></div><div><span \
style="font-family:monospace"><span style="font-family:arial,sans-serif">It's a \
confusing situation.</span> </span></div></div><br><div class="gmail_quote"><div \
dir="ltr" class="gmail_attr">Il giorno ven 3 feb 2023 alle ore 14:13 Mario Marietto \
<<a href="mailto:marietto2008@gmail.com" \
target="_blank">marietto2008@gmail.com</a>> ha scritto:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><span>I've sent an email \
to the hackers ML with this content \
:</span></div><div><span><br></span></div><div><span>> On 30 Jan 2023, at 05:37, \
Mario Marietto <<a href="mailto:marietto2008@gmail.com" \
target="_blank">marietto2008@gmail.com</a>> \
wrote:</span></div><div><span></span></div><span> > In some FreeBSD 13.1 machines \
I have the problem below,in some others I don't have it. I would like to know \
what the causes could be and how to fix it. <br>
> <br>
> root@marietto:/usr/home/marietto # devctl detach pci0:1:0:0 <br>
> <br>
> devctl: Failed to detach pci0:1:0:0: Device busy<br>
> <br>
> root@marietto:/usr/home/marietto # devctl detach pci0:2:0:0 <br>
> <br>
> devctl: Failed to detach pci0:2:0:0: Device busy <br>
> <br>
> Not even it works if instead of detach them, I try to attach them directly to \
the ppt driver :<br> > <br>
> root@marietto:/usr/home/marietto/bhyve/Files # devctl set driver pci0:2:0:0 ppt \
<br> > devctl: Failed to set pci0:2:0:0 driver to ppt: Device busy<br>
> <br>
> root@marietto:/usr/home/marietto/bhyve/Files # devctl detach pci0:2:0:0 <br>
> devctl: Failed to detach pci0:2:0:0: Device busy<br>
> <br>
> Pci addresses 1:0:0 and 2:0:0 belong to the two GPUs that I have on my PC :<br>
> <br>
> vgapci0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de \
device=0x1c02 subvendor=0x19da subdevice=0x2438 <br> > vgapci1@pci0:2:0:0: \
class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e04 subvendor=0x19da \
subdevice=0x2503<br> > <br>
> Actually I have commented this line on /boot/loader.conf,because it
makes no difference if I keep it uncommented or not. It is totally
ignored.<br>
> <br>
> #pptdevs="1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3" \
<br></span><div><br></div><div>and George replied to me like this \
:</div><div><br></div> Do you have 'vmm_load="YES"' in \
loader.conf?<br> <br><div>
If not I am not sure it will have an effect - vmm must be loaded early
otherwise the card will be grabbed by the vgapci driver rather than be
put under ppt control.<span><br></span></div><div><span><br></span></div><div><span>I \
think this is an interesting answer. Yes,I keep vmm_load="YES" in \
loader.conf,BUT putting the 4 slots under ppt control using the loader.conf did not \
work for me for months. We haven't taken this into consideration. And you \
didn't reply when I've asked you if you have modified the bhyve source \
code,because putting the pci addresses under ppt using the loader.conf always worked \
for me. I would like to understand why it does not work anymore. This is important to \
understand why I'm not able to attach / detach the nVidia driver from the host to \
the guest os. And maybe George's consideration is relevant. \
<br></span></div><div><span></span></div></div><br><div class="gmail_quote"><div \
dir="ltr" class="gmail_attr">Il giorno ven 3 feb 2023 alle ore 12:11 Corvin Köhne \
<<a href="mailto:C.Koehne@beckhoff.com" \
target="_blank">C.Koehne@beckhoff.com</a>> ha scritto:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div>
<div lang="EN-US">
<div id="m_-3276811306036492484m_-3210909820163076187m_2261178785101495437m_-1035847088375675656m_144419679703708073OLX_DISCLAIMER_MARKER" \
style="line-height:0px"> </div>
<div>
<div>
<div>
<div>
<ul style="margin-top:0in" type="disc">
<li style="margin-left:0in">---> Figure out if you can blacklist your gpu in the \
nvidia driver. So, the nVidia driver never gets attached to your \
device.<u></u><u></u></li></ul> </div>
<div>
<ul style="margin-top:0in" type="disc">
<li style="margin-left:0in">nope. I need to attach the nVidia driver to my gpu if I \
want to run stable diffusion + pytorch + cuda on the Linuxulator on FreeBSD. I'm \
thinking of installing Stable Diffusion within the Linux VM,instead of doing it on \
FreeBSD.<u></u><u></u></li></ul> </div>
<div>
<ul style="margin-top:0in" type="disc">
<li style="margin-left:0in">---> Wait for drm drivers and use the nouveau driver. \
Nouveau isn't maintained by nVidia, so it's more likely to get \
fixed.<u></u><u></u></li></ul> </div>
<div>
<ul style="margin-top:0in" type="disc">
<li style="margin-left:0in">The nouveau driver is always bugged and it never worked \
well. I wonder if it is still supported. It has even limited \
features.<u></u><u></u></li></ul> </div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Afaik, you don't have other options, sry. I'm no gpu driver \
developer, so I can't help you.<u></u><u></u></p> <p class="MsoNormal">Btw: Do you \
refer to linux nouveau or freebsd nouveau? Freebsd made much progress in porting \
linux drm to freebsd with 14.0. So, if latest linux nouveau fits your requirements, \
it's likely that freebsd 14.0 may fit them too.<u></u><u></u></p> <p \
class="MsoNormal"><u></u> <u></u></p> </div>
<div>
<ul style="margin-top:0in" type="disc">
<li style="margin-left:0in">Are the following commands enough to update the bhyve \
source code on my machine ? take in consideration that I need to grab only your new \
last patch.<u></u><u></u></li></ul> </div>
<div>
<ul style="margin-top:0in" type="disc">
<li style="margin-left:0in">cd /usr/src<u></u><u></u></li></ul>
</div>
<div>
<ul style="margin-top:0in" type="disc">
<li style="margin-left:0in">git checkout -f \
origin/phab/corvink/13.1/nvidia-wip<u></u><u></u></li></ul> </div>
<ul style="margin-top:0in" type="disc">
<li style="margin-left:0in"># cd \
/wherever/you/put/my/Makefile<u></u><u></u></li></ul> <p class="MsoNormal" \
style="margin-left:0.5in"># make -j 8<u></u><u></u></p> <p class="MsoNormal"><u></u> \
<u></u></p> <p class="MsoNormal">My latest patch is a mouse driver patch for the \
host. So, no, you have to use:<u></u><u></u></p> <p class="MsoNormal">cd \
/usr/src<u></u><u></u></p> <p class="MsoNormal">git fetch<u></u><u></u></p>
<p class="MsoNormal">git checkout -f \
origin/phab/corvink/13.1/nvidia-wip<u></u><u></u></p> <p class="MsoNormal">make \
kernel<u></u><u></u></p> <p class="MsoNormal">reboot<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Best regards<u></u><u></u></p>
<p class="MsoNormal">Corvin<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<br>
<span style="font-family:Arial,Helvetica,sans-serif;font-size:8pt">This email \
contains confidential information. If you have received it in error, you must not \
read, use, copy or pass on this e-mail or its attachments.<br> If you have received \
the e-mail in error, please inform me immediately by reply e-mail and then delete \
this e-mail from your system. Thank you<br> <br>
Diese E-Mail enthaelt vertrauliche Informationen. Sollten Sie sie irrtuemlich \
erhalten haben, duerfen Sie diese E-Mail oder ihre Anhaenge nicht lesen, verwenden, \
kopieren oder weitergeben.<br> Sollten Sie die Mail versehentlich erhalten haben, \
teilen Sie mir dies bitte umgehend per Antwort-E-Mail mit und loeschen Sie diese \
E-Mail dann aus Ihrem System. Vielen Dank<br> <br>
Beckhoff Automation GmbH & Co. KG | Managing Director: Dipl. Phys. Hans \
Beckhoff<br> Registered office: Verl, Germany | Register court: Guetersloh HRA \
7075</span> <br> <br>
</div>
</div></blockquote></div><br clear="all"><br>-- <br><div dir="ltr">Mario.<br></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr">Mario.<br></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr">Mario.<br></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic