[prev in list] [next in list] [prev in thread] [next in thread]
List: dpdk-users
Subject: DPDK occurs "failed to allocate private data" in rte_eal_init-a possibly unknown dpdk bug?
From: "=?gb18030?B?bGFubmlzdGVy?=" <2311041380 () qq ! com>
Date: 2022-10-15 15:38:38
Message-ID: tencent_E492CC2A74886E64191F8F29C69C64574009 () qq ! com
[Download RAW message or body]
[Attachment #2 (text/plain)]
When I searched for this problem, I found that almost no one else encountered \
such a problem.It's strange.By the way, I use dpdk-19.11.12 and Ubuntu-20.04
After looking at the dpdk source code, I know that the above code is in \
the rte_eth_dev_create function: if (priv_data_size) { \
ethdev->data->dev_private = **rte_zmalloc_socket**( name, \
priv_data_size, RTE_CACHE_LINE_SIZE, device->numa_node); if \
(!ethdev->data->dev_private) { RTE_LOG(ERR, EAL, "failed to \
allocate private data"); retval = -ENOMEM; goto probe_failed; \
} } It seems rte_zmalloc_socket return a NULL pointer. Why this \
happens?I allocated the relevant hugepage memory as requested.
Some Information£º
EAL: Detected 4 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: \
Selected IOVA mode 'PA' EAL: No available hugepages reported in hugepages-2048kB EAL: \
Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:31:00.0 on \
NUMA socket 3 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device \
0000:31:00.1 on NUMA socket 3 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI \
device 0000:51:00.0 on NUMA socket 5 EAL: probe driver: 8086:10fb net_ixgbe failed \
to allocate private data EAL: Requested device 0000:51:00.0 cannot be used EAL: PCI \
device 0000:51:00.1 on NUMA socket 5 EAL: probe driver: 8086:10fb net_ixgbe failed \
to allocate private data EAL: Requested device 0000:51:00.1 cannot be used
Hugepages£º (cat /proc/meminfo | grep Huge)
AnonHugePages: 0 kB ShmemHugePages: 0 kB FileHugePages: 0 kB \
HugePages_Total: 20 HugePages_Free: 19 HugePages_Rsvd: 0 \
HugePages_Surp: 0 Hugepagesize: 1048576 kB Hugetlb: 20971520 kB
and mount information: (mount | grep huge)
cgroup on /sys/fs/cgroup/hugetlb type cgroup \
(rw,nosuid,nodev,noexec,relatime,hugetlb) hugetlbfs on /dev/hugepages type hugetlbfs \
(rw,relatime,pagesize=1024M) nodev on /mnt/huge type hugetlbfs \
(rw,relatime,pagesize=1024M)
NIC:
Network devices using DPDK-compatible driver \
============================================ 0000:51:00.0 '82599ES 10-Gigabit \
SFI/SFP+ Network Connection 10fb' drv=igb_uio unused=ixgbe,vfio-pci 0000:51:00.1 \
'82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio \
unused=ixgbe,vfio-pci Network devices using kernel driver \
=================================== 0000:31:00.0 'I350 Gigabit Network Connection \
1521' if=enp49s0f0 drv=igb unused=igb_uio,vfio-pci 0000:31:00.1 'I350 Gigabit \
Network Connection 1521' if=enp49s0f1 drv=igb unused=igb_uio,vfio-pci *Active* Numa \
information:(numactl -H) node 5 cpus: 30 31 32 33 34 35 78 79 80 81 82 83 node 5 \
size: 64475 MB node 5 free: 58743 MB 82599ES NICs are on numa node 5.
I thought I did all the initialization but rte_eal_init returned the error "failed to \
allocate private data"
Any ideas on this issue? Thanks for help.
-----------------------update-----------------------------------------
Since this problem occurs in rte_zmalloc_socket(),I guess that there is something \
wrong with my configuration in Hugepages.But as posted above,i check Hugepages with \
command:
1.cat /proc/meminfo | grep Huge 2.mount | grep huge
first command checks available 1G Hugepages and second checks mount situation.
Result above seems to be normal which confuses me most.
Any clues?Thanks.
----------------------------update------------------------------------------
After using debug mode, I seem to find why this error occurs,but still got confused:
Pay attention to first parameter of rte_malloc_socket and rte_zmalloc_socket:
#0 rte_malloc_socket (type=0x17ffb0440 <error: Cannot access memory at address \
0x17ffb0440>, size=93825001207904, align=96, socket_arg=2147157000) at \
../lib/librte_eal/common/rte_malloc.c:46 #1 0x00007ffff7bdc79b in rte_zmalloc_socket \
(type=0x555555de3860 "0000:51:00.0", size=30552, align=64, socket=5) at \
../lib/librte_eal/common/rte_malloc.c:79 #2 0x00007ffff7cebc08 in rte_eth_dev_create \
(device=0x555555ddcf50, name=0x555555de3860 "0000:51:00.0", priv_data_size=30552, \
ethdev_bus_specific_init=0x7ffff59c74b5 <eth_dev_pci_specific_init>, \
bus_init_params=0x555555ddcf40, ethdev_init=0x7ffff59c8ed0 <eth_ixgbe_dev_init>, \
init_params=0x0) at ../lib/librte_ethdev/rte_ethdev.c:4279
And in dpdk source code:
/* * Allocate zero'd memory on specified heap. */ void * rte_zmalloc_socket(const \
char *type, size_t size, unsigned align, int socket) { void *ptr = \
rte_malloc_socket(type, size, align, socket); if (ptr != NULL) memset(ptr, \
0, size); return ptr; }
first parameter of rte_malloc_socket and rte_zmalloc_socket should be same, but as \
shown above, this does not happen in my case
other parameters seems to have same question!!
This confuses me, what appened?And is what happened above the cause of the error?
How can i fix it?Any clues?
It seems we need a dpdk developer's help:)
same question posted in stack overflow.
Thanks.
lannister
2311041380@qq.com
[Attachment #3 (text/html)]
<meta http-equiv="Content-Type" content="text/html; \
charset=GB18030"><div><br></div><div><p style="margin-top: 0px; margin-right: 0px; \
margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">When I searched for this problem, I found that \
almost <strong style="margin: 0px; padding: 0px; border: 0px; font-style: \
inherit; font-variant: inherit; font-stretch: inherit; line-height: inherit; \
font-family: inherit; vertical-align: baseline; box-sizing: inherit;">no one else \
encountered such a problem</strong>.It's strange.By the way, I use dpdk-19.11.12 and \
Ubuntu-20.04</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: \
var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);"><br></p><p style="margin-top: 0px; margin-right: 0px; \
margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">After looking at the dpdk source code, I know that the \
above code is in the <strong style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-stretch: inherit; line-height: \
inherit; font-family: inherit; vertical-align: baseline; box-sizing: \
inherit;">rte_eth_dev_create</strong> function:</p><pre style="margin-top: 0px; \
margin-bottom: calc(var(--s-prose-spacing) + 0.4em); padding: var(--su12); border: \
0px; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: var(--lh-md); font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; width: auto; \
max-height: 600px; overflow: auto; background-color: var(--highlight-bg); \
border-radius: var(--br-md); overflow-wrap: normal; color: \
var(--highlight-color);"><code style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: \
inherit; line-height: inherit; font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; background-color: \
transparent; white-space: inherit; color: var(--black-800); border-radius: 0px;"> \
</code></pre><pre style="margin-top: 0px; margin-bottom: calc(var(--s-prose-spacing) \
+ 0.4em); padding: var(--su12); border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: var(--lh-md); \
font-family: var(--ff-mono); font-size: var(--fs-body1); vertical-align: baseline; \
box-sizing: inherit; width: auto; max-height: 600px; overflow: auto; \
background-color: var(--highlight-bg); border-radius: var(--br-md); overflow-wrap: \
normal; color: var(--highlight-color);"><code style="margin: 0px; padding: 0px; \
border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; \
font-stretch: inherit; line-height: inherit; font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; background-color: \
transparent; white-space: inherit; color: var(--black-800); border-radius: 0px;"> \
if (priv_data_size) { ethdev->data->dev_private = **rte_zmalloc_socket**(
name, priv_data_size, RTE_CACHE_LINE_SIZE,
device->numa_node);
if (!ethdev->data->dev_private) {
RTE_LOG(ERR, EAL, "failed to allocate private data");
retval = -ENOMEM;
goto probe_failed;
}
}
</code></pre><p style="margin-top: 0px; margin-right: 0px; margin-bottom: \
var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">It seems <strong style="margin: 0px; padding: \
0px; border: 0px; font-style: inherit; font-variant: inherit; font-stretch: inherit; \
line-height: inherit; font-family: inherit; vertical-align: baseline; box-sizing: \
inherit;">rte_zmalloc_socket</strong> return a NULL pointer. Why this happens?I \
allocated the relevant hugepage memory as requested.</p><p style="margin-top: 0px; \
margin-right: 0px; margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: \
0px; border: 0px; font-variant-numeric: inherit; font-variant-east-asian: inherit; \
font-stretch: inherit; line-height: inherit; font-family: -apple-system, \
BlinkMacSystemFont, "Segoe UI Adjusted", "Segoe UI", \
"Liberation Sans", sans-serif; font-size: 15px; vertical-align: baseline; \
box-sizing: inherit; clear: both; color: rgb(35, 38, 41);"><br></p><p \
style="margin-top: 0px; margin-right: 0px; margin-bottom: var(--s-prose-spacing); \
margin-left: 0px; padding: 0px; border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: inherit; \
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI Adjusted", \
"Segoe UI", "Liberation Sans", sans-serif; font-size: 15px; \
vertical-align: baseline; box-sizing: inherit; clear: both; color: rgb(35, 38, \
41);">Some Information£º</p><pre style="margin-top: 0px; margin-bottom: \
calc(var(--s-prose-spacing) + 0.4em); padding: var(--su12); border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: var(--lh-md); font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; width: auto; \
max-height: 600px; overflow: auto; background-color: var(--highlight-bg); \
border-radius: var(--br-md); overflow-wrap: normal; color: \
var(--highlight-color);"><code style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: \
inherit; line-height: inherit; font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; background-color: \
transparent; white-space: inherit; color: var(--black-800); border-radius: 0px;">EAL: \
Detected 4 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:31:00.0 on NUMA socket 3
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:31:00.1 on NUMA socket 3
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:51:00.0 on NUMA socket 5
EAL: probe driver: 8086:10fb net_ixgbe
failed to allocate private data
EAL: Requested device 0000:51:00.0 cannot be used
EAL: PCI device 0000:51:00.1 on NUMA socket 5
EAL: probe driver: 8086:10fb net_ixgbe
failed to allocate private data
EAL: Requested device 0000:51:00.1 cannot be used
</code></pre><p style="margin-top: 0px; margin-right: 0px; margin-bottom: \
var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);"><strong style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-stretch: inherit; line-height: \
inherit; font-family: inherit; vertical-align: baseline; box-sizing: \
inherit;"><br></strong></p><p style="margin-top: 0px; margin-right: 0px; \
margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);"><strong style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-stretch: inherit; line-height: \
inherit; font-family: inherit; vertical-align: baseline; box-sizing: \
inherit;">Hugepages£º</strong> (cat /proc/meminfo | grep Huge)</p><pre \
style="margin-top: 0px; margin-bottom: calc(var(--s-prose-spacing) + 0.4em); padding: \
var(--su12); border: 0px; font-variant-numeric: inherit; font-variant-east-asian: \
inherit; font-stretch: inherit; line-height: var(--lh-md); font-family: \
var(--ff-mono); font-size: var(--fs-body1); vertical-align: baseline; box-sizing: \
inherit; width: auto; max-height: 600px; overflow: auto; background-color: \
var(--highlight-bg); border-radius: var(--br-md); overflow-wrap: normal; color: \
var(--highlight-color);"><code style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: \
inherit; line-height: inherit; font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; background-color: \
transparent; white-space: inherit; color: var(--black-800); border-radius: \
0px;">AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 20
HugePages_Free: 19
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
Hugetlb: 20971520 kB
</code></pre><p style="margin-top: 0px; margin-right: 0px; margin-bottom: \
var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);"><strong style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-stretch: inherit; line-height: \
inherit; font-family: inherit; vertical-align: baseline; box-sizing: \
inherit;"><br></strong></p><p style="margin-top: 0px; margin-right: 0px; \
margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);"><strong style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-stretch: inherit; line-height: \
inherit; font-family: inherit; vertical-align: baseline; box-sizing: inherit;">and \
mount information:</strong> (mount | grep huge)</p><pre style="margin-top: 0px; \
margin-bottom: calc(var(--s-prose-spacing) + 0.4em); padding: var(--su12); border: \
0px; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: var(--lh-md); font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; width: auto; \
max-height: 600px; overflow: auto; background-color: var(--highlight-bg); \
border-radius: var(--br-md); overflow-wrap: normal; color: \
var(--highlight-color);"><code style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: \
inherit; line-height: inherit; font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; background-color: \
transparent; white-space: inherit; color: var(--black-800); border-radius: \
0px;">cgroup on /sys/fs/cgroup/hugetlb type cgroup \
(rw,nosuid,nodev,noexec,relatime,hugetlb) hugetlbfs on /dev/hugepages type hugetlbfs \
(rw,relatime,pagesize=1024M) nodev on /mnt/huge type hugetlbfs \
(rw,relatime,pagesize=1024M) </code></pre><p style="margin-top: 0px; margin-right: \
0px; margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: \
0px; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);"><strong style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-stretch: inherit; line-height: \
inherit; font-family: inherit; vertical-align: baseline; box-sizing: \
inherit;"><br></strong></p><p style="margin-top: 0px; margin-right: 0px; \
margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);"><strong style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-stretch: inherit; line-height: \
inherit; font-family: inherit; vertical-align: baseline; box-sizing: \
inherit;">NIC:</strong></p><pre style="margin-top: 0px; margin-bottom: \
calc(var(--s-prose-spacing) + 0.4em); padding: var(--su12); border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: var(--lh-md); font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; width: auto; \
max-height: 600px; overflow: auto; background-color: var(--highlight-bg); \
border-radius: var(--br-md); overflow-wrap: normal; color: \
var(--highlight-color);"><code style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: \
inherit; line-height: inherit; font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; background-color: \
transparent; white-space: inherit; color: var(--black-800); border-radius: \
0px;">Network devices using DPDK-compatible driver \
============================================ 0000:51:00.0 '82599ES 10-Gigabit \
SFI/SFP+ Network Connection 10fb' drv=igb_uio unused=ixgbe,vfio-pci 0000:51:00.1 \
'82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio \
unused=ixgbe,vfio-pci
Network devices using kernel driver
===================================
0000:31:00.0 'I350 Gigabit Network Connection 1521' if=enp49s0f0 drv=igb \
unused=igb_uio,vfio-pci 0000:31:00.1 'I350 Gigabit Network Connection 1521' \
if=enp49s0f1 drv=igb unused=igb_uio,vfio-pci *Active* </code></pre><p \
style="margin-top: 0px; margin-right: 0px; margin-bottom: var(--s-prose-spacing); \
margin-left: 0px; padding: 0px; border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: inherit; \
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI Adjusted", \
"Segoe UI", "Liberation Sans", sans-serif; font-size: 15px; \
vertical-align: baseline; box-sizing: inherit; clear: both; color: rgb(35, 38, \
41);"><strong style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; \
font-variant: inherit; font-stretch: inherit; line-height: inherit; font-family: \
inherit; vertical-align: baseline; box-sizing: inherit;">Numa \
information:</strong>(numactl -H)</p><pre style="margin-top: 0px; margin-bottom: \
calc(var(--s-prose-spacing) + 0.4em); padding: var(--su12); border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: var(--lh-md); font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; width: auto; \
max-height: 600px; overflow: auto; background-color: var(--highlight-bg); \
border-radius: var(--br-md); overflow-wrap: normal; color: \
var(--highlight-color);"><code style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: \
inherit; line-height: inherit; font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; background-color: \
transparent; white-space: inherit; color: var(--black-800); border-radius: 0px;">node \
5 cpus: 30 31 32 33 34 35 78 79 80 81 82 83 node 5 size: 64475 MB
node 5 free: 58743 MB
</code></pre><p style="margin-top: 0px; margin-right: 0px; margin-bottom: \
var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">82599ES NICs are on numa node 5.</p><p \
style="margin-top: 0px; margin-right: 0px; margin-bottom: var(--s-prose-spacing); \
margin-left: 0px; padding: 0px; border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: inherit; \
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI Adjusted", \
"Segoe UI", "Liberation Sans", sans-serif; font-size: 15px; \
vertical-align: baseline; box-sizing: inherit; clear: both; color: rgb(35, 38, \
41);">I thought I did all the initialization but rte_eal_init returned the error \
"failed to allocate private data"</p><p style="margin-top: 0px; margin-right: 0px; \
margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">Any ideas on this issue? Thanks for help.</p><p \
style="margin-top: 0px; margin-right: 0px; margin-bottom: var(--s-prose-spacing); \
margin-left: 0px; padding: 0px; border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: inherit; \
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI Adjusted", \
"Segoe UI", "Liberation Sans", sans-serif; font-size: 15px; \
vertical-align: baseline; box-sizing: inherit; clear: both; color: rgb(35, 38, \
41);">-----------------------update-----------------------------------------</p><p \
style="margin-top: 0px; margin-right: 0px; margin-bottom: var(--s-prose-spacing); \
margin-left: 0px; padding: 0px; border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: inherit; \
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI Adjusted", \
"Segoe UI", "Liberation Sans", sans-serif; font-size: 15px; \
vertical-align: baseline; box-sizing: inherit; clear: both; color: rgb(35, 38, \
41);">Since this problem occurs in rte_zmalloc_socket(),I guess that there is \
something wrong with my configuration in Hugepages.But as posted above,i check \
Hugepages with command:</p><p style="margin-top: 0px; margin-right: 0px; \
margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);"><br></p><pre style="margin-top: 0px; margin-bottom: \
calc(var(--s-prose-spacing) + 0.4em); padding: var(--su12); border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: var(--lh-md); font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; width: auto; \
max-height: 600px; overflow: auto; background-color: var(--highlight-bg); \
border-radius: var(--br-md); overflow-wrap: normal; color: \
var(--highlight-color);"><code style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: \
inherit; line-height: inherit; font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; background-color: \
transparent; white-space: inherit; color: var(--black-800); border-radius: \
0px;">1.cat /proc/meminfo | grep Huge 2.mount | grep huge
</code></pre><pre style="margin-top: 0px; margin-bottom: calc(var(--s-prose-spacing) \
+ 0.4em); padding: var(--su12); border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: var(--lh-md); \
font-family: var(--ff-mono); font-size: var(--fs-body1); vertical-align: baseline; \
box-sizing: inherit; width: auto; max-height: 600px; overflow: auto; \
background-color: var(--highlight-bg); border-radius: var(--br-md); overflow-wrap: \
normal; color: var(--highlight-color);"><code style="margin: 0px; padding: 0px; \
border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; \
font-stretch: inherit; line-height: inherit; font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; background-color: \
transparent; white-space: inherit; color: var(--black-800); border-radius: \
0px;"><br></code></pre><p style="margin-top: 0px; margin-right: 0px; margin-bottom: \
var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">first command checks available 1G Hugepages and second \
checks mount situation.</p><p style="margin-top: 0px; margin-right: 0px; \
margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">Result above seems to be normal which confuses me \
most.</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: \
var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">Any clues?Thanks.</p><p style="margin-top: 0px; \
margin-right: 0px; margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: \
0px; border: 0px; font-variant-numeric: inherit; font-variant-east-asian: inherit; \
font-stretch: inherit; line-height: inherit; font-family: -apple-system, \
BlinkMacSystemFont, "Segoe UI Adjusted", "Segoe UI", \
"Liberation Sans", sans-serif; font-size: 15px; vertical-align: baseline; \
box-sizing: inherit; clear: both; color: rgb(35, 38, \
41);">----------------------------update------------------------------------------</p><p \
style="margin-top: 0px; margin-right: 0px; margin-bottom: var(--s-prose-spacing); \
margin-left: 0px; padding: 0px; border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: inherit; \
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI Adjusted", \
"Segoe UI", "Liberation Sans", sans-serif; font-size: 15px; \
vertical-align: baseline; box-sizing: inherit; clear: both; color: rgb(35, 38, \
41);">After using debug mode, I seem to find why this error occurs,but still got \
confused:</p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: \
var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">Pay attention to <strong style="margin: 0px; \
padding: 0px; border: 0px; font-style: inherit; font-variant: inherit; font-stretch: \
inherit; line-height: inherit; font-family: inherit; vertical-align: baseline; \
box-sizing: inherit;">first parameter of rte_malloc_socket and \
rte_zmalloc_socket</strong>:</p><p style="margin-top: 0px; margin-right: 0px; \
margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);"><br></p><pre style="margin-top: 0px; margin-bottom: \
calc(var(--s-prose-spacing) + 0.4em); padding: var(--su12); border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: var(--lh-md); font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; width: auto; \
max-height: 600px; overflow: auto; background-color: var(--highlight-bg); \
border-radius: var(--br-md); overflow-wrap: normal; color: \
var(--highlight-color);"><code style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: \
inherit; line-height: inherit; font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; background-color: \
transparent; white-space: inherit; color: var(--black-800); border-radius: 0px;">#0 \
rte_malloc_socket (type=0x17ffb0440 <error: Cannot access memory at address \
0x17ffb0440>, size=93825001207904, align=96, socket_arg=2147157000) at \
../lib/librte_eal/common/rte_malloc.c:46
#1 0x00007ffff7bdc79b in rte_zmalloc_socket (type=0x555555de3860 "0000:51:00.0", \
size=30552, align=64, socket=5) at ../lib/librte_eal/common/rte_malloc.c:79
#2 0x00007ffff7cebc08 in rte_eth_dev_create (device=0x555555ddcf50, \
name=0x555555de3860 "0000:51:00.0", priv_data_size=30552, \
ethdev_bus_specific_init=0x7ffff59c74b5 <eth_dev_pci_specific_init>, \
bus_init_params=0x555555ddcf40, ethdev_init=0x7ffff59c8ed0 \
<eth_ixgbe_dev_init>, init_params=0x0) at \
../lib/librte_ethdev/rte_ethdev.c:4279 </code></pre><pre style="margin-top: 0px; \
margin-bottom: calc(var(--s-prose-spacing) + 0.4em); padding: var(--su12); border: \
0px; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: var(--lh-md); font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; width: auto; \
max-height: 600px; overflow: auto; background-color: var(--highlight-bg); \
border-radius: var(--br-md); overflow-wrap: normal; color: \
var(--highlight-color);"><code style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: \
inherit; line-height: inherit; font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; background-color: \
transparent; white-space: inherit; color: var(--black-800); border-radius: \
0px;"><br></code></pre><p style="margin-top: 0px; margin-right: 0px; margin-bottom: \
var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">And in dpdk source code:</p><pre style="margin-top: \
0px; margin-bottom: calc(var(--s-prose-spacing) + 0.4em); padding: var(--su12); \
border: 0px; font-variant-numeric: inherit; font-variant-east-asian: inherit; \
font-stretch: inherit; line-height: var(--lh-md); font-family: var(--ff-mono); \
font-size: var(--fs-body1); vertical-align: baseline; box-sizing: inherit; width: \
auto; max-height: 600px; overflow: auto; background-color: var(--highlight-bg); \
border-radius: var(--br-md); overflow-wrap: normal; color: \
var(--highlight-color);"><code style="margin: 0px; padding: 0px; border: 0px; \
font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: \
inherit; line-height: inherit; font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; background-color: \
transparent; white-space: inherit; color: var(--black-800); \
border-radius: 0px;">/*
* Allocate zero'd memory on specified heap.
*/
void *
rte_zmalloc_socket(const char *type, size_t size, unsigned align, int socket)
{
void *ptr = rte_malloc_socket(type, size, align, socket);
if (ptr != NULL)
memset(ptr, 0, size);
return ptr;
}
</code></pre><pre style="margin-top: 0px; margin-bottom: calc(var(--s-prose-spacing) \
+ 0.4em); padding: var(--su12); border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: var(--lh-md); \
font-family: var(--ff-mono); font-size: var(--fs-body1); vertical-align: baseline; \
box-sizing: inherit; width: auto; max-height: 600px; overflow: auto; \
background-color: var(--highlight-bg); border-radius: var(--br-md); overflow-wrap: \
normal; color: var(--highlight-color);"><code style="margin: 0px; padding: 0px; \
border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; \
font-stretch: inherit; line-height: inherit; font-family: var(--ff-mono); font-size: \
var(--fs-body1); vertical-align: baseline; box-sizing: inherit; background-color: \
transparent; white-space: inherit; color: var(--black-800); border-radius: \
0px;"><br></code></pre><p style="margin-top: 0px; margin-right: 0px; margin-bottom: \
var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">first parameter of rte_malloc_socket and \
rte_zmalloc_socket should be same, but as shown above, <strong style="margin: \
0px; padding: 0px; border: 0px; font-style: inherit; font-variant: inherit; \
font-stretch: inherit; line-height: inherit; font-family: inherit; vertical-align: \
baseline; box-sizing: inherit;">this does not happen in my case</strong></p><p \
style="margin-top: 0px; margin-right: 0px; margin-bottom: var(--s-prose-spacing); \
margin-left: 0px; padding: 0px; border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: inherit; \
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI Adjusted", \
"Segoe UI", "Liberation Sans", sans-serif; font-size: 15px; \
vertical-align: baseline; box-sizing: inherit; clear: both; color: rgb(35, 38, \
41);"><br></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: \
var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">other parameters seems to have same question!!</p><p \
style="margin-top: 0px; margin-right: 0px; margin-bottom: var(--s-prose-spacing); \
margin-left: 0px; padding: 0px; border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: inherit; \
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI Adjusted", \
"Segoe UI", "Liberation Sans", sans-serif; font-size: 15px; \
vertical-align: baseline; box-sizing: inherit; clear: both; color: rgb(35, 38, \
41);"><br></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: \
var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">This confuses me, what appened?And is what happened \
above the cause of the error?</p><p style="margin-top: 0px; margin-right: 0px; \
margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);"><br></p><p style="margin-top: 0px; margin-right: 0px; \
margin-bottom: var(--s-prose-spacing); margin-left: 0px; padding: 0px; border: 0px; \
font-variant-numeric: inherit; font-variant-east-asian: inherit; font-stretch: \
inherit; line-height: inherit; font-family: -apple-system, BlinkMacSystemFont, \
"Segoe UI Adjusted", "Segoe UI", "Liberation Sans", \
sans-serif; font-size: 15px; vertical-align: baseline; box-sizing: inherit; clear: \
both; color: rgb(35, 38, 41);">How can i fix it?Any clues?</p><p style="margin: 0px; \
padding: 0px; border: 0px; font-variant-numeric: inherit; font-variant-east-asian: \
inherit; font-stretch: inherit; line-height: inherit; font-family: -apple-system, \
BlinkMacSystemFont, "Segoe UI Adjusted", "Segoe UI", \
"Liberation Sans", sans-serif; font-size: 15px; vertical-align: baseline; \
box-sizing: inherit; clear: both; color: rgb(35, 38, 41);"><br></p><p style="margin: \
0px; padding: 0px; border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: inherit; \
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI Adjusted", \
"Segoe UI", "Liberation Sans", sans-serif; font-size: 15px; \
vertical-align: baseline; box-sizing: inherit; clear: both; color: rgb(35, 38, \
41);">It seems we need a dpdk developer's help:)</p><p style="margin: 0px; padding: \
0px; border: 0px; font-variant-numeric: inherit; font-variant-east-asian: inherit; \
font-stretch: inherit; line-height: inherit; font-family: -apple-system, \
BlinkMacSystemFont, "Segoe UI Adjusted", "Segoe UI", \
"Liberation Sans", sans-serif; font-size: 15px; vertical-align: baseline; \
box-sizing: inherit; clear: both; color: rgb(35, 38, 41);"><br></p><p style="margin: \
0px; padding: 0px; border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: inherit; \
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI Adjusted", \
"Segoe UI", "Liberation Sans", sans-serif; font-size: 15px; \
vertical-align: baseline; box-sizing: inherit; clear: both; color: rgb(35, 38, \
41);">same question posted in stack overflow.</p><p style="margin: 0px; padding: \
0px; border: 0px; font-variant-numeric: inherit; font-variant-east-asian: inherit; \
font-stretch: inherit; line-height: inherit; font-family: -apple-system, \
BlinkMacSystemFont, "Segoe UI Adjusted", "Segoe UI", \
"Liberation Sans", sans-serif; font-size: 15px; vertical-align: baseline; \
box-sizing: inherit; clear: both; color: rgb(35, 38, 41);"><br></p><p style="margin: \
0px; padding: 0px; border: 0px; font-variant-numeric: inherit; \
font-variant-east-asian: inherit; font-stretch: inherit; line-height: inherit; \
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI Adjusted", \
"Segoe UI", "Liberation Sans", sans-serif; font-size: 15px; \
vertical-align: baseline; box-sizing: inherit; clear: both; color: rgb(35, 38, \
41);">Thanks.</p></div><div><hr align="left" style="margin: 0 0 10px 0;border: \
0;border-bottom:1px solid #E4E5E6;height:0;line-height:0;font-size:0;padding: 20px 0 \
0 0;width: 50px;"><div style="font-size:14px;font-family:Verdana;color:#000;"><a \
class="xm_write_card" id="in_alias" style="white-space: normal; display: \
inline-block; text-decoration: none !important;font-family: \
-apple-system,BlinkMacSystemFont,PingFang SC,Microsoft YaHei;" \
href="https://wx.mail.qq.com/home/index?t=readmail_businesscard_midpage&nocheck=tr \
ue&name=lannister&icon=http%3A%2F%2Fthirdqq.qlogo.cn%2Fg%3Fb%3Dsdk%26k%3DXibap \
jrPlQsA5alKp8x4yow%26s%3D100%26t%3D1557373576%3Frand%3D1636270658&mail=2311041380%40qq.com&code=" \
target="_blank"><table style="white-space: normal;table-layout: fixed; padding-right: \
20px;" contenteditable="false" cellpadding="0" cellspacing="0"><tbody><tr \
valign="top"><td style="width: 40px;min-width: 40px; padding-top:10px"><div \
style="width: 38px; height: 38px; border: 1px #FFF solid; border-radius:50%; margin: \
0;vertical-align: top;box-shadow: 0 0 10px 0 rgba(127,152,178,0.14);"><img \
src="http://thirdqq.qlogo.cn/g?b=sdk&k=XibapjrPlQsA5alKp8x4yow&s=100&t=1557373576?rand=1636270658" \
style="width:100%;height:100%;border-radius:50%;pointer-events: none;"></div></td><td \
style="padding: 10px 0 8px 10px;"><div class="businessCard_name" style="font-size: \
14px;color: #33312E;line-height: 20px; padding-bottom: 2px; margin:0;font-weight: \
500;">lannister</div><div class="businessCard_mail" style="font-size: 12px;color: \
#999896;line-height: 18px; \
margin:0;">2311041380@qq.com</div></td></tr></tbody></table></a></div></div><div> </div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic