[prev in list] [next in list] [prev in thread] [next in thread]
List: oss-security
Subject: [oss-security] Xen Security Advisory 343 v4 (CVE-2020-25599) - races with evtchn_reset()
From: Xen.org security team <security () xen ! org>
Date: 2020-09-22 13:38:30
Message-ID: E1kKiV4-00042p-8K () xenbits ! xenproject ! org
[Download RAW message or body]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Xen Security Advisory CVE-2020-25599 / XSA-343
version 4
races with evtchn_reset()
UPDATES IN VERSION 4
====================
Fix build of backports of patch 3. Adjust affect versions.
Public release.
ISSUE DESCRIPTION
=================
Uses of EVTCHNOP_reset (potentially by a guest on itself) or
XEN_DOMCTL_soft_reset (by itself covered by XSA-77) can lead to the
violation of various internal assumptions. This may lead to out of
bounds memory accesses or triggering of bug checks.
IMPACT
======
In particular x86 PV guests may be able to elevate their privilege to
that of the host. Host and guest crashes are also possible, leading to
a Denial of Service (DoS). Information leaks cannot be ruled out.
VULNERABLE SYSTEMS
==================
All Xen versions from 4.5 onwards are vulnerable. Xen versions 4.4 and
earlier are not vulnerable.
MITIGATION
==========
There is no known mitigation.
CREDITS
=======
Different aspects of this issue were discovered by Julien Grall of
Amazon and by Jan Beulich of SUSE.
RESOLUTION
==========
Applying the appropriate set of attached patches resolves this issue.
Note that patches for released versions are generally prepared to
apply to the stable branches, and may not apply cleanly to the most
recent release tarball. Downstreams are encouraged to update to the
tip of the stable branch before applying these patches.
xsa343/xsa343-?.patch Xen 4.13 - xen-unstable
xsa343/xsa343-4.12-?.patch Xen 4.12
xsa343/xsa343-4.11-?.patch Xen 4.11
xsa343/xsa343-4.10-?.patch Xen 4.10
$ sha256sum xsa343* xsa343*/*
097d5fa32e22fc7a18fddd757f950699e823202bbae67245eece783d6d06f4eb xsa343.meta
d714a542bae9d96b6a061c5a8f754549d699dcfb7bf2a766b721f6bbe33aefd2 xsa343/xsa343-1.patch
657c44c8ea13523d2e59776531237bbc20166c9b7c3960e0e9ad381fce927344 xsa343/xsa343-2.patch
2b275e3fa559167c1b59e6fd4a20bc4d1df9d9cb0cbd0050a3db9c3d0299b233 xsa343/xsa343-3.patch
9aec124e2afcba57f8adaf7374ecebffc4a8ed1913512a7456f87761bb115f68 xsa343/xsa343-4.10-1.patch
54d9ce9acdb8dcc6aa81928037afbb081a6cd579127aa225833767e285e30ea2 xsa343/xsa343-4.10-2.patch
3801300cddd8d138c800dc45eeff111e313eb40cea3aa94e2e045ac8956ab9d3 xsa343/xsa343-4.10-3.patch
7abbec828f77c427a53182db820fc19bdf34e37882fc6ae51351ed6027c56da1 xsa343/xsa343-4.11-1.patch
5c90a53333e9c81ce938deddfc690f474d61e083d2a43b859d3227100f793aff xsa343/xsa343-4.11-2.patch
0e12cfe8e505b9685912c61a740b98084d62e4ba0670d51a47345739f463a039 xsa343/xsa343-4.11-3.patch
f3462b4e672f69a9fa951b1c04a50d754c64d18aadf444ef248587b3ac7f635a xsa343/xsa343-4.12-1.patch
d99cbbc3792755c4998b73460bbeaef5612a8942f98adcaea0762950e5a07c2a xsa343/xsa343-4.12-2.patch
cf23d3b61d4f07efc7057035c45e53e32a0b0f8fc3b9bc6c05f0f5bc71204914 xsa343/xsa343-4.12-3.patch
$
DEPLOYMENT DURING EMBARGO
=========================
Deployment of the patches and/or mitigations described above (or
others which are substantially similar) is permitted during the
embargo, even on public-facing systems with untrusted guest users and
administrators.
But: Distribution of updated software is prohibited (except to other
members of the predisclosure list).
Predisclosure list members who wish to deploy significantly different
patches and/or mitigations, please contact the Xen Project Security
Team.
(Note: this during-embargo deployment notice is retained in
post-embargo publicly released Xen Project advisories, even though it
is then no longer applicable. This is to enable the community to have
oversight of the Xen Project Security Team's decisionmaking.)
For more information about permissible uses of embargoed information,
consult the Xen Project community's agreed Security Policy:
http://www.xenproject.org/security-policy.html
-----BEGIN PGP SIGNATURE-----
iQFABAEBCAAqFiEEI+MiLBRfRHX6gGCng/4UyVfoK9kFAl9p/k0MHHBncEB4ZW4u
b3JnAAoJEIP+FMlX6CvZQvkIAKG74lMZpma+6A2V3eZAWUFBWeUOXwgD9+UorfvT
01zdpXBcVh5alrZ+ZSzmuAa/3tREUOnJHIHJD8fBpg+Yywy3dMduuxvv6RWzuQGj
rnf0X4LNklQYzOQXsYO5HkF6hxkPyZsgJPnOO9Zwb8zB8nqpSQkq9NMJ2Ochgfmm
+aZqWe7YXCNReBfMUzqyEpSwK2tNPNNE29IerxwEooqwUV5i9mHX3lYh1UYjwush
7OrvZzTLl8csjWIenxNuXX+STxUGdS81UDAbxEENmqSLoG1djRrUkAkCu5pNphxK
dkgAk9k8wAs2fc1BOYabCNeatZEUJ11n0dxJ7nn+AsnQ5YY=
=rqkV
-----END PGP SIGNATURE-----
["xsa343.meta" (application/octet-stream)]
["xsa343/xsa343-1.patch" (application/octet-stream)]
From: Jan Beulich <jbeulich@suse.com>
Subject: evtchn: evtchn_reset() shouldn't succeed with still-open ports
While the function closes all ports, it does so without holding any
lock, and hence racing requests may be issued causing new ports to get
opened. This would have been problematic in particular if such a newly
opened port had a port number above the new implementation limit (i.e.
when switching from FIFO to 2-level) after the reset, as prior to
"evtchn: relax port_is_valid()" this could have led to e.g.
evtchn_close()'s "BUG_ON(!port_is_valid(d2, port2))" to trigger.
Introduce a counter of active ports and check that it's (still) no
larger then the number of Xen internally used ones after obtaining the
necessary lock in evtchn_reset().
As to the access model of the new {active,xen}_evtchns fields - while
all writes get done using write_atomic(), reads ought to use
read_atomic() only when outside of a suitably locked region.
Note that as of now evtchn_bind_virq() and evtchn_bind_ipi() don't have
a need to call check_free_port().
This is part of XSA-343.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Reviewed-by: Julien Grall <jgrall@amazon.com>
---
v7: Drop optimization from evtchn_reset().
v6: Fix loop exit condition in evtchn_reset(). Use {read,write}_atomic()
also for xen_evtchns.
v5: Move increment in alloc_unbound_xen_event_channel() out of the inner
locked region.
v4: Account for Xen internal ports.
v3: Document intended access next to new struct field.
v2: Add comment to check_free_port(). Drop commented out calls.
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -188,6 +188,8 @@ int evtchn_allocate_port(struct domain *
write_atomic(&d->valid_evtchns, d->valid_evtchns + EVTCHNS_PER_BUCKET);
}
+ write_atomic(&d->active_evtchns, d->active_evtchns + 1);
+
return 0;
}
@@ -211,11 +213,26 @@ static int get_free_port(struct domain *
return -ENOSPC;
}
+/*
+ * Check whether a port is still marked free, and if so update the domain
+ * counter accordingly. To be used on function exit paths.
+ */
+static void check_free_port(struct domain *d, evtchn_port_t port)
+{
+ if ( port_is_valid(d, port) &&
+ evtchn_from_port(d, port)->state == ECS_FREE )
+ write_atomic(&d->active_evtchns, d->active_evtchns - 1);
+}
+
void evtchn_free(struct domain *d, struct evtchn *chn)
{
/* Clear pending event to avoid unexpected behavior on re-bind. */
evtchn_port_clear_pending(d, chn);
+ if ( consumer_is_xen(chn) )
+ write_atomic(&d->xen_evtchns, d->xen_evtchns - 1);
+ write_atomic(&d->active_evtchns, d->active_evtchns - 1);
+
/* Reset binding to vcpu0 when the channel is freed. */
chn->state = ECS_FREE;
chn->notify_vcpu_id = 0;
@@ -258,6 +275,7 @@ static long evtchn_alloc_unbound(evtchn_
alloc->port = port;
out:
+ check_free_port(d, port);
spin_unlock(&d->event_lock);
rcu_unlock_domain(d);
@@ -351,6 +369,7 @@ static long evtchn_bind_interdomain(evtc
bind->local_port = lport;
out:
+ check_free_port(ld, lport);
spin_unlock(&ld->event_lock);
if ( ld != rd )
spin_unlock(&rd->event_lock);
@@ -488,7 +507,7 @@ static long evtchn_bind_pirq(evtchn_bind
struct domain *d = current->domain;
struct vcpu *v = d->vcpu[0];
struct pirq *info;
- int port, pirq = bind->pirq;
+ int port = 0, pirq = bind->pirq;
long rc;
if ( (pirq < 0) || (pirq >= d->nr_pirqs) )
@@ -536,6 +555,7 @@ static long evtchn_bind_pirq(evtchn_bind
arch_evtchn_bind_pirq(d, pirq);
out:
+ check_free_port(d, port);
spin_unlock(&d->event_lock);
return rc;
@@ -1011,10 +1031,10 @@ int evtchn_unmask(unsigned int port)
return 0;
}
-
int evtchn_reset(struct domain *d)
{
unsigned int i;
+ int rc = 0;
if ( d != current->domain && !d->controller_pause_count )
return -EINVAL;
@@ -1024,7 +1044,9 @@ int evtchn_reset(struct domain *d)
spin_lock(&d->event_lock);
- if ( d->evtchn_fifo )
+ if ( d->active_evtchns > d->xen_evtchns )
+ rc = -EAGAIN;
+ else if ( d->evtchn_fifo )
{
/* Switching back to 2-level ABI. */
evtchn_fifo_destroy(d);
@@ -1033,7 +1055,7 @@ int evtchn_reset(struct domain *d)
spin_unlock(&d->event_lock);
- return 0;
+ return rc;
}
static long evtchn_set_priority(const struct evtchn_set_priority *set_priority)
@@ -1219,10 +1241,9 @@ int alloc_unbound_xen_event_channel(
spin_lock(&ld->event_lock);
- rc = get_free_port(ld);
+ port = rc = get_free_port(ld);
if ( rc < 0 )
goto out;
- port = rc;
chn = evtchn_from_port(ld, port);
rc = xsm_evtchn_unbound(XSM_TARGET, ld, chn, remote_domid);
@@ -1238,7 +1259,10 @@ int alloc_unbound_xen_event_channel(
spin_unlock(&chn->lock);
+ write_atomic(&ld->xen_evtchns, ld->xen_evtchns + 1);
+
out:
+ check_free_port(ld, port);
spin_unlock(&ld->event_lock);
return rc < 0 ? rc : port;
@@ -1314,6 +1338,7 @@ int evtchn_init(struct domain *d, unsign
return -EINVAL;
}
evtchn_from_port(d, 0)->state = ECS_RESERVED;
+ write_atomic(&d->active_evtchns, 0);
#if MAX_VIRT_CPUS > BITS_PER_LONG
d->poll_mask = xzalloc_array(unsigned long, BITS_TO_LONGS(d->max_vcpus));
@@ -1340,6 +1365,8 @@ void evtchn_destroy(struct domain *d)
for ( i = 0; port_is_valid(d, i); i++ )
evtchn_close(d, i, 0);
+ ASSERT(!d->active_evtchns);
+
clear_global_virq_handlers(d);
evtchn_fifo_destroy(d);
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -361,6 +361,16 @@ struct domain
struct evtchn **evtchn_group[NR_EVTCHN_GROUPS]; /* all other buckets */
unsigned int max_evtchn_port; /* max permitted port number */
unsigned int valid_evtchns; /* number of allocated event channels */
+ /*
+ * Number of in-use event channels. Writers should use write_atomic().
+ * Readers need to use read_atomic() only when not holding event_lock.
+ */
+ unsigned int active_evtchns;
+ /*
+ * Number of event channels used internally by Xen (not subject to
+ * EVTCHNOP_reset). Read/write access like for active_evtchns.
+ */
+ unsigned int xen_evtchns;
spinlock_t event_lock;
const struct evtchn_port_ops *evtchn_port_ops;
struct evtchn_fifo_domain *evtchn_fifo;
["xsa343/xsa343-2.patch" (application/octet-stream)]
From: Jan Beulich <jbeulich@suse.com>
Subject: evtchn: convert per-channel lock to be IRQ-safe
... in order for send_guest_{global,vcpu}_virq() to be able to make use
of it.
This is part of XSA-343.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Julien Grall <jgrall@amazon.com>
---
v6: New.
---
TBD: This is the "dumb" conversion variant. In a couple of cases the
slightly simpler spin_{,un}lock_irq() could apparently be used.
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -248,6 +248,7 @@ static long evtchn_alloc_unbound(evtchn_
int port;
domid_t dom = alloc->dom;
long rc;
+ unsigned long flags;
d = rcu_lock_domain_by_any_id(dom);
if ( d == NULL )
@@ -263,14 +264,14 @@ static long evtchn_alloc_unbound(evtchn_
if ( rc )
goto out;
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_UNBOUND;
if ( (chn->u.unbound.remote_domid = alloc->remote_dom) == DOMID_SELF )
chn->u.unbound.remote_domid = current->domain->domain_id;
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
alloc->port = port;
@@ -283,26 +284,32 @@ static long evtchn_alloc_unbound(evtchn_
}
-static void double_evtchn_lock(struct evtchn *lchn, struct evtchn *rchn)
+static unsigned long double_evtchn_lock(struct evtchn *lchn,
+ struct evtchn *rchn)
{
- if ( lchn < rchn )
+ unsigned long flags;
+
+ if ( lchn <= rchn )
{
- spin_lock(&lchn->lock);
- spin_lock(&rchn->lock);
+ spin_lock_irqsave(&lchn->lock, flags);
+ if ( lchn != rchn )
+ spin_lock(&rchn->lock);
}
else
{
- if ( lchn != rchn )
- spin_lock(&rchn->lock);
+ spin_lock_irqsave(&rchn->lock, flags);
spin_lock(&lchn->lock);
}
+
+ return flags;
}
-static void double_evtchn_unlock(struct evtchn *lchn, struct evtchn *rchn)
+static void double_evtchn_unlock(struct evtchn *lchn, struct evtchn *rchn,
+ unsigned long flags)
{
- spin_unlock(&lchn->lock);
if ( lchn != rchn )
- spin_unlock(&rchn->lock);
+ spin_unlock(&lchn->lock);
+ spin_unlock_irqrestore(&rchn->lock, flags);
}
static long evtchn_bind_interdomain(evtchn_bind_interdomain_t *bind)
@@ -312,6 +319,7 @@ static long evtchn_bind_interdomain(evtc
int lport, rport = bind->remote_port;
domid_t rdom = bind->remote_dom;
long rc;
+ unsigned long flags;
if ( rdom == DOMID_SELF )
rdom = current->domain->domain_id;
@@ -347,7 +355,7 @@ static long evtchn_bind_interdomain(evtc
if ( rc )
goto out;
- double_evtchn_lock(lchn, rchn);
+ flags = double_evtchn_lock(lchn, rchn);
lchn->u.interdomain.remote_dom = rd;
lchn->u.interdomain.remote_port = rport;
@@ -364,7 +372,7 @@ static long evtchn_bind_interdomain(evtc
*/
evtchn_port_set_pending(ld, lchn->notify_vcpu_id, lchn);
- double_evtchn_unlock(lchn, rchn);
+ double_evtchn_unlock(lchn, rchn, flags);
bind->local_port = lport;
@@ -387,6 +395,7 @@ int evtchn_bind_virq(evtchn_bind_virq_t
struct domain *d = current->domain;
int virq = bind->virq, vcpu = bind->vcpu;
int rc = 0;
+ unsigned long flags;
if ( (virq < 0) || (virq >= ARRAY_SIZE(v->virq_to_evtchn)) )
return -EINVAL;
@@ -424,14 +433,14 @@ int evtchn_bind_virq(evtchn_bind_virq_t
chn = evtchn_from_port(d, port);
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_VIRQ;
chn->notify_vcpu_id = vcpu;
chn->u.virq = virq;
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
v->virq_to_evtchn[virq] = bind->port = port;
@@ -448,6 +457,7 @@ static long evtchn_bind_ipi(evtchn_bind_
struct domain *d = current->domain;
int port, vcpu = bind->vcpu;
long rc = 0;
+ unsigned long flags;
if ( domain_vcpu(d, vcpu) == NULL )
return -ENOENT;
@@ -459,13 +469,13 @@ static long evtchn_bind_ipi(evtchn_bind_
chn = evtchn_from_port(d, port);
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_IPI;
chn->notify_vcpu_id = vcpu;
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
bind->port = port;
@@ -509,6 +519,7 @@ static long evtchn_bind_pirq(evtchn_bind
struct pirq *info;
int port = 0, pirq = bind->pirq;
long rc;
+ unsigned long flags;
if ( (pirq < 0) || (pirq >= d->nr_pirqs) )
return -EINVAL;
@@ -541,14 +552,14 @@ static long evtchn_bind_pirq(evtchn_bind
goto out;
}
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_PIRQ;
chn->u.pirq.irq = pirq;
link_pirq_port(port, chn, v);
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
bind->port = port;
@@ -569,6 +580,7 @@ int evtchn_close(struct domain *d1, int
struct evtchn *chn1, *chn2;
int port2;
long rc = 0;
+ unsigned long flags;
again:
spin_lock(&d1->event_lock);
@@ -668,14 +680,14 @@ int evtchn_close(struct domain *d1, int
BUG_ON(chn2->state != ECS_INTERDOMAIN);
BUG_ON(chn2->u.interdomain.remote_dom != d1);
- double_evtchn_lock(chn1, chn2);
+ flags = double_evtchn_lock(chn1, chn2);
evtchn_free(d1, chn1);
chn2->state = ECS_UNBOUND;
chn2->u.unbound.remote_domid = d1->domain_id;
- double_evtchn_unlock(chn1, chn2);
+ double_evtchn_unlock(chn1, chn2, flags);
goto out;
@@ -683,9 +695,9 @@ int evtchn_close(struct domain *d1, int
BUG();
}
- spin_lock(&chn1->lock);
+ spin_lock_irqsave(&chn1->lock, flags);
evtchn_free(d1, chn1);
- spin_unlock(&chn1->lock);
+ spin_unlock_irqrestore(&chn1->lock, flags);
out:
if ( d2 != NULL )
@@ -705,13 +717,14 @@ int evtchn_send(struct domain *ld, unsig
struct evtchn *lchn, *rchn;
struct domain *rd;
int rport, ret = 0;
+ unsigned long flags;
if ( !port_is_valid(ld, lport) )
return -EINVAL;
lchn = evtchn_from_port(ld, lport);
- spin_lock(&lchn->lock);
+ spin_lock_irqsave(&lchn->lock, flags);
/* Guest cannot send via a Xen-attached event channel. */
if ( unlikely(consumer_is_xen(lchn)) )
@@ -746,7 +759,7 @@ int evtchn_send(struct domain *ld, unsig
}
out:
- spin_unlock(&lchn->lock);
+ spin_unlock_irqrestore(&lchn->lock, flags);
return ret;
}
@@ -1238,6 +1251,7 @@ int alloc_unbound_xen_event_channel(
{
struct evtchn *chn;
int port, rc;
+ unsigned long flags;
spin_lock(&ld->event_lock);
@@ -1250,14 +1264,14 @@ int alloc_unbound_xen_event_channel(
if ( rc )
goto out;
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_UNBOUND;
chn->xen_consumer = get_xen_consumer(notification_fn);
chn->notify_vcpu_id = lvcpu;
chn->u.unbound.remote_domid = remote_domid;
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
write_atomic(&ld->xen_evtchns, ld->xen_evtchns + 1);
@@ -1280,11 +1294,12 @@ void notify_via_xen_event_channel(struct
{
struct evtchn *lchn, *rchn;
struct domain *rd;
+ unsigned long flags;
ASSERT(port_is_valid(ld, lport));
lchn = evtchn_from_port(ld, lport);
- spin_lock(&lchn->lock);
+ spin_lock_irqsave(&lchn->lock, flags);
if ( likely(lchn->state == ECS_INTERDOMAIN) )
{
@@ -1294,7 +1309,7 @@ void notify_via_xen_event_channel(struct
evtchn_port_set_pending(rd, rchn->notify_vcpu_id, rchn);
}
- spin_unlock(&lchn->lock);
+ spin_unlock_irqrestore(&lchn->lock, flags);
}
void evtchn_check_pollers(struct domain *d, unsigned int port)
["xsa343/xsa343-3.patch" (application/octet-stream)]
From: Jan Beulich <jbeulich@suse.com>
Subject: evtchn: address races with evtchn_reset()
Neither d->evtchn_port_ops nor max_evtchns(d) may be used in an entirely
lock-less manner, as both may change by a racing evtchn_reset(). In the
common case, at least one of the domain's event lock or the per-channel
lock needs to be held. In the specific case of the inter-domain sending
by evtchn_send() and notify_via_xen_event_channel() holding the other
side's per-channel lock is sufficient, as the channel can't change state
without both per-channel locks held. Without such a channel changing
state, evtchn_reset() can't complete successfully.
Lock-free accesses continue to be permitted for the shim (calling some
otherwise internal event channel functions), as this happens while the
domain is in effectively single-threaded mode. Special care also needs
taking for the shim's marking of in-use ports as ECS_RESERVED (allowing
use of such ports in the shim case is okay because switching into and
hence also out of FIFO mode is impossihble there).
As a side effect, certain operations on Xen bound event channels which
were mistakenly permitted so far (e.g. unmask or poll) will be refused
now.
This is part of XSA-343.
Reported-by: Julien Grall <jgrall@amazon.com>
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Julien Grall <jgrall@amazon.com>
---
v9: Add arch_evtchn_is_special() to fix PV shim.
v8: Add BUILD_BUG_ON() in evtchn_usable().
v7: Add locking related comment ahead of struct evtchn_port_ops.
v6: New.
---
TBD: I've been considering to move some of the wrappers from xen/event.h
into event_channel.c (or even drop them altogether), when they
require external locking (e.g. evtchn_port_init() or
evtchn_port_set_priority()). Does anyone have a strong opinion
either way?
--- a/xen/arch/x86/irq.c
+++ b/xen/arch/x86/irq.c
@@ -2488,14 +2488,24 @@ static void dump_irqs(unsigned char key)
for ( i = 0; i < action->nr_guests; )
{
+ struct evtchn *evtchn;
+ unsigned int pending = 2, masked = 2;
+
d = action->guest[i++];
pirq = domain_irq_to_pirq(d, irq);
info = pirq_info(d, pirq);
+ evtchn = evtchn_from_port(d, info->evtchn);
+ local_irq_disable();
+ if ( spin_trylock(&evtchn->lock) )
+ {
+ pending = evtchn_is_pending(d, evtchn);
+ masked = evtchn_is_masked(d, evtchn);
+ spin_unlock(&evtchn->lock);
+ }
+ local_irq_enable();
printk("d%d:%3d(%c%c%c)%c",
- d->domain_id, pirq,
- evtchn_port_is_pending(d, info->evtchn) ? 'P' : '-',
- evtchn_port_is_masked(d, info->evtchn) ? 'M' : '-',
- info->masked ? 'M' : '-',
+ d->domain_id, pirq, "-P?"[pending],
+ "-M?"[masked], info->masked ? 'M' : '-',
i < action->nr_guests ? ',' : '\n');
}
}
--- a/xen/arch/x86/pv/shim.c
+++ b/xen/arch/x86/pv/shim.c
@@ -660,8 +660,11 @@ void pv_shim_inject_evtchn(unsigned int
if ( port_is_valid(guest, port) )
{
struct evtchn *chn = evtchn_from_port(guest, port);
+ unsigned long flags;
+ spin_lock_irqsave(&chn->lock, flags);
evtchn_port_set_pending(guest, chn->notify_vcpu_id, chn);
+ spin_unlock_irqrestore(&chn->lock, flags);
}
}
--- a/xen/common/event_2l.c
+++ b/xen/common/event_2l.c
@@ -63,8 +63,10 @@ static void evtchn_2l_unmask(struct doma
}
}
-static bool evtchn_2l_is_pending(const struct domain *d, evtchn_port_t port)
+static bool evtchn_2l_is_pending(const struct domain *d,
+ const struct evtchn *evtchn)
{
+ evtchn_port_t port = evtchn->port;
unsigned int max_ports = BITS_PER_EVTCHN_WORD(d) * BITS_PER_EVTCHN_WORD(d);
ASSERT(port < max_ports);
@@ -72,8 +74,10 @@ static bool evtchn_2l_is_pending(const s
guest_test_bit(d, port, &shared_info(d, evtchn_pending)));
}
-static bool evtchn_2l_is_masked(const struct domain *d, evtchn_port_t port)
+static bool evtchn_2l_is_masked(const struct domain *d,
+ const struct evtchn *evtchn)
{
+ evtchn_port_t port = evtchn->port;
unsigned int max_ports = BITS_PER_EVTCHN_WORD(d) * BITS_PER_EVTCHN_WORD(d);
ASSERT(port < max_ports);
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -156,8 +156,9 @@ int evtchn_allocate_port(struct domain *
if ( port_is_valid(d, port) )
{
- if ( evtchn_from_port(d, port)->state != ECS_FREE ||
- evtchn_port_is_busy(d, port) )
+ const struct evtchn *chn = evtchn_from_port(d, port);
+
+ if ( chn->state != ECS_FREE || evtchn_is_busy(d, chn) )
return -EBUSY;
}
else
@@ -774,6 +775,7 @@ void send_guest_vcpu_virq(struct vcpu *v
unsigned long flags;
int port;
struct domain *d;
+ struct evtchn *chn;
ASSERT(!virq_is_global(virq));
@@ -784,7 +786,10 @@ void send_guest_vcpu_virq(struct vcpu *v
goto out;
d = v->domain;
- evtchn_port_set_pending(d, v->vcpu_id, evtchn_from_port(d, port));
+ chn = evtchn_from_port(d, port);
+ spin_lock(&chn->lock);
+ evtchn_port_set_pending(d, v->vcpu_id, chn);
+ spin_unlock(&chn->lock);
out:
spin_unlock_irqrestore(&v->virq_lock, flags);
@@ -813,7 +818,9 @@ void send_guest_global_virq(struct domai
goto out;
chn = evtchn_from_port(d, port);
+ spin_lock(&chn->lock);
evtchn_port_set_pending(d, chn->notify_vcpu_id, chn);
+ spin_unlock(&chn->lock);
out:
spin_unlock_irqrestore(&v->virq_lock, flags);
@@ -823,6 +830,7 @@ void send_guest_pirq(struct domain *d, c
{
int port;
struct evtchn *chn;
+ unsigned long flags;
/*
* PV guests: It should not be possible to race with __evtchn_close(). The
@@ -837,7 +845,9 @@ void send_guest_pirq(struct domain *d, c
}
chn = evtchn_from_port(d, port);
+ spin_lock_irqsave(&chn->lock, flags);
evtchn_port_set_pending(d, chn->notify_vcpu_id, chn);
+ spin_unlock_irqrestore(&chn->lock, flags);
}
static struct domain *global_virq_handlers[NR_VIRQS] __read_mostly;
@@ -1034,12 +1044,15 @@ int evtchn_unmask(unsigned int port)
{
struct domain *d = current->domain;
struct evtchn *evtchn;
+ unsigned long flags;
if ( unlikely(!port_is_valid(d, port)) )
return -EINVAL;
evtchn = evtchn_from_port(d, port);
+ spin_lock_irqsave(&evtchn->lock, flags);
evtchn_port_unmask(d, evtchn);
+ spin_unlock_irqrestore(&evtchn->lock, flags);
return 0;
}
@@ -1449,8 +1462,8 @@ static void domain_dump_evtchn_info(stru
printk(" %4u [%d/%d/",
port,
- evtchn_port_is_pending(d, port),
- evtchn_port_is_masked(d, port));
+ evtchn_is_pending(d, chn),
+ evtchn_is_masked(d, chn));
evtchn_port_print_state(d, chn);
printk("]: s=%d n=%d x=%d",
chn->state, chn->notify_vcpu_id, chn->xen_consumer);
--- a/xen/common/event_fifo.c
+++ b/xen/common/event_fifo.c
@@ -296,23 +296,26 @@ static void evtchn_fifo_unmask(struct do
evtchn_fifo_set_pending(v, evtchn);
}
-static bool evtchn_fifo_is_pending(const struct domain *d, evtchn_port_t port)
+static bool evtchn_fifo_is_pending(const struct domain *d,
+ const struct evtchn *evtchn)
{
- const event_word_t *word = evtchn_fifo_word_from_port(d, port);
+ const event_word_t *word = evtchn_fifo_word_from_port(d, evtchn->port);
return word && guest_test_bit(d, EVTCHN_FIFO_PENDING, word);
}
-static bool_t evtchn_fifo_is_masked(const struct domain *d, evtchn_port_t port)
+static bool_t evtchn_fifo_is_masked(const struct domain *d,
+ const struct evtchn *evtchn)
{
- const event_word_t *word = evtchn_fifo_word_from_port(d, port);
+ const event_word_t *word = evtchn_fifo_word_from_port(d, evtchn->port);
return !word || guest_test_bit(d, EVTCHN_FIFO_MASKED, word);
}
-static bool_t evtchn_fifo_is_busy(const struct domain *d, evtchn_port_t port)
+static bool_t evtchn_fifo_is_busy(const struct domain *d,
+ const struct evtchn *evtchn)
{
- const event_word_t *word = evtchn_fifo_word_from_port(d, port);
+ const event_word_t *word = evtchn_fifo_word_from_port(d, evtchn->port);
return word && guest_test_bit(d, EVTCHN_FIFO_LINKED, word);
}
--- a/xen/include/asm-x86/event.h
+++ b/xen/include/asm-x86/event.h
@@ -47,4 +47,10 @@ static inline bool arch_virq_is_global(u
return true;
}
+#ifdef CONFIG_PV_SHIM
+# include <asm/pv/shim.h>
+# define arch_evtchn_is_special(chn) \
+ (pv_shim && (chn)->port && (chn)->state == ECS_RESERVED)
+#endif
+
#endif
--- a/xen/include/xen/event.h
+++ b/xen/include/xen/event.h
@@ -133,6 +133,24 @@ static inline struct evtchn *evtchn_from
return bucket_from_port(d, p) + (p % EVTCHNS_PER_BUCKET);
}
+/*
+ * "usable" as in "by a guest", i.e. Xen consumed channels are assumed to be
+ * taken care of separately where used for Xen's internal purposes.
+ */
+static bool evtchn_usable(const struct evtchn *evtchn)
+{
+ if ( evtchn->xen_consumer )
+ return false;
+
+#ifdef arch_evtchn_is_special
+ if ( arch_evtchn_is_special(evtchn) )
+ return true;
+#endif
+
+ BUILD_BUG_ON(ECS_FREE > ECS_RESERVED);
+ return evtchn->state > ECS_RESERVED;
+}
+
/* Wait on a Xen-attached event channel. */
#define wait_on_xen_event_channel(port, condition) \
do { \
@@ -165,19 +183,24 @@ int evtchn_reset(struct domain *d);
/*
* Low-level event channel port ops.
+ *
+ * All hooks have to be called with a lock held which prevents the channel
+ * from changing state. This may be the domain event lock, the per-channel
+ * lock, or in the case of sending interdomain events also the other side's
+ * per-channel lock. Exceptions apply in certain cases for the PV shim.
*/
struct evtchn_port_ops {
void (*init)(struct domain *d, struct evtchn *evtchn);
void (*set_pending)(struct vcpu *v, struct evtchn *evtchn);
void (*clear_pending)(struct domain *d, struct evtchn *evtchn);
void (*unmask)(struct domain *d, struct evtchn *evtchn);
- bool (*is_pending)(const struct domain *d, evtchn_port_t port);
- bool (*is_masked)(const struct domain *d, evtchn_port_t port);
+ bool (*is_pending)(const struct domain *d, const struct evtchn *evtchn);
+ bool (*is_masked)(const struct domain *d, const struct evtchn *evtchn);
/*
* Is the port unavailable because it's still being cleaned up
* after being closed?
*/
- bool (*is_busy)(const struct domain *d, evtchn_port_t port);
+ bool (*is_busy)(const struct domain *d, const struct evtchn *evtchn);
int (*set_priority)(struct domain *d, struct evtchn *evtchn,
unsigned int priority);
void (*print_state)(struct domain *d, const struct evtchn *evtchn);
@@ -193,38 +216,67 @@ static inline void evtchn_port_set_pendi
unsigned int vcpu_id,
struct evtchn *evtchn)
{
- d->evtchn_port_ops->set_pending(d->vcpu[vcpu_id], evtchn);
+ if ( evtchn_usable(evtchn) )
+ d->evtchn_port_ops->set_pending(d->vcpu[vcpu_id], evtchn);
}
static inline void evtchn_port_clear_pending(struct domain *d,
struct evtchn *evtchn)
{
- d->evtchn_port_ops->clear_pending(d, evtchn);
+ if ( evtchn_usable(evtchn) )
+ d->evtchn_port_ops->clear_pending(d, evtchn);
}
static inline void evtchn_port_unmask(struct domain *d,
struct evtchn *evtchn)
{
- d->evtchn_port_ops->unmask(d, evtchn);
+ if ( evtchn_usable(evtchn) )
+ d->evtchn_port_ops->unmask(d, evtchn);
}
-static inline bool evtchn_port_is_pending(const struct domain *d,
- evtchn_port_t port)
+static inline bool evtchn_is_pending(const struct domain *d,
+ const struct evtchn *evtchn)
{
- return d->evtchn_port_ops->is_pending(d, port);
+ return evtchn_usable(evtchn) && d->evtchn_port_ops->is_pending(d, evtchn);
}
-static inline bool evtchn_port_is_masked(const struct domain *d,
- evtchn_port_t port)
+static inline bool evtchn_port_is_pending(struct domain *d, evtchn_port_t port)
{
- return d->evtchn_port_ops->is_masked(d, port);
+ struct evtchn *evtchn = evtchn_from_port(d, port);
+ bool rc;
+ unsigned long flags;
+
+ spin_lock_irqsave(&evtchn->lock, flags);
+ rc = evtchn_is_pending(d, evtchn);
+ spin_unlock_irqrestore(&evtchn->lock, flags);
+
+ return rc;
+}
+
+static inline bool evtchn_is_masked(const struct domain *d,
+ const struct evtchn *evtchn)
+{
+ return !evtchn_usable(evtchn) || d->evtchn_port_ops->is_masked(d, evtchn);
+}
+
+static inline bool evtchn_port_is_masked(struct domain *d, evtchn_port_t port)
+{
+ struct evtchn *evtchn = evtchn_from_port(d, port);
+ bool rc;
+ unsigned long flags;
+
+ spin_lock_irqsave(&evtchn->lock, flags);
+ rc = evtchn_is_masked(d, evtchn);
+ spin_unlock_irqrestore(&evtchn->lock, flags);
+
+ return rc;
}
-static inline bool evtchn_port_is_busy(const struct domain *d,
- evtchn_port_t port)
+static inline bool evtchn_is_busy(const struct domain *d,
+ const struct evtchn *evtchn)
{
return d->evtchn_port_ops->is_busy &&
- d->evtchn_port_ops->is_busy(d, port);
+ d->evtchn_port_ops->is_busy(d, evtchn);
}
static inline int evtchn_port_set_priority(struct domain *d,
@@ -233,6 +285,8 @@ static inline int evtchn_port_set_priori
{
if ( !d->evtchn_port_ops->set_priority )
return -ENOSYS;
+ if ( !evtchn_usable(evtchn) )
+ return -EACCES;
return d->evtchn_port_ops->set_priority(d, evtchn, priority);
}
["xsa343/xsa343-4.10-1.patch" (application/octet-stream)]
From: Jan Beulich <jbeulich@suse.com>
Subject: evtchn: evtchn_reset() may not succeed with still-open ports
While the function closes all ports, it does so without holding any
lock, and hence racing requests may be issued causing new ports to get
opened. This would have been problematic in particular if such a newly
opened port had a port number above the new implementation limit (i.e.
when switching from FIFO to 2-level) after the reset, as prior to
"evtchn: relax port_is_valid()" this could have led to e.g.
evtchn_close()'s "BUG_ON(!port_is_valid(d2, port2))" to trigger.
Introduce a counter of active ports and check that it's (still) no
larger then the number of Xen internally used ones after obtaining the
necessary lock in evtchn_reset().
As to the access model of the new {active,xen}_evtchns fields - while
all writes get done using write_atomic(), reads ought to use
read_atomic() only when outside of a suitably locked region.
Note that as of now evtchn_bind_virq() and evtchn_bind_ipi() don't have
a need to call check_free_port().
This is part of XSA-343.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Reviewed-by: Julien Grall <jgrall@amazon.com>
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -195,6 +195,8 @@ int evtchn_allocate_port(struct domain *
write_atomic(&d->valid_evtchns, d->valid_evtchns + EVTCHNS_PER_BUCKET);
}
+ write_atomic(&d->active_evtchns, d->active_evtchns + 1);
+
return 0;
}
@@ -218,11 +220,26 @@ static int get_free_port(struct domain *
return -ENOSPC;
}
+/*
+ * Check whether a port is still marked free, and if so update the domain
+ * counter accordingly. To be used on function exit paths.
+ */
+static void check_free_port(struct domain *d, evtchn_port_t port)
+{
+ if ( port_is_valid(d, port) &&
+ evtchn_from_port(d, port)->state == ECS_FREE )
+ write_atomic(&d->active_evtchns, d->active_evtchns - 1);
+}
+
void evtchn_free(struct domain *d, struct evtchn *chn)
{
/* Clear pending event to avoid unexpected behavior on re-bind. */
evtchn_port_clear_pending(d, chn);
+ if ( consumer_is_xen(chn) )
+ write_atomic(&d->xen_evtchns, d->xen_evtchns - 1);
+ write_atomic(&d->active_evtchns, d->active_evtchns - 1);
+
/* Reset binding to vcpu0 when the channel is freed. */
chn->state = ECS_FREE;
chn->notify_vcpu_id = 0;
@@ -265,6 +282,7 @@ static long evtchn_alloc_unbound(evtchn_
alloc->port = port;
out:
+ check_free_port(d, port);
spin_unlock(&d->event_lock);
rcu_unlock_domain(d);
@@ -358,6 +376,7 @@ static long evtchn_bind_interdomain(evtc
bind->local_port = lport;
out:
+ check_free_port(ld, lport);
spin_unlock(&ld->event_lock);
if ( ld != rd )
spin_unlock(&rd->event_lock);
@@ -491,7 +510,7 @@ static long evtchn_bind_pirq(evtchn_bind
struct domain *d = current->domain;
struct vcpu *v = d->vcpu[0];
struct pirq *info;
- int port, pirq = bind->pirq;
+ int port = 0, pirq = bind->pirq;
long rc;
if ( (pirq < 0) || (pirq >= d->nr_pirqs) )
@@ -539,6 +558,7 @@ static long evtchn_bind_pirq(evtchn_bind
arch_evtchn_bind_pirq(d, pirq);
out:
+ check_free_port(d, port);
spin_unlock(&d->event_lock);
return rc;
@@ -1013,10 +1033,10 @@ int evtchn_unmask(unsigned int port)
return 0;
}
-
int evtchn_reset(struct domain *d)
{
unsigned int i;
+ int rc = 0;
if ( d != current->domain && !d->controller_pause_count )
return -EINVAL;
@@ -1026,7 +1046,9 @@ int evtchn_reset(struct domain *d)
spin_lock(&d->event_lock);
- if ( d->evtchn_fifo )
+ if ( d->active_evtchns > d->xen_evtchns )
+ rc = -EAGAIN;
+ else if ( d->evtchn_fifo )
{
/* Switching back to 2-level ABI. */
evtchn_fifo_destroy(d);
@@ -1035,7 +1057,7 @@ int evtchn_reset(struct domain *d)
spin_unlock(&d->event_lock);
- return 0;
+ return rc;
}
static long evtchn_set_priority(const struct evtchn_set_priority *set_priority)
@@ -1221,10 +1243,9 @@ int alloc_unbound_xen_event_channel(
spin_lock(&ld->event_lock);
- rc = get_free_port(ld);
+ port = rc = get_free_port(ld);
if ( rc < 0 )
goto out;
- port = rc;
chn = evtchn_from_port(ld, port);
rc = xsm_evtchn_unbound(XSM_TARGET, ld, chn, remote_domid);
@@ -1240,7 +1261,10 @@ int alloc_unbound_xen_event_channel(
spin_unlock(&chn->lock);
+ write_atomic(&ld->xen_evtchns, ld->xen_evtchns + 1);
+
out:
+ check_free_port(ld, port);
spin_unlock(&ld->event_lock);
return rc < 0 ? rc : port;
@@ -1316,6 +1340,7 @@ int evtchn_init(struct domain *d)
return -EINVAL;
}
evtchn_from_port(d, 0)->state = ECS_RESERVED;
+ write_atomic(&d->active_evtchns, 0);
#if MAX_VIRT_CPUS > BITS_PER_LONG
d->poll_mask = xzalloc_array(unsigned long,
@@ -1343,6 +1368,8 @@ void evtchn_destroy(struct domain *d)
for ( i = 0; port_is_valid(d, i); i++ )
evtchn_close(d, i, 0);
+ ASSERT(!d->active_evtchns);
+
clear_global_virq_handlers(d);
evtchn_fifo_destroy(d);
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -338,6 +338,16 @@ struct domain
struct evtchn **evtchn_group[NR_EVTCHN_GROUPS]; /* all other buckets */
unsigned int max_evtchn_port; /* max permitted port number */
unsigned int valid_evtchns; /* number of allocated event channels */
+ /*
+ * Number of in-use event channels. Writers should use write_atomic().
+ * Readers need to use read_atomic() only when not holding event_lock.
+ */
+ unsigned int active_evtchns;
+ /*
+ * Number of event channels used internally by Xen (not subject to
+ * EVTCHNOP_reset). Read/write access like for active_evtchns.
+ */
+ unsigned int xen_evtchns;
spinlock_t event_lock;
const struct evtchn_port_ops *evtchn_port_ops;
struct evtchn_fifo_domain *evtchn_fifo;
["xsa343/xsa343-4.10-2.patch" (application/octet-stream)]
From: Jan Beulich <jbeulich@suse.com>
Subject: evtchn: convert per-channel lock to be IRQ-safe
... in order for send_guest_{global,vcpu}_virq() to be able to make use
of it.
This is part of XSA-343.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Julien Grall <jgrall@amazon.com>
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -255,6 +255,7 @@ static long evtchn_alloc_unbound(evtchn_
int port;
domid_t dom = alloc->dom;
long rc;
+ unsigned long flags;
d = rcu_lock_domain_by_any_id(dom);
if ( d == NULL )
@@ -270,14 +271,14 @@ static long evtchn_alloc_unbound(evtchn_
if ( rc )
goto out;
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_UNBOUND;
if ( (chn->u.unbound.remote_domid = alloc->remote_dom) == DOMID_SELF )
chn->u.unbound.remote_domid = current->domain->domain_id;
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
alloc->port = port;
@@ -290,26 +291,32 @@ static long evtchn_alloc_unbound(evtchn_
}
-static void double_evtchn_lock(struct evtchn *lchn, struct evtchn *rchn)
+static unsigned long double_evtchn_lock(struct evtchn *lchn,
+ struct evtchn *rchn)
{
- if ( lchn < rchn )
+ unsigned long flags;
+
+ if ( lchn <= rchn )
{
- spin_lock(&lchn->lock);
- spin_lock(&rchn->lock);
+ spin_lock_irqsave(&lchn->lock, flags);
+ if ( lchn != rchn )
+ spin_lock(&rchn->lock);
}
else
{
- if ( lchn != rchn )
- spin_lock(&rchn->lock);
+ spin_lock_irqsave(&rchn->lock, flags);
spin_lock(&lchn->lock);
}
+
+ return flags;
}
-static void double_evtchn_unlock(struct evtchn *lchn, struct evtchn *rchn)
+static void double_evtchn_unlock(struct evtchn *lchn, struct evtchn *rchn,
+ unsigned long flags)
{
- spin_unlock(&lchn->lock);
if ( lchn != rchn )
- spin_unlock(&rchn->lock);
+ spin_unlock(&lchn->lock);
+ spin_unlock_irqrestore(&rchn->lock, flags);
}
static long evtchn_bind_interdomain(evtchn_bind_interdomain_t *bind)
@@ -319,6 +326,7 @@ static long evtchn_bind_interdomain(evtc
int lport, rport = bind->remote_port;
domid_t rdom = bind->remote_dom;
long rc;
+ unsigned long flags;
if ( rdom == DOMID_SELF )
rdom = current->domain->domain_id;
@@ -354,7 +362,7 @@ static long evtchn_bind_interdomain(evtc
if ( rc )
goto out;
- double_evtchn_lock(lchn, rchn);
+ flags = double_evtchn_lock(lchn, rchn);
lchn->u.interdomain.remote_dom = rd;
lchn->u.interdomain.remote_port = rport;
@@ -371,7 +379,7 @@ static long evtchn_bind_interdomain(evtc
*/
evtchn_port_set_pending(ld, lchn->notify_vcpu_id, lchn);
- double_evtchn_unlock(lchn, rchn);
+ double_evtchn_unlock(lchn, rchn, flags);
bind->local_port = lport;
@@ -394,6 +402,7 @@ int evtchn_bind_virq(evtchn_bind_virq_t
struct domain *d = current->domain;
int virq = bind->virq, vcpu = bind->vcpu;
int rc = 0;
+ unsigned long flags;
if ( (virq < 0) || (virq >= ARRAY_SIZE(v->virq_to_evtchn)) )
return -EINVAL;
@@ -426,14 +435,14 @@ int evtchn_bind_virq(evtchn_bind_virq_t
chn = evtchn_from_port(d, port);
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_VIRQ;
chn->notify_vcpu_id = vcpu;
chn->u.virq = virq;
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
v->virq_to_evtchn[virq] = bind->port = port;
@@ -450,6 +459,7 @@ static long evtchn_bind_ipi(evtchn_bind_
struct domain *d = current->domain;
int port, vcpu = bind->vcpu;
long rc = 0;
+ unsigned long flags;
if ( (vcpu < 0) || (vcpu >= d->max_vcpus) ||
(d->vcpu[vcpu] == NULL) )
@@ -462,13 +472,13 @@ static long evtchn_bind_ipi(evtchn_bind_
chn = evtchn_from_port(d, port);
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_IPI;
chn->notify_vcpu_id = vcpu;
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
bind->port = port;
@@ -512,6 +522,7 @@ static long evtchn_bind_pirq(evtchn_bind
struct pirq *info;
int port = 0, pirq = bind->pirq;
long rc;
+ unsigned long flags;
if ( (pirq < 0) || (pirq >= d->nr_pirqs) )
return -EINVAL;
@@ -544,14 +555,14 @@ static long evtchn_bind_pirq(evtchn_bind
goto out;
}
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_PIRQ;
chn->u.pirq.irq = pirq;
link_pirq_port(port, chn, v);
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
bind->port = port;
@@ -572,6 +583,7 @@ int evtchn_close(struct domain *d1, int
struct evtchn *chn1, *chn2;
int port2;
long rc = 0;
+ unsigned long flags;
again:
spin_lock(&d1->event_lock);
@@ -671,14 +683,14 @@ int evtchn_close(struct domain *d1, int
BUG_ON(chn2->state != ECS_INTERDOMAIN);
BUG_ON(chn2->u.interdomain.remote_dom != d1);
- double_evtchn_lock(chn1, chn2);
+ flags = double_evtchn_lock(chn1, chn2);
evtchn_free(d1, chn1);
chn2->state = ECS_UNBOUND;
chn2->u.unbound.remote_domid = d1->domain_id;
- double_evtchn_unlock(chn1, chn2);
+ double_evtchn_unlock(chn1, chn2, flags);
goto out;
@@ -686,9 +698,9 @@ int evtchn_close(struct domain *d1, int
BUG();
}
- spin_lock(&chn1->lock);
+ spin_lock_irqsave(&chn1->lock, flags);
evtchn_free(d1, chn1);
- spin_unlock(&chn1->lock);
+ spin_unlock_irqrestore(&chn1->lock, flags);
out:
if ( d2 != NULL )
@@ -708,13 +720,14 @@ int evtchn_send(struct domain *ld, unsig
struct evtchn *lchn, *rchn;
struct domain *rd;
int rport, ret = 0;
+ unsigned long flags;
if ( !port_is_valid(ld, lport) )
return -EINVAL;
lchn = evtchn_from_port(ld, lport);
- spin_lock(&lchn->lock);
+ spin_lock_irqsave(&lchn->lock, flags);
/* Guest cannot send via a Xen-attached event channel. */
if ( unlikely(consumer_is_xen(lchn)) )
@@ -749,7 +762,7 @@ int evtchn_send(struct domain *ld, unsig
}
out:
- spin_unlock(&lchn->lock);
+ spin_unlock_irqrestore(&lchn->lock, flags);
return ret;
}
@@ -1240,6 +1253,7 @@ int alloc_unbound_xen_event_channel(
{
struct evtchn *chn;
int port, rc;
+ unsigned long flags;
spin_lock(&ld->event_lock);
@@ -1252,14 +1266,14 @@ int alloc_unbound_xen_event_channel(
if ( rc )
goto out;
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_UNBOUND;
chn->xen_consumer = get_xen_consumer(notification_fn);
chn->notify_vcpu_id = lvcpu;
chn->u.unbound.remote_domid = remote_domid;
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
write_atomic(&ld->xen_evtchns, ld->xen_evtchns + 1);
@@ -1282,11 +1296,12 @@ void notify_via_xen_event_channel(struct
{
struct evtchn *lchn, *rchn;
struct domain *rd;
+ unsigned long flags;
ASSERT(port_is_valid(ld, lport));
lchn = evtchn_from_port(ld, lport);
- spin_lock(&lchn->lock);
+ spin_lock_irqsave(&lchn->lock, flags);
if ( likely(lchn->state == ECS_INTERDOMAIN) )
{
@@ -1296,7 +1311,7 @@ void notify_via_xen_event_channel(struct
evtchn_port_set_pending(rd, rchn->notify_vcpu_id, rchn);
}
- spin_unlock(&lchn->lock);
+ spin_unlock_irqrestore(&lchn->lock, flags);
}
void evtchn_check_pollers(struct domain *d, unsigned int port)
["xsa343/xsa343-4.10-3.patch" (application/octet-stream)]
From: Jan Beulich <jbeulich@suse.com>
Subject: evtchn: address races with evtchn_reset()
Neither d->evtchn_port_ops nor max_evtchns(d) may be used in an entirely
lock-less manner, as both may change by a racing evtchn_reset(). In the
common case, at least one of the domain's event lock or the per-channel
lock needs to be held. In the specific case of the inter-domain sending
by evtchn_send() and notify_via_xen_event_channel() holding the other
side's per-channel lock is sufficient, as the channel can't change state
without both per-channel locks held. Without such a channel changing
state, evtchn_reset() can't complete successfully.
Lock-free accesses continue to be permitted for the shim (calling some
otherwise internal event channel functions), as this happens while the
domain is in effectively single-threaded mode. Special care also needs
taking for the shim's marking of in-use ports as ECS_RESERVED (allowing
use of such ports in the shim case is okay because switching into and
hence also out of FIFO mode is impossible there).
As a side effect, certain operations on Xen bound event channels which
were mistakenly permitted so far (e.g. unmask or poll) will be refused
now.
This is part of XSA-343.
Reported-by: Julien Grall <jgrall@amazon.com>
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Julien Grall <jgrall@amazon.com>
--- a/xen/arch/x86/irq.c
+++ b/xen/arch/x86/irq.c
@@ -2323,14 +2323,24 @@ static void dump_irqs(unsigned char key)
for ( i = 0; i < action->nr_guests; i++ )
{
+ struct evtchn *evtchn;
+ unsigned int pending = 2, masked = 2;
+
d = action->guest[i];
pirq = domain_irq_to_pirq(d, irq);
info = pirq_info(d, pirq);
+ evtchn = evtchn_from_port(d, info->evtchn);
+ local_irq_disable();
+ if ( spin_trylock(&evtchn->lock) )
+ {
+ pending = evtchn_is_pending(d, evtchn);
+ masked = evtchn_is_masked(d, evtchn);
+ spin_unlock(&evtchn->lock);
+ }
+ local_irq_enable();
printk("%u:%3d(%c%c%c)",
- d->domain_id, pirq,
- evtchn_port_is_pending(d, info->evtchn) ? 'P' : '-',
- evtchn_port_is_masked(d, info->evtchn) ? 'M' : '-',
- (info->masked ? 'M' : '-'));
+ d->domain_id, pirq, "-P?"[pending],
+ "-M?"[masked], info->masked ? 'M' : '-');
if ( i != action->nr_guests )
printk(",");
}
--- a/xen/arch/x86/pv/shim.c
+++ b/xen/arch/x86/pv/shim.c
@@ -603,8 +603,11 @@ void pv_shim_inject_evtchn(unsigned int
if ( port_is_valid(guest, port) )
{
struct evtchn *chn = evtchn_from_port(guest, port);
+ unsigned long flags;
+ spin_lock_irqsave(&chn->lock, flags);
evtchn_port_set_pending(guest, chn->notify_vcpu_id, chn);
+ spin_unlock_irqrestore(&chn->lock, flags);
}
}
--- a/xen/common/event_2l.c
+++ b/xen/common/event_2l.c
@@ -63,8 +63,10 @@ static void evtchn_2l_unmask(struct doma
}
}
-static bool evtchn_2l_is_pending(const struct domain *d, evtchn_port_t port)
+static bool evtchn_2l_is_pending(const struct domain *d,
+ const struct evtchn *evtchn)
{
+ evtchn_port_t port = evtchn->port;
unsigned int max_ports = BITS_PER_EVTCHN_WORD(d) * BITS_PER_EVTCHN_WORD(d);
ASSERT(port < max_ports);
@@ -72,8 +74,10 @@ static bool evtchn_2l_is_pending(const s
guest_test_bit(d, port, &shared_info(d, evtchn_pending)));
}
-static bool evtchn_2l_is_masked(const struct domain *d, evtchn_port_t port)
+static bool evtchn_2l_is_masked(const struct domain *d,
+ const struct evtchn *evtchn)
{
+ evtchn_port_t port = evtchn->port;
unsigned int max_ports = BITS_PER_EVTCHN_WORD(d) * BITS_PER_EVTCHN_WORD(d);
ASSERT(port < max_ports);
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -163,8 +163,9 @@ int evtchn_allocate_port(struct domain *
if ( port_is_valid(d, port) )
{
- if ( evtchn_from_port(d, port)->state != ECS_FREE ||
- evtchn_port_is_busy(d, port) )
+ const struct evtchn *chn = evtchn_from_port(d, port);
+
+ if ( chn->state != ECS_FREE || evtchn_is_busy(d, chn) )
return -EBUSY;
}
else
@@ -777,6 +778,7 @@ void send_guest_vcpu_virq(struct vcpu *v
unsigned long flags;
int port;
struct domain *d;
+ struct evtchn *chn;
ASSERT(!virq_is_global(virq));
@@ -787,7 +789,10 @@ void send_guest_vcpu_virq(struct vcpu *v
goto out;
d = v->domain;
- evtchn_port_set_pending(d, v->vcpu_id, evtchn_from_port(d, port));
+ chn = evtchn_from_port(d, port);
+ spin_lock(&chn->lock);
+ evtchn_port_set_pending(d, v->vcpu_id, chn);
+ spin_unlock(&chn->lock);
out:
spin_unlock_irqrestore(&v->virq_lock, flags);
@@ -816,7 +821,9 @@ static void send_guest_global_virq(struc
goto out;
chn = evtchn_from_port(d, port);
+ spin_lock(&chn->lock);
evtchn_port_set_pending(d, chn->notify_vcpu_id, chn);
+ spin_unlock(&chn->lock);
out:
spin_unlock_irqrestore(&v->virq_lock, flags);
@@ -826,6 +833,7 @@ void send_guest_pirq(struct domain *d, c
{
int port;
struct evtchn *chn;
+ unsigned long flags;
/*
* PV guests: It should not be possible to race with __evtchn_close(). The
@@ -840,7 +848,9 @@ void send_guest_pirq(struct domain *d, c
}
chn = evtchn_from_port(d, port);
+ spin_lock_irqsave(&chn->lock, flags);
evtchn_port_set_pending(d, chn->notify_vcpu_id, chn);
+ spin_unlock_irqrestore(&chn->lock, flags);
}
static struct domain *global_virq_handlers[NR_VIRQS] __read_mostly;
@@ -1036,12 +1046,15 @@ int evtchn_unmask(unsigned int port)
{
struct domain *d = current->domain;
struct evtchn *evtchn;
+ unsigned long flags;
if ( unlikely(!port_is_valid(d, port)) )
return -EINVAL;
evtchn = evtchn_from_port(d, port);
+ spin_lock_irqsave(&evtchn->lock, flags);
evtchn_port_unmask(d, evtchn);
+ spin_unlock_irqrestore(&evtchn->lock, flags);
return 0;
}
@@ -1454,8 +1467,8 @@ static void domain_dump_evtchn_info(stru
printk(" %4u [%d/%d/",
port,
- evtchn_port_is_pending(d, port),
- evtchn_port_is_masked(d, port));
+ evtchn_is_pending(d, chn),
+ evtchn_is_masked(d, chn));
evtchn_port_print_state(d, chn);
printk("]: s=%d n=%d x=%d",
chn->state, chn->notify_vcpu_id, chn->xen_consumer);
--- a/xen/common/event_fifo.c
+++ b/xen/common/event_fifo.c
@@ -295,23 +295,26 @@ static void evtchn_fifo_unmask(struct do
evtchn_fifo_set_pending(v, evtchn);
}
-static bool evtchn_fifo_is_pending(const struct domain *d, evtchn_port_t port)
+static bool evtchn_fifo_is_pending(const struct domain *d,
+ const struct evtchn *evtchn)
{
- const event_word_t *word = evtchn_fifo_word_from_port(d, port);
+ const event_word_t *word = evtchn_fifo_word_from_port(d, evtchn->port);
return word && guest_test_bit(d, EVTCHN_FIFO_PENDING, word);
}
-static bool_t evtchn_fifo_is_masked(const struct domain *d, evtchn_port_t port)
+static bool_t evtchn_fifo_is_masked(const struct domain *d,
+ const struct evtchn *evtchn)
{
- const event_word_t *word = evtchn_fifo_word_from_port(d, port);
+ const event_word_t *word = evtchn_fifo_word_from_port(d, evtchn->port);
return !word || guest_test_bit(d, EVTCHN_FIFO_MASKED, word);
}
-static bool_t evtchn_fifo_is_busy(const struct domain *d, evtchn_port_t port)
+static bool_t evtchn_fifo_is_busy(const struct domain *d,
+ const struct evtchn *evtchn)
{
- const event_word_t *word = evtchn_fifo_word_from_port(d, port);
+ const event_word_t *word = evtchn_fifo_word_from_port(d, evtchn->port);
return word && guest_test_bit(d, EVTCHN_FIFO_LINKED, word);
}
--- a/xen/include/asm-x86/event.h
+++ b/xen/include/asm-x86/event.h
@@ -47,4 +47,10 @@ static inline int arch_virq_is_global(ui
return 1;
}
+#ifdef CONFIG_PV_SHIM
+# include <asm/pv/shim.h>
+# define arch_evtchn_is_special(chn) \
+ (pv_shim && (chn)->port && (chn)->state == ECS_RESERVED)
+#endif
+
#endif
--- a/xen/include/xen/event.h
+++ b/xen/include/xen/event.h
@@ -125,6 +125,24 @@ static inline struct evtchn *evtchn_from
return bucket_from_port(d, p) + (p % EVTCHNS_PER_BUCKET);
}
+/*
+ * "usable" as in "by a guest", i.e. Xen consumed channels are assumed to be
+ * taken care of separately where used for Xen's internal purposes.
+ */
+static bool evtchn_usable(const struct evtchn *evtchn)
+{
+ if ( evtchn->xen_consumer )
+ return false;
+
+#ifdef arch_evtchn_is_special
+ if ( arch_evtchn_is_special(evtchn) )
+ return true;
+#endif
+
+ BUILD_BUG_ON(ECS_FREE > ECS_RESERVED);
+ return evtchn->state > ECS_RESERVED;
+}
+
/* Wait on a Xen-attached event channel. */
#define wait_on_xen_event_channel(port, condition) \
do { \
@@ -157,19 +175,24 @@ int evtchn_reset(struct domain *d);
/*
* Low-level event channel port ops.
+ *
+ * All hooks have to be called with a lock held which prevents the channel
+ * from changing state. This may be the domain event lock, the per-channel
+ * lock, or in the case of sending interdomain events also the other side's
+ * per-channel lock. Exceptions apply in certain cases for the PV shim.
*/
struct evtchn_port_ops {
void (*init)(struct domain *d, struct evtchn *evtchn);
void (*set_pending)(struct vcpu *v, struct evtchn *evtchn);
void (*clear_pending)(struct domain *d, struct evtchn *evtchn);
void (*unmask)(struct domain *d, struct evtchn *evtchn);
- bool (*is_pending)(const struct domain *d, evtchn_port_t port);
- bool (*is_masked)(const struct domain *d, evtchn_port_t port);
+ bool (*is_pending)(const struct domain *d, const struct evtchn *evtchn);
+ bool (*is_masked)(const struct domain *d, const struct evtchn *evtchn);
/*
* Is the port unavailable because it's still being cleaned up
* after being closed?
*/
- bool (*is_busy)(const struct domain *d, evtchn_port_t port);
+ bool (*is_busy)(const struct domain *d, const struct evtchn *evtchn);
int (*set_priority)(struct domain *d, struct evtchn *evtchn,
unsigned int priority);
void (*print_state)(struct domain *d, const struct evtchn *evtchn);
@@ -185,38 +208,67 @@ static inline void evtchn_port_set_pendi
unsigned int vcpu_id,
struct evtchn *evtchn)
{
- d->evtchn_port_ops->set_pending(d->vcpu[vcpu_id], evtchn);
+ if ( evtchn_usable(evtchn) )
+ d->evtchn_port_ops->set_pending(d->vcpu[vcpu_id], evtchn);
}
static inline void evtchn_port_clear_pending(struct domain *d,
struct evtchn *evtchn)
{
- d->evtchn_port_ops->clear_pending(d, evtchn);
+ if ( evtchn_usable(evtchn) )
+ d->evtchn_port_ops->clear_pending(d, evtchn);
}
static inline void evtchn_port_unmask(struct domain *d,
struct evtchn *evtchn)
{
- d->evtchn_port_ops->unmask(d, evtchn);
+ if ( evtchn_usable(evtchn) )
+ d->evtchn_port_ops->unmask(d, evtchn);
}
-static inline bool evtchn_port_is_pending(const struct domain *d,
- evtchn_port_t port)
+static inline bool evtchn_is_pending(const struct domain *d,
+ const struct evtchn *evtchn)
{
- return d->evtchn_port_ops->is_pending(d, port);
+ return evtchn_usable(evtchn) && d->evtchn_port_ops->is_pending(d, evtchn);
}
-static inline bool evtchn_port_is_masked(const struct domain *d,
- evtchn_port_t port)
+static inline bool evtchn_port_is_pending(struct domain *d, evtchn_port_t port)
{
- return d->evtchn_port_ops->is_masked(d, port);
+ struct evtchn *evtchn = evtchn_from_port(d, port);
+ bool rc;
+ unsigned long flags;
+
+ spin_lock_irqsave(&evtchn->lock, flags);
+ rc = evtchn_is_pending(d, evtchn);
+ spin_unlock_irqrestore(&evtchn->lock, flags);
+
+ return rc;
+}
+
+static inline bool evtchn_is_masked(const struct domain *d,
+ const struct evtchn *evtchn)
+{
+ return !evtchn_usable(evtchn) || d->evtchn_port_ops->is_masked(d, evtchn);
+}
+
+static inline bool evtchn_port_is_masked(struct domain *d, evtchn_port_t port)
+{
+ struct evtchn *evtchn = evtchn_from_port(d, port);
+ bool rc;
+ unsigned long flags;
+
+ spin_lock_irqsave(&evtchn->lock, flags);
+ rc = evtchn_is_masked(d, evtchn);
+ spin_unlock_irqrestore(&evtchn->lock, flags);
+
+ return rc;
}
-static inline bool evtchn_port_is_busy(const struct domain *d,
- evtchn_port_t port)
+static inline bool evtchn_is_busy(const struct domain *d,
+ const struct evtchn *evtchn)
{
return d->evtchn_port_ops->is_busy &&
- d->evtchn_port_ops->is_busy(d, port);
+ d->evtchn_port_ops->is_busy(d, evtchn);
}
static inline int evtchn_port_set_priority(struct domain *d,
@@ -225,6 +277,8 @@ static inline int evtchn_port_set_priori
{
if ( !d->evtchn_port_ops->set_priority )
return -ENOSYS;
+ if ( !evtchn_usable(evtchn) )
+ return -EACCES;
return d->evtchn_port_ops->set_priority(d, evtchn, priority);
}
["xsa343/xsa343-4.11-1.patch" (application/octet-stream)]
From: Jan Beulich <jbeulich@suse.com>
Subject: evtchn: evtchn_reset() shouldn't succeed with still-open ports
While the function closes all ports, it does so without holding any
lock, and hence racing requests may be issued causing new ports to get
opened. This would have been problematic in particular if such a newly
opened port had a port number above the new implementation limit (i.e.
when switching from FIFO to 2-level) after the reset, as prior to
"evtchn: relax port_is_valid()" this could have led to e.g.
evtchn_close()'s "BUG_ON(!port_is_valid(d2, port2))" to trigger.
Introduce a counter of active ports and check that it's (still) no
larger then the number of Xen internally used ones after obtaining the
necessary lock in evtchn_reset().
As to the access model of the new {active,xen}_evtchns fields - while
all writes get done using write_atomic(), reads ought to use
read_atomic() only when outside of a suitably locked region.
Note that as of now evtchn_bind_virq() and evtchn_bind_ipi() don't have
a need to call check_free_port().
This is part of XSA-343.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Reviewed-by: Julien Grall <jgrall@amazon.com>
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -188,6 +188,8 @@ int evtchn_allocate_port(struct domain *
write_atomic(&d->valid_evtchns, d->valid_evtchns + EVTCHNS_PER_BUCKET);
}
+ write_atomic(&d->active_evtchns, d->active_evtchns + 1);
+
return 0;
}
@@ -211,11 +213,26 @@ static int get_free_port(struct domain *
return -ENOSPC;
}
+/*
+ * Check whether a port is still marked free, and if so update the domain
+ * counter accordingly. To be used on function exit paths.
+ */
+static void check_free_port(struct domain *d, evtchn_port_t port)
+{
+ if ( port_is_valid(d, port) &&
+ evtchn_from_port(d, port)->state == ECS_FREE )
+ write_atomic(&d->active_evtchns, d->active_evtchns - 1);
+}
+
void evtchn_free(struct domain *d, struct evtchn *chn)
{
/* Clear pending event to avoid unexpected behavior on re-bind. */
evtchn_port_clear_pending(d, chn);
+ if ( consumer_is_xen(chn) )
+ write_atomic(&d->xen_evtchns, d->xen_evtchns - 1);
+ write_atomic(&d->active_evtchns, d->active_evtchns - 1);
+
/* Reset binding to vcpu0 when the channel is freed. */
chn->state = ECS_FREE;
chn->notify_vcpu_id = 0;
@@ -258,6 +275,7 @@ static long evtchn_alloc_unbound(evtchn_
alloc->port = port;
out:
+ check_free_port(d, port);
spin_unlock(&d->event_lock);
rcu_unlock_domain(d);
@@ -351,6 +369,7 @@ static long evtchn_bind_interdomain(evtc
bind->local_port = lport;
out:
+ check_free_port(ld, lport);
spin_unlock(&ld->event_lock);
if ( ld != rd )
spin_unlock(&rd->event_lock);
@@ -484,7 +503,7 @@ static long evtchn_bind_pirq(evtchn_bind
struct domain *d = current->domain;
struct vcpu *v = d->vcpu[0];
struct pirq *info;
- int port, pirq = bind->pirq;
+ int port = 0, pirq = bind->pirq;
long rc;
if ( (pirq < 0) || (pirq >= d->nr_pirqs) )
@@ -532,6 +551,7 @@ static long evtchn_bind_pirq(evtchn_bind
arch_evtchn_bind_pirq(d, pirq);
out:
+ check_free_port(d, port);
spin_unlock(&d->event_lock);
return rc;
@@ -1005,10 +1025,10 @@ int evtchn_unmask(unsigned int port)
return 0;
}
-
int evtchn_reset(struct domain *d)
{
unsigned int i;
+ int rc = 0;
if ( d != current->domain && !d->controller_pause_count )
return -EINVAL;
@@ -1018,7 +1038,9 @@ int evtchn_reset(struct domain *d)
spin_lock(&d->event_lock);
- if ( d->evtchn_fifo )
+ if ( d->active_evtchns > d->xen_evtchns )
+ rc = -EAGAIN;
+ else if ( d->evtchn_fifo )
{
/* Switching back to 2-level ABI. */
evtchn_fifo_destroy(d);
@@ -1027,7 +1049,7 @@ int evtchn_reset(struct domain *d)
spin_unlock(&d->event_lock);
- return 0;
+ return rc;
}
static long evtchn_set_priority(const struct evtchn_set_priority *set_priority)
@@ -1213,10 +1235,9 @@ int alloc_unbound_xen_event_channel(
spin_lock(&ld->event_lock);
- rc = get_free_port(ld);
+ port = rc = get_free_port(ld);
if ( rc < 0 )
goto out;
- port = rc;
chn = evtchn_from_port(ld, port);
rc = xsm_evtchn_unbound(XSM_TARGET, ld, chn, remote_domid);
@@ -1232,7 +1253,10 @@ int alloc_unbound_xen_event_channel(
spin_unlock(&chn->lock);
+ write_atomic(&ld->xen_evtchns, ld->xen_evtchns + 1);
+
out:
+ check_free_port(ld, port);
spin_unlock(&ld->event_lock);
return rc < 0 ? rc : port;
@@ -1308,6 +1332,7 @@ int evtchn_init(struct domain *d)
return -EINVAL;
}
evtchn_from_port(d, 0)->state = ECS_RESERVED;
+ write_atomic(&d->active_evtchns, 0);
#if MAX_VIRT_CPUS > BITS_PER_LONG
d->poll_mask = xzalloc_array(unsigned long,
@@ -1335,6 +1360,8 @@ void evtchn_destroy(struct domain *d)
for ( i = 0; port_is_valid(d, i); i++ )
evtchn_close(d, i, 0);
+ ASSERT(!d->active_evtchns);
+
clear_global_virq_handlers(d);
evtchn_fifo_destroy(d);
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -345,6 +345,16 @@ struct domain
struct evtchn **evtchn_group[NR_EVTCHN_GROUPS]; /* all other buckets */
unsigned int max_evtchn_port; /* max permitted port number */
unsigned int valid_evtchns; /* number of allocated event channels */
+ /*
+ * Number of in-use event channels. Writers should use write_atomic().
+ * Readers need to use read_atomic() only when not holding event_lock.
+ */
+ unsigned int active_evtchns;
+ /*
+ * Number of event channels used internally by Xen (not subject to
+ * EVTCHNOP_reset). Read/write access like for active_evtchns.
+ */
+ unsigned int xen_evtchns;
spinlock_t event_lock;
const struct evtchn_port_ops *evtchn_port_ops;
struct evtchn_fifo_domain *evtchn_fifo;
["xsa343/xsa343-4.11-2.patch" (application/octet-stream)]
From: Jan Beulich <jbeulich@suse.com>
Subject: evtchn: convert per-channel lock to be IRQ-safe
... in order for send_guest_{global,vcpu}_virq() to be able to make use
of it.
This is part of XSA-343.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Julien Grall <jgrall@amazon.com>
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -248,6 +248,7 @@ static long evtchn_alloc_unbound(evtchn_
int port;
domid_t dom = alloc->dom;
long rc;
+ unsigned long flags;
d = rcu_lock_domain_by_any_id(dom);
if ( d == NULL )
@@ -263,14 +264,14 @@ static long evtchn_alloc_unbound(evtchn_
if ( rc )
goto out;
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_UNBOUND;
if ( (chn->u.unbound.remote_domid = alloc->remote_dom) == DOMID_SELF )
chn->u.unbound.remote_domid = current->domain->domain_id;
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
alloc->port = port;
@@ -283,26 +284,32 @@ static long evtchn_alloc_unbound(evtchn_
}
-static void double_evtchn_lock(struct evtchn *lchn, struct evtchn *rchn)
+static unsigned long double_evtchn_lock(struct evtchn *lchn,
+ struct evtchn *rchn)
{
- if ( lchn < rchn )
+ unsigned long flags;
+
+ if ( lchn <= rchn )
{
- spin_lock(&lchn->lock);
- spin_lock(&rchn->lock);
+ spin_lock_irqsave(&lchn->lock, flags);
+ if ( lchn != rchn )
+ spin_lock(&rchn->lock);
}
else
{
- if ( lchn != rchn )
- spin_lock(&rchn->lock);
+ spin_lock_irqsave(&rchn->lock, flags);
spin_lock(&lchn->lock);
}
+
+ return flags;
}
-static void double_evtchn_unlock(struct evtchn *lchn, struct evtchn *rchn)
+static void double_evtchn_unlock(struct evtchn *lchn, struct evtchn *rchn,
+ unsigned long flags)
{
- spin_unlock(&lchn->lock);
if ( lchn != rchn )
- spin_unlock(&rchn->lock);
+ spin_unlock(&lchn->lock);
+ spin_unlock_irqrestore(&rchn->lock, flags);
}
static long evtchn_bind_interdomain(evtchn_bind_interdomain_t *bind)
@@ -312,6 +319,7 @@ static long evtchn_bind_interdomain(evtc
int lport, rport = bind->remote_port;
domid_t rdom = bind->remote_dom;
long rc;
+ unsigned long flags;
if ( rdom == DOMID_SELF )
rdom = current->domain->domain_id;
@@ -347,7 +355,7 @@ static long evtchn_bind_interdomain(evtc
if ( rc )
goto out;
- double_evtchn_lock(lchn, rchn);
+ flags = double_evtchn_lock(lchn, rchn);
lchn->u.interdomain.remote_dom = rd;
lchn->u.interdomain.remote_port = rport;
@@ -364,7 +372,7 @@ static long evtchn_bind_interdomain(evtc
*/
evtchn_port_set_pending(ld, lchn->notify_vcpu_id, lchn);
- double_evtchn_unlock(lchn, rchn);
+ double_evtchn_unlock(lchn, rchn, flags);
bind->local_port = lport;
@@ -387,6 +395,7 @@ int evtchn_bind_virq(evtchn_bind_virq_t
struct domain *d = current->domain;
int virq = bind->virq, vcpu = bind->vcpu;
int rc = 0;
+ unsigned long flags;
if ( (virq < 0) || (virq >= ARRAY_SIZE(v->virq_to_evtchn)) )
return -EINVAL;
@@ -419,14 +428,14 @@ int evtchn_bind_virq(evtchn_bind_virq_t
chn = evtchn_from_port(d, port);
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_VIRQ;
chn->notify_vcpu_id = vcpu;
chn->u.virq = virq;
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
v->virq_to_evtchn[virq] = bind->port = port;
@@ -443,6 +452,7 @@ static long evtchn_bind_ipi(evtchn_bind_
struct domain *d = current->domain;
int port, vcpu = bind->vcpu;
long rc = 0;
+ unsigned long flags;
if ( (vcpu < 0) || (vcpu >= d->max_vcpus) ||
(d->vcpu[vcpu] == NULL) )
@@ -455,13 +465,13 @@ static long evtchn_bind_ipi(evtchn_bind_
chn = evtchn_from_port(d, port);
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_IPI;
chn->notify_vcpu_id = vcpu;
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
bind->port = port;
@@ -505,6 +515,7 @@ static long evtchn_bind_pirq(evtchn_bind
struct pirq *info;
int port = 0, pirq = bind->pirq;
long rc;
+ unsigned long flags;
if ( (pirq < 0) || (pirq >= d->nr_pirqs) )
return -EINVAL;
@@ -537,14 +548,14 @@ static long evtchn_bind_pirq(evtchn_bind
goto out;
}
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_PIRQ;
chn->u.pirq.irq = pirq;
link_pirq_port(port, chn, v);
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
bind->port = port;
@@ -565,6 +576,7 @@ int evtchn_close(struct domain *d1, int
struct evtchn *chn1, *chn2;
int port2;
long rc = 0;
+ unsigned long flags;
again:
spin_lock(&d1->event_lock);
@@ -664,14 +676,14 @@ int evtchn_close(struct domain *d1, int
BUG_ON(chn2->state != ECS_INTERDOMAIN);
BUG_ON(chn2->u.interdomain.remote_dom != d1);
- double_evtchn_lock(chn1, chn2);
+ flags = double_evtchn_lock(chn1, chn2);
evtchn_free(d1, chn1);
chn2->state = ECS_UNBOUND;
chn2->u.unbound.remote_domid = d1->domain_id;
- double_evtchn_unlock(chn1, chn2);
+ double_evtchn_unlock(chn1, chn2, flags);
goto out;
@@ -679,9 +691,9 @@ int evtchn_close(struct domain *d1, int
BUG();
}
- spin_lock(&chn1->lock);
+ spin_lock_irqsave(&chn1->lock, flags);
evtchn_free(d1, chn1);
- spin_unlock(&chn1->lock);
+ spin_unlock_irqrestore(&chn1->lock, flags);
out:
if ( d2 != NULL )
@@ -701,13 +713,14 @@ int evtchn_send(struct domain *ld, unsig
struct evtchn *lchn, *rchn;
struct domain *rd;
int rport, ret = 0;
+ unsigned long flags;
if ( !port_is_valid(ld, lport) )
return -EINVAL;
lchn = evtchn_from_port(ld, lport);
- spin_lock(&lchn->lock);
+ spin_lock_irqsave(&lchn->lock, flags);
/* Guest cannot send via a Xen-attached event channel. */
if ( unlikely(consumer_is_xen(lchn)) )
@@ -742,7 +755,7 @@ int evtchn_send(struct domain *ld, unsig
}
out:
- spin_unlock(&lchn->lock);
+ spin_unlock_irqrestore(&lchn->lock, flags);
return ret;
}
@@ -1232,6 +1245,7 @@ int alloc_unbound_xen_event_channel(
{
struct evtchn *chn;
int port, rc;
+ unsigned long flags;
spin_lock(&ld->event_lock);
@@ -1244,14 +1258,14 @@ int alloc_unbound_xen_event_channel(
if ( rc )
goto out;
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_UNBOUND;
chn->xen_consumer = get_xen_consumer(notification_fn);
chn->notify_vcpu_id = lvcpu;
chn->u.unbound.remote_domid = remote_domid;
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
write_atomic(&ld->xen_evtchns, ld->xen_evtchns + 1);
@@ -1274,11 +1288,12 @@ void notify_via_xen_event_channel(struct
{
struct evtchn *lchn, *rchn;
struct domain *rd;
+ unsigned long flags;
ASSERT(port_is_valid(ld, lport));
lchn = evtchn_from_port(ld, lport);
- spin_lock(&lchn->lock);
+ spin_lock_irqsave(&lchn->lock, flags);
if ( likely(lchn->state == ECS_INTERDOMAIN) )
{
@@ -1288,7 +1303,7 @@ void notify_via_xen_event_channel(struct
evtchn_port_set_pending(rd, rchn->notify_vcpu_id, rchn);
}
- spin_unlock(&lchn->lock);
+ spin_unlock_irqrestore(&lchn->lock, flags);
}
void evtchn_check_pollers(struct domain *d, unsigned int port)
["xsa343/xsa343-4.11-3.patch" (application/octet-stream)]
From: Jan Beulich <jbeulich@suse.com>
Subject: evtchn: address races with evtchn_reset()
Neither d->evtchn_port_ops nor max_evtchns(d) may be used in an entirely
lock-less manner, as both may change by a racing evtchn_reset(). In the
common case, at least one of the domain's event lock or the per-channel
lock needs to be held. In the specific case of the inter-domain sending
by evtchn_send() and notify_via_xen_event_channel() holding the other
side's per-channel lock is sufficient, as the channel can't change state
without both per-channel locks held. Without such a channel changing
state, evtchn_reset() can't complete successfully.
Lock-free accesses continue to be permitted for the shim (calling some
otherwise internal event channel functions), as this happens while the
domain is in effectively single-threaded mode. Special care also needs
taking for the shim's marking of in-use ports as ECS_RESERVED (allowing
use of such ports in the shim case is okay because switching into and
hence also out of FIFO mode is impossible there).
As a side effect, certain operations on Xen bound event channels which
were mistakenly permitted so far (e.g. unmask or poll) will be refused
now.
This is part of XSA-343.
Reported-by: Julien Grall <jgrall@amazon.com>
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Julien Grall <jgrall@amazon.com>
--- a/xen/arch/x86/irq.c
+++ b/xen/arch/x86/irq.c
@@ -2367,14 +2367,24 @@ static void dump_irqs(unsigned char key)
for ( i = 0; i < action->nr_guests; i++ )
{
+ struct evtchn *evtchn;
+ unsigned int pending = 2, masked = 2;
+
d = action->guest[i];
pirq = domain_irq_to_pirq(d, irq);
info = pirq_info(d, pirq);
+ evtchn = evtchn_from_port(d, info->evtchn);
+ local_irq_disable();
+ if ( spin_trylock(&evtchn->lock) )
+ {
+ pending = evtchn_is_pending(d, evtchn);
+ masked = evtchn_is_masked(d, evtchn);
+ spin_unlock(&evtchn->lock);
+ }
+ local_irq_enable();
printk("%u:%3d(%c%c%c)",
- d->domain_id, pirq,
- evtchn_port_is_pending(d, info->evtchn) ? 'P' : '-',
- evtchn_port_is_masked(d, info->evtchn) ? 'M' : '-',
- (info->masked ? 'M' : '-'));
+ d->domain_id, pirq, "-P?"[pending],
+ "-M?"[masked], info->masked ? 'M' : '-');
if ( i != action->nr_guests )
printk(",");
}
--- a/xen/arch/x86/pv/shim.c
+++ b/xen/arch/x86/pv/shim.c
@@ -616,8 +616,11 @@ void pv_shim_inject_evtchn(unsigned int
if ( port_is_valid(guest, port) )
{
struct evtchn *chn = evtchn_from_port(guest, port);
+ unsigned long flags;
+ spin_lock_irqsave(&chn->lock, flags);
evtchn_port_set_pending(guest, chn->notify_vcpu_id, chn);
+ spin_unlock_irqrestore(&chn->lock, flags);
}
}
--- a/xen/common/event_2l.c
+++ b/xen/common/event_2l.c
@@ -63,8 +63,10 @@ static void evtchn_2l_unmask(struct doma
}
}
-static bool evtchn_2l_is_pending(const struct domain *d, evtchn_port_t port)
+static bool evtchn_2l_is_pending(const struct domain *d,
+ const struct evtchn *evtchn)
{
+ evtchn_port_t port = evtchn->port;
unsigned int max_ports = BITS_PER_EVTCHN_WORD(d) * BITS_PER_EVTCHN_WORD(d);
ASSERT(port < max_ports);
@@ -72,8 +74,10 @@ static bool evtchn_2l_is_pending(const s
guest_test_bit(d, port, &shared_info(d, evtchn_pending)));
}
-static bool evtchn_2l_is_masked(const struct domain *d, evtchn_port_t port)
+static bool evtchn_2l_is_masked(const struct domain *d,
+ const struct evtchn *evtchn)
{
+ evtchn_port_t port = evtchn->port;
unsigned int max_ports = BITS_PER_EVTCHN_WORD(d) * BITS_PER_EVTCHN_WORD(d);
ASSERT(port < max_ports);
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -156,8 +156,9 @@ int evtchn_allocate_port(struct domain *
if ( port_is_valid(d, port) )
{
- if ( evtchn_from_port(d, port)->state != ECS_FREE ||
- evtchn_port_is_busy(d, port) )
+ const struct evtchn *chn = evtchn_from_port(d, port);
+
+ if ( chn->state != ECS_FREE || evtchn_is_busy(d, chn) )
return -EBUSY;
}
else
@@ -770,6 +771,7 @@ void send_guest_vcpu_virq(struct vcpu *v
unsigned long flags;
int port;
struct domain *d;
+ struct evtchn *chn;
ASSERT(!virq_is_global(virq));
@@ -780,7 +782,10 @@ void send_guest_vcpu_virq(struct vcpu *v
goto out;
d = v->domain;
- evtchn_port_set_pending(d, v->vcpu_id, evtchn_from_port(d, port));
+ chn = evtchn_from_port(d, port);
+ spin_lock(&chn->lock);
+ evtchn_port_set_pending(d, v->vcpu_id, chn);
+ spin_unlock(&chn->lock);
out:
spin_unlock_irqrestore(&v->virq_lock, flags);
@@ -809,7 +814,9 @@ static void send_guest_global_virq(struc
goto out;
chn = evtchn_from_port(d, port);
+ spin_lock(&chn->lock);
evtchn_port_set_pending(d, chn->notify_vcpu_id, chn);
+ spin_unlock(&chn->lock);
out:
spin_unlock_irqrestore(&v->virq_lock, flags);
@@ -819,6 +826,7 @@ void send_guest_pirq(struct domain *d, c
{
int port;
struct evtchn *chn;
+ unsigned long flags;
/*
* PV guests: It should not be possible to race with __evtchn_close(). The
@@ -833,7 +841,9 @@ void send_guest_pirq(struct domain *d, c
}
chn = evtchn_from_port(d, port);
+ spin_lock_irqsave(&chn->lock, flags);
evtchn_port_set_pending(d, chn->notify_vcpu_id, chn);
+ spin_unlock_irqrestore(&chn->lock, flags);
}
static struct domain *global_virq_handlers[NR_VIRQS] __read_mostly;
@@ -1028,12 +1038,15 @@ int evtchn_unmask(unsigned int port)
{
struct domain *d = current->domain;
struct evtchn *evtchn;
+ unsigned long flags;
if ( unlikely(!port_is_valid(d, port)) )
return -EINVAL;
evtchn = evtchn_from_port(d, port);
+ spin_lock_irqsave(&evtchn->lock, flags);
evtchn_port_unmask(d, evtchn);
+ spin_unlock_irqrestore(&evtchn->lock, flags);
return 0;
}
@@ -1446,8 +1459,8 @@ static void domain_dump_evtchn_info(stru
printk(" %4u [%d/%d/",
port,
- evtchn_port_is_pending(d, port),
- evtchn_port_is_masked(d, port));
+ evtchn_is_pending(d, chn),
+ evtchn_is_masked(d, chn));
evtchn_port_print_state(d, chn);
printk("]: s=%d n=%d x=%d",
chn->state, chn->notify_vcpu_id, chn->xen_consumer);
--- a/xen/common/event_fifo.c
+++ b/xen/common/event_fifo.c
@@ -295,23 +295,26 @@ static void evtchn_fifo_unmask(struct do
evtchn_fifo_set_pending(v, evtchn);
}
-static bool evtchn_fifo_is_pending(const struct domain *d, evtchn_port_t port)
+static bool evtchn_fifo_is_pending(const struct domain *d,
+ const struct evtchn *evtchn)
{
- const event_word_t *word = evtchn_fifo_word_from_port(d, port);
+ const event_word_t *word = evtchn_fifo_word_from_port(d, evtchn->port);
return word && guest_test_bit(d, EVTCHN_FIFO_PENDING, word);
}
-static bool_t evtchn_fifo_is_masked(const struct domain *d, evtchn_port_t port)
+static bool_t evtchn_fifo_is_masked(const struct domain *d,
+ const struct evtchn *evtchn)
{
- const event_word_t *word = evtchn_fifo_word_from_port(d, port);
+ const event_word_t *word = evtchn_fifo_word_from_port(d, evtchn->port);
return !word || guest_test_bit(d, EVTCHN_FIFO_MASKED, word);
}
-static bool_t evtchn_fifo_is_busy(const struct domain *d, evtchn_port_t port)
+static bool_t evtchn_fifo_is_busy(const struct domain *d,
+ const struct evtchn *evtchn)
{
- const event_word_t *word = evtchn_fifo_word_from_port(d, port);
+ const event_word_t *word = evtchn_fifo_word_from_port(d, evtchn->port);
return word && guest_test_bit(d, EVTCHN_FIFO_LINKED, word);
}
--- a/xen/include/asm-x86/event.h
+++ b/xen/include/asm-x86/event.h
@@ -47,4 +47,10 @@ static inline bool arch_virq_is_global(u
return true;
}
+#ifdef CONFIG_PV_SHIM
+# include <asm/pv/shim.h>
+# define arch_evtchn_is_special(chn) \
+ (pv_shim && (chn)->port && (chn)->state == ECS_RESERVED)
+#endif
+
#endif
--- a/xen/include/xen/event.h
+++ b/xen/include/xen/event.h
@@ -125,6 +125,24 @@ static inline struct evtchn *evtchn_from
return bucket_from_port(d, p) + (p % EVTCHNS_PER_BUCKET);
}
+/*
+ * "usable" as in "by a guest", i.e. Xen consumed channels are assumed to be
+ * taken care of separately where used for Xen's internal purposes.
+ */
+static bool evtchn_usable(const struct evtchn *evtchn)
+{
+ if ( evtchn->xen_consumer )
+ return false;
+
+#ifdef arch_evtchn_is_special
+ if ( arch_evtchn_is_special(evtchn) )
+ return true;
+#endif
+
+ BUILD_BUG_ON(ECS_FREE > ECS_RESERVED);
+ return evtchn->state > ECS_RESERVED;
+}
+
/* Wait on a Xen-attached event channel. */
#define wait_on_xen_event_channel(port, condition) \
do { \
@@ -157,19 +175,24 @@ int evtchn_reset(struct domain *d);
/*
* Low-level event channel port ops.
+ *
+ * All hooks have to be called with a lock held which prevents the channel
+ * from changing state. This may be the domain event lock, the per-channel
+ * lock, or in the case of sending interdomain events also the other side's
+ * per-channel lock. Exceptions apply in certain cases for the PV shim.
*/
struct evtchn_port_ops {
void (*init)(struct domain *d, struct evtchn *evtchn);
void (*set_pending)(struct vcpu *v, struct evtchn *evtchn);
void (*clear_pending)(struct domain *d, struct evtchn *evtchn);
void (*unmask)(struct domain *d, struct evtchn *evtchn);
- bool (*is_pending)(const struct domain *d, evtchn_port_t port);
- bool (*is_masked)(const struct domain *d, evtchn_port_t port);
+ bool (*is_pending)(const struct domain *d, const struct evtchn *evtchn);
+ bool (*is_masked)(const struct domain *d, const struct evtchn *evtchn);
/*
* Is the port unavailable because it's still being cleaned up
* after being closed?
*/
- bool (*is_busy)(const struct domain *d, evtchn_port_t port);
+ bool (*is_busy)(const struct domain *d, const struct evtchn *evtchn);
int (*set_priority)(struct domain *d, struct evtchn *evtchn,
unsigned int priority);
void (*print_state)(struct domain *d, const struct evtchn *evtchn);
@@ -185,38 +208,67 @@ static inline void evtchn_port_set_pendi
unsigned int vcpu_id,
struct evtchn *evtchn)
{
- d->evtchn_port_ops->set_pending(d->vcpu[vcpu_id], evtchn);
+ if ( evtchn_usable(evtchn) )
+ d->evtchn_port_ops->set_pending(d->vcpu[vcpu_id], evtchn);
}
static inline void evtchn_port_clear_pending(struct domain *d,
struct evtchn *evtchn)
{
- d->evtchn_port_ops->clear_pending(d, evtchn);
+ if ( evtchn_usable(evtchn) )
+ d->evtchn_port_ops->clear_pending(d, evtchn);
}
static inline void evtchn_port_unmask(struct domain *d,
struct evtchn *evtchn)
{
- d->evtchn_port_ops->unmask(d, evtchn);
+ if ( evtchn_usable(evtchn) )
+ d->evtchn_port_ops->unmask(d, evtchn);
}
-static inline bool evtchn_port_is_pending(const struct domain *d,
- evtchn_port_t port)
+static inline bool evtchn_is_pending(const struct domain *d,
+ const struct evtchn *evtchn)
{
- return d->evtchn_port_ops->is_pending(d, port);
+ return evtchn_usable(evtchn) && d->evtchn_port_ops->is_pending(d, evtchn);
}
-static inline bool evtchn_port_is_masked(const struct domain *d,
- evtchn_port_t port)
+static inline bool evtchn_port_is_pending(struct domain *d, evtchn_port_t port)
{
- return d->evtchn_port_ops->is_masked(d, port);
+ struct evtchn *evtchn = evtchn_from_port(d, port);
+ bool rc;
+ unsigned long flags;
+
+ spin_lock_irqsave(&evtchn->lock, flags);
+ rc = evtchn_is_pending(d, evtchn);
+ spin_unlock_irqrestore(&evtchn->lock, flags);
+
+ return rc;
+}
+
+static inline bool evtchn_is_masked(const struct domain *d,
+ const struct evtchn *evtchn)
+{
+ return !evtchn_usable(evtchn) || d->evtchn_port_ops->is_masked(d, evtchn);
+}
+
+static inline bool evtchn_port_is_masked(struct domain *d, evtchn_port_t port)
+{
+ struct evtchn *evtchn = evtchn_from_port(d, port);
+ bool rc;
+ unsigned long flags;
+
+ spin_lock_irqsave(&evtchn->lock, flags);
+ rc = evtchn_is_masked(d, evtchn);
+ spin_unlock_irqrestore(&evtchn->lock, flags);
+
+ return rc;
}
-static inline bool evtchn_port_is_busy(const struct domain *d,
- evtchn_port_t port)
+static inline bool evtchn_is_busy(const struct domain *d,
+ const struct evtchn *evtchn)
{
return d->evtchn_port_ops->is_busy &&
- d->evtchn_port_ops->is_busy(d, port);
+ d->evtchn_port_ops->is_busy(d, evtchn);
}
static inline int evtchn_port_set_priority(struct domain *d,
@@ -225,6 +277,8 @@ static inline int evtchn_port_set_priori
{
if ( !d->evtchn_port_ops->set_priority )
return -ENOSYS;
+ if ( !evtchn_usable(evtchn) )
+ return -EACCES;
return d->evtchn_port_ops->set_priority(d, evtchn, priority);
}
["xsa343/xsa343-4.12-1.patch" (application/octet-stream)]
From: Jan Beulich <jbeulich@suse.com>
Subject: evtchn: evtchn_reset() shouldn't succeed with still-open ports
While the function closes all ports, it does so without holding any
lock, and hence racing requests may be issued causing new ports to get
opened. This would have been problematic in particular if such a newly
opened port had a port number above the new implementation limit (i.e.
when switching from FIFO to 2-level) after the reset, as prior to
"evtchn: relax port_is_valid()" this could have led to e.g.
evtchn_close()'s "BUG_ON(!port_is_valid(d2, port2))" to trigger.
Introduce a counter of active ports and check that it's (still) no
larger then the number of Xen internally used ones after obtaining the
necessary lock in evtchn_reset().
As to the access model of the new {active,xen}_evtchns fields - while
all writes get done using write_atomic(), reads ought to use
read_atomic() only when outside of a suitably locked region.
Note that as of now evtchn_bind_virq() and evtchn_bind_ipi() don't have
a need to call check_free_port().
This is part of XSA-343.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Reviewed-by: Julien Grall <jgrall@amazon.com>
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -188,6 +188,8 @@ int evtchn_allocate_port(struct domain *
write_atomic(&d->valid_evtchns, d->valid_evtchns + EVTCHNS_PER_BUCKET);
}
+ write_atomic(&d->active_evtchns, d->active_evtchns + 1);
+
return 0;
}
@@ -211,11 +213,26 @@ static int get_free_port(struct domain *
return -ENOSPC;
}
+/*
+ * Check whether a port is still marked free, and if so update the domain
+ * counter accordingly. To be used on function exit paths.
+ */
+static void check_free_port(struct domain *d, evtchn_port_t port)
+{
+ if ( port_is_valid(d, port) &&
+ evtchn_from_port(d, port)->state == ECS_FREE )
+ write_atomic(&d->active_evtchns, d->active_evtchns - 1);
+}
+
void evtchn_free(struct domain *d, struct evtchn *chn)
{
/* Clear pending event to avoid unexpected behavior on re-bind. */
evtchn_port_clear_pending(d, chn);
+ if ( consumer_is_xen(chn) )
+ write_atomic(&d->xen_evtchns, d->xen_evtchns - 1);
+ write_atomic(&d->active_evtchns, d->active_evtchns - 1);
+
/* Reset binding to vcpu0 when the channel is freed. */
chn->state = ECS_FREE;
chn->notify_vcpu_id = 0;
@@ -258,6 +275,7 @@ static long evtchn_alloc_unbound(evtchn_
alloc->port = port;
out:
+ check_free_port(d, port);
spin_unlock(&d->event_lock);
rcu_unlock_domain(d);
@@ -351,6 +369,7 @@ static long evtchn_bind_interdomain(evtc
bind->local_port = lport;
out:
+ check_free_port(ld, lport);
spin_unlock(&ld->event_lock);
if ( ld != rd )
spin_unlock(&rd->event_lock);
@@ -488,7 +507,7 @@ static long evtchn_bind_pirq(evtchn_bind
struct domain *d = current->domain;
struct vcpu *v = d->vcpu[0];
struct pirq *info;
- int port, pirq = bind->pirq;
+ int port = 0, pirq = bind->pirq;
long rc;
if ( (pirq < 0) || (pirq >= d->nr_pirqs) )
@@ -536,6 +555,7 @@ static long evtchn_bind_pirq(evtchn_bind
arch_evtchn_bind_pirq(d, pirq);
out:
+ check_free_port(d, port);
spin_unlock(&d->event_lock);
return rc;
@@ -1011,10 +1031,10 @@ int evtchn_unmask(unsigned int port)
return 0;
}
-
int evtchn_reset(struct domain *d)
{
unsigned int i;
+ int rc = 0;
if ( d != current->domain && !d->controller_pause_count )
return -EINVAL;
@@ -1024,7 +1044,9 @@ int evtchn_reset(struct domain *d)
spin_lock(&d->event_lock);
- if ( d->evtchn_fifo )
+ if ( d->active_evtchns > d->xen_evtchns )
+ rc = -EAGAIN;
+ else if ( d->evtchn_fifo )
{
/* Switching back to 2-level ABI. */
evtchn_fifo_destroy(d);
@@ -1033,7 +1055,7 @@ int evtchn_reset(struct domain *d)
spin_unlock(&d->event_lock);
- return 0;
+ return rc;
}
static long evtchn_set_priority(const struct evtchn_set_priority *set_priority)
@@ -1219,10 +1241,9 @@ int alloc_unbound_xen_event_channel(
spin_lock(&ld->event_lock);
- rc = get_free_port(ld);
+ port = rc = get_free_port(ld);
if ( rc < 0 )
goto out;
- port = rc;
chn = evtchn_from_port(ld, port);
rc = xsm_evtchn_unbound(XSM_TARGET, ld, chn, remote_domid);
@@ -1238,7 +1259,10 @@ int alloc_unbound_xen_event_channel(
spin_unlock(&chn->lock);
+ write_atomic(&ld->xen_evtchns, ld->xen_evtchns + 1);
+
out:
+ check_free_port(ld, port);
spin_unlock(&ld->event_lock);
return rc < 0 ? rc : port;
@@ -1314,6 +1338,7 @@ int evtchn_init(struct domain *d, unsign
return -EINVAL;
}
evtchn_from_port(d, 0)->state = ECS_RESERVED;
+ write_atomic(&d->active_evtchns, 0);
#if MAX_VIRT_CPUS > BITS_PER_LONG
d->poll_mask = xzalloc_array(unsigned long, BITS_TO_LONGS(d->max_vcpus));
@@ -1340,6 +1365,8 @@ void evtchn_destroy(struct domain *d)
for ( i = 0; port_is_valid(d, i); i++ )
evtchn_close(d, i, 0);
+ ASSERT(!d->active_evtchns);
+
clear_global_virq_handlers(d);
evtchn_fifo_destroy(d);
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -346,6 +346,16 @@ struct domain
struct evtchn **evtchn_group[NR_EVTCHN_GROUPS]; /* all other buckets */
unsigned int max_evtchn_port; /* max permitted port number */
unsigned int valid_evtchns; /* number of allocated event channels */
+ /*
+ * Number of in-use event channels. Writers should use write_atomic().
+ * Readers need to use read_atomic() only when not holding event_lock.
+ */
+ unsigned int active_evtchns;
+ /*
+ * Number of event channels used internally by Xen (not subject to
+ * EVTCHNOP_reset). Read/write access like for active_evtchns.
+ */
+ unsigned int xen_evtchns;
spinlock_t event_lock;
const struct evtchn_port_ops *evtchn_port_ops;
struct evtchn_fifo_domain *evtchn_fifo;
["xsa343/xsa343-4.12-2.patch" (application/octet-stream)]
From: Jan Beulich <jbeulich@suse.com>
Subject: evtchn: convert per-channel lock to be IRQ-safe
... in order for send_guest_{global,vcpu}_virq() to be able to make use
of it.
This is part of XSA-343.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Julien Grall <jgrall@amazon.com>
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -248,6 +248,7 @@ static long evtchn_alloc_unbound(evtchn_
int port;
domid_t dom = alloc->dom;
long rc;
+ unsigned long flags;
d = rcu_lock_domain_by_any_id(dom);
if ( d == NULL )
@@ -263,14 +264,14 @@ static long evtchn_alloc_unbound(evtchn_
if ( rc )
goto out;
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_UNBOUND;
if ( (chn->u.unbound.remote_domid = alloc->remote_dom) == DOMID_SELF )
chn->u.unbound.remote_domid = current->domain->domain_id;
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
alloc->port = port;
@@ -283,26 +284,32 @@ static long evtchn_alloc_unbound(evtchn_
}
-static void double_evtchn_lock(struct evtchn *lchn, struct evtchn *rchn)
+static unsigned long double_evtchn_lock(struct evtchn *lchn,
+ struct evtchn *rchn)
{
- if ( lchn < rchn )
+ unsigned long flags;
+
+ if ( lchn <= rchn )
{
- spin_lock(&lchn->lock);
- spin_lock(&rchn->lock);
+ spin_lock_irqsave(&lchn->lock, flags);
+ if ( lchn != rchn )
+ spin_lock(&rchn->lock);
}
else
{
- if ( lchn != rchn )
- spin_lock(&rchn->lock);
+ spin_lock_irqsave(&rchn->lock, flags);
spin_lock(&lchn->lock);
}
+
+ return flags;
}
-static void double_evtchn_unlock(struct evtchn *lchn, struct evtchn *rchn)
+static void double_evtchn_unlock(struct evtchn *lchn, struct evtchn *rchn,
+ unsigned long flags)
{
- spin_unlock(&lchn->lock);
if ( lchn != rchn )
- spin_unlock(&rchn->lock);
+ spin_unlock(&lchn->lock);
+ spin_unlock_irqrestore(&rchn->lock, flags);
}
static long evtchn_bind_interdomain(evtchn_bind_interdomain_t *bind)
@@ -312,6 +319,7 @@ static long evtchn_bind_interdomain(evtc
int lport, rport = bind->remote_port;
domid_t rdom = bind->remote_dom;
long rc;
+ unsigned long flags;
if ( rdom == DOMID_SELF )
rdom = current->domain->domain_id;
@@ -347,7 +355,7 @@ static long evtchn_bind_interdomain(evtc
if ( rc )
goto out;
- double_evtchn_lock(lchn, rchn);
+ flags = double_evtchn_lock(lchn, rchn);
lchn->u.interdomain.remote_dom = rd;
lchn->u.interdomain.remote_port = rport;
@@ -364,7 +372,7 @@ static long evtchn_bind_interdomain(evtc
*/
evtchn_port_set_pending(ld, lchn->notify_vcpu_id, lchn);
- double_evtchn_unlock(lchn, rchn);
+ double_evtchn_unlock(lchn, rchn, flags);
bind->local_port = lport;
@@ -387,6 +395,7 @@ int evtchn_bind_virq(evtchn_bind_virq_t
struct domain *d = current->domain;
int virq = bind->virq, vcpu = bind->vcpu;
int rc = 0;
+ unsigned long flags;
if ( (virq < 0) || (virq >= ARRAY_SIZE(v->virq_to_evtchn)) )
return -EINVAL;
@@ -424,14 +433,14 @@ int evtchn_bind_virq(evtchn_bind_virq_t
chn = evtchn_from_port(d, port);
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_VIRQ;
chn->notify_vcpu_id = vcpu;
chn->u.virq = virq;
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
v->virq_to_evtchn[virq] = bind->port = port;
@@ -448,6 +457,7 @@ static long evtchn_bind_ipi(evtchn_bind_
struct domain *d = current->domain;
int port, vcpu = bind->vcpu;
long rc = 0;
+ unsigned long flags;
if ( domain_vcpu(d, vcpu) == NULL )
return -ENOENT;
@@ -459,13 +469,13 @@ static long evtchn_bind_ipi(evtchn_bind_
chn = evtchn_from_port(d, port);
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_IPI;
chn->notify_vcpu_id = vcpu;
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
bind->port = port;
@@ -509,6 +519,7 @@ static long evtchn_bind_pirq(evtchn_bind
struct pirq *info;
int port = 0, pirq = bind->pirq;
long rc;
+ unsigned long flags;
if ( (pirq < 0) || (pirq >= d->nr_pirqs) )
return -EINVAL;
@@ -541,14 +552,14 @@ static long evtchn_bind_pirq(evtchn_bind
goto out;
}
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_PIRQ;
chn->u.pirq.irq = pirq;
link_pirq_port(port, chn, v);
evtchn_port_init(d, chn);
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
bind->port = port;
@@ -569,6 +580,7 @@ int evtchn_close(struct domain *d1, int
struct evtchn *chn1, *chn2;
int port2;
long rc = 0;
+ unsigned long flags;
again:
spin_lock(&d1->event_lock);
@@ -668,14 +680,14 @@ int evtchn_close(struct domain *d1, int
BUG_ON(chn2->state != ECS_INTERDOMAIN);
BUG_ON(chn2->u.interdomain.remote_dom != d1);
- double_evtchn_lock(chn1, chn2);
+ flags = double_evtchn_lock(chn1, chn2);
evtchn_free(d1, chn1);
chn2->state = ECS_UNBOUND;
chn2->u.unbound.remote_domid = d1->domain_id;
- double_evtchn_unlock(chn1, chn2);
+ double_evtchn_unlock(chn1, chn2, flags);
goto out;
@@ -683,9 +695,9 @@ int evtchn_close(struct domain *d1, int
BUG();
}
- spin_lock(&chn1->lock);
+ spin_lock_irqsave(&chn1->lock, flags);
evtchn_free(d1, chn1);
- spin_unlock(&chn1->lock);
+ spin_unlock_irqrestore(&chn1->lock, flags);
out:
if ( d2 != NULL )
@@ -705,13 +717,14 @@ int evtchn_send(struct domain *ld, unsig
struct evtchn *lchn, *rchn;
struct domain *rd;
int rport, ret = 0;
+ unsigned long flags;
if ( !port_is_valid(ld, lport) )
return -EINVAL;
lchn = evtchn_from_port(ld, lport);
- spin_lock(&lchn->lock);
+ spin_lock_irqsave(&lchn->lock, flags);
/* Guest cannot send via a Xen-attached event channel. */
if ( unlikely(consumer_is_xen(lchn)) )
@@ -746,7 +759,7 @@ int evtchn_send(struct domain *ld, unsig
}
out:
- spin_unlock(&lchn->lock);
+ spin_unlock_irqrestore(&lchn->lock, flags);
return ret;
}
@@ -1238,6 +1251,7 @@ int alloc_unbound_xen_event_channel(
{
struct evtchn *chn;
int port, rc;
+ unsigned long flags;
spin_lock(&ld->event_lock);
@@ -1250,14 +1264,14 @@ int alloc_unbound_xen_event_channel(
if ( rc )
goto out;
- spin_lock(&chn->lock);
+ spin_lock_irqsave(&chn->lock, flags);
chn->state = ECS_UNBOUND;
chn->xen_consumer = get_xen_consumer(notification_fn);
chn->notify_vcpu_id = lvcpu;
chn->u.unbound.remote_domid = remote_domid;
- spin_unlock(&chn->lock);
+ spin_unlock_irqrestore(&chn->lock, flags);
write_atomic(&ld->xen_evtchns, ld->xen_evtchns + 1);
@@ -1280,11 +1294,12 @@ void notify_via_xen_event_channel(struct
{
struct evtchn *lchn, *rchn;
struct domain *rd;
+ unsigned long flags;
ASSERT(port_is_valid(ld, lport));
lchn = evtchn_from_port(ld, lport);
- spin_lock(&lchn->lock);
+ spin_lock_irqsave(&lchn->lock, flags);
if ( likely(lchn->state == ECS_INTERDOMAIN) )
{
@@ -1294,7 +1309,7 @@ void notify_via_xen_event_channel(struct
evtchn_port_set_pending(rd, rchn->notify_vcpu_id, rchn);
}
- spin_unlock(&lchn->lock);
+ spin_unlock_irqrestore(&lchn->lock, flags);
}
void evtchn_check_pollers(struct domain *d, unsigned int port)
["xsa343/xsa343-4.12-3.patch" (application/octet-stream)]
From: Jan Beulich <jbeulich@suse.com>
Subject: evtchn: address races with evtchn_reset()
Neither d->evtchn_port_ops nor max_evtchns(d) may be used in an entirely
lock-less manner, as both may change by a racing evtchn_reset(). In the
common case, at least one of the domain's event lock or the per-channel
lock needs to be held. In the specific case of the inter-domain sending
by evtchn_send() and notify_via_xen_event_channel() holding the other
side's per-channel lock is sufficient, as the channel can't change state
without both per-channel locks held. Without such a channel changing
state, evtchn_reset() can't complete successfully.
Lock-free accesses continue to be permitted for the shim (calling some
otherwise internal event channel functions), as this happens while the
domain is in effectively single-threaded mode. Special care also needs
taking for the shim's marking of in-use ports as ECS_RESERVED (allowing
use of such ports in the shim case is okay because switching into and
hence also out of FIFO mode is impossible there).
As a side effect, certain operations on Xen bound event channels which
were mistakenly permitted so far (e.g. unmask or poll) will be refused
now.
This is part of XSA-343.
Reported-by: Julien Grall <jgrall@amazon.com>
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Julien Grall <jgrall@amazon.com>
--- a/xen/arch/x86/irq.c
+++ b/xen/arch/x86/irq.c
@@ -2364,14 +2364,24 @@ static void dump_irqs(unsigned char key)
for ( i = 0; i < action->nr_guests; i++ )
{
+ struct evtchn *evtchn;
+ unsigned int pending = 2, masked = 2;
+
d = action->guest[i];
pirq = domain_irq_to_pirq(d, irq);
info = pirq_info(d, pirq);
+ evtchn = evtchn_from_port(d, info->evtchn);
+ local_irq_disable();
+ if ( spin_trylock(&evtchn->lock) )
+ {
+ pending = evtchn_is_pending(d, evtchn);
+ masked = evtchn_is_masked(d, evtchn);
+ spin_unlock(&evtchn->lock);
+ }
+ local_irq_enable();
printk("%u:%3d(%c%c%c)",
- d->domain_id, pirq,
- evtchn_port_is_pending(d, info->evtchn) ? 'P' : '-',
- evtchn_port_is_masked(d, info->evtchn) ? 'M' : '-',
- (info->masked ? 'M' : '-'));
+ d->domain_id, pirq, "-P?"[pending],
+ "-M?"[masked], info->masked ? 'M' : '-');
if ( i != action->nr_guests )
printk(",");
}
--- a/xen/arch/x86/pv/shim.c
+++ b/xen/arch/x86/pv/shim.c
@@ -662,8 +662,11 @@ void pv_shim_inject_evtchn(unsigned int
if ( port_is_valid(guest, port) )
{
struct evtchn *chn = evtchn_from_port(guest, port);
+ unsigned long flags;
+ spin_lock_irqsave(&chn->lock, flags);
evtchn_port_set_pending(guest, chn->notify_vcpu_id, chn);
+ spin_unlock_irqrestore(&chn->lock, flags);
}
}
--- a/xen/common/event_2l.c
+++ b/xen/common/event_2l.c
@@ -63,8 +63,10 @@ static void evtchn_2l_unmask(struct doma
}
}
-static bool evtchn_2l_is_pending(const struct domain *d, evtchn_port_t port)
+static bool evtchn_2l_is_pending(const struct domain *d,
+ const struct evtchn *evtchn)
{
+ evtchn_port_t port = evtchn->port;
unsigned int max_ports = BITS_PER_EVTCHN_WORD(d) * BITS_PER_EVTCHN_WORD(d);
ASSERT(port < max_ports);
@@ -72,8 +74,10 @@ static bool evtchn_2l_is_pending(const s
guest_test_bit(d, port, &shared_info(d, evtchn_pending)));
}
-static bool evtchn_2l_is_masked(const struct domain *d, evtchn_port_t port)
+static bool evtchn_2l_is_masked(const struct domain *d,
+ const struct evtchn *evtchn)
{
+ evtchn_port_t port = evtchn->port;
unsigned int max_ports = BITS_PER_EVTCHN_WORD(d) * BITS_PER_EVTCHN_WORD(d);
ASSERT(port < max_ports);
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -156,8 +156,9 @@ int evtchn_allocate_port(struct domain *
if ( port_is_valid(d, port) )
{
- if ( evtchn_from_port(d, port)->state != ECS_FREE ||
- evtchn_port_is_busy(d, port) )
+ const struct evtchn *chn = evtchn_from_port(d, port);
+
+ if ( chn->state != ECS_FREE || evtchn_is_busy(d, chn) )
return -EBUSY;
}
else
@@ -774,6 +775,7 @@ void send_guest_vcpu_virq(struct vcpu *v
unsigned long flags;
int port;
struct domain *d;
+ struct evtchn *chn;
ASSERT(!virq_is_global(virq));
@@ -784,7 +786,10 @@ void send_guest_vcpu_virq(struct vcpu *v
goto out;
d = v->domain;
- evtchn_port_set_pending(d, v->vcpu_id, evtchn_from_port(d, port));
+ chn = evtchn_from_port(d, port);
+ spin_lock(&chn->lock);
+ evtchn_port_set_pending(d, v->vcpu_id, chn);
+ spin_unlock(&chn->lock);
out:
spin_unlock_irqrestore(&v->virq_lock, flags);
@@ -813,7 +818,9 @@ void send_guest_global_virq(struct domai
goto out;
chn = evtchn_from_port(d, port);
+ spin_lock(&chn->lock);
evtchn_port_set_pending(d, chn->notify_vcpu_id, chn);
+ spin_unlock(&chn->lock);
out:
spin_unlock_irqrestore(&v->virq_lock, flags);
@@ -823,6 +830,7 @@ void send_guest_pirq(struct domain *d, c
{
int port;
struct evtchn *chn;
+ unsigned long flags;
/*
* PV guests: It should not be possible to race with __evtchn_close(). The
@@ -837,7 +845,9 @@ void send_guest_pirq(struct domain *d, c
}
chn = evtchn_from_port(d, port);
+ spin_lock_irqsave(&chn->lock, flags);
evtchn_port_set_pending(d, chn->notify_vcpu_id, chn);
+ spin_unlock_irqrestore(&chn->lock, flags);
}
static struct domain *global_virq_handlers[NR_VIRQS] __read_mostly;
@@ -1034,12 +1044,15 @@ int evtchn_unmask(unsigned int port)
{
struct domain *d = current->domain;
struct evtchn *evtchn;
+ unsigned long flags;
if ( unlikely(!port_is_valid(d, port)) )
return -EINVAL;
evtchn = evtchn_from_port(d, port);
+ spin_lock_irqsave(&evtchn->lock, flags);
evtchn_port_unmask(d, evtchn);
+ spin_unlock_irqrestore(&evtchn->lock, flags);
return 0;
}
@@ -1449,8 +1462,8 @@ static void domain_dump_evtchn_info(stru
printk(" %4u [%d/%d/",
port,
- evtchn_port_is_pending(d, port),
- evtchn_port_is_masked(d, port));
+ evtchn_is_pending(d, chn),
+ evtchn_is_masked(d, chn));
evtchn_port_print_state(d, chn);
printk("]: s=%d n=%d x=%d",
chn->state, chn->notify_vcpu_id, chn->xen_consumer);
--- a/xen/common/event_fifo.c
+++ b/xen/common/event_fifo.c
@@ -296,23 +296,26 @@ static void evtchn_fifo_unmask(struct do
evtchn_fifo_set_pending(v, evtchn);
}
-static bool evtchn_fifo_is_pending(const struct domain *d, evtchn_port_t port)
+static bool evtchn_fifo_is_pending(const struct domain *d,
+ const struct evtchn *evtchn)
{
- const event_word_t *word = evtchn_fifo_word_from_port(d, port);
+ const event_word_t *word = evtchn_fifo_word_from_port(d, evtchn->port);
return word && guest_test_bit(d, EVTCHN_FIFO_PENDING, word);
}
-static bool_t evtchn_fifo_is_masked(const struct domain *d, evtchn_port_t port)
+static bool_t evtchn_fifo_is_masked(const struct domain *d,
+ const struct evtchn *evtchn)
{
- const event_word_t *word = evtchn_fifo_word_from_port(d, port);
+ const event_word_t *word = evtchn_fifo_word_from_port(d, evtchn->port);
return !word || guest_test_bit(d, EVTCHN_FIFO_MASKED, word);
}
-static bool_t evtchn_fifo_is_busy(const struct domain *d, evtchn_port_t port)
+static bool_t evtchn_fifo_is_busy(const struct domain *d,
+ const struct evtchn *evtchn)
{
- const event_word_t *word = evtchn_fifo_word_from_port(d, port);
+ const event_word_t *word = evtchn_fifo_word_from_port(d, evtchn->port);
return word && guest_test_bit(d, EVTCHN_FIFO_LINKED, word);
}
--- a/xen/include/asm-x86/event.h
+++ b/xen/include/asm-x86/event.h
@@ -47,4 +47,10 @@ static inline bool arch_virq_is_global(u
return true;
}
+#ifdef CONFIG_PV_SHIM
+# include <asm/pv/shim.h>
+# define arch_evtchn_is_special(chn) \
+ (pv_shim && (chn)->port && (chn)->state == ECS_RESERVED)
+#endif
+
#endif
--- a/xen/include/xen/event.h
+++ b/xen/include/xen/event.h
@@ -133,6 +133,24 @@ static inline struct evtchn *evtchn_from
return bucket_from_port(d, p) + (p % EVTCHNS_PER_BUCKET);
}
+/*
+ * "usable" as in "by a guest", i.e. Xen consumed channels are assumed to be
+ * taken care of separately where used for Xen's internal purposes.
+ */
+static bool evtchn_usable(const struct evtchn *evtchn)
+{
+ if ( evtchn->xen_consumer )
+ return false;
+
+#ifdef arch_evtchn_is_special
+ if ( arch_evtchn_is_special(evtchn) )
+ return true;
+#endif
+
+ BUILD_BUG_ON(ECS_FREE > ECS_RESERVED);
+ return evtchn->state > ECS_RESERVED;
+}
+
/* Wait on a Xen-attached event channel. */
#define wait_on_xen_event_channel(port, condition) \
do { \
@@ -165,19 +183,24 @@ int evtchn_reset(struct domain *d);
/*
* Low-level event channel port ops.
+ *
+ * All hooks have to be called with a lock held which prevents the channel
+ * from changing state. This may be the domain event lock, the per-channel
+ * lock, or in the case of sending interdomain events also the other side's
+ * per-channel lock. Exceptions apply in certain cases for the PV shim.
*/
struct evtchn_port_ops {
void (*init)(struct domain *d, struct evtchn *evtchn);
void (*set_pending)(struct vcpu *v, struct evtchn *evtchn);
void (*clear_pending)(struct domain *d, struct evtchn *evtchn);
void (*unmask)(struct domain *d, struct evtchn *evtchn);
- bool (*is_pending)(const struct domain *d, evtchn_port_t port);
- bool (*is_masked)(const struct domain *d, evtchn_port_t port);
+ bool (*is_pending)(const struct domain *d, const struct evtchn *evtchn);
+ bool (*is_masked)(const struct domain *d, const struct evtchn *evtchn);
/*
* Is the port unavailable because it's still being cleaned up
* after being closed?
*/
- bool (*is_busy)(const struct domain *d, evtchn_port_t port);
+ bool (*is_busy)(const struct domain *d, const struct evtchn *evtchn);
int (*set_priority)(struct domain *d, struct evtchn *evtchn,
unsigned int priority);
void (*print_state)(struct domain *d, const struct evtchn *evtchn);
@@ -193,38 +216,67 @@ static inline void evtchn_port_set_pendi
unsigned int vcpu_id,
struct evtchn *evtchn)
{
- d->evtchn_port_ops->set_pending(d->vcpu[vcpu_id], evtchn);
+ if ( evtchn_usable(evtchn) )
+ d->evtchn_port_ops->set_pending(d->vcpu[vcpu_id], evtchn);
}
static inline void evtchn_port_clear_pending(struct domain *d,
struct evtchn *evtchn)
{
- d->evtchn_port_ops->clear_pending(d, evtchn);
+ if ( evtchn_usable(evtchn) )
+ d->evtchn_port_ops->clear_pending(d, evtchn);
}
static inline void evtchn_port_unmask(struct domain *d,
struct evtchn *evtchn)
{
- d->evtchn_port_ops->unmask(d, evtchn);
+ if ( evtchn_usable(evtchn) )
+ d->evtchn_port_ops->unmask(d, evtchn);
}
-static inline bool evtchn_port_is_pending(const struct domain *d,
- evtchn_port_t port)
+static inline bool evtchn_is_pending(const struct domain *d,
+ const struct evtchn *evtchn)
{
- return d->evtchn_port_ops->is_pending(d, port);
+ return evtchn_usable(evtchn) && d->evtchn_port_ops->is_pending(d, evtchn);
}
-static inline bool evtchn_port_is_masked(const struct domain *d,
- evtchn_port_t port)
+static inline bool evtchn_port_is_pending(struct domain *d, evtchn_port_t port)
{
- return d->evtchn_port_ops->is_masked(d, port);
+ struct evtchn *evtchn = evtchn_from_port(d, port);
+ bool rc;
+ unsigned long flags;
+
+ spin_lock_irqsave(&evtchn->lock, flags);
+ rc = evtchn_is_pending(d, evtchn);
+ spin_unlock_irqrestore(&evtchn->lock, flags);
+
+ return rc;
+}
+
+static inline bool evtchn_is_masked(const struct domain *d,
+ const struct evtchn *evtchn)
+{
+ return !evtchn_usable(evtchn) || d->evtchn_port_ops->is_masked(d, evtchn);
+}
+
+static inline bool evtchn_port_is_masked(struct domain *d, evtchn_port_t port)
+{
+ struct evtchn *evtchn = evtchn_from_port(d, port);
+ bool rc;
+ unsigned long flags;
+
+ spin_lock_irqsave(&evtchn->lock, flags);
+ rc = evtchn_is_masked(d, evtchn);
+ spin_unlock_irqrestore(&evtchn->lock, flags);
+
+ return rc;
}
-static inline bool evtchn_port_is_busy(const struct domain *d,
- evtchn_port_t port)
+static inline bool evtchn_is_busy(const struct domain *d,
+ const struct evtchn *evtchn)
{
return d->evtchn_port_ops->is_busy &&
- d->evtchn_port_ops->is_busy(d, port);
+ d->evtchn_port_ops->is_busy(d, evtchn);
}
static inline int evtchn_port_set_priority(struct domain *d,
@@ -233,6 +285,8 @@ static inline int evtchn_port_set_priori
{
if ( !d->evtchn_port_ops->set_priority )
return -ENOSYS;
+ if ( !evtchn_usable(evtchn) )
+ return -EACCES;
return d->evtchn_port_ops->set_priority(d, evtchn, priority);
}
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic