[prev in list] [next in list] [prev in thread] [next in thread]
List: imap
Subject: Re: [Imap-protocol] Best way to support HTTP PUT based 'push' notifications from IMAP?
From: Alexey Melnikov <alexey.melnikov () isode ! com>
Date: 2013-09-08 20:22:32
Message-ID: 522CDC88.4070607 () isode ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On 03/09/2013 11:53, Andrew Sutherland wrote:
> On devices that either have difficulty maintaining persistent TCP
> connections or have power concerns about doing so, maintaining an IMAP
> connection for IDLE/NOTIFY purposes is undesirable and it would be
> great if the IMAP server could generate a notification via other means.
>
> For the Firefox OS e-mail client, one of the Web APIs the Firefox OS
> platform makes available and that Mozilla and other contributors are
> trying to standardize is the "Push API"
> (http://www.w3.org/TR/push-api/) under the auspices of the webapps WG
> (http://www.w3.org/2008/webapps/). The API is somewhat decoupled from
> the actual notification mechanism. Mozilla's reference server
> protocol is documented at
> https://wiki.mozilla.org/WebAPI/SimplePush/Protocol.
>
> From the IMAP server's perspective, the process is that the client
> uses the Push API to get a URL and then the client tells the IMAP
> server the URL. To notify the client, the IMAP server issues a PUT
> request to the URL. The only data payload is the 'version' which is a
> monotonically increasing 64-bit integer and allows for a degree of
> idempotency. The only key thing for the version is that if the server
> doesn't increase the value since the last time the client heard it,
> the client won't receive a new notification.
>
> *Our application-level goal is that rather than checking every N
> minutes, after each synchronization attempt we refresh our
> notification request before disconnecting from the IMAP server. The
> IMAP server would then send a ping notification when new messages are
> received for the account/inbox/whatever. *
>
> Details/thoughts from my investigation are found below. In general,
> I'm very interested in what server implementers think and would be
> willing to adopt, especially for things that might be
> enabled-by-default in deployments. It's been indicated to me that we
> (Mozilla) have engineering resources that could be used to help
> provide patches to open source implementation to help make this happen.
>
> Note that any solution that satisfies our needs could probably also
> satisfy similar needs. For example, the Firefox OS Calendar App makes
> use of the Google Calendar API's push notifications mechanism as
> documented at https://developers.google.com/google-apps/calendar/v3/push
>
>
> # Existing Work
>
> ## XAPPLEPUSHSERVICE
>
> The only thing I've seen in the wild seems to be XAPPLEPUSHSERVICE
> advertised by Yahoo's IMAP server and Apple's dovecot fork. The most
> useful information I've found on this is
> http://www.dovecot.org/list/dovecot/2012-August/067682.html which
> leads us to the implementation of XAPPLEPUSHSERVICE at
> http://www.opensource.apple.com/source/dovecot/dovecot-239.8/dovecot/src/imap/cmd-x-apple-push-service.c \
> which tells us the syntax of the command. While the idea is similar,
> XAPPLEPUSHSERVICE does not seem to support alternate endpoints to
> Apple's Push Notification Service and so is not useful for our purposes.
>
> ## sieve/enotify
>
> The sieve enotify extension (http://tools.ietf.org/html/rfc5435) seems
> like the only standadized mechanism capable of implementing such
> logic. Unfortunately, there is no HTTP/URL notification listed in the
> registry at
> https://www.iana.org/assignments/sieve-notification/sieve-notification.xhtml.
> 'mailto' is a defined mechanism that has some support (ex: Pigeonhole
> on Dovecot), and it's possible for Mozilla to stand-up a proxy to help
> bridge this gap.
mailto: can be sufficient for some gatewaying cases, but I suggest you
define a new "http" notification mechanism. I would be happy to help you
out with that.
> The main problem with sieve is that adoption does not seem
> particularly wide-spread and discoverability when supported is
> limited, although I'm hoping that http://tools.ietf.org/html/rfc6785
> could help improve that.
If you only care about notifications generated due to new mail
deliveries, than you can use ManageSieve:
<http://www.rfc-editor.org/rfc/rfc5804.txt>. This is quite common when
Sieve is supported on delivery.
> # Potential Workarounds
>
> I think the following options exist to make this work today:
>
> ## sieve mailto bridge
>
> Stand up an SMTP-receiving proxy so that sieve/enotify/mailto can be
> used to trigger instead of HTTP PUT. Use sieve to set up a rule to
> generate mail notifications every time a message is received. No
> message contents would be sent. This might result in a large number
> of notifications since it's not obvious to me how to write a stateful
> sieve script.
Metadata can be used, as you suggested below.
> ## Hosted IMAP Proxy (Undesirable due to inherent credential/data
> exposure)
>
> A proposal from those working on the back-end of Mozilla's push
> notification service is an HTTPS-accessible proxy. Instead of having
> the device maintain the long-lived IMAP connection, the device has a
> server do that on its behalf. The very undesirable aspect of such a
> solution is that while the proxy need not persist the credentials to
> disk, it does need to know them to establish the connection using
> traditional LOGIN. And even if a challenge-response authentication
> mechanism was used in such a way that the proxy server could not
> establish additional connections without device support, the proxy (or
> whatever pretends to be the proxy) would still have a fully authorized
> IMAP connection to use as it chooses until the connection is dropped.
>
>
> # Potential Standards Approaches
>
> ## Create a new capability, ex: PUSH-NOTIFY-HTTP=PUT
>
> Create a "PUSH-NOTIFY-HTTP PUT <URL> <BODY PAYLOAD>" capability with
> very simple semantics:
Did you mean an IMAP capability?
> - Each request is one-shot. The request lives until it is fired.
> - At once most one request per URL; new requests replace the old request.
> - An HTTP PUT is triggered to the given URL with content-type
> application/x-www-form-urlencoded with the provided body payload.
> - Body payload is limited to something really small; we only need to
> be able to encode "version=<64bitnumber>"
> - Delivery to the Inbox or maybe just delivery to the account in
> general in case a filter files the message somewhere. Messages filed
> to the junk folder would not trigger the notification.
>
> ## sieve enotify HTTP PUT extension
This would be my preferred method.
> Since it's not likely for adoption to happen immediately anyways and
> sieve is extremely powerful it seems like a reasonable choice to avoid
> introducing a new feature which is doomed to feature creep.
>
> Because sieve is not particularly stateful the main issue would be how
> to achieve single-shot semantics. This might be best addressed by
> doing something like the servermetadata example at
> http://tools.ietf.org/html/rfc5490#section-4.1 where the client would
> use metadata to control the process and the URL notification function
> would have built-in idempotent semantics.
A particular notification mechanism can support duplicate suppression,
although I am not sure that that is what you are asking for.
Metadata works otherwise though.
> Each time the client connected it would refresh the 'verson' it uses
> and which is used in the notification invocation. Then when the sieve
> rule next fires, the URL/payload changes so that the notify action
> actually goes through.
>
> Multiple consumers of such notifications would be handled during
> rewrites of the sieve rules. A new device setting up for push
> notifications would identify its rule based on the URL the push API
> tells it. It would write its rule to the sieve store along with the
> current timestamp for the rule. It would also cull any other rules of
> the same variety whose timestamp more than twice as old as the
> rule-rewrite-interval based on the timestamp.
>
> ## custom sieve extension
>
> Realize the above state management logic is too crazy. Internalize the
> one-shot state management, still using metadata somehow.
It is better to extend enotify, unless you can see some severe
limitations in the enotify framework.
> ## Extend NOTIFY somehow
>
> This seems like it would be a mismatch since the NOTIFY subscription
> is inherently associated with the connection lifetime and any other
> lifetime would be weird.
Agreed.
> ## Define CONTEXT=pushnotify
>
> Similar to NOTIFY, this seems like a mismatch.
>
>
> Thanks for any/all responses,
> Andrew Sutherland
Best Regards,
Alexey
[Attachment #5 (text/html)]
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 03/09/2013 11:53, Andrew Sutherland
wrote:<br>
</div>
<blockquote cite="mid:5225BF8C.7060602@mozilla.com" type="cite">On
devices that either have difficulty maintaining persistent TCP
connections or have power concerns about doing so, maintaining an
IMAP connection for IDLE/NOTIFY purposes is undesirable and it
would be great if the IMAP server could generate a notification
via other means. <br>
<br>
For the Firefox OS e-mail client, one of the Web APIs the Firefox
OS platform makes available and that Mozilla and other
contributors are trying to standardize is the "Push API" (<a
class="moz-txt-link-freetext"
href="http://www.w3.org/TR/push-api/">http://www.w3.org/TR/push-api/</a>)
under the auspices of the webapps WG (<a
class="moz-txt-link-freetext"
href="http://www.w3.org/2008/webapps/">http://www.w3.org/2008/webapps/</a>).
The API is somewhat decoupled from the actual notification
mechanism. Mozilla's reference server protocol is documented at <a
class="moz-txt-link-freetext"
href="https://wiki.mozilla.org/WebAPI/SimplePush/Protocol">https://wiki.mozilla.org/WebAPI/SimplePush/Protocol</a>.
<br>
<br>
From the IMAP server's perspective, the process is that the client
uses the Push API to get a URL and then the client tells the IMAP
server the URL. To notify the client, the IMAP server issues a
PUT request to the URL. The only data payload is the 'version'
which is a monotonically increasing 64-bit integer and allows for
a degree of idempotency. The only key thing for the version is
that if the server doesn't increase the value since the last time
the client heard it, the client won't receive a new notification.
<br>
<br>
<b>Our application-level goal is that rather than checking every N
minutes, after each synchronization attempt we refresh our
notification request before disconnecting from the IMAP server.
The IMAP server would then send a ping notification when new
messages are received for the account/inbox/whatever. </b><br>
<br>
Details/thoughts from my investigation are found below. In
general, I'm very interested in what server implementers think and
would be willing to adopt, especially for things that might be
enabled-by-default in deployments. It's been indicated to me that
we (Mozilla) have engineering resources that could be used to help
provide patches to open source implementation to help make this
happen. <br>
<br>
Note that any solution that satisfies our needs could probably
also satisfy similar needs. For example, the Firefox OS Calendar
App makes use of the Google Calendar API's push notifications
mechanism as documented at <a class="moz-txt-link-freetext"
href="https://developers.google.com/google-apps/calendar/v3/push">https://developers.google.com/google-apps/calendar/v3/push</a>
<br>
<br>
<br>
# Existing Work <br>
<br>
## XAPPLEPUSHSERVICE <br>
<br>
The only thing I've seen in the wild seems to be XAPPLEPUSHSERVICE
advertised by Yahoo's IMAP server and Apple's dovecot fork. The
most useful information I've found on this is <a
class="moz-txt-link-freetext"
href="http://www.dovecot.org/list/dovecot/2012-August/067682.html">http://www.dovecot.org/list/dovecot/2012-August/067682.html</a>
which leads us to the implementation of XAPPLEPUSHSERVICE at <a
class="moz-txt-link-freetext"
href="http://www.opensource.apple.com/source/dovecot/dovecot-239.8/dovecot/src/imap/cm \
d-x-apple-push-service.c">http://www.opensource.apple.com/source/dovecot/dovecot-239.8/dovecot/src/imap/cmd-x-apple-push-service.c</a>
which tells us the syntax of the command. While the idea is
similar, XAPPLEPUSHSERVICE does not seem to support alternate
endpoints to Apple's Push Notification Service and so is not
useful for our purposes. <br>
<br>
## sieve/enotify <br>
<br>
The sieve enotify extension (<a class="moz-txt-link-freetext"
href="http://tools.ietf.org/html/rfc5435">http://tools.ietf.org/html/rfc5435</a>)
seems like the only standadized mechanism capable of implementing
such logic. Unfortunately, there is no HTTP/URL notification
listed in the registry at <a class="moz-txt-link-freetext"
href="https://www.iana.org/assignments/sieve-notification/sieve-notification.xhtml">https://www.iana.org/assignments/sieve-notification/sieve-notification.xhtml</a>.
'mailto' is a defined mechanism that has some support (ex:
Pigeonhole on Dovecot), and it's possible for Mozilla to stand-up
a proxy to help bridge this gap.<br>
</blockquote>
<br>
mailto: can be sufficient for some gatewaying cases, but I suggest
you define a new "http" notification mechanism. I would be happy to
help you out with that.<br>
<br>
<blockquote cite="mid:5225BF8C.7060602@mozilla.com" type="cite">The
main problem with sieve is that adoption does not seem
particularly wide-spread and discoverability when supported is
limited, although I'm hoping that <a
class="moz-txt-link-freetext"
href="http://tools.ietf.org/html/rfc6785">http://tools.ietf.org/html/rfc6785</a>
could help improve that. <br>
</blockquote>
If you only care about notifications generated due to new mail
deliveries, than you can use ManageSieve:
<a class="moz-txt-link-rfc2396E" \
href="http://www.rfc-editor.org/rfc/rfc5804.txt"><http://www.rfc-editor.org/rfc/rfc5804.txt></a>. \
This is quite common when Sieve is supported on delivery.<br>
<br>
<blockquote cite="mid:5225BF8C.7060602@mozilla.com" type="cite">#
Potential Workarounds <br>
<br>
I think the following options exist to make this work today: <br>
<br>
## sieve mailto bridge <br>
<br>
Stand up an SMTP-receiving proxy so that sieve/enotify/mailto can
be used to trigger instead of HTTP PUT. Use sieve to set up a
rule to generate mail notifications every time a message is
received. No message contents would be sent. This might result
in a large number of notifications since it's not obvious to me
how to write a stateful sieve script. <br>
</blockquote>
<br>
Metadata can be used, as you suggested below.<br>
<br>
<blockquote cite="mid:5225BF8C.7060602@mozilla.com" type="cite">##
Hosted IMAP Proxy (Undesirable due to inherent credential/data
exposure) <br>
<br>
A proposal from those working on the back-end of Mozilla's push
notification service is an HTTPS-accessible proxy. Instead of
having the device maintain the long-lived IMAP connection, the
device has a server do that on its behalf. The very undesirable
aspect of such a solution is that while the proxy need not persist
the credentials to disk, it does need to know them to establish
the connection using traditional LOGIN. And even if a
challenge-response authentication mechanism was used in such a way
that the proxy server could not establish additional connections
without device support, the proxy (or whatever pretends to be the
proxy) would still have a fully authorized IMAP connection to use
as it chooses until the connection is dropped. <br>
<br>
<br>
# Potential Standards Approaches <br>
<br>
## Create a new capability, ex: PUSH-NOTIFY-HTTP=PUT <br>
<br>
Create a "PUSH-NOTIFY-HTTP PUT <URL> <BODY PAYLOAD>"
capability with very simple semantics: <br>
</blockquote>
Did you mean an IMAP capability?<br>
<blockquote cite="mid:5225BF8C.7060602@mozilla.com" type="cite"> -
Each request is one-shot. The request lives until it is fired. <br>
- At once most one request per URL; new requests replace the old
request. <br>
- An HTTP PUT is triggered to the given URL with content-type
application/x-www-form-urlencoded with the provided body payload.
<br>
- Body payload is limited to something really small; we only need
to be able to encode "version=<64bitnumber>" <br>
- Delivery to the Inbox or maybe just delivery to the account in
general in case a filter files the message somewhere. Messages
filed to the junk folder would not trigger the notification. <br>
<br>
## sieve enotify HTTP PUT extension <br>
</blockquote>
<br>
This would be my preferred method.<br>
<br>
<blockquote cite="mid:5225BF8C.7060602@mozilla.com" type="cite">
Since it's not likely for adoption to happen immediately anyways
and sieve is extremely powerful it seems like a reasonable choice
to avoid introducing a new feature which is doomed to feature
creep. <br>
<br>
Because sieve is not particularly stateful the main issue would be
how to achieve single-shot semantics. This might be best
addressed by doing something like the servermetadata example at <a
class="moz-txt-link-freetext"
href="http://tools.ietf.org/html/rfc5490#section-4.1">http://tools.ietf.org/html/rfc5490#section-4.1</a>
where the client would use metadata to control the process and the
URL notification function would have built-in idempotent
semantics.</blockquote>
A particular notification mechanism can support duplicate
suppression, although I am not sure that that is what you are asking
for.<br>
<br>
Metadata works otherwise though.<br>
<blockquote cite="mid:5225BF8C.7060602@mozilla.com" type="cite">Each
time the client connected it would refresh the 'verson' it uses
and which is used in the notification invocation. Then when the
sieve rule next fires, the URL/payload changes so that the notify
action actually goes through. <br>
<br>
Multiple consumers of such notifications would be handled during
rewrites of the sieve rules. A new device setting up for push
notifications would identify its rule based on the URL the push
API tells it. It would write its rule to the sieve store along
with the current timestamp for the rule. It would also cull any
other rules of the same variety whose timestamp more than twice as
old as the rule-rewrite-interval based on the timestamp. <br>
<br>
## custom sieve extension <br>
<br>
Realize the above state management logic is too crazy.
Internalize the one-shot state management, still using metadata
somehow. <br>
</blockquote>
<br>
It is better to extend enotify, unless you can see some severe
limitations in the enotify framework.<br>
<br>
<blockquote cite="mid:5225BF8C.7060602@mozilla.com" type="cite"> ##
Extend NOTIFY somehow <br>
<br>
This seems like it would be a mismatch since the NOTIFY
subscription is inherently associated with the connection lifetime
and any other lifetime would be weird. <br>
</blockquote>
<br>
Agreed.<br>
<br>
<blockquote cite="mid:5225BF8C.7060602@mozilla.com" type="cite"> ##
Define CONTEXT=pushnotify <br>
<br>
Similar to NOTIFY, this seems like a mismatch. <br>
<br>
<br>
Thanks for any/all responses, <br>
Andrew Sutherland<br>
</blockquote>
<br>
Best Regards,<br>
Alexey<br>
<br>
</body>
</html>
_______________________________________________
Imap-protocol mailing list
Imap-protocol@u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-protocol
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic