[prev in list] [next in list] [prev in thread] [next in thread]
List: pykde
Subject: Re: [PyQt] Logging UTF-8 when pyqtdeployed [CLOSED]
From: Barry <barry () barrys-emacs ! org>
Date: 2017-02-11 11:57:32
Message-ID: AD5E2578-303B-4291-BEF8-EAC083E2DAAC () barrys-emacs ! org
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
> On 11 Feb 2017, at 00:45, Dave Gradwell <davegradwell@yahoo.co.uk> wrote:
>
> Hi Barry,
>
> Thanks for your message and advice.
> Your mention of py2app led me to find that environment variables can also be set in \
> .app/Contents/Info.plist. So this is yet another way of dodging the problem - and \
> possibly in a better way than my previous workaround.
> <key>LSEnvironment</key>
> <dict>
> <key>LC_CTYPE</key>
> <string>en_US.UTF-8</string>
> </dict>
>
> Thanks, Dave.
Nice to know about that.
Beware that having a fixed locale means you cannot internationalise your app
with translations etc.
Barry
>
>
>
>
> > > On 10 Feb 2017, at 07:18, Barry Scott <barry@barrys-emacs.org> wrote:
> > >
> > >
> > > On 9 Feb 2017, at 00:24, Dave Gradwell <davegradwell@yahoo.co.uk> wrote:
> > >
> > > Just for the archives…
> > > To workaround this, I changed my Python Logging configuration:
> > > from : logging.basicConfig(filename=APP_LOG_PATH, level=logging.DEBU…..
> > > to : logging.basicConfig(handlers=[logging.FileHandler(filename=APP_LOG_PATH, \
> > > encoding='utf-8')], level=logging.DEBU…... With the explicit encoding \
> > > parameter, UTF-8 characters are now logged without complaint.
> > > During investigation, I dumped os.environ in the different launching contexts \
> > > and noticed:
> > > - When launched from PyCharm, PYTHONIOENCODING and LC_CTYPE are set to "UTF-8" \
> > > and "en_US.UTF-8" respectively.
> > > - When invoked by launching .app/Contents/MacOS/<binary> from the Terminal.app, \
> > > LANG is set to "en_GB.UTF-8".
> > > - When launched by pyqtdeploy, LANG is set to "en_GB.UTF-8" (perhaps \
> > > unsurprisingly, as pyqtdeploy was itself launched from Terminal.app and \
> > > presumably confers its environment onto the binary it makes and launches).
> >
> >
> > > - When the app is just double-clicked, however, no environment variables are \
> > > set relating to any of PYTHONIOENCODING, LC_CTYPE or LANG. I see that \
> > > __CF_USER_TEXT_ENCODING is set but I don't think Python is particularly wowed \
> > > by this.
> >
> > True python does not look a __CF_USER_TEXT_ENCODING.
> >
> > >
> > > I checked pyqtdeploy_start.cpp.
> > > locale_codec_name is "UTF-8" (which is good).
> > > I also hacked-in Py_SetStandardStreamEncoding("UTF-8", "strict"); just \
> > > before Py_Initialize(); but this had no discernible effect.
> > > I wonder if the lack of 'LANGUAGE', 'LC_ALL', 'LC_CTYPE' or ‘LANG' is causing \
> > > Python to fall back on a more primitive encoding…
> > > locale.getdefaultlocale() consistently returns None, None.
> >
> > My double clicked app has LC_CTYPE=en_US.UTF-8, which is not my locale, I'm \
> > guessing that it is put there by py2app that I use to create my apps, to get \
> > around these issues maybe.
> > You could add code that looks at __CF_USER_TEXT_ENCODING and do a local.setlocale \
> > early on in main. I used to do this to make wxPython apps work well on Mac.
> >
> > As yet I have not coded the conversion code for PyQt apps. But the shape of the \
> > solution is to decode the __CF_USER_TEXT_ENCODING Into a pair of ints. Then use \
> > them in a look up table to get the country code to use in setlocale. Always using \
> > UTF-8 as the encoding.
> > Note that if your locale is setup then you would not have a problem with the log \
> > file. The docs for open state that the default encoding is \
> > locale.getpreferredencoding(False) which should be utf-8 on macOS if the locale \
> > is setup.
> > If you do handle __CF_USER_TEXT_ENCODING be aware that some versions of macOS hex \
> > for the values and others decimal. 0x1f or 31 for example.
> >
> > Of course the simplest fix is to add setlocale( ‘en_US.utf-8' ) at the start of \
> > main() until you care about the locale in detail.
> > >
> > > Possibly of interest:
> > > - https://docs.python.org/2/library/locale.html#locale.getdefaultlocale
> > > - http://stackoverflow.com/questions/25661295/why-does-qcoreapplication-call-setlocalelc-all-by-default-on-unix-linux
> > >
> > > Anyway I'm just sending this into the ether so maybe it can sit in the archives \
> > > and possibly help someone else one day… — Dave
> >
> > Barry
> >
> >
>
[Attachment #5 (text/html)]
<html><head><meta http-equiv="content-type" content="text/html; \
charset=utf-8"></head><body dir="auto"><div><br></div><div>On 11 Feb 2017, at 00:45, \
Dave Gradwell <<a \
href="mailto:davegradwell@yahoo.co.uk">davegradwell@yahoo.co.uk</a>> \
wrote:<br><br></div><blockquote type="cite"><div><meta http-equiv="Content-Type" \
content="text/html charset=utf-8"><div class="">Hi Barry,</div><div class=""><br \
class=""></div><div class="">Thanks for your message and advice.</div><div \
class="">Your mention of py2app led me to find that environment variables can \
also be set in .app/Contents/Info.plist.</div><div class="">So this is yet another \
way of dodging the problem - and possibly in a better way than my previous \
workaround.</div><div class=""><br class=""></div><div class=""><div class="" \
style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: \
after-white-space;"><div class=""><div \
class=""><key>LSEnvironment</key><div class=""> \
<dict></div><div class=""> \
<key>LC_CTYPE</key></div><div class=""> \
<string>en_US.UTF-8</string></div><div class=""> \
</dict></div><div class=""><br class=""></div></div><div class="">Thanks, \
Dave.</div></div></div></div></div></blockquote><div><br></div><div><br></div>Nice to \
know about that.<div><br></div><div>Beware that having a fixed locale means you \
cannot internationalise your app</div><div>with translations \
etc.</div><div><br></div><div>Barry</div><div><br><blockquote type="cite"><div><div \
class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; \
-webkit-line-break: after-white-space;"><div class=""><div class=""><br \
class=""></div><div class=""><br class=""></div><div class=""><br \
class=""></div></div></div></div><br class=""><div><blockquote type="cite" \
class=""><div class="">On 10 Feb 2017, at 07:18, Barry Scott <<a \
href="mailto:barry@barrys-emacs.org" class="">barry@barrys-emacs.org</a>> \
wrote:</div><br class="Apple-interchange-newline"><div class=""><meta \
http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div \
style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: \
after-white-space;" class=""><br class=""><div class=""><blockquote type="cite" \
class=""><div class="">On 9 Feb 2017, at 00:24, Dave Gradwell <<a \
href="mailto:davegradwell@yahoo.co.uk" class="">davegradwell@yahoo.co.uk</a>> \
wrote:</div><br class="Apple-interchange-newline"><div class=""><meta \
http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div \
style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: \
after-white-space;" class=""><div class="">Just for the archives…</div><div \
class="">To workaround this, I changed my Python Logging configuration:</div><div \
class="">from<span class="Apple-tab-span" style="white-space:pre"> </span>:<span \
class="Apple-tab-span" \
style="white-space:pre"> </span>logging.basicConfig(filename=APP_LOG_PATH, level=logging.DEBU…..</div><div \
class="">to<span class="Apple-tab-span" style="white-space:pre"> </span>:<span \
class="Apple-tab-span" \
style="white-space:pre"> </span>logging.basicConfig(handlers=[logging.FileHandler(file \
name=APP_LOG_PATH, encoding='utf-8')], level=logging.DEBU…...</div><div \
class="">With the explicit encoding parameter, UTF-8 characters are now logged \
without complaint.</div><div class=""><br class=""></div><div class="">During \
investigation, I dumped os.environ in the different launching contexts and \
noticed:</div><div class="">- When launched from PyCharm, PYTHONIOENCODING \
and LC_CTYPE are set to "UTF-8" and "en_US.UTF-8" respectively.</div><div \
class="">- When invoked by launching .app/Contents/MacOS/<binary> from the \
Terminal.app, LANG is set to "en_GB.UTF-8".</div><div class="">- When launched \
by pyqtdeploy, LANG is set to "en_GB.UTF-8" (perhaps unsurprisingly, as \
pyqtdeploy was itself launched from Terminal.app and presumably confers its \
environment onto the binary it makes and \
launches).</div></div></div></blockquote><div class=""><br class=""></div><br \
class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: \
break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" \
class=""><div class="">- When the app is just double-clicked, however, no environment \
variables are set relating to any of PYTHONIOENCODING, LC_CTYPE or LANG. I see \
that __CF_USER_TEXT_ENCODING is set but I don't think Python is particularly wowed by \
this.</div></div></div></blockquote><div class=""><br class=""></div><div \
class="">True python does not look a __CF_USER_TEXT_ENCODING.</div><br \
class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: \
break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" \
class=""><div class=""><br class=""></div><div class="">I checked \
pyqtdeploy_start.cpp. </div><div class="">locale_codec_name is "UTF-8" (which \
is good). </div><div class="">I also hacked-in \
Py_SetStandardStreamEncoding("UTF-8", "strict"); just before \
<span style="color: rgb(192, 192, 192);" class=""> </span>Py_Initialize(); \
but this had no discernible effect.</div><div class=""><br class=""></div><div \
class="">I wonder if the lack of \
'LANGUAGE', 'LC_ALL', 'LC_CTYPE' or ‘LANG' is causing Python to \
fall back on a more primitive encoding…</div></div></div></blockquote><blockquote \
type="cite" class=""><div class=""><div style="word-wrap: break-word; \
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div \
class=""><br class=""></div><div class="">locale.getdefaultlocale() \
consistently returns None, None.</div></div></div></blockquote><div \
class=""><br class=""></div><div class="">My double clicked app \
has LC_CTYPE=en_US.UTF-8, which is not my locale, I'm guessing that it is put \
there by py2app that I use</div><div class="">to create my apps, to get around these \
issues maybe.</div><div class=""><br class=""></div><div class="">You could add code \
that looks at __CF_USER_TEXT_ENCODING and do a local.setlocale early on in \
main.</div><div class="">I used to do this to make wxPython apps work well on \
Mac.</div><div class=""><br class=""></div><div class="">As yet I have not coded the \
conversion code for PyQt apps. But the shape of the solution is to decode the \
__CF_USER_TEXT_ENCODING</div><div class="">Into a pair of ints. Then use them in a \
look up table to get the country code to use in setlocale. Always using UTF-8 as the \
encoding.</div><div class=""><br class=""></div><div class="">Note that if your \
locale is setup then you would not have a problem with the log file. The docs for \
open state that the default encoding</div><div class="">is \
locale.getpreferredencoding(False) which should be utf-8 on macOS if the locale is \
setup.</div><div class=""><br class=""></div><div class="">If you do handle \
__CF_USER_TEXT_ENCODING be aware that some versions of macOS hex for the values and \
others decimal.</div><div class="">0x1f or 31 for example.</div><div class=""><br \
class=""></div><div class="">Of course the simplest fix is to add setlocale( \
‘en_US.utf-8' ) at the start of main() until you care about the locale in \
detail.</div><div class=""><br class=""></div><blockquote type="cite" class=""><div \
class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; \
-webkit-line-break: after-white-space;" class=""><div class=""><br \
class=""></div><div class="">Possibly of interest:</div><div class="">- <a \
href="https://docs.python.org/2/library/locale.html#locale.getdefaultlocale" \
class="">https://docs.python.org/2/library/locale.html#locale.getdefaultlocale</a></div><div \
class="">- <a href="http://stackoverflow.com/questions/25661295/why-does-qcoreapplication-call-setlocalelc-all-by-default-on-unix-linux" \
class="">http://stackoverflow.com/questions/25661295/why-does-qcoreapplication-call-setlocalelc-all-by-default-on-unix-linux</a></div><div \
class=""><br class=""></div><div class="">Anyway I'm just sending this into the ether \
so maybe it can sit in the archives and possibly help someone else one \
day…</div><div class="">— Dave</div></div></div></blockquote><br \
class=""></div><div class="">Barry</div><div class=""><br class=""></div><br \
class=""></div></div></blockquote></div><br \
class=""></div></blockquote></div></body></html>
[Attachment #6 (text/plain)]
_______________________________________________
PyQt mailing list PyQt@riverbankcomputing.com
https://www.riverbankcomputing.com/mailman/listinfo/pyqt
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic