tag:blogger.com,1999:blog-10067973083901372602024-03-20T00:59:49.918-07:00poquito picanteabsurd alliterationsbwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.comBlogger197125tag:blogger.com,1999:blog-1006797308390137260.post-28432307065826361412017-04-05T09:58:00.000-07:002017-04-05T09:58:20.955-07:00The EndThis is the end of <a href="https://poquitopicante.blogspot.com/" target="_blank">Poquito Picante</a>. I am now posting my stories at <a href="https://breakingbytes.github.io/" target="_blank">Breaking Bytes</a>.<br />
<br />
The URL of my new blog is:<br />
<blockquote class="tr_bq">
"<a href="https://breakingbytes.github.io/">https://breakingbytes.github.io/</a>".</blockquote>
For the how and why I am making this move from Google Blogger to <a href="https://blog.getpelican.com/" target="_blank">Pelican</a> and <a href="https://pages.github.com/" target="_blank">GitHub Pages</a>, please read my first post.<br />
<blockquote class="tr_bq">
<a href="https://breakingbytes.github.io/moving-to-pelican-at-github-pages.html#moving-to-pelican-at-github-pages" target="_blank">Moving to Pelican at GitHub Pages</a></blockquote>
Good bye Google Blogger, you've served me well. And for anyone following Poquito Picante, I hope to see you soon at Breaking Bytes.<br />
<br />
Sincerely Yours,<br />
<a class="g-profile" href="https://plus.google.com/114749747116603825948" target="_blank">+Mark Mikofski</a> (AKA BwanaMarko, AKA BreakingBytes)bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-86231854667136801972017-01-12T13:29:00.000-08:002018-09-28T15:31:30.661-07:00PV Power<link href="https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shCore.css" rel="stylesheet" type="text/css">
<link href="https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shThemeFadeToGrey.css" rel="stylesheet" type="text/css">
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shCore.js" type="text/javascript"></script>
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shAutoloader.js" type="text/javascript"></script>
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPython.js" type="text/javascript"></script>
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushXml.js" type="text/javascript"></script>
<script language='javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all();
</script>
<p>Please read my
<a href="https://poquitopicante.blogspot.com/p/darn-disclaimer-and-litigious-license.html">
disclaimer</a>.</p>
<p><a href="https://us.sunpower.com/">SunPower Corp.</a>, my employer, has open
sourced a solar panel mismatch estimation tool called
<a href="https://pypi.python.org/pypi/pvmismatch">PVMismatch</a> at the
<a href="https://pypi.python.org/pypi">Python Package Index</a> with a
<a href="https://opensource.org/licenses/BSD-3-Clause">standard 3-clause BSD
license</a>. The <a href="https://sunpower.github.io/PVMismatch/">documentation</a>,
<a href="https://github.com/SunPower/PVMismatch/">source code</a> and
<a href="https://github.com/SunPower/PVMismatch/releases">releases</a> are also
available at the <a href="https://sunpower.github.io/">SunPower Organization
GitHub page</a>.</p>
<p>PVMismatch can be used to simulate mismatch between cells, modules and
strings in a PV system. Mismatch can be caused be differences in irradiance, say
by shading or by changes to the electrical characteristics of the cells such as
series resistance or dark current, and these changes can lead to a difference in
the current-voltage relation of the cell (<i>aka</i> its I-V curve). For a
module made from solar cells in series, the cells must carry the same current,
leading to different voltages in each cell, and so the effective I-V curve of
the panel will differ from an individual cell. In some cases a cell may be
forced to operate in reverse bias in order to pass the current imposed on it.</p>
<p>There is a detailed
<a href="https://sunpower.github.io/PVMismatch/tutorials/quickstart.html#example">
example in the Quickstart section of the documentation</a>, but here's a shorter
demonstration of the basic features and usage:</p>
<pre class="brush:python">
from pvmismatch import * # this imports pvcell, pvmodule, pvstring and pvsystem
# create a default PV system from 10 strings of 10 300[W] panels each
pvsys = pvsystem.PVsystem()
f = pvsys.plotSys() # plot the system
f.show()
</pre>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUyg3xsaB0e3As8-Fpivou0mdC371VmuXotpZAqt_VsjDiTA70ivlIYJIGxNLhKLPhlZdTYMXVGC03DpDMB_W10B2FmSQFo9TkWKYqso0Nj6phpa5rk3ltgNQFedkGR5bMRXon_0nNlw/s1600/PVMismatch.pvsystem.PVsystem.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUyg3xsaB0e3As8-Fpivou0mdC371VmuXotpZAqt_VsjDiTA70ivlIYJIGxNLhKLPhlZdTYMXVGC03DpDMB_W10B2FmSQFo9TkWKYqso0Nj6phpa5rk3ltgNQFedkGR5bMRXon_0nNlw/s640/PVMismatch.pvsystem.PVsystem.png" width="640" height="480" /></a>
<p>If you find any <a href="https://github.com/SunPower/PVMismatch/issues">issues</a>, please report them on GitHub. Also, if you want to
contribute, there's lots to do, so please fork the repo.</p>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-85758769895904513802016-11-08T17:03:00.003-08:002018-09-28T15:32:29.971-07:00Bypassing Box Upload Limits by API<link href='https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href="https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shThemeFadeToGrey.css" rel="stylesheet" type="text/css" />
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shCore.js" type="text/javascript"></script>
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shAutoloader.js" type="text/javascript"></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPython.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushXml.js' type='text/javascript'></script>
<script language='javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all();
</script>
<h1>Box for large files</h1>
<p><a href="https://www.box.com/pricing/personal">Box offers 10gb of online storage for free, double what anyone else offers, with an individual max size of 2gb, but you can only upload 250mb files.</a> So how do you upload that 2gb file? The <a href="https://developer.box.com/">Box API</a> that's how, either with <a href="https://docs.python-requests.org/en/master/user/quickstart/">regular ol' requests</a> or their <a href="https://github.com/box/box-python-sdk">fancy smancy sdk</a>. First follow the <a href="https://docs.box.com/docs/getting-started-box-integration#section-using-the-box-api">Getting Started instructions, sign up for a developer account and create a temporary key</a>. Then in Python, try this out:</p>
<pre class="brush: python">
# import the requests package
import requests
# copy your token here
TOKEN = "<your developer token>"
# try to get the top level folder, id: "0", using this command exactly as below:
r = requests.get(url='https://api.box.com/2.0/folders/0',
headers={'Authorization': 'Bearer %s' % TOKEN})
# check the response
r
# <Response [200]>
# success!
# get the output
r.json()
# lots of stuff
# upload a file, using the commands exactly as below, except put the actual id number
# of the desired folder
FILES = {'file': open('path/to/myfile','rb')}
PAYLOAD = {'attributes': '{"name":"myfile", "parent":{"id":"<id # of desired folder>"}}'}
r = requests.post(url='https://upload.box.com/api/2.0/files/content',
headers={'Authorization': 'Bearer %s' % TOKEN},
files=FILES,
data=PAYLOAD)
# check the response
r
# <Response [201]>
# success!
</pre>
<h2>References</h2>
<p>Check the online <a href="https://docs.box.com/reference">Content API reference</a> for full documentation.</p>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-42017971679179415102016-11-07T14:59:00.000-08:002017-01-04T09:41:46.373-08:00Panda Pop<h1>Pandas Offset Aliases</h1>
<p>Memorize this table - or just bookmark this link: <a href="http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases">Pandas Offset Aliases</a><p>
<div class="section" id="offset-aliases">
<span id="timeseries-offset-aliases"></span><h3>Offset Aliases<a class="headerlink" href="http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases" title="Permalink to this headline">¶</a></h3>
<p>A number of string aliases are given to useful common time series
frequencies. We will refer to these aliases as <em>offset aliases</em>
(referred to as <em>time rules</em> prior to v0.8.0).</p>
<table border="1" class="docutils" id="myTable">
<colgroup>
<col width="13%">
<col width="87%">
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Alias</th>
<th class="head">Description</th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td>B</td>
<td>business day frequency</td>
</tr>
<tr class="row-odd"><td>C</td>
<td>custom business day frequency (experimental)</td>
</tr>
<tr class="row-even"><td>D</td>
<td>calendar day frequency</td>
</tr>
<tr class="row-odd"><td>W</td>
<td>weekly frequency</td>
</tr>
<tr class="row-even"><td>M</td>
<td>month end frequency</td>
</tr>
<tr class="row-odd"><td>SM</td>
<td>semi-month end frequency (15th and end of month)</td>
</tr>
<tr class="row-even"><td>BM</td>
<td>business month end frequency</td>
</tr>
<tr class="row-odd"><td>CBM</td>
<td>custom business month end frequency</td>
</tr>
<tr class="row-even"><td>MS</td>
<td>month start frequency</td>
</tr>
<tr class="row-odd"><td>SMS</td>
<td>semi-month start frequency (1st and 15th)</td>
</tr>
<tr class="row-even"><td>BMS</td>
<td>business month start frequency</td>
</tr>
<tr class="row-odd"><td>CBMS</td>
<td>custom business month start frequency</td>
</tr>
<tr class="row-even"><td>Q</td>
<td>quarter end frequency</td>
</tr>
<tr class="row-odd"><td>BQ</td>
<td>business quarter endfrequency</td>
</tr>
<tr class="row-even"><td>QS</td>
<td>quarter start frequency</td>
</tr>
<tr class="row-odd"><td>BQS</td>
<td>business quarter start frequency</td>
</tr>
<tr class="row-even"><td>A</td>
<td>year end frequency</td>
</tr>
<tr class="row-odd"><td>BA</td>
<td>business year end frequency</td>
</tr>
<tr class="row-even"><td>AS</td>
<td>year start frequency</td>
</tr>
<tr class="row-odd"><td>BAS</td>
<td>business year start frequency</td>
</tr>
<tr class="row-even"><td>BH</td>
<td>business hour frequency</td>
</tr>
<tr class="row-odd"><td>H</td>
<td>hourly frequency</td>
</tr>
<tr class="row-even"><td>T, min</td>
<td>minutely frequency</td>
</tr>
<tr class="row-odd"><td>S</td>
<td>secondly frequency</td>
</tr>
<tr class="row-even"><td>L, ms</td>
<td>milliseconds</td>
</tr>
<tr class="row-odd"><td>U, us</td>
<td>microseconds</td>
</tr>
<tr class="row-even"><td>N</td>
<td>nanoseconds</td>
</tr>
</tbody>
</table>
</div>
bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-9799165755369551012016-11-01T12:03:00.000-07:002017-03-02T13:37:30.934-08:00robotic releases<!-- override code class to add white text on black background -->
<style>
code {
background-color: black;
color: lightgreen
}
pre {
background-color: black;
color: lightgreen
}
</style>
<!-- body -->
<h1>Basic Auto-Versioning from Git</h1>
<p>If you're using the <a href="http://poquitopicante.blogspot.com/2016/10/winning-workflow.html">winning workflow</a> and the <a href="http://poquitopicante.blogspot.com/2015/04/recommended-python-project-layout.html">recommended Python project layout</a> then you've set up a CI server to build releases when you tag them in Git, and you set your version in the <code>__init__.py</code> file of your package. But, "Oh, No!" you did it again. You created the Git tag, but forgot to update your code's <code>__version__</code> string.</p>
<p>Okay, there is a Python package called <a href="https://pypi.python.org/pypi/versioneer/">Versioneer</a> that handles this for you, and it's pretty awesome. But it turns out it's also pretty easy to roll your own, especially if you're just using <a href="https://git-scm.com/">Git</a>, because Python has a Git implementation called <a href="https://pypi.python.org/pypi/dulwich">Dulwich</a> that can do this in just a few lines. <strike>Maybe it will get integrated into a future version of Dulwich -</strike> I've submitted a PR (<a href="https://github.com/jelmer/dulwich/pull/462">#462</a>) <b>which was merged into v0.16.3</b> and an update (<a href="https://github.com/jelmer/dulwich/pull/489">#489</a>) <b>which was also merged into v0.17</b> to also list tags that are not objects. <strike>Anyway, for now, the easiest way to use this is to copy this file into your package at the top level,</strike> Install <a href="https://pypi.python.org/pypi/dulwich">the latest version of dulwich (>=0.17.1)</a>, import it and then add something like this to your package dunder init module so it works both in your repo during dev and then later when deployed to users.</p>
<pre>
"""
Example package dunder init module implementing
``dulwich.contrib.release_robot`` to get current version.
"""
import os
import importlib
# try to import Dulwich or create dummies
try:
from dulwich.contrib.release_robot import get_current_version
from dulwich.repo import NotGitRepository
except ImportError:
NotGitRepository = NotImplementedError
def get_current_version():
raise NotGitRepository
BASEDIR = os.path.dirname(__file__) # this directory
VER_FILE = 'version' # name of file to store version
# use release robot to try to get current Git tag
try:
GIT_TAG = get_current_version()
except NotGitRepository:
GIT_TAG = None
# check version file
try:
version = importlib.import_module('%s.%s' % (__name__, VER_FILE))
except ImportError:
VERSION = None
else:
VERSION = version.VERSION
# update version file if it differs from Git tag
if GIT_TAG is not None and VERSION != GIT_TAG:
with open(os.path.join(BASEDIR, VER_FILE + '.py'), 'w') as vf:
vf.write('VERSION = "%s"\n' % GIT_TAG)
else:
GIT_TAG = VERSION # if Git tag is none use version file
VERSION = GIT_TAG # version
__author__ = u'your name'
__email__ = u'your.email@your.company.com'
__url__ = u'https://github.com/your-org/your-project'
__version__ = VERSION
__release__ = u'your release name'
</pre>
<p>Or you can also use it to get all recent tags.</p>
<pre>get_recent_tags()[0][0]</pre>
<p>assuming your tags all use <a href="http://semver.org/">semantic versions</a> like "v0.3". Enjoy!</p>
<script src="https://gist.github.com/mikofski/e923750b415e4e4961b65a8eb42999e8.js"></script>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-84537594619248411582016-10-31T17:08:00.000-07:002016-10-31T21:48:26.780-07:00Carousel Cotton Candy<!-- gray backgrounds for code and pre -->
<style>
code {
background-color: silver;
}
pre {
background-color: silver;
}
</style>
<h1>Version 0.3</h1>
<p>I'm super excited to announce the Cotton Candy release of Carousel, version 0.3 on <a href="https://pypi.python.org/pypi/Carousel">PyPI</a> and <a href="https://github.com/SunPower/Carousel/releases">GitHub</a>. There were a few more issues I really wanted to close with this release, but I decided to push it forward anyway. So the remaining <a href="https://github.com/SunPower/Carousel/milestone/2">milestones for v0.3</a> will get pushed to v0.3.1.</p>
<ul>
<li><a href="https://github.com/SunPower/Carousel/issues/62">issue #62</a> use <code>Meta</code> class for all layers - currently usage is spotty and inconsistent. I wanted to keep the number of commits to close <a href="https://github.com/SunPower/Carousel/pull/68">PR #68</a> to a minimum (following the <a href="http://poquitopicante.blogspot.com/2016/10/winning-workflow.html">winning workflow</a>) so I only implemented <code>Meta</code> classes where I had to. In fact it's not even implemented in the <code>DataSource</code> example below.</li>
<li><a href="https://github.com/SunPower/Carousel/issues/25">issue #25</a> move all folders into project package - this is already how I have set up the PVPower demo. It just makes more sense with new style models to have them all in the same package.</li>
<li><a href="https://github.com/SunPower/Carousel/issues/63">issue #63</a> and <a href="https://github.com/SunPower/Carousel/issues/22">issue #22</a> split calculations into separate parameters - I knew this couldn't be done by v0.3, it was a stretch goal, but I'm super excited about this. By moving <code>dependencies</code> to an attribute of each parameter, the DAG shows which calculations are orthogonal, so we can run them simultaneously. According to issue #22, I had this idea already, but when I saw this <a href="https://youtu.be/cHATHSB_450">presentation at PyData SF 2016</a> on <a href="https://airflow.incubator.apache.org/">Airflow</a> by <a href="https://twitter.com/jiffyclub">Matt Davis</a> I was even more motivated to make it happen.</li>
<li><a href="https://github.com/SunPower/Carousel/issues/59">issue #59</a> and <a href="https://github.com/SunPower/Carousel/issues/73">issue #73</a> which don't seem like they're relevant, but they both have to do with implementing a <code>Calculator</code> class whose job it is to crunch through the calculations, somewhat similar to what <code>DataReaders</code> and <code>FormulaImporter</code> do for their layers. Then the boiler plate uncertainty propagation code in the static class could be applied to any calculator such as a dynamic calculator, a linear system solver for an acyclic DAG of linear equations or a non-linear system solver for an acyclic DAG of non-linear equations.</li>
</ul>
<h2>The <code>Parameter</code> class</h2>
<p>What's new in Carousel-0.3 (Cotton Candy)? The biggest difference is the introduction of the <code>Parameter</code> class which is now used to specify parameters for data, formulas, outputs, calculations and simulation settings. For example, previously data parameters would be entered as a dictionary of attributes.</p>
<h2>Bicycle Bears</h2>
<pre style="background-color:pink; color:red">
class PVPowerData(DataSource):
"""
Data sources for PV Power demo.
"""
data_reader = ArgumentReader
latitude = {"units": "degrees", "uncertainty": 1.0}
longitude = {"units": "degrees", "uncertainty": 1.0}
elevation = {"units": "meters", "uncertainty": 1.0}
</pre>
<h2>Cotton Candy</h2>
<pre style="color:green">
class PVPowerData(DataSource):
"""
Data sources for PV Power demo.
"""
latitude = DataParameter(units="degrees", uncertainty=1.0)
longitude = DataParameter(units="degrees", uncertainty=1.0)
elevation = DataParameter(units="meters", uncertainty=1.0)
class Meta:
data_reader = ArgumentReader
</pre>
<p>Why the change? The <i>Bicycle Bear</i> version did not have any way to distinguish parameters, like <code>latitude</code> from attributes of the <code>DataSource</code> like <code>data_reader</code>. This had two unfortunate side-effects:</p>
<ul>
<li>each layer attribute had to be hardcoded in the base metaclass so that they wouldn't be misinterpreted as parameters</li>
<li>and users could not define any custom class attributes, because they would be misinterpreted as parameters and stripped from the class by the metaclass.</li>
</ul>
<p>The <i>Cotton Candy</i> version makes it easy for the metaclass to determine which class attributes are parameters, which are attributes of the layer and then leaves everything else alone. Every layer now has a corresponding <code>Parameter</code> subclass which also defines some base attributes corresponding to that layer. Any extra parameter attributes are saved in <code>extras</code>. Attributes that apply to the entire layer are now specified in the `Meta` class attribute, similar to <a href="https://www.djangoproject.com/">Django</a>, <a href="http://marshmallow.readthedocs.io/en/latest/index.html">Marshmallow</a> and <a href="http://www.django-rest-framework.org/">DRF</a>. The similarities are completely intentional as I have been strongly inspired by those project code bases. Unfortunately, the <code>Meta</code> class is only partially implemented, but will be the major focus of v0.3.1.
<h2>Separated Formulas</h2>
<p>The <code>Formula</code> class is also improved. Now each formulas is a <code>Parameter</code> with attributes, rather than a giant dictionary. This improvement is still on the roadmap for the <code>Calculation</code> class. As I said above, it was a stretch goal for this release.</p>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-60587867273995421522016-10-12T03:05:00.000-07:002018-09-13T13:24:29.268-07:00Winning Workflow<style>
code {
background-color: silver;
}
pre {
background-color: silver;
}
</style>
<h1>Intro</h1>
<p>There are many blog posts on the topic of <a href="http://nvie.com/posts/a-successful-git-branching-model/">effective Git workflows</a>, <a href="http://stackoverflow.com/questions/35419185/git-workflow-example">SO questions and answers</a>, <a href="https://www.atlassian.com/git/tutorials/comparing-workflows">BitBucket tutorials</a> and <a href="https://guides.github.com/introduction/flow/">GitHub guides</a> and <a href="https://www.integralist.co.uk/posts/github-workflow/">an article that has been archived by former BBC Mark McDonnell</a>. So why another post on git workflow? None of these workflows seemed right for us, but recently it's just clicked, and I feel like we've finally found the process that works for us. The key was finding the simplest workflow that included the most valuable best practices. In particular, we found that complicated multi-branch strategies were unnecessary, but test driven development (TDD) and continuous integration (CI) were a must.</p>
<h2>Winning Workflow</h2>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5pojD9y6BS041yta-D_0yNQ7j49cf0xN0i73kF3c-759aj8MdmoBAsX4JJrWMKAMAxal-ck5teSWAuwsni8EtR3SXCswApgqAZwzV3OyLJhw0tqo8o0o1mciSoO0l-zz5EHtfIss6sg/s1600/workflow-allcolor.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5pojD9y6BS041yta-D_0yNQ7j49cf0xN0i73kF3c-759aj8MdmoBAsX4JJrWMKAMAxal-ck5teSWAuwsni8EtR3SXCswApgqAZwzV3OyLJhw0tqo8o0o1mciSoO0l-zz5EHtfIss6sg/s640/workflow-allcolor.png" width="640" height="261" /></a></div>
<br />
<h2>Setting up Remotes</h2>
<p>We start with the assumption that all of collaborators fork the upstream repository to their personal profile. Then each person clones their profile to their laptop as origin and adds another remote pointing to the upstream repository. For convenience, they may also create remotes to the forks of their most frequent collaborators.</p>
<pre>
[myusername@mycomputer ~/Projects]
$ git clone git@github.com:myusername/myrepo.git
[myusername@mycomputer ~/Projects]
$ cd myrepo
[myusername@mycomputer ~/Projects/myrepo]
$ git remote add upstream git@github.com:mycompany/myrepo.git
[myusername@mycomputer ~/Projects/myrepo]
$ git remote add mycollaborator git@github.com:mycollaborator/myrepo.git
[myusername@mycomputer ~/Projects/myrepo]
$ git remote show
origin
upstream
mycollaborator
</pre>
<h2>Ground Rules</h2>
<p>The next assumption is that we all keep our version of master synchronized with upstream master. And we <b>never</b> work out of our own master branch! Basically this means at the start of any new work we do the following:
<ol>
<li>I like to do <code>git fetch --all</code> to get the lay of the land. This combined with <br/>
<code>git log --all --graph --date=short --pretty=format:"%ad %h %s%d [%an]"</code>
<br /> let's me know what everyone is working on, assuming that I've made remotes to their forks.</li>
<li>Then I <b>pull from upstream master</b> to get the latest nightly or release,</li>
<li>and <b>push to origin master</b> to keep my fork current.</li>
</ol>
<h2>Recommended Project Layout</h2>
<p>I'm also going to assume that everyone is following the <a href="http://poquitopicante.blogspot.com/2015/04/recommended-python-project-layout.html">recommended project layout</a>. This means that their project has all dependencies listed in <code>requirements.txt</code>, is developed and deployed in its own virtual environment, includes testing and documentation that aims for >80% coverage, has a boilerplate design that allows testing, documentation and package data to be bundled into a distribution and enables use with a test runner with self discovery, and is written with docstrings for autodocumentation. Nothing is ever perfect, so being diligent of path clashes, aware of the arcana of Mac OS X<sup><a href="#macosx-matplotlib-venv">1</a></sup> or Windows<sup><a href="#win32-nose-venv">2</a></sup> and able to use <a href="http://stackoverflow.com/">Stack Overflow</a> to find answers is still important.</p>
<h2>Branching, Testing, Pull Requests and Collaboration</h2>
<ol>
<li>Now I <b>switch to a new feature branch with a meaningful name</b> - I'll delete this branch everywhere later so it can be verbose.</li>
<li>The very first code <b>I write</b> <i>is</i> <b>a test or two that demonstrates more or less exactly what we want the feature or bug fix to do</b>. This is one of the most valuable steps because it clearly defines the acceptance criteria. Although it's also important to be thoughtful and flexible - just because your tests pass doesn't necessarily mean the feature is implemented as intended. Some new tests or adjustments may be needed along the way.</li>
<li>Now, before I write any more code, is when I <b>submit a pull request (PR) from my fork's feature branch to upstream/master</b>. So many people are surprised by this. Many collaborators have told me they thought that PR's should be submitted after their work is complete and passing all tests. But in my opinion that defeats the entire point of collaborating on a short iteration cycle.
<ul>
<li>If you wait until the end to submit your work you risk diverging from the feature's intended goals especially if the feature's requirements shift or you've misinterpreted the goals even slightly.</li>
<li>Waiting also means you're missing out on collaborating with your teammates and soliciting their feedback mid-project.</li>
</ul>
On the other hand, by submitting your PR right after you write your tests means:
<ul>
<li>Every push to your fork will trigger a build that runs your tests.</li>
<li>Your teammates will get continuous updates so they can monitor your progress in real-time but also on their time so you won't have to hold a formal review, since collaborators can review your work anytime as the commits will all be queued in the PR.</li>
</ul>
I think the reason people wait until the end to submit PR's is the same reason they like to write tests at the end. I used to hate seeing my tests fail because it made me feel like I was failing. I think people delay submitting their PR's because they're nervous about having incomplete work reviewed out of context and receiving unfair criticism or harsh judgment. <b>IMO, punitive behavior is dysfunctional and a collaboration killer</b> and should be rooted out with a frank discussion about what mutual success looks like. I also think some people aren't natural collaborators and don't want other's interfering with their work. Again, a constructive discussion can help promote new habits, although don't expect people to change overnight. You can take a hard stance on punitive behavior but you can't expect an introvert to feel comfortable sharing themselves freely without some accommodations.</li>
<li>Now comes the really fun part. We hack and collaborate until the tests all pass. But we don't have too much fun - there should be at most 10 commits before we realize we've embarked on an epic that needs to be re-organized, otherwise the PR will become difficult to merge. That will sap moral and waste time. So keep it simple.</li>
<li>The final bit of collaboration is the code review and merging the PR into upstream master. This is fairly easy, since there are
<ul>
<li>already tests that demonstrate what the code should do,</li>
<li>only a few commits,</li>
<li>and all of the collaborators have been following the commits as they've been queuing in the PR.</li>
</ul>
So really the review and merge is a sanity check. Do these tests really demonstrate the feature as intended? Anything else major would have stood out already.</li>
<li>Whoever the repository owner or maintainer is should add the tag and push it to upstream. This triggers the CI to test, build and deploy a new release.</li>
</ol>
</p>
<h2>Continuous Integration</h2>
<p>This is key. Set up Travis, Circle, AppVeyor or Jenkins on upstream master to test and build every commit, every commit to an open PR and to deploy on every tag. Easy!</p>
<h2>Wrapping Up</h2>
<p>
There are some features of this style that stand out:
<ul>
<li>There is only one master branch. Using CI to deploy only on tags eliminates our need for a dev or staging branch because any commits on master not tagged are the equivalent of the bleeding edge.</li>
<li>This method depends heavily on an online hosted Git repo like GitHub or BitBucket, use of TDD, strong collaboration and a CI server like Travis.</li>
</ul>
</p>
<h2>Happy Coding!</h2>
<hr>
<h4>footnotes</h4>
<ol>
<li id="macosx-matplotlib-venv">On Mac OS X matplotlib will not work in a virtual environment unless a framework interpreter is used. The easiest way to do this is to run python as <code>PYTHONHOME=/home/you/path/to/project/venv/ python</code> instead of using <code>source venv/bin/activate</code>.</li>
<li id="win32-nose-venv">On Windows pip often creates an executable for scripts that is bound to the Python interpreter it was installed with. If the virtual environments was created with system site packages or if the package is not installed in the virtual environment then you may get a confusing path clash. For example running the <code>nosetests</code> script will use your system Python and therefore the Python path will not include your virtual environment. The solution is to never use system site packages and install all dependencies directly in your virtual environment.</li>
</ol>
bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-37583979861255856422016-07-19T01:05:00.000-07:002016-07-19T01:05:26.968-07:00Derived Django Database Field<style>
code{background-color:silver};
</style>
<p>The trick to this is creating a <a href="https://docs.djangoproject.com/en/1.9/howto/custom-model-fields/">custom field</a> and overloading <a href="https://docs.djangoproject.com/en/1.9/ref/models/fields/#django.db.models.Field.pre_save"><code>pre_save</code></a>. Pay special attention to the <code>self.attname</code> member that is set to the value. The <a href="https://docs.djangoproject.com/en/1.9/_modules/django/db/models/fields/#DateField">source for <code>DateField</code></a> is a good example. Make sure that if you add any new attributes to the field in it's <code>__init__</code> method you also add a corresponding <a href="https://docs.djangoproject.com/en/1.9/howto/custom-model-fields/#field-deconstruction"><code>deconstruct</code></a> method.</p>
<script src="https://gist.github.com/mikofski/d2452658d21ef22ab423fb7ba58745a0.js"></script>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-47118832558816886202016-07-18T18:32:00.002-07:002022-11-04T14:09:00.242-07:00Mocking Django App<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/styles/shCore.css" integrity="sha512-G2vGLyD0mR1jxSAJvcuHEbSzeAZDhtem+wdYeBSr2M+wrAFo/VFZmRDsIluBnhdlq242U83cFEEvXhN3jkRv1w==" crossorigin="anonymous" referrerpolicy="no-referrer" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/styles/shThemeFadeToGrey.css" integrity="sha512-r2G85JYd+RGzDz5MMdNP1yJlvlimIa/ncBIOmO2hbCmfM1S60IuHV/4B3ixl+9eNZKevESmEITwvqtiyOAojtQ==" crossorigin="anonymous" referrerpolicy="no-referrer" />
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shCore.min.js" integrity="sha512-Z5dAQyvO8EyY1cHQcqYTYL8z6PDjM0URql6AioNodsSxxTJS5Fqfujr+o/4ye2dLp0he1qAVTiZABTunv6oLow==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shAutoloader.js" integrity="sha512-hsbtM1IhPwoNY2OToen11Ln2kDn0YnH+7VZluA5Abekh4kWOuNO1RKnePdNirA8JyliQx0iOAKMcvbrAuGHQNw==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushPython.js" integrity="sha512-nsiCxqo4QFYKSS09HPK0+HJK4qfHOvJDE0ZlIeB1Ls8YL+qW7BjTYKdoX48+kf2ATYLyaHFqvcfIuxXOoHoLZg==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushXml.js" integrity="sha512-TJi3OozXt3jOt1OJFV0wxldwBNL1Ng6QVdx4NuLa9Ni8pZGi0NKMzslD9jPi7Yz2V20v/wuRhPUwWfthZCZz3g==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushPlain.js" integrity="sha512-JHhokiQPA/SRIlZXstJ6OIEWlvO6z9S1dJkCKwpbqwD1Gimso0O+rlSANtpQ6pU1bZ/hgeLzMoKU5N7B+zfiqA==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushJScript.js" integrity="sha512-xck9yKEbi3rhQD7i/hpg43x8Z2htoeyATgaLGWRxWETAcVzbI7bw7Xs2qR0GxcBBnIdIR4uIn/pWI7Rivqu6dA==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushCss.js" integrity="sha512-sO0p1vBq6eTgn+1UDU267FGW0pPao1s0OahxuDFzH3wgb4eYtB4mfePQkA+Ps+VjSkKsv09ibpH5nFOge8TlmQ==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script language='javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all();
</script>
<style type="text/css">
code {
background-color: silver
}
</style>
<p>
I'm sure this is completely wrong. I needed a Django model for testing, but I don't have a Django app or even a Django project. I'm developing a Django model reader for <a href="http://poquitopicante.blogspot.com/2016/07/carousel-python-model-simulation.html">Carousel</a>, and so I needed a model to test it out with. Sure I could have created a quick django project, but that seemed silly, and my first instinct was to import <code>django.db.models</code>, make a model and use it, but this raised:
</p>
<pre style="background-color: silver">
ImproperlyConfigured: Requested setting DEFAULT_INDEX_TABLESPACE, but settings are not
configured. You must either define the environment variable
DJANGO_SETTINGS_MODULE or call settings.configure() before accessing
settings.
</pre>
<p>
Most normal people would turn back now, but instead I imported <code>django.conf.settings</code> and called <code>settings.configure()</code> just like it said to do. Now I got this error:
</p>
<pre style="background-color: silver">
AppRegistryNotReady: Apps aren't loaded yet.
</pre>
<p>
So now I felt like I was getting somewhere. But where? Googling told me to import <code>django</code> and run <code>setup</code> which I did and that raised:
</p>
<pre style="background-color: silver">
RuntimeError: Model class __main__.MyModel doesn't declare an explicit app_label and isn't
in an application in INSTALLED_APPS.
</pre>
<p>
Wow! Normally <code>RuntimeError</code> is a scary warning, like you dumped your core, but this just said I needed to add the app to <code>settings.INSTALLED_APPS</code>, which makes perfect sense, and it also complained that my model wasn't actually part of an app and even explained how to explicitly declare it. Some more Googling and I discovered that <code>app_label</code> is a model option that can be set in <code>class Meta</code>. So I did as told, and it worked!
</p>
<pre class="brush:python">
from django.db import models
from django.conf import settings
import django
MYAPP = 'myapp.MyApp'
settings.configure()
django.setup()
settings.INSTALLED_APPS.append(MYAPP)
class MyModel(models.Model):
air_temp = models.FloatField()
latitude = models.FloatField()
longitude = models.FloatField()
timezone = models.FloatField()
pvmodule = models.CharField(max_length=20)
class Meta:
app_label = MYAPP
mymodel = MyModel(air_temp=25.0, latitude=38.0, longitude=-122.0,
timezone=-8.0, pvmodule='SPR E20-327')
mymodel.__dict__
#{'_state': <django.db.models.base.ModelState at 0x496b2b0>,
# 'air_temp': 25.0,
# 'id': None,
# 'latitude': 38.0,
# 'longitude': -122.0,
# 'pvmodule': 'SPR E20-327',
# 'timezone': -8.0}
</pre>
<h1>
Caveats
</h1>
<p>
So I should stop here and point out that that evidently the order of these commands matters, because if I add the fake app to <code>INSTALLED_APPS</code> before calling <code>django.setup()</code> then I get this:
</p>
<pre style="background-color: silver">
ImportError: No module named myapp
</pre>
<p>
And unfortunately, I just figured this out now, in this post. But this isn't what I originally did. Yes, I'm completely crazy. First I added a fake module called <code>'myapp'</code> to <code>sys.modules</code> setting it to a mock object, but that didn't work. I got back <code>TypeError: 'Mock' object is not iterable</code> because, as I found out later, there has to be an <code>AppConfig</code> subclass in the app module. But since I didn't know that yet, I did the only logical thing and put the module in a list. What? Yes, did I mention I'm an idiot? This nonsense yielded the following stern warning:</p>
<pre style="background-color: silver">
ImproperlyConfigured: The app module [<Mock id='75109208'>] has no filesystem location, you
must configure this app with an AppConfig subclass with a 'path' class
attribute.
</pre>
<p>
But this is where I found out about <a href="https://docs.djangoproject.com/en/1.9/ref/applications/"><code>AppConfig</code> in the Django docs</a> which is covered quite nicely. Following the nice directions, I did as told and subclassed <code>AppConfig</code>, added <code>path</code> and also <code>name</code> which I learned from the docs, monkeypatched my mock module with it, and used the dotted name of the app <code>myapp.MyApp</code> now. I felt like I was getting closer, since I only got: <code>AttributeError: __name__</code> which seemed like a problem with my pretend module. Another monkeypatch and we have my final ludicrously ridiculous hack.
</p>
<pre class="brush:python">
from django.db import models
from django.conf import settings
import django
from django.apps import AppConfig
import sys
import mock
class MyApp(AppConfig):
"""
Apps subclass ``AppConfig`` and define ``name`` and ``path``
"""
path = '.' # path to app
name = 'myapp' # name of app
# make a mock module with ``__name__`` and ``MyApp`` member
myapp_module = mock.Mock(__name__='myapp', MyApp=MyApp)
MYAPP = 'myapp.MyApp' # full path to app
sys.modules['myapp'] = myapp_module # register module
settings.configure()
settings.INSTALLED_APPS.append(MYAPP)
django.setup()
class MyModel(models.Model):
air_temp = models.FloatField()
latitude = models.FloatField()
longitude = models.FloatField()
timezone = models.FloatField()
pvmodule = models.CharField(max_length=20)
class Meta:
app_label = MYAPP
mymodel = MyModel(air_temp=25.0, latitude=38.0, longitude=-122.0,
timezone=-8.0, pvmodule='SPR E20-327')
mymodel.__dict__
#{'_state': <django.db.models.base.ModelState at 0x496b2b0>,
# 'air_temp': 25.0,
# 'id': None,
# 'latitude': 38.0,
# 'longitude': -122.0,
# 'pvmodule': 'SPR E20-327',
# 'timezone': -8.0}
</pre>
<p>Yay?</p>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-69949654225234104122016-07-18T15:37:00.000-07:002016-07-18T15:37:58.621-07:00Carousel Python Model Simulation Framework<h1>Carousel - A Python Model Simulation Framework</h1>
<p>I want to introduce Carousel, a project that my employer, <a href="https://us.sunpower.com/">SunPower</a> has been supporting for use in prediction models. Carousel is an extensible framework for mathematical models that handles generic routines such as loading and saving data, generating reports, converting units, propagating uncertainty and running simulations so developers can focus on creating complex algorithms that are easy to share and maintain.</p>
<h1>Introduction</h1>
<p>Mathematical models consist of algorithms glued together with generic routines. While the algorithms may sometimes be unique and complex, the rest of the code is often simple and routine. Sometimes mathematical models developed by teams of developers over time become difficult to update because there is no framework for how new data, calculations and outputs are integrated into the existing models. Carousel allows developers to focus on creating complex mathematical models that are robust and easy to maintain by abstracting generic routines and establishing a simple but extensible framework.</p>
<h1>The Framework</h1>
<p>A Carousel basic model consists of 5 built in layers:</p>
<ul>
<li>Data</li>
<li>Formulas</li>
<li>Calculations</li>
<li>Simulations</li>
<li>Outputs</li>
</ul>
<h2>Layers</h2>
<p>Carousel is extensible by creating more advanced models and layers. A Carousel model is a collection of layers. Carousel layers share a common base class. Each layer also has a corresponding object and a registry where objects are stored. All layers have a load method that loads all of the layer objects specified into the model. When a model is loaded it loads the objects specified for each layer.</p>
<h3>Example</h3>
<p>Consider a load shifting algorithm for residential or commercial rooftop solar power. The model might have a performance calculation, a load calculation, a cost calculation and an optimization algorithm that determines how home or business appliances are operated to minimize overall yearly cost of the system.</p>
<p>The performance calculation contains several formulas which require input data from an internal database of solar panel parameters and an online API of weather conditions, so the user creates a data source and reader for each of these. There are some data readers already included in Carousel and once a data reader is created it can be reused in many different projects. Maybe the user submits a pull request to Carousel to add the new API and database reader. The load calculation contains formulas for how the appliances are used. The input data for the appliances are entered into a generic worksheet so the user creates a data source for appliances and uses the <code>XLRDReader</code> to collect the data for each appliance from their worksheets.</p>
<p>The user organizes the formulas into 4 modules, that correspond to each of the calculations, but some formulas are reused since they are generic. For example, data frame summation formulas are used with different time series to create daily, monthly and annual outputs. The user maps out the calculations and specifies their interdependence to other formulas. For example, the cost calculation depends on the load and performance calculations and the optimization algorithm depends on the cost.</p>
<p>The user specifies each output name, initial value and other attributes. Specifications for each layer can be in a JSON parameter file or directly in the code as class attributes; Carousel will interpret either at runtime when it creates the model. Finally the user creates a simulation which in this case is unique because instead of marching through time or space, the simulation iterates over potential load shifting solutions from the algorithm. The user decides which data or outputs to log during the simulation and which to save in reports. Now that the model is created, the user loads the model and sends it the "start" command. After the simulation is complete, the user can examine the outputs and their variances. The outputs will have been automatically converted to the units specified in the model.</p>
<h2>Data Layer</h2>
<p>The data layer handles all inputs to the models. The data layer object is a data source. Each data source has a data reader. A data source is a document, API, database or other place from which input data for the model can be obtained. The data source and reader provide a framework for specifying how data is acquired. For example a data source for stock market prices might be a public API. An implementation of the stock market API data source specifies the names and attributes of each input data that will be read from the API and how the data reader should read them. The data source is similar to a deserializer because it describes how the data from the source should be interpreted by the model and creates an object in the data registry.</p>
<h2>Formula Layer</h2>
<p>The formula layer handles operations on input data that generate new outputs. It differs from the calculation layer which handles how formulas are combined together. The formula layer object is a formula, and each formula has a formula importer. For example the Python formula importer can import formulas that are written as Python functions.</p>
<h2>Calculation Layer</h2>
<p>Calculations are combinations of formulas. Each calculation also has a map of what data and output are used as arguments and what outputs the return values will be. Calculations also implement calculators. Currently there is a static calculator and a dynamic calculator, but new calculators can be implemented that can be reused in other models. The calculation also implements indexing into data and output registries in order retrieve items by index or at a specific time.</p>
<h2>Output Layer</h2>
<p>Outputs are just like data except they don't need a reader because they are only generated from calculations. Each output is like a serializer because it determines how output objects will be reported or displayed to the user.</p>
<h2>Simulation Layer</h2>
<p>The simulation layer determines the order of calculations based on the calculation dependencies. It first executes all of the static calculations and then loops over dynamic calculations, displaying logs and saving periodic reports as specified.</p>
<h2>Model</h2>
<p>The model is a low level class that can be extended to add new layers or implement new simulation commands. Currently only the basic model is implemented. A new model might contain a post processing layer that generates plots and reports from outputs.</p>
<h2>Registry</h2>
<p>Every layer has a dictionary called a registry that contains all of the layer objects and metadata corresponding to the layer attributes. The registry implements a register method that doesn't allow an item to registered more than once. Each layer registry is subclassed from the base registry so that specific layer attributes can be associated to each key. For example, data sources and outputs have a <code>variance</code> attribute while formulas have an <code>args</code> attribute.</p>
<h1>Running Model Simulations</h1>
<p>After a model has been described using the framework, it can be loaded. Then any or all of the model simulations can be executed from the model. The simulation specifies <code>commands</code> to the model that user sends using the models <code>command</code> method. Currently the basic model can execute the simulation <code>start</code> command.</p>
<h1>Units and Uncertainty</h1>
<p>Carousel uses the <a href="http://pint.readthedocs.io/en/latest/wrapping.html">Pint units wrapper</a> to convert units as specified. Uncertainty is propagated using the <a href="https://pypi.python.org/pypi/UncertaintyWrapper">UncertaintyWrapper</a> package which was developed for Carousel. It can wrap Python extensions and non-linear algorithms without changing any code. It propagates covariance and sensitivity across all formulas. </p>
<h1>Future Work</h1>
<p>A basic version of Carousel is ready now. There is an example of a photovoltaic module performance model in the documentation online at GitHub. Some ideas for new features are listed in the Carousel wiki on GitHub.</p>
<ul>
<li>Data validation</li>
<li>Reuse 3rd party serializer/deserializer for data layer</li>
<li>Model integrity check</li>
<li>Database data reader</li>
<li>REST API data reader</li>
<li>Online repository to share data readers, simualtions, formulas and layers</li>
<li>Automatic solver selection</li>
<li>Post processing layer</li>
<li>Testing tools</li>
<li>Concurrency and speedups</li>
<li>Remote process and Carousel client</li>
</ul>
<h1>Source, Docs, Issues and Wiki</h1>
<ul>
<li><a href="https://github.com/SunPower/Carousel">Carousel</a></li>
<li><a href="https://github.com/SunPower/UncertaintyWrapper">UncertaintyWrapper</a></li>
</ul>
<h1>Previous Presentations</h1>
<p>Carousel was presented at the <a href="https://pvpmc.sandia.gov/resources/events/2016-pvlib-user-group-meeting/">5th Sandia PVPMC Workshop hosted by EPRI in Santa Clara in May 2016</a></p>
<h1>Acknowledgement</h1>
<p>Carousel and UncertaintyWrapper were developed with the support of SunPower Corp. They are distributed with a BSD 3-clause license.</p>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-73763430727232445212016-04-22T17:41:00.002-07:002020-06-15T16:00:57.780-07:00Uncertainty Unwrapped<link href='https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href="https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shThemeFadeToGrey.css" rel="stylesheet" type="text/css" />
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shCore.js" type="text/javascript"></script>
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shAutoloader.js" type="text/javascript"></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPython.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushXml.js' type='text/javascript'></script>
<script language='javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all();
</script>
<p>Please read my <a href="https://poquitopicante.blogspot.com/p/darn-disclaimer-and-litigious-license.html">disclaimer</a>.</p>
<p>I'm proud to announce <a href="https://pypi.python.org/pypi/uncertainty_wrapper">UncertaintyWrapper</a> at the <a href="https://pypi.python.org/pypi">Cheese Shop</a>. This work was supported by my employer, <a href="https://us.sunpower.com/">SunPower Corp.</a> and is currently offered with a <a href="https://opensource.org/licenses/BSD-3-Clause">standard 3-clause BSD license</a>. The <a href="https://sunpower.github.io/UncertaintyWrapper/">documentation</a>, <a href="https://github.com/SunPower/UncertaintyWrapper/">source code</a> and <a href="https://github.com/SunPower/UncertaintyWrapper/releases">releases</a> are also available <a href="https://github.com/SunPower">our SunPower org GitHub</a> page.</p>
<p>So what does <code>uncertainty_wrapper</code> do? Let's say you have a Python function, to calculate solar position, and the function uses a C/C++ library via the <a href="https://docs.python.org/2/library/ctypes.html">Python <code>ctypes</code> library</a>. Or maybe you just have a really complicated set of calculations that you repeat 8760 times, and you want it to run super fast, so you don't want it to calculate derivatives and uncertainty at every internal step, just the final output. Oh and by the way, you want all 8760 calculations vectorized, _ie_: done concurrently as much as possible.</p>
<p>Heres an example using <a href="http://pvlib-python.readthedocs.org/en/latest/index.html">PVLIB</a> of just the first 24 hours.</p>
<pre class="brush:python">
#
import numpy as np # v1.11.0
import pandas as pd # v0.18.0
import pytz # v2016.1
import pvlib # v0.3.2
from uncertainty_wrapper import unc_wrapper_args # v0.4.1
PST = pytz.timezone('US/Pacific') # Pacific Standard Time
times = pd.DatetimeIndex(start='2015/1/1', end='2015/1/2', freq='1h', tz=PST) # date range
# arguments for the number of observations
# new in UncertaintyWrapper==0.4.1, jagged arrays are okay
latitude, longitude, pressure, altitude, temperature = 37., -122., 101325., 0., 22.
# standard deviation of 1% assuming normal distribution
covariance = np.tile(np.diag([0.0001] * 5), (times.size, 1, 1)) # tile this for the number of observations
@unc_wrapper_args(1, 2, 3, 4, 5)
# indices specify positions of independent variables:
# 1: latitude, 2: longitude, 3: pressure, 4: altitude, 5: temperature
def spa(times, latitude, longitude, pressure, altitude, temperature):
# location class only used prior to pvlib-0.3
dataframe = pvlib.solarposition.spa_c(times, latitude, longitude, pressure, altitude, temperature)
retvals = dataframe.to_records()
zenith = retvals['apparent_zenith']
zenith = np.where(zenith<90, zenith, np.nan)
azimuth = retvals['azimuth']
return zenith, azimuth
ze, az, cov, jac = spa(times, latitude, longitude, pressure, altitude, temperature, __covariance__=covariance)
df = pd.DataFrame({'zenith': ze, 'az': az}, index=times) # easier to view as dataframe
print df
# az zenith
# 2015-01-01 00:00:00-08:00 349.297715 NaN
# 2015-01-01 01:00:00-08:00 40.210628 NaN
# 2015-01-01 02:00:00-08:00 66.719304 NaN
# 2015-01-01 03:00:00-08:00 80.930185 NaN
# 2015-01-01 04:00:00-08:00 90.852887 NaN
# 2015-01-01 05:00:00-08:00 99.212426 NaN
# 2015-01-01 06:00:00-08:00 107.181217 NaN
# 2015-01-01 07:00:00-08:00 115.450451 NaN
# 2015-01-01 08:00:00-08:00 124.564183 84.113440
# 2015-01-01 09:00:00-08:00 135.023137 74.984664
# 2015-01-01 10:00:00-08:00 147.247403 67.475783
# 2015-01-01 11:00:00-08:00 161.371578 62.273878
# 2015-01-01 12:00:00-08:00 176.922804 60.008978
# 2015-01-01 13:00:00-08:00 192.742327 61.017538
# 2015-01-01 14:00:00-08:00 207.519768 65.144340
# 2015-01-01 15:00:00-08:00 220.494108 71.839001
# 2015-01-01 16:00:00-08:00 231.600910 80.422988
# 2015-01-01 17:00:00-08:00 241.184075 89.948123
# 2015-01-01 18:00:00-08:00 249.726361 NaN
# 2015-01-01 19:00:00-08:00 257.751550 NaN
# 2015-01-01 20:00:00-08:00 265.873170 NaN
# 2015-01-01 21:00:00-08:00 275.014534 NaN
# 2015-01-01 22:00:00-08:00 287.078877 NaN
# 2015-01-01 23:00:00-08:00 307.283646 NaN
# 2015-01-02 00:00:00-08:00 348.921385 NaN
# covariance at 8AM
idx = 8
print times[idx]
# Timestamp('2015-01-01 08:00:00-0800', tz='US/Pacific', offset='H')
nf = 2 # number of dependent variables: [ze, az]
print cov[idx]
# [[ 0.6617299 -0.6152971 ]
# [-0.6152971 0.62483904]]
# standard deviation
print np.sqrt(cov[8].diagonal())
# [ 0.81346782, 0.79046761]
# Jacobian at 8AM
nargs = 5 # number of independent args
print jac[idx]
# [[ 5.56456716e-01 -6.45065654e-01 -1.37538277e-06 0.00000000e+00 4.72409055e-04]
# [ 8.29163154e-02 6.47436098e-01 0.00000000e+00 0.00000000e+00 0.00000000e+00]]
#
</pre>
<p>First this tells us that the standard deviation of the zenith is 1% if the input has a standard deviation of 1%. That's reasonable.
This also tells that zenith is more sensitive to latitude and longitude than pressure or temperature and more sensitive to latitude than azimuth is.</p>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-42616358800284926332015-11-05T01:14:00.001-08:002020-06-15T16:04:33.251-07:00Wrangling Django ArrayField Migrations<link href='https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href="https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shThemeFadeToGrey.css" rel="stylesheet" type="text/css" />
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shCore.js" type="text/javascript"></script>
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shAutoloader.js" type="text/javascript"></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPython.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushXml.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushSql.js' type='text/javascript'></script>
<script language='javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all();
</script>
<p>Unfortunately you can't depend on <code>makemigrations</code> to generate the correct SQL to migrate and cast data from a scalar field to a PostgreSQL ARRAY. But Django provides a nifty <a href="https://docs.djangoproject.com/en/1.8/ref/migration-operations/#runsql"><code>RunSQL</code></a> that's also described in this post, <a href="https://www.aeracode.org/2013/9/25/down-and-dirty/">"Down and Dirty - 9/25/2013"</a> by <a href="https://www.aeracode.org/about/">Aeracode</a>, the original creator of <a href="http://south.aeracode.org/">South</a> predecessor of <a href="https://docs.djangoproject.com/en/stable/topics/migrations/">Django migrations</a>. That post even mentions using <code>RunSQL</code> to alter a column using CAST.</p>
<p>The issue and trick to migrating a column to an <code>ArrayField</code> is given by PostgreSQL in the traceback, which says:
<blockquote><pre>column "my_field " cannot be cast automatically to type double precision[]
HINT: Specify a USING expression to perform the conversion.</pre></blockquote>
Further hints can be found by <a href="http://www.postgresql.org/docs/9.4/static/sql-altertable.html">rtfm</a> and searching the internet, <a href="http://stackoverflow.com/a/13879850/1020470">such this stackoverflow Q&A</a>.
My procedure was to use <code>makemigrations</code> to get the <code>state_operations</code> and then wrap each one into a <code>RunSQL</code> migration operation.</p>
<pre class="brush: python">
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import migrations, models
import datetime
from django.utils.timezone import utc
import django.contrib.postgres.fields
import simengapi_app.models
import django.core.validators
class Migration(migrations.Migration):
dependencies = [
('my_app', '0XYZ_auto_YYYYMMDD_hhmm'),
]
operations = [
migrations.RunSQL(
"""
ALTER TABLE my_app_mymodel
ALTER COLUMN "my_field"
TYPE double precision[]
USING array["my_field"]::double precision[];
""",
state_operations=[
migrations.AlterField(
model_name='mymodel',
name='my_field',
field=django.contrib.postgres.fields.ArrayField(
base_field=models.FloatField(), default=list,
verbose_name=b'my field', size=None
),
)
],
),
]
</pre>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-78553034833797309352015-10-20T11:00:00.001-07:002020-06-15T16:05:29.360-07:00REST-ful revelations<link href='https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href="https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shThemeFadeToGrey.css" rel="stylesheet" type="text/css" />
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shCore.js" type="text/javascript"></script>
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shAutoloader.js" type="text/javascript"></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPython.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushXml.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushSql.js' type='text/javascript'></script>
<script language='javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all();
</script>
<p>I've started using <a href="http://www.django-rest-framework.org/">Django REST Framework</a>, and it is simply magic!</p>
<p>Here is a technique I've used to input lists of primitive types and serializers with <code>many=True</code></p>
<pre class="brush: python">
from functools import partial
from rest_framework import viewsets
from rest_framework.response import Response
from rest_framework import status
from my_app.serializers import MyNestedModelSerializer
...
class MyNestedModelViewSet(viewsets.ViewSet):
serializer_class = MyNestedModelSerializer
def create(self, request):
serializer = self.serializer_class(data=request.data)
# get the submodel list serializer since it can't render/parse html
submodel_list_serializer = serializer.fields['submodels']
# make a partial function by setting the submodel list serializer
partial_get_value = partial(custom_get_value, submodel_list_serializer)
# monkey patch submodel_list_serializer.get_value() with partial function
submodel_list_serializer.get_value = partial_get_value
if serializer.is_valid():
simulate_data = serializer.save()
# do stuff ...
return Response(serializer.data, status=status.HTTP_201_CREATED)
return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
</pre>
<p>The function `custom_get_value()` uses JSON to parse the input:</p>
<pre class="brush: python">
def custom_get_value(serializer, dictionary):
if serializer.field_name not in dictionary:
if getattr(serializer.root, 'partial', False):
return empty
# We override the default field access in order to support
# lists in HTML forms.
if html.is_html_input(dictionary):
listval = dictionary.getlist(serializer.field_name)
if len(listval) == 1 and isinstance(listval[0], basestring):
# get only item in value list, strip leading/trailing whitespace
listval = listval[0].strip()
# add brackets if missing so that it's a JSON list
if not (listval.startswith('[') and listval.endswith(']')):
listval = '[' + listval + ']'
# try to deserialize JSON string
try:
listval = json.loads(listval)
except ValueError as err:
# return original string and log error
pass
# set the field with the new value list
dictionary.setlist(serializer.field_name, listval)
val = dictionary.getlist(serializer.field_name, [])
if len(val) > 0:
# Support QueryDict lists in HTML input.
return val
return html.parse_html_list(dictionary, prefix=serializer.field_name)
return dictionary.get(serializer.field_name, empty)
</pre>
bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-75422828500496178822015-05-26T10:07:00.001-07:002020-06-15T16:13:37.994-07:00Geek & Sundry<p>If you enjoyed <a href="http://drhorrible.com/">Dr. Horrible's Sing Along Blog</a> then you may also like <a href="http://www.imdb.com/name/nm1260407/">Felicia Day</a>'s <a href="http://watchtheguild.com/">The Guild</a> from her <a href="https://www.youtube.com/user/geekandsundry">Geek & Sundry YouTube channel</a>. Irresistible, and pretty amazing production values for a web series.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/grCTXGW3sxQ" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-13525402108992246282015-05-21T09:44:00.001-07:002016-04-08T15:52:29.591-07:00Dr. Horrible's Sing Along BlogLove struck super villain (Neil Patrick Harris) loses to super hero (Nathan Fillion) musical. This just never get’s old. Almost as good as the <a href="https://youtu.be/ngElkyQ6Rhs">official Star Wars trailer</a>.<br />
<a href="http://www.drhorrible.com"><img src="http://www.drhorrible.com/images/banners/banner2.gif" border="0"></a>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-59325739324951559552015-05-14T00:26:00.002-07:002020-06-15T16:16:29.731-07:00[MATLAB] Do *not* use `obj.empty` to preallocate object array<!-- begin js -->
<link href='https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href="https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shThemeFadeToGrey.css" rel="stylesheet" type="text/css" />
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shCore.js" type="text/javascript"></script>
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shAutoloader.js" type="text/javascript"></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPython.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushXml.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushBash.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushJava.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPowerShell.js' type='text/javascript'></script>
<script language="javascript">
/**
* Author: Will Schleter
* based on: http://www.jamesrohal.com
*/
SyntaxHighlighter.brushes.MatlabKey = function()
{
var keywords = 'break case catch classdef continue else elseif end for function global if otherwise parfor persistent return spmd switch try while';
var functions = ' ';
this.regexList = [
{ regex: /%.*$/gm, css: 'comments' }, // one line comments
{ regex: /\%\{[\s\S]*?\%\}/gm, css: 'comments'}, // multiline comments
{ regex: SyntaxHighlighter.regexLib.singleQuotedString, css: 'string' },
{ regex: SyntaxHighlighter.regexLib.doubleQuotedString, css: 'string'},
{ regex: new RegExp(this.getKeywords(keywords), 'gm'), css: 'keyword' }
];
};
SyntaxHighlighter.brushes.MatlabKey.prototype = new SyntaxHighlighter.Highlighter();
SyntaxHighlighter.brushes.MatlabKey.aliases = ['matlabkey'];
</script>
<script language='javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all();
</script>
<!-- end js -->
<h1>FYI: You do not need to use `obj.empty` to preallocate an object array.</h1>
<p>In fact as soon as you assign a value to any element in the object array it <i>grows</i> the array to that size, which allocates (or reallocates) RAM for the new object array, therefore defeating the point of <i>preallocating</i> space.</p>
<p>From <a href="http://www.mathworks.com/help/matlab/matlab_oop/empty-arrays.html">Empty Arrays section of OOP documentation</a>:
<blockquote style="background-color: lightgray; padding: 5px; border-style: none none none solid; border-color: white black;">“If you make an assignment to a property value, MATLAB calls the SimpleClass constructor to <b>grow</b> the array to the require size:”</blockquote></p>
<p>Instead if you want to preallocate space for an object array, grow the array once by assigning the last object first. This requires the class to have a no-arg constructor. Each time you grow your array you will reallocate RAM for it, wasting time and space, so do it once with the max expected size of the array. See <a href="http://www.mathworks.com/help/matlab/matlab_oop/initialize-object-arrays.html">Initialize Object Arrays</a> and <a href="http://www.mathworks.com/help/matlab/matlab_oop/initializing-arrays-of-handle-objects.html">Initializing Arrays of Handle Objects</a> in the OOP documentation.</p>
<pre class="brush: matlabkey; gutter: false">
>> S(max_size) = MyClass(args)
</pre>
<p>Another option is to preallocate any other container like a cell array (best IMHO), structure or <a href="http://poquitopicante.blogspot.com/2013/09/matlab-containersmap-is-10x-more.html"><code>containers.Map</code></a> and then fill in the class objects as they are created. An advantage to this is you don’t have to subclass <a href="http://www.mathworks.com/help/matlab/ref/matlab.mixin.heterogeneous-class.html"><code>matlab.mixin.Heterogeneous</code></a> to group different classes together.</p>
<pre class="brush: matlabkey; gutter: false">
>> S = cell(max_size); args = {1,2,3;4,5,6;7,8,9};
>> for x = 1:size(args,1), S(x) = MyClass(args{x,:});end
</pre>
<p>The only time to use an empty object is if you want it as a default for the situation where nothing gets instantiated, and you need the it be an instance of the class. Of course any empty array will do this, IE: <code>''</code>, <code>[]</code> and <code>{}</code> are also empty.</p>
<pre class="brush: matlabkey; gutter: false">
>> S = MyClass.empty
>> if blah,S = MyClass(args);end
>> if isa(S, 'MyClass') && isempty(S),do stuff; end
</pre>
<p><a href="http://www.mathworks.com/matlabcentral/answers/12947-why-do-i-have-to-create-empty-object-array-before-calling-constructor-in-constructor">Another reason might be to clear defaults if the constructor is called recursively, although <code>obj.delete</code> will do the same thing.</a></p>
<p>I hope this helps someone; it definitely helped me understand the odd nature of MATLAB. This behavior is because everything in MATLAB is an array, even a scalar is a <1x1 double> read the C-API <a href="http://www.mathworks.com/help/pdf_doc/matlab/apiref.pdf "><code>mxArray</code></a> for external references and <a href="http://www.mathworks.com/help/compiler_sdk/cxx/mwarray.html"><code>mwArray</code></a> for compiled/deployed MATLAB for more info.
<blockquote style="background-color: lightgray; padding: 5px; border-style: none none none solid; border-color: white black;">MATLAB = Matrix Laboratory</blockquote>Class definitions didn’t appear until 2008. Other languages like C++, Java, Python and Ruby are object first. So the <a href="
http://www.mathworks.com/help/matlab/ref/empty.html">empty method</a> is meant to duplicate the ability to be empty similar to other MATLAB datatypes such as double, cell, struct, etc. IMO outside of MATLAB it's a very artificial and somewhat meaningless construct.</p>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-70095091743604665632015-04-22T23:37:00.001-07:002020-06-15T16:45:41.776-07:00Git big media on Windows<style type="text/css">
code {
background-color: lightgray;
}
</style>
<p><i>[UPDATE 2015-06-09]</i> I have switched to <a href="https://pypi.python.org/pypi/git-fat/0.5.0">git-fat-0.5.0 [2015-05-06]</a>; the fork on <a href="https://pypi.python.org/pypi">PyPI</a> <a href="https://github.com/cyaninc/git-fat">maintained on GitHub by Alan Braithwaite of Cyan Inc.</a> Unfortunately, I could not figure out how to clone a git-media repo, <a href="#git-media-sucks">therefore git-media sucks</a>.</p>
<h1>Git-Fat - just works</h1>
<p>Basically this comes with everything you need to work on Windows, Mac or Linux. It uses <a href="http://en.wikipedia.org/wiki/Rsync"><code>rsync</code></a> for transport, and the rest is mostly written in Python but does depend on some libraries that are standard in Linux and have mature ports in Windows and Mac. One of the major benefits of git-fat over git-media is that it uses a <code>.gitfat</code> config file which updates your <code>.git/config</code> when <code>git fat init</code> is run. This is similar to git submodules and makes repos portable. In general there's more functionality and features than git-media. For example, you can list the files managed by git-fat, check for orphans and pull data from or push data to your remote storage. The only catch is that the wheel file at PyPI has metatags for win32 not amd64. This is easy to fix, but I think there are a couple of use cases that might differ from how the distribution was implemented.</p>
<h3>Bootstrap</h3>
<p>If you look at a Linux install, the repository has a symlink to <code>git_fat.py</code> in <code>bin</code> called <code>git-fat</code>. Why not just bootstrap git-fat if we only really need one file. Just dump everything in a single folder, change the file name to <code>git-fat</code> and make sure there's a shebang that uses <code>#! /usr/bin/env python</code> which git seems to prefer, then stick it on your path. This works for both msys-git and Windows cmd.</p>
<h3>MSYS-Git</h3>
<p><a href="http://msysgit.github.io/">msysgit</a> comes with Git Bash, a posix shell which includes many Linux libraries ported to Windows, such as gawk and ssh. Unfortunately it does not come with rsync, however you can get rsync from the <a href="http://sourceforge.net/p/mingw-w64/wiki2/MSYS/">msys</a> source either from <a href="http://sourceforge.net/projects/mingw-w64/files/External%20binary%20packages%20%28Win64%20hosted%29/MSYS%20%2832-bit%29/">mingw-w64</a> (that's where I got it), from <a href="http://msys2.github.io/">msys2</a>, from <a href="http://sourceforge.net/projects/mingw/files/">the original mingw project</a>, from <a href="http://sourceforge.net/projects/mingwbuilds/files/external-binary-packages/">mingw builds</a> and from lots of places. You could even get it from <a href="https://www.cygwin.com/">cygwin</a>. I usually stick files like this in my local bin folder which is always first on my path in git bash. You'll need to also grab the iconv, intl, lzma and popt msys libraries which rsync depends on. Anyway, since you have these libraries, you don't need the ones bundled in the wheel, however, git-fat is written to look for those bundled files if it detects that your platform is Windows, so just comment out those lines. You will need to change awk to gawk, since awk is a shell script that calls gawk. Again you can bootstrap this file, ie: put it in your local bin folder or install it into your Python site-packages and/or scripts folder.</p>
<h3>__main__</h3>
<p>This is the way I ended up using it. You can <a href="https://www.dropbox.com/s/czsf8170u215zuz/git_fat-0.5.0-main.zip">download my version here</a> and <a href="https://pip.pypa.io/en/latest/reference/pip_install.html#usage">install it with pip</a>. I put the windows libraries into the site-packages git-fat folder instead of in scripts and then in the git-fat script, added the site-packages git-fat folder to the shell's path. Then I called the git-fat module as a script by adding a <code>__main__.py</code> file to it which basically imports <code>git_fat.py</code> and calls it using Python with the <code>-m</code> option, but you could just as easily call the module as a script. This just keeps these extra libraries bundled together rather than dumping them into the scripts folder with everything else. Also since I mostly use git bash it doesn't put git-fat's libraries ahead of git's since they both use gawk and ssh.</p>
<h1>Usage</h1>
<p>Usage is extremely easy compared to git-media, which is a plus! Note these instructions are for msysgit git bash. For Windows cmd window replace <code>git fat</code> with <code>git-fat</code> everywhere. Both methods should work fine.</p>
<ol><li>Clone a repo that uses git-fat: <code>git clone my://remote/repo.git</code>
</li><li>At this point there are only placeholders for your files with the same names, but just sha numbers that tell git-fat which file to grab from your remote storage
</li><li>Run <code>git fat init</code> which sets up the filters and smudges that tell your local repo how to use git-fat with the .gitattributes file which is part of the repo already.
</li><li>Run <code>git fat pull</code> which downloads your files from the remote storage specified in the .gitfat, which is also already in the repo
</li><li>Run <code>git fat list</code> to see a list of managed files
</li><li>Run <code>git fat status</code> to see a list of orphans waiting to be pulled/pushed?
</li></ol>
<p>Creating a repo and setting it up to use git-fat is also easy. There is great help in the <a href="https://pypi.python.org/pypi/git-fat/0.5.0">readme at PyPI</a>
and <a href="https://github.com/cyaninc/git-fat">the readme at GitHub</a>.</p>
<ol><li>Create a .gitfat file that specifies where rsync should store files. Note there are <b>no indents</b>. A windows UNC path seems to work fine.
<pre style="background-color: lightgray;">
[rsync]
remote = //server/share/repo/fat
</pre>
</li><li>Create a .gitattributes file to specify which files to store at the remote
</li><li>Commit the .gitfat and .gitattribute files
</li><li>Run <code>git fat init</code> to set up your local .git/config
</li><li>Hack, commit, push, etc.
</li><li>Run <code>git fat push</code> to send stuff to your remote
</li></ol>
</p>
<h1 id=git-media-sucks>Git-Media - sucky</h1>
<p>Finally time to install <a href="http://rubyinstaller.org/">Ruby</a>.
You're going to need it if you want to use
<a href="https://github.com/alebedev/git-media">git-media</a> which let's
you mix big biinary files within your git repo, but store them in some
remote host, which could be google-drive, amazon s3, another server via
ssh/scp or a network share. Why don't you want to store big binary files
in your git repo? Since Git stores each revision instead of deltas, that
means that it will quickly blow up as you make new commits.</p>
<a href="https://rubyinstaller.org/" imageanchor="1" ><img border="0" src="https://raw.githubusercontent.com/oneclick/rubyinstaller/master/resources/installer/images/wizard-logo.bmp" /></a>
<h3>RubyInstaller</h3>
<p>Super easy, they recommend 2.1, no admin rights required, unzips into c:\
just like python, I checked all of the options: tk/tcl, add ruby to path
and what was the last option? Then I ran <code>gem update</code>.</p>
<a href="http://git-scm.com/images/logos/downloads/Git-Logo-2Color.png" imageanchor="1" ><img border="0" height="100" width="237" src="http://git-scm.com/images/logos/downloads/Git-Logo-2Color.png" /></a>
<h3>git-media gem</h3>
<pre>gem install git-media trollup right_aws</pre>
<h3>right_aws OpenSSL::Digest issue</h3>
<p>There is a tiny issue with right_aws where it outputs the message:
<blockquote style="background-color: lightgray; padding: 5px; border-style: none none none solid; border-color: white black;"><code>Digest::Digest is deprecated; use Digest</code></blockquote>
which is easily fixed by following the comments in <a href="https://github.com/alebedev/git-media/issues/3">git-media issue #3</a> or <a href="https://github.com/rightscale/right_aws/pull/176/files">right_aws pull request #176</a>.</p>
<h3>git-media setup</h3>
<p>The <a href="https://github.com/alebedev/git-media/blob/master/README.md">readme</a> on the github overview page has everything you need to know.</p>
<h1>other large file storage</h1>
<ul>
<li><a href="https://git-lfs.github.com">git-lfs</a>: relative newcomer, developed by <a href="https://github.com/">github</a> for <a href="https://github.com/blog/1986-announcing-git-large-file-storage-lfs">Github large file storage</a>. Git-LFS appears to require an LFS server which GitHub intends to monetize. Sounds promising, it is open sourced, but not sure if it can be implemented easily as a standalone system unfortunately.</li>
<li><a href="https://github.com/jedbrown/git-fat">git-fat by Jed Brown</a>: an older repo that the cyaninc one is forked from, <a href="https://github.com/jedbrown/git-fat/pull/19">there is an open PR</a></li>
<li><a href="https://pypi.python.org/pypi/git-fat">git-fat on PyPI</a>: just works!</li>
<li><a href="https://git-annex.branchable.com/">git-annex</a>: written in <a href="https://www.haskell.org/">Haskell</a>, there is a <a href="https://git-annex.branchable.com/install/Windows/">beta Windows installer</a>.</li>
</ul>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-69685259030594654592015-04-15T13:35:00.000-07:002018-09-04T08:48:57.783-07:00Recommended Python Project Layout<style type="text/css">
ol li ol {
list-style-type: lower-latin;
}
ol li ol li ol {
list-style-type: lower-roman;
}
</style>
<p><i>[UPDATE 2018-09-04] Links to <a href="https://cookiecutter.readthedocs.io/en/latest/readme.html">Cookiecutter</a> and <a href="https://nsls-ii.github.io/scientific-python-cookiecutter/index.html#">Bootstrap a Scientific Python Library</a> from <a href="https://nsls-ii.github.io/">the National Synchrotron Light Source II (NLSL-II)</a>.</i></p>
<p><i>[UPDATE 2016-07-19] Lately I've preferred using <code style="background-color:silver">core</code> instead of <code style="background-color:silver">lib</code> for the main package modules.</i></p>
<p><i>[UPDATE 2015-06-04] Create top level package to bundle all sub-packages and package-data together for install.</i></p>
<p>Been looking for a good, comprehensive, credible guide:</p>
<ol>
<li>Pretty good links in this SO Q&A:
<ol>
<li><a href="http://stackoverflow.com/questions/193161/what-is-the-best-project-structure-for-a-python-application">What is the best project structure for a Python application?</a></li>
<li>Especially this one:
<ol>
<li><a href="http://www.jeffknupp.com/blog/2013/08/16/open-sourcing-a-python-project-the-right-way/">Open Sourcing a Python Project the Right Way</a></li>
</ol></li>
</ol></li>
<li>And maybe, maybe theses ones:
<ol>
<li><a href="http://learnpythonthehardway.org/book/ex46.html"><i>Learn Python The Hard Way</i> Exercise 46: A Project Skeleton</a></li>
<li><a href="http://docs.python-guide.org/en/latest/writing/structure/"><i>The Hitchhiker’s Guide to Python!</i> Structuring Your Project</a> by <a href="http://www.kennethreitz.org/">Kenneth Reitz</a>
<ol>
<li><a href="http://www.kennethreitz.org/essays/repository-structure-and-python">Repository Structure and Python</a> also by <a href="http://www.kennethreitz.org/">Kenneth Reitz</a></li>
</ol></li>
<li><a href="http://www.scotttorborg.com/python-packaging/minimal.html">How to Package Your Python Code: Minimal Structure</a> by <a href="http://www.scotttorborg.com/">Scott Torborg</a></li>
<li><a href="http://as.ynchrono.us/2007/12/filesystem-structure-of-python-project_21.html">Interesting Things, Largely Python and Twisted Related: Filesystem structure of a Python project</a> by <a href="http://as.ynchrono.us/">Jean-Paul Calderone</a></li>
</ol></li>
<li>Of course understanding Python Modules and Packages
<ol>
<li><a href="https://docs.python.org/2/tutorial/modules.html">The Python Tutorial: 6. Modules</a></li>
</ol></li>
<li>An understanding of how to install packages, and roughly I guess how pip and setuptools interact with distutils is good
<ol>
<li><a href="https://docs.python.org/2/install/">Python Documentation: Installing Python Modules</a></li>
</ol></li>
<li>Way later down the line it helps to understand distutils and setuptools for deploying packages
<ol>
<li><a href="https://python-packaging-user-guide.readthedocs.org/en/latest/">Python Packaging User Guide</a></li>
<li><a href="http://pythonhosted.org/setuptools/">Setuptools</a></li>
<li><a href="https://docs.python.org/2/distutils/">Python Documentation: Distributing Python Modules</a></li>
<li><a href="http://www.scotttorborg.com/python-packaging/">How to Package Your Python Code</a> by <a href="http://www.scotttorborg.com/">Scott Torborg</a></li>
<li><a href="https://the-hitchhikers-guide-to-packaging.readthedocs.org/en/latest/">The Hitchhiker’s Guide to Packaging</a></li>
</ol></li>
<li>There are also a packages that will create a boiler plate project layout for you <b>but I wouldn't recommend them except as reference guides</b> - the tutorial by NSLS-2 being the notable exception, PTAL!
<ol>
<li><a href="https://nsls-ii.github.io/scientific-python-cookiecutter/index.html#">Bootstrap a Scientific Python Library</a>: <q>This is a tutorial with a template for packaging, testing, documenting, and publishing scientific Python code.</q></li>
<li><a href="https://cookiecutter.readthedocs.io/en/latest/readme.html">Cookiecutter</a>: <q>A command-line utility that creates projects from cookiecutters (project templates), e.g. creating a Python package project from a Python package project template.</q></li>
<li><a href="https://pypi.python.org/pypi/python_boilerplate_template">PyPI: Python Boilerplate Template</a></li>
</ol></li>
</ol>
<p>It's hard to pin a standard style down. Here’s mine:</p>
<pre style="padding: 10px; background-color: lightgray">
MyProject/ <- git repository
|
+- .gitignore <- *.pyc, IDE files, venv/, build/, dist/, doc/_build, etc.
|
+- requirements.txt <- to install into a virtualenv
|
+- setup.py <- use setuptools, include packages, extensions, scripts and data
|
+- MANIFEST.in <- files to include in or exclude from sdist
|
+- readme.rst <- incorporate into setup.py and docs
|
+- changes.rst <- release notes, incorporate into setup.py and docs
|
+- myproject_script.py <- script to run myproject from command line, use Python
| argparse for command line arguments put shebang
| `#! /usr/bin/env python` on 1st line and end with a
| `if __name__ == "__main__":` section, include in
| setup.py scripts section for install
|
+- any_other_scripts.py <- scripts for configuration, documentation generation
| or downloading assets, etc., include in setup.py
|
+- venv/ <- virtual environment to run tests, validate setup.py, development
|
+- myproject/ <- top level package keeps sub-packages and package-data together
| for install
|
+- __init__.py <- contains __version__, an API by importing key modules,
| classes, functions and constants, __all__ for easy import
|
+- docs/ <- use Sphinx to auto-generate documentation
|
+- tests/ <- use nose to perform unit tests
|
+- other_package_data/ <- images, data files, include in setup.py
|
+- core/ <- main source code for myproject, sometimes called `lib`
| |
| +- __init__.py <- necessary to make mypoject_lib a sub-package
| |
| +- … <- the rest of the folders and files in myproject
|
+- related_project/ <- a GUI library that uses myproject_lib or tools that
| myproject_lib depends on that's bundled together, etc.
|
+- __init__.py <- necessary to make related_project a sub-package
|
+- … <- the rest of the folders and files in your the related project
</pre>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-3330287221653822822015-03-26T00:05:00.001-07:002020-06-15T16:20:49.697-07:00Building Python x64 on Windows 7 with SDK 7.0<link href='https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href="https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shThemeFadeToGrey.css" rel="stylesheet" type="text/css" />
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shCore.js" type="text/javascript"></script>
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shAutoloader.js" type="text/javascript"></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPython.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushXml.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPowerShell.js' type='text/javascript'></script>
<script language='javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all();
</script>
<p>[UPDATE 2015-07-02] <i>Check out <a href="http://breakingbytes.bitbucket.io/">Python Bootstrap</a> a continuously integrated build of Python-2.7 for Windows that can be installed without admin rights.</i></p>
<p>I know I was just <a href="http://poquitopicante.blogspot.com/2015/03/python-issue-22516-administrator-rights.html">ranting about the inability to distribute Python27 w/o admin rights</a>, but surprise! This is a piece of cake, thanks to the whoever the amazing Python developers are who maintain the PCbuild and external buildbot tools for Windows. <a href="http://www.clemens-sielaff.com/create-a-portable-python-with-pip-on-windows/">There are also a few sites out there that have similar information on building Python for windows</a>, but honestly everything you need is in <a href="https://hg.python.org/cpython/file/v2.7.10/PCbuild/readme.txt">PCbuild/readme.txt</a>. Read it, then read it again. Seriously. Also check out the <a href="https://docs.python.org/devguide/">Python docs developer's guide</a>. Hmm, thinking of setting up an <a href="http://www.appveyor.com/">AppVeyor</a> buildbot for this.</p>
<div>OK, let's do this:
<ol>
<li>Get Python and install it on your system. You may need a working binary to bootstrap the amd64 build.</li>
<li>Get a working version of <a href="http://www.microsoft.com/en-us/download/details.aspx?id=3138">Microsoft SDK for Windows 7 (7.0)</a>. AFAIK Visual Studio 2013 Express Desktop or Community editions include both SDK 7.0 and 7.1, so alternately install that. Make sure that you include the redistributables in when installing the SDK because you will need them to distribute your Python build. See <a href="http://poquitopicante.blogspot.com/2015/03/upgrade-to-visual-studio-2013-express.html">upgrade to vs2013</a> for fixes to some issues you may encounter especially if you have some other VC components already installed.</li>
<li>Get a working <a href="https://subversion.apache.org/packages.html#windows"><code>svn</code> binary for windows</a> and put it on your path. I use <a href="http://www.collab.net/downloads/subversion">CollabNet SubVersion</a> commandline binaries. The easiest way to get and build all of the external libraries (bzip2, sqlite3, tk/tcl, etc.) is to use the <a href="https://hg.python.org/cpython/file/v2.7.10/Tools/buildbot">Tools/buildbot batch scripts</a> which call <code>svn.exe</code>.</li>
<li>Either download the <a href="https://www.python.org/downloads/release/python-2710/">gzipped source tarball from python.org</a> or clone the <a href="https://hg.python.org/cpython/tag/v2.7.10/">tag v2.7.10 of the cpython mercurial repository</a>. Archives of the Hg repo are also conveniently available from the repo viewer.</li>
<li>Read the PCbuild/readme.txt again</li>
<li>Open the SDK command shell from the Startmenu. Change the target to <b>Release x86</b>, the default is <i>Debug x64</i>, by typing the following:</li>
<pre style="background-color:black; color:yellow">C:\Program Files\Microsoft SDKs\Windows\v7.0>setenv /Release /x86</pre>
<li>Change to the directory where the source tarball is extracted.</li>
<li>Patch the <code>Tools/buildbot/externals</code> batch script exactly as described in the PCbuild readme. I added the release build immediately after the debug fields.</li>
<pre class="brush:powershell">
if not exist tcltk\bin\tcl85.dll (
@rem all and install need to be separate invocations, otherwise nmakehlp is not found on install
cd tcl-8.5.15.0\win
nmake -f makefile.vc INSTALLDIR=..\..\tcltk clean all
nmake -f makefile.vc INSTALLDIR=..\..\tcltk install
cd ..\..
)
if not exist tcltk\bin\tk85.dll (
cd tk-8.5.15.0\win
nmake -f makefile.vc INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.15.0 clean
nmake -f makefile.vc INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.15.0 all
nmake -f makefile.vc INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.15.0 install
cd ..\..
)
if not exist tcltk\lib\tix8.4.3\tix84.dll (
cd tix-8.4.3.5\win
nmake -f python.mak DEBUG=0 MACHINE=IX86 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk clean
nmake -f python.mak DEBUG=0 MACHINE=IX86 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk all
nmake -f python.mak DEBUG=0 MACHINE=IX86 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk install
cd ..\..
)
</pre>
<li>From the archive root (<code>Python-2.7.9</code>) call the <code>externals</code> batch script. It will copy and build all of the externals from svn.python.org in a folder called <code>externals/</code>.</li>
<li>Now cd to PCbuild and call <code>build.bat</code>. Voila, python.exe for x86.</li>
<li>Almost there. go back up to the root of the extracted tarball and rename externals to externals-x86.</li>
<li>Change the target to <b>Release x64</b> by typing the following:</li>
<pre style="background-color:black; color:green">C:\Users\myname\downloads\Python-2.7.9>setenv /Release /x64</pre>
<li>Set an environment variable <code>HOST_PYTHON=C:\Python27\python.exe</code>. You may not need this at all or you might be able to use the 32-bit version just built.</li>
<li>Patch the buildbot externals-amd64 batch script just like the x86 script.</li>
<pre class="brush:powershell">
if not exist tcltk64\bin\tcl85.dll (
cd tcl-8.5.15.0\win
nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 clean all
nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 install
cd ..\..
)
if not exist tcltk64\bin\tk85.dll (
cd tk-8.5.15.0\win
nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 TCLDIR=..\..\tcl-8.5.15.0 clean
nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 TCLDIR=..\..\tcl-8.5.15.0 all
nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 TCLDIR=..\..\tcl-8.5.15.0 install
cd ..\..
)
if not exist tcltk64\lib\tix8.4.3\tix84.dll (
cd tix-8.4.3.5\win
nmake -f python.mak DEBUG=0 MACHINE=AMD64 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk64 clean
nmake -f python.mak DEBUG=0 MACHINE=AMD64 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk64 all
nmake -f python.mak DEBUG=0 MACHINE=AMD64 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk64 install
cd ..\..
)
</pre>
<li>From the archive root (<code>Python-2.7.9</code>) call the <code>externals-amd64</code> batch script</li>
<li>Finally cd back to PCbuild and call <code>build.bat -p x64</code>. Voila, python.exe for x64.</li>
<li>Add <code>externals/tcltk</code> to your path and run the tests</li>
<pre class="brush:powershell, gutter:false">C:\Users\myname\downloads\Python-2.7.9\PCbuild>set PATH=C:\Users\myname\downloads\Python-2.7.9\externals;%PATH%
C:\Users\myname\downloads\Python-2.7.9\PCbuild>rt
</pre>
<li>Copy the VC runtime from <code>Program Files (x86)\Microsoft Visual Studio 9.0\VC\redist\x86\Microsoft.VC90.CRT\</code>to PCbuild folder, and
<code>Program Files (x86)\Microsoft Visual Studio 9.0\VC\redist\amd64\Microsoft.VC90.CRT\</code>to PCbuild\amd64 folder.</li>
<li>To distribute create a similar file structure for both archtypes and copy the files into the folders</li>
<pre>
Python27
|
+-python.exe
|
+-pythonw.exe
|
+-python27.dll
|
+-msvcr90.dll <- from VC/redist/MICROSOFT.VC90.CRT
|
+-msvcp90.dll <- from VC/redist/MICROSOFT.VC90.CRT
|
+-msvcm90.dll <- from VC/redist/MICROSOFT.VC90.CRT
|
+-MICROSOFT.VC90.CRT.manifest <- from VC/redist/MICROSOFT.VC90.CRT
|
+-DLLs <- all externals/tcltk/bin, PCbuild/*.dll & PCbuild/*.pyd files
| & PC/py.ico, PC/pyc.ico & PC/pycon.ico.
|
+-Lib <- same as source archive except all *.pyc files, all plat-* folders
| & ensurepip folder
|
+-libs <- all PCbuild/*.lib files
|
+-include <- same as source archive + PC/pyconfig.h
|
+-tcl <- everything in tcltk/lib
|
+-Scripts <- PCbuild/idle.bat
|
+-Doc <- set %PYTHON%=python.exe and build Sphinx docs with Sphinx if you have it
</pre>
</ol>
<p><i>Note</i>: <code>wish85.exe</code> and <code>tclsh85.exe</code> won't work with this Python installed file structure although it will work in the externals bin folder because they look for the <code>tcl85.dll</code> in <code>../lib</code>. Also note that <code>idle.bat</code> needs some fixin. And also it's very important to know that most executables in Scripts created by installer packages have the path to <code>python.exe</code> hardwired, IE: they are initially not portable, however <a href="http://www.clemens-sielaff.com/create-a-portable-python-with-pip-on-windows/">check out this blog for a few quick tricks to fix them</a>.</p>
<p>You can <a href="https://www.dropbox.com/s/bcmh11kha9go8t6/Python27-x64_asbuilt.zip?dl=0">download my x64 build</a> and <a href="https://www.dropbox.com/s/wc2x2or2477f4as/Python27-x86_asbuilt.zip?dl=0">x86 build</a> from dropbox. Congratulations!</p>
</div>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-25013934363632395732015-03-19T14:37:00.000-07:002017-03-17T10:22:00.275-07:00Python issue 22516: Administrator rights required for local installation<p>[UPDATE 2015-07-02] <i>Check out <a href="http://breakingbytes.bitbucket.io/">Python Bootstrap</a> a continuously integrated build of Python-2.7 for Windows that can be installed without admin rights.</i></p>
<div><blockquote>
<a href="http://bugs.python.org/issue22516">Windows Installer won't - even when using "just for me"option</a>
</blockquote></div>
<p>This issue starts simple enough, but then unravels to reveal some very interesting insights. Evidently creating a Python for Windows installer that does not depend on administrator rights is not as easy as it seems. The question comes down to what we really need. Steve Dover from Microsoft breaks it down like this:</p>
<ol>
<li><i>Python as application</i></li>
This is when someone wants to use python to write scripts, do analysis, etc. Python is an application on their windows machine just like MATLAB or Excel. This could be installed per-user and possible without administrative rights.
<li><i>Python as library</i></li>
This version of Python could be used to embed Python in an application. Something similar to what <a href="http://www.py2exe.org/">pyexe</a> and other python-freezing packages do. This could be a zip file.
<li><i>Python as-if system</i></li>
This version would be installed on a system, possibly by system admins, in a custom windows build, similar to the way that Python is integrated into Mac and Linux. It would require admin rights, and could be used by system admins to add custom functionality to the corporate OS.
</li>
</ol>
<p>Let me say 1st off that I think that #3 is absurd. I can't imagine Windows system admins <b>ever</b> using Python in this way. Most of them have never even heard of Python. And there is an entire .NET infrastructure to do exactly this. Why would you use Python instead of C#? Windows will never, ever be like Linux. It is not a POSIX environment, it does not use GNU tools and it does not need Python.</p>
<p>I think that #1 and #2 could serve the same purpose and should really be the only option available. Users who want to use Python on Windows should unzip the Python Windows distro to their System Root (ie: C:\Python27) and use it. No admin rights required. End of story. It should contain the msvcrt90 redistributable and all libraries it needs. There should be no other dependencies.</p>
<p>There is also a 0th option - do not distribute a binary for Windows at all. Let Windows users build from source themselves, or recommend alternate distribution, which Python.org already does on its <a href="https://www.python.org/download/alternatives/">alternative distribution page.</a>. This is what Ruby does, and perhaps it's the best way to satisfy everyone. But the fact that official Python is available for windows is a very nice thing. Althoght Enthought and ActiveState have been around for a long time, they are private and could go out of business. Nevertheless, this does seem to be the path people are taking.</p>
<p><a href="https://store.continuum.io/cshop/anaconda/">Anaconda, from Continuum Analytics</a>, a relative newcomer, founded by Peter Wang and Travis Oliphant formerly from Enthought, seems to have become, almost overnight, the most popular source of Python on windows. It's baby sister, Miniconda, is less know but merely installs the conda package/environment manager and Python-2.7 which can be used to install more Python version and packages, whereas Anaconda preinstalls most sci-data packages. The only major concern for me with Anaconda is that it is closed source. Is it the python.org version built out of the box?WinPython on the other hand is open source on github and offers both 32 and 64 bit versions that do not require admin rights to instal. Enthought is also closed source and PortablePython only offers an older out of date 32bit version. There is also PythonXY but for me it seemed buggy.</p>
<p>Not sure what the future of Python on Windows will look like. If you are interested in shaping that future, I suggest you contact one of the Python devs and let them know what your use case is.</p>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-31549931362946026992015-03-16T17:11:00.001-07:002015-04-24T21:15:52.738-07:00Upgrade to Visual Studio 2013 Express<style>
/* Define look of the table
* use class "aqua_border" so other tables in the template don't get styled unintentionally */
table.aqua_border {
border-collapse: collapse;
}
table.aqua_border tr, table.aqua_border tr th, table.aqua_border tr td {
padding: 3px;
border-width: 1px;
border-style: solid;
text-align: left;
}
table.aqua_border tr th {
background: aqua;
}
</style>
<div>Welcome to the future. It's nice of you to join us. Have you been limping along with very old outdated C/C++/C# toolsets? Still using Visual Studio 2010? 2008? Afraid to uninstall them for fear you will lose the ability to compile your projects. Let's take care of that right now, don't worry about a thing. In about 1-2 hours, you will be happily in the future, enjoying the modern conveniences of Visual Studio 2013. It's very nice here. Won't you join us?</div>
<ol>
<li>Remove Visual Studio 2010 SP1 and run the uninstall utility</li>
<table class=aqua_border>
<tr><th>Name</th><th>Size</th><th>Version</th></tr>
<tr><td>Microsoft Visual Studio 2010 Service Pack 1</td><td>75.9 MB</td><td>10.0.40219</td></tr>
<tr><td>Microsoft Visual C++ 2010 Express - ENU</td><td></td><td>10.0.40219</td></tr>
<tr><td>Microsoft Visual C# 2010 Express - ENU</td><td></td><td>10.0.40219</td></tr>
<tr><td>Microsoft Visual Studio 2010 Express Prerequisites x64 - ENU</td><td>21.6 MB</td><td>10.0.40219</td></tr>
</table>
<p>You can follow the instructions I posted in the 3/13/2015 update to <a href="http://poquitopicante.blogspot.com/2012/12/download-sites-for-old-free-ms.html">Download sites for old, free MS compilers and IDEs</a> but the system restore point has dubious value. In fact it didn't help at all. When I was in trouble I found myself reinstalling the application then removing it again in the correct order. <strong>The key here is to uninstall SP1 first, then use <a href="http://blogs.msdn.com/b/heaths/archive/2010/08/23/visual-studio-2010-uninstall-utility.aspx">Stewart Heath's VS2010 uninstall utility</a> in default mode.</strong> If you find yourself in trouble, reinstall VS2010 and VS2010-SP1. If you need installers I have them here in my <a href="https://www.dropbox.com/sh/tpw763qiqokn837/AAAdBeiTh93ZYqqhG1BkriY3a?dl=0">dropbox</a>.</p>
<li>Remove Visual Studio 2008 SP1.</li>
<table class=aqua_border>
<tr><th>Name</th><th>Size</th><th>Version</th></tr>
<tr><td>Microsoft Visual C++ 2008 Express Edition with SP1 - ENU</td><td></td><td></td></tr>
</table>
<p>Same here, make sure you have an installer. The web installer in my <a href="https://www.dropbox.com/sh/bfj158f5ewf657g/AACX3TbmjciB5WpfjxrBhGN7a?dl=0">dropbox</a> still works surprisingly. Any trouble, reinstall and uninstall again.</p>
<li>Remove both SDKs for Windows 7 and .NET Frameworks 3.5 and 4.0</li>
<table class=aqua_border>
<tr><th>Name</th><th>Size</th><th>Version</th></tr>
<tr><td>Microsoft SDK for Windows 7 (7.1)</td><td></td><td>7.0.7600.16385.40715</td></tr>
<tr><td>Microsoft SDK for Windows 7 (7.0)</td><td></td><td>7.1.7600.0.30514</td></tr>
</table>
<p>If you have any issues with this, look at the last entry in the log. There's a <span style=" border-radius: 5px; border-width: 1px; border-style: solid; padding: 3px; background: gray; color: white">View Log</span> button when setup fails next to the <span style=" border-radius: 5px; border-width: 1px; border-style: solid; padding: 3px; background: gray; color: white">Finish</span> button. A part of the SDK that the installer is looking for may be missing. Search for the keyword "fail" and "unknown source". If you find an unknown source in the log file, download and extract the ISO from the <a href="https://msdn.microsoft.com/en-us/windows/desktop/ff851942.aspx">SDK archives page</a> and run the installer for the missing component. Any archive client will work. I use <a href="http://www.7-zip.org/">7-zip 9-22beta</a>. For SDK 7.0 I had to reinstall Intellidocs before I could completely remove the SDK. And for SDK 7.1 I had to install the Windows Performance Toolkit to remove the SDK completely. <b>Only the ISO will work here</b>, not the redistributables.<p>
<p>Also beware of the <a href="https://support.microsoft.com/en-us/mats/program_install_and_uninstall">Microsoft Install/Uninstall Fixit</a> it doesn't actually do anything but clean your registry. It removed both SDKs but then wouldn't let me reinstall them, all of the files were still in <code>C:\Program Files\Microsoft SDKs\Windows\v7.x</code> all that was different was that they were not in the add/remove programs control panel.</p>
<li>Remove everything else</li>
<table class=aqua_border>
<tr><th>Name</th><th>Size</th><th>Version</th></tr>
<tr><td>Microsoft Document Explorer 2008</td><td></td><td></td></tr>
<tr><td>Microsoft Help Viewer 1.1</td><td>3.97 MB</td><td>1.1.40219</td></tr>
<tr><td>Microsoft SQL Server 2008 R2 Management Objects</td><td>12.4 MB</td><td>10.50.1750.9</td></tr>
<tr><td>Microsoft SQL Server Compact 3.5 SP2 ENU</td><td>3.39 MB</td><td>3.5.8080.0</td></tr>
<tr><td>Microsoft SQL Server Compact 3.5 SP2 x64 ENU</td><td>4.50 MB</td><td>3.5.8080.0</td></tr>
<tr><td>Microsoft SQL Server System CLR Types</td><td>930 KB</td><td>10.50.1750.9</td></tr>
<tr><td>Application Verifier (x64)</td><td>55.3 MB</td><td>4.1.1078</td></tr>
<tr><td>Debugging Tools for Windows (x64)</td><td>39.8 MB</td><td>6.12.2.633</td></tr>
<tr><td>Microsoft Visual Studio 2008 Remote Debugger light (x64) - ENU</td><td></td><td></td></tr>
<tr><td>Microsoft Visual Studio 2010 ADO.NET Entity Framework Tools</td><td>34.2 MB</td><td>10.0.40219</td></tr>
<tr><td>Microsoft Visual Studio 2010 Tools for Office Runtime (x64)</td><td></td><td>10.0.50903</td></tr>
<tr><td>Microsoft Windows Performance Toolkit</td><td>26.1 MB</td><td>4.8.0</td></tr>
<tr><td>Microsoft Windows SDK for Visual Studio 2008 Headers and Libraries</td><td>114 MB</td><td>6.1.5288.17011</td></tr>
<tr><td>Microsoft Windows SDK for Visual Studio 2008 SP1 Express<br />Tools for .NET Framework - enu</td><td>4.41 MB</td><td>3.5.30729</td></tr>
<tr><td>Microsoft Windows SDK for Visual Studio 2008 SP1 Express<br />Tools for Win32</td><td>2.61 MB</td><td>6.1.5295.17011</td></tr>
</table>
<p>As you can see there is a lot of detritus left behind.</p>
<li>Remove the 2008 & 2010 C++ compilers and the Visual C++ 2010 SP1 redistributables.</li>
<table class=aqua_border>
<tr><th>Name</th><th>Size</th><th>Version</th></tr>
<tr><td>Microsoft Visual C++ Compilers 2008 Standard Edition - enu - x64</td><td>127 MB</td><td>9.0.30729</td></tr>
<tr><td>Microsoft Visual C++ Compilers 2008 Standard Edition - enu - x86</td><td>321 MB</td><td>9.0.30729</td></tr>
<tr><td>Microsoft Visual C++ Compilers 2010 SP1 Standard - x64</td><td>206 MB</td><td>10.0.40219</td></tr>
<tr><td>Microsoft Visual C++ Compilers 2010 SP1 Standard - x86</td><td>613 MB</td><td>10.0.40219</td></tr>
<tr><td>Microsoft Visual C++ 2010 x64 Redistributable - 10.0.40219</td><td>6.86 MB</td><td>10.0.40219</td></tr>
<tr><td>Microsoft Visual C++ 2010 x86 Redistributable - 10.0.40219</td><td>5.44 MB</td><td>10.0.40219</td></tr>
</table>
<p>These will be reinstalled later. You can not install Windows SDK for Windows 7 (7.1) with NET 4.0 Framework if you already have the Visual C++ 2010 SP1 redistributable installed or you will get <a href="http://support.microsoft.com/en-us/kb/2717426">the dreaded error 5100</a>.</p>
<li>Reinstall both SDKs</li>
<p>Use the web installers linked from the <a href="https://msdn.microsoft.com/en-us/windows/desktop/ff851942.aspx">SDK archives page</a>.</p>
<li>Install <a href="http://www.microsoft.com/en-us/download/details.aspx?id=44266">C++ compiler for Python 2.7</a> and patch <code>vcvarsall.bat</code></li>
<p>The standalone Python compiler will install the VS2008 (VC90) compilers and headers as well as the <code>vcvarsall.bat</code> batch file that sets environment variables necessary to build Python packages on the fly using <a href="https://pip.pypa.io/en/latest/">pip</a> and <a href="https://pythonhosted.org/setuptools/">setuptools>=6.0</a>. <b>However to build packages using distutils, <i>i.e.</i>: <code>python setup.py build</code> you will need to <a href="http://poquitopicante.blogspot.com/2014/04/fix-vcvarsallbat-to-install-python-27.html">patch vcvarsall.bat</a> in your <code>C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC</code> directory. To do this copy the <code>vcvarsall.txt</code> file that the SDK created as <code>vcvarsall.bat</code>, then patch it with the Gist in my post. <i>i.e.</i>: <code>patch vcvarsall.bat vcvarsall.bat.patch</code> in bash after downloading and extracting the Gist.</b></p>
<li>Reinstall the <a href="https://www.microsoft.com/en-za/download/details.aspx?id=13523">Visual C++ 2010 redistributables for x64</a> and <a href="https://www.microsoft.com/en-za/download/details.aspx?id=5555">x86</a></li>
<li>Install the <a href="https://www.microsoft.com/en-us/download/confirmation.aspx?id=4422">Microsoft Visual C++ 2010 SP1 compiler update for Windows SDK 7.1</a></li>
<li>Install <a href="http://www.microsoft.com/en-us/download/details.aspx?id=44914">VS2013 Express with Update 4</a> or the free <a href="https://www.visualstudio.com/en-us/products/visual-studio-community-vs.aspx">VS2013 Community</a> edition.</li>
<p>The Express edition only has VB, C# and C/C++ compilers, and does not allow <a href="https://visualstudiogallery.msdn.microsoft.com/">extensions</a>, whereas the community edition has everything, but is restricted for commercial use in large corporations.</p>
</ol>
Voila!bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-4377501847867711322015-03-12T15:12:00.002-07:002020-06-15T16:29:27.264-07:00sqlite in MATLAB<!-- begin js -->
<link href='https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href="https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shThemeFadeToGrey.css" rel="stylesheet" type="text/css" />
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shCore.js" type="text/javascript"></script>
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shAutoloader.js" type="text/javascript"></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPython.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushXml.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushBash.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushJava.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPowerShell.js' type='text/javascript'></script>
<script language="javascript">
/**
* Author: Will Schleter
* based on: http://www.jamesrohal.com
*/
SyntaxHighlighter.brushes.MatlabKey = function()
{
var keywords = 'break case catch classdef continue else elseif end for function global if otherwise parfor persistent return spmd switch try while';
var functions = ' ';
this.regexList = [
{ regex: /%.*$/gm, css: 'comments' }, // one line comments
{ regex: /\%\{[\s\S]*?\%\}/gm, css: 'comments'}, // multiline comments
{ regex: SyntaxHighlighter.regexLib.singleQuotedString, css: 'string' },
{ regex: SyntaxHighlighter.regexLib.doubleQuotedString, css: 'string'},
{ regex: new RegExp(this.getKeywords(keywords), 'gm'), css: 'keyword' }
];
};
SyntaxHighlighter.brushes.MatlabKey.prototype = new SyntaxHighlighter.Highlighter();
SyntaxHighlighter.brushes.MatlabKey.aliases = ['matlabkey'];
</script>
<script language="javascript">
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all();
</script>
<!-- end js -->
It turns out that MATLAB has <a href="https://www.sqlite.org/docs.html" target="_blank">sqlite</a> builtin
<ul>
<li><a href="https://code.google.com/p/sqlite4java/">https://code.google.com/p/sqlite4java/</a></li>
<li><a href="http://almworks.com/sqlite4java/javadoc/index.html">http://almworks.com/sqlite4java/javadoc/index.html</a></li>
<li><a href="https://www.sqlite.org/c3ref/intro.html ">https://www.sqlite.org/c3ref/intro.html </a></li>
</ul>
<pre class="brush:matlabkey">
% get or create a new database
db = com.almworks.sqlite4java.SQLiteConnection(java.io.File('sample.db'))
db.open % open database
% create a table called “person” with 2 columns, name and id
db.exec('create table person (id integer, name string)')
% add rows to “person” table
db.exec('insert into person values(1, "leo")')
db.exec('insert into person values(2, "yui")')
db.dispose % dispose of db handle
% optionally close and reopen database to see it persists
db = com.almworks.sqlite4java.SQLiteConnection(java.io.File('sample.db'))
db.open
% create a prepared statement with ? wildcard
st = db.prepare('select * from person where id>?')
st.bind(1,0) % bind 1st ? wildcard to any number greater than 0
% binding the prepared statment also works for strings
% st = db.prepare('select * from person where name>=?')
% st.bind(1,'') % bind 1st ? wildcard
% note: all string are greater than or equal to ''
% step through matching rows
while st.step
% returning the data type from the desired column
st.columnInt(0) % get IDs from column 0
st.columnString(1) % get name from column 1
end
% disposed of used up statement container
st.dispose
st.isDisposed
% ditto for db connection
db.dispose
db.isDisposed
% output
ans = 1
ans = leo
ans = 2
ans = yui
</pre>
Although IMO xerial’s jdbc driver (with sqlite included) is much easier
<ul>
<li><a href="https://bitbucket.org/xerial/sqlite-jdbc">https://bitbucket.org/xerial/sqlite-jdbc</a></li>
<li><a href="http://docs.oracle.com/javase/tutorial/jdbc/index.html">http://docs.oracle.com/javase/tutorial/jdbc/index.html</a></li>
<li><a href="http://docs.oracle.com/javase/7/docs/technotes/guides/jdbc/">http://docs.oracle.com/javase/7/docs/technotes/guides/jdbc/</a></li>
</ul>
<pre class="brush:matlabkey">
% https://bitbucket.org/xerial/sqlite-jdbc/wiki/Usage
javaaddpath('C:\Users\mmikofski\Documents\MATLAB\sqlite\sqlite-jdbc-3.8.7.jar')
d = org.sqlite.JDBC
p = java.util.Properties()
c = d.createConnection('jdbc:sqlite:sample.db',p) % named file
% optional connections
% c = d.createConnection('jdbc:sqlite:C:/full/path/to/sample.db',p) % full path
% c = d.createConnection('jdbc:sqlite::memory:',p) % memory db
% c = d.createConnection('jdbc:sqlite:',p) % default
s = c.createStatement() % create a statement
% create a table, insert rows, etc.
% s.executeUpdate('create table person (id integer, name string)');
% s.executeUpdate('insert into person values(1, "leo")');
% s.executeUpdate('insert into person values(2, "yui")');
% execute query, get id and name
rs = s.executeQuery('select * from person')
while rs.next
rs.getString('id')
rs.getString('name')
end
c.close % close connection
c.isClosed
% output
ans = 1
ans = leo
ans = 2
ans = yui
</pre>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-57306008446264672042015-03-04T02:03:00.000-08:002015-03-04T02:26:34.893-08:00Bootstrap & Syntax Highlighting<p><a href="http://getbootstrap.com/getting-started/#" target="_blank">Bootstrap</a> is probably the hottest web framework out there right now, but try using it with the extremely popular <a target="_blank" href="http://alexgorbatchev.com/SyntaxHighlighter/#">SyntaxHightlighter by Alex Gorbatchev</a>, which I wrote about in <a href="http://poquitopicante.blogspot.com/2013/06/syntax-sensation.html">syntax sensation</a>. There are at least two issues:
<ol><li>Y scrollbars appear for no reason and</li><li>there is a conflict between Bootstrap's and SyntaxHighlighter's <code>.container</code> class.</li></ol></p>
<p>Enter our latest contenders:<ul><li><a href="http://prismjs.com/index.html" target="_blank">Prism</a> and</li><li><a target="_blank" href="https://highlightjs.org/">highlight.js</a>.</li></ul>These are both extremely light and fast but offer somewhat more quality than <a target="_blank" href="https://code.google.com/p/google-code-prettify/">google pretify</a> which I also mentioned in <a href="http://poquitopicante.blogspot.com/2013/06/syntax-sensation.html">syntax sensation</a>. One thing I will say about both of these is that looking at the resulting DOM, highlight.js prepends <code>hljs-</code> to all of its classes, almost like a namespace, so that it's unlikely that there will ever be conflicts with any other plugin. highlight.js has many languages and styles while Prism has many extra plugins like line-numbers that you can add to your build from their download page. Finally both of these new syntax highlighter's conform to the <code><pre><code class="language-blah"></code></pre></code> style that evidently is the standard for putting code into HTML documents. Who knew? SyntaxHighlighter only uses <code><pre class="brush: blah"></pre></code> which is non-standard, I guess. Nit-pick much?</p>
<p>Check out for yourself how Bootstrap interacts with each syntax highlighter in the iframe below, or click the link to open in a new tab. The option menu on the right side of the navbar lets you choose which syntax highlighter to see. The template is <a target="_blank" href="http://getbootstrap.com/examples/theme/">Bootstrap's theme example</a> which you can return to by clicking the brand on the left side of the navbar. Since highlight.hs comes with 49 styles, you can peruse them from the dropdown menu. Let me know if you find anything amiss anywhere. Of course you will see the extra y-scrollbar in the SyntaxHightlighter rendition.</p>
<div><h2><a target="_blank" href="http://bwanamarko.alwaysdata.net/bootstrap/theme.html">Bootstrap & Syntax Highlighting 3-Way</a></h2>
<iframe height="635" src="http://bwanamarko.alwaysdata.net/bootstrap/theme.html" width="100%">
Bootstrap & Syntax Highlighting
</iframe></div>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-34234582751730296202015-01-21T17:20:00.005-08:002022-11-04T14:04:19.378-07:00Single Sign On from Apache in Django using Active Directory and LDAP<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/styles/shCore.css" integrity="sha512-G2vGLyD0mR1jxSAJvcuHEbSzeAZDhtem+wdYeBSr2M+wrAFo/VFZmRDsIluBnhdlq242U83cFEEvXhN3jkRv1w==" crossorigin="anonymous" referrerpolicy="no-referrer" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/styles/shThemeFadeToGrey.css" integrity="sha512-r2G85JYd+RGzDz5MMdNP1yJlvlimIa/ncBIOmO2hbCmfM1S60IuHV/4B3ixl+9eNZKevESmEITwvqtiyOAojtQ==" crossorigin="anonymous" referrerpolicy="no-referrer" />
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shCore.min.js" integrity="sha512-Z5dAQyvO8EyY1cHQcqYTYL8z6PDjM0URql6AioNodsSxxTJS5Fqfujr+o/4ye2dLp0he1qAVTiZABTunv6oLow==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shAutoloader.js" integrity="sha512-hsbtM1IhPwoNY2OToen11Ln2kDn0YnH+7VZluA5Abekh4kWOuNO1RKnePdNirA8JyliQx0iOAKMcvbrAuGHQNw==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushPython.js" integrity="sha512-nsiCxqo4QFYKSS09HPK0+HJK4qfHOvJDE0ZlIeB1Ls8YL+qW7BjTYKdoX48+kf2ATYLyaHFqvcfIuxXOoHoLZg==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushXml.js" integrity="sha512-TJi3OozXt3jOt1OJFV0wxldwBNL1Ng6QVdx4NuLa9Ni8pZGi0NKMzslD9jPi7Yz2V20v/wuRhPUwWfthZCZz3g==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushPlain.js" integrity="sha512-JHhokiQPA/SRIlZXstJ6OIEWlvO6z9S1dJkCKwpbqwD1Gimso0O+rlSANtpQ6pU1bZ/hgeLzMoKU5N7B+zfiqA==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushJScript.js" integrity="sha512-xck9yKEbi3rhQD7i/hpg43x8Z2htoeyATgaLGWRxWETAcVzbI7bw7Xs2qR0GxcBBnIdIR4uIn/pWI7Rivqu6dA==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushCss.js" integrity="sha512-sO0p1vBq6eTgn+1UDU267FGW0pPao1s0OahxuDFzH3wgb4eYtB4mfePQkA+Ps+VjSkKsv09ibpH5nFOge8TlmQ==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script language='javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all();
</script>
<p><i><b>UPDATE: 2015-02-25</b> Today I nearly crapped a cow. I was testing out a custom <code>ErrorDocument 401</code> directive that would redirect back to the sign in page (BTW: that's a bad idea, IE & Firefox sign on windows are modal). I clicked OK with empty username and empty password fields, and I got the dreaded <a href="http://en.wikipedia.org/wiki/List_of_HTTP_status_codes#5xx_Server_Error">Internal Sever Error HTTP/1.1 500</a> page. Then because the browser had cached the empty creds, I could not get back on the server. Clearing the cache and browser history had no effect. I actually thought I had broken Apache! Stack Exchange <a href="http://serverfault.com/questions/568434/apache-mod-authnz-ldap-500-error-when-blank-username-used">ServerFault</a> to the rescue. The fix is to set <code>AuthLDAPBindAuthoritative off</code> in <coed>httpd.conf</code>.</i></p>
So you have a nice and shiny new Django application, <a href="http://poquitopicante.blogspot.com/2014/07/django-from-development-to-production.html">you successfully transitioned from development to production</a>, and now you want to add <a href="http://en.wikipedia.org/wiki/Single_sign-on">Single Sign On (SSO)</a><a class="ref"></a> so users can use the same credentials they already use somewhere else. Sounds good, how do you do it?
<h2>TL;DR</h2>
This is surprisingly easy, although there is some new syntax to learn, and you will need to get some info from your system administrator. Here are some steps for <a href="http://www.apachelounge.com/download/">Apache-2.4 from ApacheLounge</a>.
<ol>
<li>Follow the directions in the Django documentation on <a href="https://docs.djangoproject.com/en/dev/howto/auth-remote-user/">Authentication using REMOTE_USER</a><a class="ref"></a> and add <a href="https://docs.djangoproject.com/en/dev/ref/middleware/#django.contrib.auth.middleware.RemoteUserMiddleware"><code>RemoteUserMiddleware</code></a><a class="ref"></a> and <a href="https://docs.djangoproject.com/en/dev/ref/contrib/auth/#django.contrib.auth.backends.RemoteUserBackend"><code>RemoteUserBackend</code></a><a class="ref"></a> to <a href="https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-AUTHENTICATION_BACKENDS"><code>AUTHENTICATION_BACKENDS</code></a><a class="ref"></a> to your settings file. This will use the <code>REMOTE_USER</code> environment variable set by Apache when it authorizes users and use it for authentication on the Django website.</li>
<blockquote><p><i>Note: This will change how Django works; for example, any authorized user not in the Django Users model will have their username automatically added and set to <b>active</b>, but their password and the <code>is_staff</code> attribute will <b>not</b> be set.</i></p></blockquote>
<li>Get the URL or IP address of your <a href="http://en.wikipedia.org/wiki/Active_Directory">Active Directory</a><a class="ref"></a> server from your system administrator. For <a href="http://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol">LDAP</a><a class="ref"></a> with <a href="http://en.wikipedia.org/wiki/Basic_access_authentication">basic authentication</a><a class="ref"></a>, the port is usually 389, but check to make sure.</li>
<li>Also get the "Distringuished Name" of the <a href="http://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol#LDAP_URLs">"search base"</a> from your system administrator. A "Distringuished Name" is LDAP lingo for a string made up of several components, usually the "Organizational Unit (OU)" and the "Domain Components (DC)", that distinguish entries in the Active Directory.</li>
<li>Finally ask your system administrator to set up a "binding" distinguished name and password to authorize searches of the Active Directory.</li>
<li>Then in <a href="http://httpd.apache.org/docs/2.4/configuring.html"><code>httpd.conf</code></a><a class="ref"></a> enable <a href="http://httpd.apache.org/docs/2.4/mod/mod_authnz_ldap.html"><code>mod_authnz_ldap</code></a><a class="ref"></a> and <a href="http://httpd.apache.org/docs/2.4/mod/mod_ldap.html"><code>mod_ldap</code></a><a class="ref"></a>.</li>
<li>Also in <code>httpd.conf</code> add a <a href="http://httpd.apache.org/docs/current/mod/core.html#location"><code>Location</code></a><a class="ref"></a> for the URL endpoint, <i>EG</i>: <code>/</code> for the entire website, to be password protected.</li>
<li>You must set <a href="http://httpd.apache.org/docs/2.4/mod/mod_authn_core.html#authname"><code>AuthName</code></a><a class="ref"></a>. This will be displayed to the user when they are prompted to enter their credentials.</li>
<li>Also must also set <a href="http://httpd.apache.org/docs/2.4/mod/mod_authn_core.html#authtype"><code>AuthType</code></a>, <a href="http://httpd.apache.org/docs/2.4/mod/mod_auth_basic.html#authbasicprovider"><code>AuthBasicProvider</code></a><a class="ref"></a>, <a href="http://httpd.apache.org/docs/2.4/mod/mod_authnz_ldap.html#authldapurl"><code>AuthLDAPUrl</code></a> and <a href="http://httpd.apache.org/docs/2.4/mod/mod_authz_core.html#require"><code>Require</code></a><a class="ref"></a>. Prepend <code>ldap://</code> to your AD server name and append the port, base DN, scope, attribute and search filter. The port is separated by a colon (:), the base DN by a slash (/) and the other parameters by question marks (?) such as:
<blockquote><code>ldap://host:port/basedn?attribute?scope?filter</code></blockquote></li>
<pre class="brush: text">
<Location />
AuthName "Please enter your SSO credentials."
AuthBasicProvider ldap
AuthType basic
AuthLDAPUrl "ldap://my.activedirectory.com:389/OU=Offices,DC=activedirectory,DC=com?sAMAccountName"
AuthLDAPBindDN "CN=binding_account,OU=Administrators,DC=activedirectory,DC=com"
AuthLDAPBindPassword binding_password
AuthLDAPBindAuthoritative off
LDAPReferrals off
Require valid-user
</Location>
</pre>
<li>The "attribute" to search for in Windows Active Directory is <a href="https://msdn.microsoft.com/en-us/library/ms679635.aspx">"SAM-Account-Name" or <code>sAMAccountName</code></a><a class="ref"></a>. This is the equivalent of a user name.</li>
<li>The default "scope" is <code>sub</code> which means it will search the base DN and everything below it in the Active Directory. And the default "filter" is <code>(objectClass=*)</code> which is the equivalent of no filter.</li>
<li><a href="http://httpd.apache.org/docs/2.4/mod/mod_authnz_ldap.html#requiredirectives">There are several options for limiting users and groups.</a> If you set <code>Require</code> to <code>valid-user</code> then any user in the AD who can authenticate will be authorized.</li>
<li>Set <a href="http://httpd.apache.org/docs/2.4/mod/mod_authnz_ldap.html#authldapbinddn"><code>AuthLDAPBindDN </code></a> and <a href="http://httpd.apache.org/docs/2.4/mod/mod_authnz_ldap.html#authldapbindpassword"><code>AuthLDAPBindPassword </code></a> to the binding account's DN and password.</li>
<li><a href="https://www.apachelounge.com/viewtopic.php?t=4697">It has been reported</a><a class="ref"></a> that <a href="http://httpd.apache.org/docs/2.4/mod/mod_ldap.html#ldapreferrals"><code>LDAPReferrals</code></a><a class="ref"></a> <a href="http://serverfault.com/questions/432844/authenticate-users-with-ldap-and-apache-2-4">should be set to off</a><a class="ref"></a> or <a href="http://stackoverflow.com/questions/14242734/apache-http-server-2-4-ldap-active-directory-authentication">you may get the following error</a><a class="ref"></a>.
<blockquote>
<p style="color: red">(70023)This function has not been implemented on this platform: AH01277: LDAP: Unable to add rebind cross reference entry. Out of memory?</p>
</blockquote>
<li>Finally, restart your Apache httpd server and test out your site.</li>
</ol>
Now when users go to your Django site, when they open the location that requires authentication they will see a pop up that asks for their credentials.
<div style="text-align: center"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi78xrke4mDTYij-7v_4DOQ7vA3yweYNGwQL2F3TFxW6tQGAg_3240vn8LkRBlHQuV5cA3E-qtBCLQV5J_V_914segkzOyjOIVlInTwuIU_7mLvK8MKp_u7RN5x1ksOXDu318w2NhxJGw/s1600/ApacheLDAPCrendentialsWindows.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi78xrke4mDTYij-7v_4DOQ7vA3yweYNGwQL2F3TFxW6tQGAg_3240vn8LkRBlHQuV5cA3E-qtBCLQV5J_V_914segkzOyjOIVlInTwuIU_7mLvK8MKp_u7RN5x1ksOXDu318w2NhxJGw/s320/ApacheLDAPCrendentialsWindows.png"></a></div>
<h2>Loggout</h2>
In addition to adding authenticated users to the Django Users model, the users credentials are stored in the browser. This makes logging out akward since the user will need to close their browser to logout. There are several approaches to get Django to logout a user.
<ul><li>redirect the user to a URL with <i>fake</i> basic authentication prepended to the path.</li>
<blockquote><code>http://log:out@example.com</code></blockquote>
<li>render a template with status set to 401 which is the code for unauthorized that will clear the credentials in browser cache.</li>
<pre class="brush:python">
from django.shortcuts import render
from django.contrib.auth import logout as auth_logout
import logging # import the logging library
logger = logging.getLogger(__name__) # Get an instance of a logger
def logout(request):
"""
Replaces ``django.contrib.auth.views.logout``.
"""
logger.debug('user %s logging out', request.user.username)
auth_logout(request)
return render(request, 'index.html', status=401)
</pre>
</ul>
<h2>Using Telnet to ping AD server</h2>
A lot of sites suggest this. First you will need to enable Telnet on your Windows PC. This can be done from <b>Uninstall a program</b> in the <b>Control Panel</b> by selecting <b>Turn Windows features on or off</b> and checking <b>Telnet Client</b>. Then opening a command terminal and typing <code>telnet</code> followed by <code>open my.activedirectory.com 389</code>. Surprise! If it works you will only see the output:
<blockquote>
<code>Connecting to my.activedirectory.com...</code>
</blockquote>
If it does not work then you will see this additional output:
<blockquote>
<code>Could not open connection to the host, on port 389: Connect failed</code>
</blockquote>
Now treat yourself and try <code>open towel.blinkenlights.nl</code>. Use control + ] to kill the connection, then type <code>quit</code> to quit telnet.
<h2>Testing LDAP using Python</h2>
<ul>
<li>Python-LDAP</li>
So to learn more about LDAP there are a couple of packages that you can use to interrogate and authenticate with and AD server using LDAP. <a href="http://www.python-ldap.org/doc/html/index.html">Python-LDAP</a><a class="ref"></a> seems to be common and easy to use. It's based on <a href="http://www.openldap.org/">OpenLDAP</a><a class="ref"></a> Here's a <a href="https://www.google.com/support/enterprise/static/gapps/docs/admin/en/gads/admin/ldap.5.4.html">list of common LDAP Queries</a> from Google.
<pre class="brush:python">
>>> import ldap
>>> server = ldap.initialize('ldap://my.activedirectory.com:389')
>>> server.simple_bind('CN=bind_user,OU=Administrators,DC=activedirectory,DC=com','bind_password') # returns 1 on success
1
>>> user = server.search_s('OU=Users,DC=activedirectory,DC=com',ldap.SCOPE_SUBTREE,'(&(sAMAccountName=my_username)(ObjectClass=user))',('cn','sAMAccountName','mail'))
>>> user
[('CN=My Name,OU=Super-Users,OU=USA,OU=California,OU=Sites,DC=activedirectory,DC=com',
{'cn': ['My Name'],
'sAMAccountName': ['my_username'],
'mail': ['my_username@activedirectory.com']})]
>> users = server.search_s('OU=Users,DC=activedirectory,DC=com',ldap.SCOPE_SUBTREE,'(&(memberOf=CN=@my_group,OU=Groups,OU=Users,DC=activedirectory,DC=com)(ObjectClass=user))',('cn','sAMAccountName','mail'))
[('CN=My Name,OU=Super-Users,OU=USA,OU=California,OU=Sites,DC=activedirectory,DC=com',
{'cn': ['My Name'],
'sAMAccountName': ['my_username'],
'mail': ['my_username@activedirectory.com']}),
('CN=Somebody_Else,OU=Super-Users,OU=USA,OU=California,OU=Sites,DC=activedirectory,DC=com',
{'cn': ['Their name'],
'sAMAccountName': ['their_username'],
'mail': ['their_username@activedirectory.com']})]
</pre>
<li>PyAD</li>
Another Python package that can use LDAP to search an active directory is <a href="https://zakird.com/pyad/">PyAD</a><a class="ref"></a> which uses <a href="http://sourceforge.net/projects/pywin32/">PyWin32</a><a class="ref"></a> and ADSI on Windows.
<li>PyWin32</li>
<a href="http://timgolden.me.uk/pywin32-docs/">The only decent documentation for this is Tim Golden's website.</a>
</ul>
<h2 id="alternatives">Alternatives</h2>
<ul>
<li>SSPI/NTLM</li>
If users will only use the Django application on a Windows PC which they already have been authorized, EG through windows logon, then using either <a href="http://mod-auth-sspi.sourceforge.net/"><code>mod_authnz_sspi</code></a> or <a href="http://modntlm.sourceforge.net/"><code>mod_authnz_ntlm</code></a> to acquire those credentials from your Windows session is also an option.
<ul>
<li>Apache-2.4</li>
<ul>
<li><a href="https://www.apachehaus.com/cgi-bin/download.plx">search for mod auth NTLM</a></li>
<li><a href="https://www.apachehaus.net/modules/mod_authnz_sspi/">mod_authnz_sspi</a></li>
</ul>
<li>Apache-2.2</li>
<ul>
<li><a href="http://sourceforge.net/projects/mod-auth-sspi/">mod_auth_sspi</a></li>
</ul>
</ul>
<li>Django Extensions and Snippets</li>
<p>There are several Django extensions and snippets that use Python-LDAP and override <code>ModelBackend</code> so that Django handles authorization and authentication instead of Apache.</p>
<ul>
<li><a href="https://djangosnippets.org/snippets/501/">snippet-501</a></li>
<li><a href="https://djangosnippets.org/snippets/901/">snippet-901</a></li>
<li><a href="https://pythonhosted.org/django-auth-ldap/">Django-AuthLDAP Extension</a></li>
</ul>
<p>Some Django extensions and snippets also exist to subclass <code>ModelBackends</code> to use PyWin32 to use local credentials from the current windows machine for authorization and authentication from within Django.</p>
<ul>
<li><a href="http://stackoverflow.com/a/141729/1020470">win32security snippet</a></li>
<li><a href="https://djangosnippets.org/snippets/517/">snippet-517</a></li>
</ul>
<li>SAML and OAuth</li>
Sure you could do this. You can also use SSL with LDAP or Kerebos with SSPI/NTLM. But, alas, I did not research these options althought I did come across a few references.
</ul>
<div id="references"></div>
<h2>CSS and JS</h2>
The references section loosely based on <a href="http://poquitopicante.blogspot.com/2014/10/javascript-toc-robot.html">Javascript TOC robot</a>. It could also use the <a href="https://developer.mozilla.org/en-US/docs/Web/Guide/CSS/Getting_started/Lists#Counters">counters</a> and the <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/::before"><code>::before</code></a> style <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/Pseudo-elements">pseudo-element</a>, but since I'm using JavaScript it doesn't make sense. But here's what that looked like anyway.
<pre class="brush: css">
<style>
h2.references {counter-reset: refs;}
p.references:before {
content: "[" counter(refs) "]";
counter-increment: refs;
margin-right: 5px;
font-weight: bold;
}
</style>
<h2 class="references">Example</h2>
<p class="references"><a href="#">first reference</a></p>
<p class="references"><a href="#">second reference</a></p>
</pre>
<style>
h2.references {counter-reset: refs;}
p.references:before {
content: "[" counter(refs) "]";
counter-increment: refs;
margin-right: 5px;
font-weight: bold;
}
</style>
<h2 class="references">Example</h2>
<p class="references"><a href="#">first reference</a></p>
<p class="references"><a href="#">second reference</a></p>
<p>In case it wasn't clear above the JavaScript below is <b>not</b> what I'm using on this page. It was for a different approach using <code>counters</code> which I scratched, so these examples are very contrived and don't really make sense anymore.</p>
<script src="https://gist.github.com/mikofski/19aa3c8a7775a2af158b.js"></script>
<script>
document.body.onload = gen_refs("a", "references", "ref");
function gen_refs(tagname, refsname, classname) {
var alltags = document.getElementsByTagName(tagname);
var refs = document.getElementById(refsname);
var content = '<h2>References</h2>';
var ctr = 0
for (var i = 0; i < alltags.length; i++) {
var tag_class = alltags[i].getAttribute("class");
if (tag_class == classname) {
ctr++;
console.log(alltags[i]);
content += '<a href="#src-' + ctr + '"><b>[' + ctr + '] </b></a><a id="ref-' + ctr + '" href="';
content += alltags[i].previousSibling.getAttribute("href");
content += '">' + alltags[i].previousSibling.getAttribute("href") + '</a><br/>';
alltags[i].setAttribute("href", "#ref-" + ctr);
alltags[i].setAttribute("id", "src-" + ctr);
alltags[i].innerHTML = '<b>[' + ctr + ']</b>';
console.log(alltags[i].getAttribute("href"));
}
}
console.log(content);
refs.innerHTML = content;
}
</script>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0tag:blogger.com,1999:blog-1006797308390137260.post-9251507591571258412015-01-09T10:34:00.001-08:002020-06-15T16:34:57.930-07:00Questionable Quantities in MATLAB<!-- BEGIN SYNTAX HIGHLIGHTING -->
<link href='https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href="https://agorbatchev.typepad.com/pub/sh/3_0_83/styles/shThemeFadeToGrey.css" rel="stylesheet" type="text/css" />
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shCore.js" type="text/javascript"></script>
<script src="https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shAutoloader.js" type="text/javascript"></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushPython.js' type='text/javascript'></script>
<script src='https://agorbatchev.typepad.com/pub/sh/3_0_83/scripts/shBrushXml.js' type='text/javascript'></script>
<script>
/**
* Author: Will Schleter
* based on: http://www.jamesrohal.com
*/
SyntaxHighlighter.brushes.MatlabKey = function()
{
var keywords = 'break case catch classdef continue else elseif end for function global if otherwise parfor persistent return spmd switch try while';
var functions = ' ';
this.regexList = [
{ regex: /%.*$/gm, css: 'comments' }, // one line comments
{ regex: /\%\{[\s\S]*?\%\}/gm, css: 'comments'}, // multiline comments
{ regex: SyntaxHighlighter.regexLib.singleQuotedString, css: 'string' },
{ regex: SyntaxHighlighter.regexLib.doubleQuotedString, css: 'string'},
{ regex: new RegExp(this.getKeywords(keywords), 'gm'), css: 'keyword' }
];
};
SyntaxHighlighter.brushes.MatlabKey.prototype = new SyntaxHighlighter.Highlighter();
SyntaxHighlighter.brushes.MatlabKey.aliases = ['matlabkey'];
</script>
<script language='javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all();
</script>
<!-- END SYNTAX HIGHLIGHTING -->
I am proud to introduce <a href="http://www.mathworks.com/matlabcentral/fileexchange/48988-mikofski-quantities">Quantities for MATLAB</a>. Quantities is an units and uncertainties package for MATLAB. It is inspired by <a href="http://pint.readthedocs.org/">Pint</a>, a Python package for quantities.
<h1>Installation</h1>
<a href="https://github.com/mikofski/Quantities">Clone</a> or <a href="https://github.com/mikofski/Quantities/archive/master.zip">download</a> the Quantities package to your MATLAB folder as <code>+Quantities</code>.
<h1>Usage</h1>
<ol>
<li> Construct a units registry, which contains all units, constants, prefixes and dimensions.</li>
<pre class="brush: matlabkey highlight: [1]">
>> ureg = Quantities.unitRegistry
ureg =
Map with properties:
Count: 279
KeyType: char
ValueType: any
</pre>
<li> Optionally pass <code>verbosity</code> parameter to <code>unitRegistry</code> to see list of units loaded.</li>
<pre class="brush: matlabkey gutter: false highlight: [1]">
>> ureg = Quantities.unitRegistry('v',2)
</pre>
<li> Units and constants can be indexed from the <code>unitRegsitry</code> using their name or alias in parentheses or as dot-notation. The <code>unit</code>, <code>constant</code> and <code>quantity</code> class all subclass to <code>double</code> so you can perform any operation on them. Combining a <code>double</code> with a <code>unit</code> creates a <code>quantity</code> class object.</li>
<pre class="brush: matlabkey highlight: [1, 5, 9]">
>> T1 = 45*ureg('celsius') % index units using parentheses or dot notation
T1 =
45 ± 0 [degC];
>> T2 = 123.3*ureg.degC % index units by name or by alias
T2 =
123.3 ± 0 [degC];
>> heat_loss = ureg.stefan_boltzmann_constant*(T1^4 - T2^4)
heat_loss =
-819814 ± 0 [gram*second^-3];
</pre>
<li> Perform operations. All units are converted to base.</li>
<pre class="brush: matlabkey highlight: [1, 5]">
>> T2.to_base
ans =
396.45 ± 0 [kelvin];
>> heat_loss = ureg.stefan_boltzmann_constant*(T1.to_base^4 - T2.to_base^4)
heat_loss =
-819814 ± 0 [gram*second^-3];
</pre>
<li> Add uncertainty to quantities by calling constructor. <a href="http://en.wikipedia.org/wiki/Propagation_of_uncertainty">Uncertainty is propagated using 1st order linear combinations</a>.</li>
<pre class="brush: matlabkey highlight: [1, 5]">
>> T3 = Quantities.quantity(56.2, 1.23, ureg.degC)
T3 =
56.2 ± 1.23 [degC];
>> heat_loss = ureg.stefan_boltzmann_constant*(T1^4 - T3^4)
heat_loss =
-86228.1 ± 9966.66 [gram*second^-3];
</pre>
<li> Convert output to different units.</li>
<pre class="brush: matlabkey highlight: [1]">
>> heat_loss_kg = heat_loss.convert(ureg.kg/ureg.s^3)
heat_loss_kg =
-819.814 ± 0 [kilogram*second^-3];
</pre>
<li> Determine arbitrary conversion factor.</li>
<pre class="brush: matlabkey highlight: [1]">
>> conversion_factor = ureg.mile.convert(ureg.km)
conversion_factor =
1.60934 ± 0 [kilometer];
</pre>
</ol>
<a href="http://www.undermyhat.org/blog/wp-content/uploads/2009/09/shBrushMatlabSimple.js">MATLAB Syntax Highlighter brush by Will Schleter</a>bwanamarkohttp://www.blogger.com/profile/10855014295815183523noreply@blogger.com0