Bypassing Box Upload Limits by API

Box for large files

Box offers 10gb of online storage for free, double what anyone else offers, with an individual max size of 2gb, but you can only upload 250mb files. So how do you upload that 2gb file? The Box API that's how, either with regular ol' requests or their fancy smancy sdk. First follow the Getting Started instructions, sign up for a developer account and create a temporary key. Then in Python, try this out:

# import the requests package
import requests

# copy your token here
TOKEN = "<your developer token>"

# try to get the top level folder, id: "0", using this command exactly as below:
r = requests.get(url='https://api.box.com/2.0/folders/0',
                 headers={'Authorization': 'Bearer %s' % TOKEN})

# check the response
r
#  <Response [200]>
# success!

# get the output
r.json()
# lots of stuff

# upload a file, using the commands exactly as below, except put the actual id number
# of the desired folder
FILES = {'file': open('path/to/myfile','rb')}
PAYLOAD = {'attributes': '{"name":"myfile", "parent":{"id":"<id # of desired folder>"}}'}
r = requests.post(url='https://upload.box.com/api/2.0/files/content',
                  headers={'Authorization': 'Bearer %s' % TOKEN},
                  files=FILES,
                  data=PAYLOAD)

# check the response
r
#  <Response [201]>
# success!

References

Check the online Content API reference for full documentation.

Panda Pop

Pandas Offset Aliases

Memorize this table - or just bookmark this link: Pandas Offset Aliases

Offset Aliases¶

A number of string aliases are given to useful common time series frequencies. We will refer to these aliases as offset aliases (referred to as time rules prior to v0.8.0).

Alias	Description
B	business day frequency
C	custom business day frequency (experimental)
D	calendar day frequency
W	weekly frequency
M	month end frequency
SM	semi-month end frequency (15th and end of month)
BM	business month end frequency
CBM	custom business month end frequency
MS	month start frequency
SMS	semi-month start frequency (1st and 15th)
BMS	business month start frequency
CBMS	custom business month start frequency
Q	quarter end frequency
BQ	business quarter endfrequency
QS	quarter start frequency
BQS	business quarter start frequency
A	year end frequency
BA	business year end frequency
AS	year start frequency
BAS	business year start frequency
BH	business hour frequency
H	hourly frequency
T, min	minutely frequency
S	secondly frequency
L, ms	milliseconds
U, us	microseconds
N	nanoseconds

robotic releases

Basic Auto-Versioning from Git

If you're using the winning workflow and the recommended Python project layout then you've set up a CI server to build releases when you tag them in Git, and you set your version in the __init__.py file of your package. But, "Oh, No!" you did it again. You created the Git tag, but forgot to update your code's __version__ string.

Okay, there is a Python package called Versioneer that handles this for you, and it's pretty awesome. But it turns out it's also pretty easy to roll your own, especially if you're just using Git, because Python has a Git implementation called Dulwich that can do this in just a few lines. Maybe it will get integrated into a future version of Dulwich - I've submitted a PR (#462) which was merged into v0.16.3 and an update (#489) which was also merged into v0.17 to also list tags that are not objects. Anyway, for now, the easiest way to use this is to copy this file into your package at the top level, Install the latest version of dulwich (>=0.17.1), import it and then add something like this to your package dunder init module so it works both in your repo during dev and then later when deployed to users.

"""
Example package dunder init module implementing
``dulwich.contrib.release_robot`` to get current version.
"""

import os
import importlib

# try to import Dulwich or create dummies
try:
    from dulwich.contrib.release_robot import get_current_version
    from dulwich.repo import NotGitRepository
except ImportError:
    NotGitRepository = NotImplementedError

    def get_current_version():
        raise NotGitRepository

BASEDIR = os.path.dirname(__file__)  # this directory
VER_FILE = 'version'  # name of file to store version
# use release robot to try to get current Git tag
try:
    GIT_TAG = get_current_version()
except NotGitRepository:
    GIT_TAG = None
# check version file
try:
    version = importlib.import_module('%s.%s' % (__name__, VER_FILE))
except ImportError:
    VERSION = None
else:
    VERSION = version.VERSION
# update version file if it differs from Git tag
if GIT_TAG is not None and VERSION != GIT_TAG:
    with open(os.path.join(BASEDIR, VER_FILE + '.py'), 'w') as vf:
        vf.write('VERSION = "%s"\n' % GIT_TAG)
else:
    GIT_TAG = VERSION  # if Git tag is none use version file
VERSION = GIT_TAG  # version

__author__ = u'your name'
__email__ = u'your.email@your.company.com'
__url__ = u'https://github.com/your-org/your-project'
__version__ = VERSION
__release__ = u'your release name'

Or you can also use it to get all recent tags.

get_recent_tags()[0][0]

assuming your tags all use semantic versions like "v0.3". Enjoy!

Carousel Cotton Candy

Version 0.3

I'm super excited to announce the Cotton Candy release of Carousel, version 0.3 on PyPI and GitHub. There were a few more issues I really wanted to close with this release, but I decided to push it forward anyway. So the remaining milestones for v0.3 will get pushed to v0.3.1.

• issue #62 use Meta class for all layers - currently usage is spotty and inconsistent. I wanted to keep the number of commits to close PR #68 to a minimum (following the winning workflow) so I only implemented Meta classes where I had to. In fact it's not even implemented in the DataSource example below.
• issue #25 move all folders into project package - this is already how I have set up the PVPower demo. It just makes more sense with new style models to have them all in the same package.
• issue #63 and issue #22 split calculations into separate parameters - I knew this couldn't be done by v0.3, it was a stretch goal, but I'm super excited about this. By moving dependencies to an attribute of each parameter, the DAG shows which calculations are orthogonal, so we can run them simultaneously. According to issue #22, I had this idea already, but when I saw this presentation at PyData SF 2016 on Airflow by Matt Davis I was even more motivated to make it happen.
• issue #59 and issue #73 which don't seem like they're relevant, but they both have to do with implementing a Calculator class whose job it is to crunch through the calculations, somewhat similar to what DataReaders and FormulaImporter do for their layers. Then the boiler plate uncertainty propagation code in the static class could be applied to any calculator such as a dynamic calculator, a linear system solver for an acyclic DAG of linear equations or a non-linear system solver for an acyclic DAG of non-linear equations.

The Parameter class

What's new in Carousel-0.3 (Cotton Candy)? The biggest difference is the introduction of the Parameter class which is now used to specify parameters for data, formulas, outputs, calculations and simulation settings. For example, previously data parameters would be entered as a dictionary of attributes.

Bicycle Bears

class PVPowerData(DataSource):
    """
    Data sources for PV Power demo.
    """
    data_reader = ArgumentReader
    latitude = {"units": "degrees", "uncertainty": 1.0}
    longitude = {"units": "degrees", "uncertainty": 1.0}
    elevation = {"units": "meters", "uncertainty": 1.0}

Cotton Candy

class PVPowerData(DataSource):
    """
    Data sources for PV Power demo.
    """
    latitude = DataParameter(units="degrees", uncertainty=1.0)
    longitude = DataParameter(units="degrees", uncertainty=1.0)
    elevation = DataParameter(units="meters", uncertainty=1.0)

    class Meta:
        data_reader = ArgumentReader

Why the change? The Bicycle Bear version did not have any way to distinguish parameters, like latitude from attributes of the DataSource like data_reader. This had two unfortunate side-effects:

• each layer attribute had to be hardcoded in the base metaclass so that they wouldn't be misinterpreted as parameters
• and users could not define any custom class attributes, because they would be misinterpreted as parameters and stripped from the class by the metaclass.

The Cotton Candy version makes it easy for the metaclass to determine which class attributes are parameters, which are attributes of the layer and then leaves everything else alone. Every layer now has a corresponding Parameter subclass which also defines some base attributes corresponding to that layer. Any extra parameter attributes are saved in extras. Attributes that apply to the entire layer are now specified in the `Meta` class attribute, similar to Django, Marshmallow and DRF. The similarities are completely intentional as I have been strongly inspired by those project code bases. Unfortunately, the Meta class is only partially implemented, but will be the major focus of v0.3.1.

Separated Formulas

The Formula class is also improved. Now each formulas is a Parameter with attributes, rather than a giant dictionary. This improvement is still on the roadmap for the Calculation class. As I said above, it was a stretch goal for this release.

Winning Workflow

Intro

There are many blog posts on the topic of effective Git workflows, SO questions and answers, BitBucket tutorials and GitHub guides and an article that has been archived by former BBC Mark McDonnell. So why another post on git workflow? None of these workflows seemed right for us, but recently it's just clicked, and I feel like we've finally found the process that works for us. The key was finding the simplest workflow that included the most valuable best practices. In particular, we found that complicated multi-branch strategies were unnecessary, but test driven development (TDD) and continuous integration (CI) were a must.

Winning Workflow

Setting up Remotes

We start with the assumption that all of collaborators fork the upstream repository to their personal profile. Then each person clones their profile to their laptop as origin and adds another remote pointing to the upstream repository. For convenience, they may also create remotes to the forks of their most frequent collaborators.

[myusername@mycomputer ~/Projects]
$ git clone git@github.com:myusername/myrepo.git
[myusername@mycomputer ~/Projects]
$ cd myrepo
[myusername@mycomputer ~/Projects/myrepo]
$ git remote add upstream git@github.com:mycompany/myrepo.git
[myusername@mycomputer ~/Projects/myrepo]
$ git remote add mycollaborator git@github.com:mycollaborator/myrepo.git
[myusername@mycomputer ~/Projects/myrepo]
$ git remote show
  origin
  upstream
  mycollaborator

Ground Rules

The next assumption is that we all keep our version of master synchronized with upstream master. And we never work out of our own master branch! Basically this means at the start of any new work we do the following:

1. I like to do git fetch --all to get the lay of the land. This combined with
   git log --all --graph --date=short --pretty=format:"%ad %h %s%d [%an]"
   let's me know what everyone is working on, assuming that I've made remotes to their forks.
2. Then I pull from upstream master to get the latest nightly or release,
3. and push to origin master to keep my fork current.

Recommended Project Layout

I'm also going to assume that everyone is following the recommended project layout. This means that their project has all dependencies listed in requirements.txt, is developed and deployed in its own virtual environment, includes testing and documentation that aims for >80% coverage, has a boilerplate design that allows testing, documentation and package data to be bundled into a distribution and enables use with a test runner with self discovery, and is written with docstrings for autodocumentation. Nothing is ever perfect, so being diligent of path clashes, aware of the arcana of Mac OS X¹ or Windows² and able to use Stack Overflow to find answers is still important.

Branching, Testing, Pull Requests and Collaboration

1. Now I switch to a new feature branch with a meaningful name - I'll delete this branch everywhere later so it can be verbose.
2. The very first code I write is a test or two that demonstrates more or less exactly what we want the feature or bug fix to do. This is one of the most valuable steps because it clearly defines the acceptance criteria. Although it's also important to be thoughtful and flexible - just because your tests pass doesn't necessarily mean the feature is implemented as intended. Some new tests or adjustments may be needed along the way.
3. Now, before I write any more code, is when I submit a pull request (PR) from my fork's feature branch to upstream/master. So many people are surprised by this. Many collaborators have told me they thought that PR's should be submitted after their work is complete and passing all tests. But in my opinion that defeats the entire point of collaborating on a short iteration cycle.
   • If you wait until the end to submit your work you risk diverging from the feature's intended goals especially if the feature's requirements shift or you've misinterpreted the goals even slightly.
   • Waiting also means you're missing out on collaborating with your teammates and soliciting their feedback mid-project.
   On the other hand, by submitting your PR right after you write your tests means:
   • Every push to your fork will trigger a build that runs your tests.
   • Your teammates will get continuous updates so they can monitor your progress in real-time but also on their time so you won't have to hold a formal review, since collaborators can review your work anytime as the commits will all be queued in the PR.
   I think the reason people wait until the end to submit PR's is the same reason they like to write tests at the end. I used to hate seeing my tests fail because it made me feel like I was failing. I think people delay submitting their PR's because they're nervous about having incomplete work reviewed out of context and receiving unfair criticism or harsh judgment. IMO, punitive behavior is dysfunctional and a collaboration killer and should be rooted out with a frank discussion about what mutual success looks like. I also think some people aren't natural collaborators and don't want other's interfering with their work. Again, a constructive discussion can help promote new habits, although don't expect people to change overnight. You can take a hard stance on punitive behavior but you can't expect an introvert to feel comfortable sharing themselves freely without some accommodations.
4. Now comes the really fun part. We hack and collaborate until the tests all pass. But we don't have too much fun - there should be at most 10 commits before we realize we've embarked on an epic that needs to be re-organized, otherwise the PR will become difficult to merge. That will sap moral and waste time. So keep it simple.
5. The final bit of collaboration is the code review and merging the PR into upstream master. This is fairly easy, since there are
   • already tests that demonstrate what the code should do,
   • only a few commits,
   • and all of the collaborators have been following the commits as they've been queuing in the PR.
   So really the review and merge is a sanity check. Do these tests really demonstrate the feature as intended? Anything else major would have stood out already.
6. Whoever the repository owner or maintainer is should add the tag and push it to upstream. This triggers the CI to test, build and deploy a new release.

Continuous Integration

This is key. Set up Travis, Circle, AppVeyor or Jenkins on upstream master to test and build every commit, every commit to an open PR and to deploy on every tag. Easy!

Wrapping Up

There are some features of this style that stand out:

• There is only one master branch. Using CI to deploy only on tags eliminates our need for a dev or staging branch because any commits on master not tagged are the equivalent of the bleeding edge.
• This method depends heavily on an online hosted Git repo like GitHub or BitBucket, use of TDD, strong collaboration and a CI server like Travis.

Happy Coding!

---

footnotes

1. On Mac OS X matplotlib will not work in a virtual environment unless a framework interpreter is used. The easiest way to do this is to run python as PYTHONHOME=/home/you/path/to/project/venv/ python instead of using source venv/bin/activate.
2. On Windows pip often creates an executable for scripts that is bound to the Python interpreter it was installed with. If the virtual environments was created with system site packages or if the package is not installed in the virtual environment then you may get a confusing path clash. For example running the nosetests script will use your system Python and therefore the Python path will not include your virtual environment. The solution is to never use system site packages and install all dependencies directly in your virtual environment.

Derived Django Database Field

The trick to this is creating a custom field and overloading pre_save. Pay special attention to the self.attname member that is set to the value. The source for DateField is a good example. Make sure that if you add any new attributes to the field in it's __init__ method you also add a corresponding deconstruct method.

Mocking Django App

I'm sure this is completely wrong. I needed a Django model for testing, but I don't have a Django app or even a Django project. I'm developing a Django model reader for Carousel, and so I needed a model to test it out with. Sure I could have created a quick django project, but that seemed silly, and my first instinct was to import django.db.models, make a model and use it, but this raised:

ImproperlyConfigured: Requested setting DEFAULT_INDEX_TABLESPACE, but settings are not
                      configured. You must either define the environment variable
                      DJANGO_SETTINGS_MODULE or call settings.configure() before accessing
                      settings.

Most normal people would turn back now, but instead I imported django.conf.settings and called settings.configure() just like it said to do. Now I got this error:

AppRegistryNotReady: Apps aren't loaded yet.

So now I felt like I was getting somewhere. But where? Googling told me to import django and run setup which I did and that raised:

RuntimeError: Model class __main__.MyModel doesn't declare an explicit app_label and isn't
              in an application in INSTALLED_APPS.

Wow! Normally RuntimeError is a scary warning, like you dumped your core, but this just said I needed to add the app to settings.INSTALLED_APPS, which makes perfect sense, and it also complained that my model wasn't actually part of an app and even explained how to explicitly declare it. Some more Googling and I discovered that app_label is a model option that can be set in class Meta. So I did as told, and it worked!

from django.db import models
from django.conf import settings
import django

MYAPP = 'myapp.MyApp'
settings.configure()
django.setup()
settings.INSTALLED_APPS.append(MYAPP)


class MyModel(models.Model):
    air_temp = models.FloatField()
    latitude = models.FloatField()
    longitude = models.FloatField()
    timezone = models.FloatField()
    pvmodule = models.CharField(max_length=20)

    class Meta:
        app_label = MYAPP


mymodel = MyModel(air_temp=25.0, latitude=38.0, longitude=-122.0,
                  timezone=-8.0, pvmodule='SPR E20-327')

mymodel.__dict__
#{'_state': <django.db.models.base.ModelState at 0x496b2b0>,
# 'air_temp': 25.0,
# 'id': None,
# 'latitude': 38.0,
# 'longitude': -122.0,
# 'pvmodule': 'SPR E20-327',
# 'timezone': -8.0}

Caveats

So I should stop here and point out that that evidently the order of these commands matters, because if I add the fake app to INSTALLED_APPS before calling django.setup() then I get this:

ImportError: No module named myapp

And unfortunately, I just figured this out now, in this post. But this isn't what I originally did. Yes, I'm completely crazy. First I added a fake module called 'myapp' to sys.modules setting it to a mock object, but that didn't work. I got back TypeError: 'Mock' object is not iterable because, as I found out later, there has to be an AppConfig subclass in the app module. But since I didn't know that yet, I did the only logical thing and put the module in a list. What? Yes, did I mention I'm an idiot? This nonsense yielded the following stern warning:

ImproperlyConfigured: The app module [] has no filesystem location, you
                      must configure this app with an AppConfig subclass with a 'path' class
                      attribute.

But this is where I found out about AppConfig in the Django docs which is covered quite nicely. Following the nice directions, I did as told and subclassed AppConfig, added path and also name which I learned from the docs, monkeypatched my mock module with it, and used the dotted name of the app myapp.MyApp now. I felt like I was getting closer, since I only got: AttributeError: __name__ which seemed like a problem with my pretend module. Another monkeypatch and we have my final ludicrously ridiculous hack.

from django.db import models
from django.conf import settings
import django
from django.apps import AppConfig
import sys
import mock

class MyApp(AppConfig):
    """
    Apps subclass ``AppConfig`` and define ``name`` and ``path``
    """
    path = '.'  # path to app
    name = 'myapp'  # name of app


# make a mock module with ``__name__`` and ``MyApp`` member
myapp_module = mock.Mock(__name__='myapp', MyApp=MyApp)
MYAPP = 'myapp.MyApp'  # full path to app
sys.modules['myapp'] = myapp_module  # register module
settings.configure()
settings.INSTALLED_APPS.append(MYAPP)
django.setup()


class MyModel(models.Model):
    air_temp = models.FloatField()
    latitude = models.FloatField()
    longitude = models.FloatField()
    timezone = models.FloatField()
    pvmodule = models.CharField(max_length=20)

    class Meta:
        app_label = MYAPP


mymodel = MyModel(air_temp=25.0, latitude=38.0, longitude=-122.0,
                  timezone=-8.0, pvmodule='SPR E20-327')

mymodel.__dict__
#{'_state': <django.db.models.base.ModelState at 0x496b2b0>,
# 'air_temp': 25.0,
# 'id': None,
# 'latitude': 38.0,
# 'longitude': -122.0,
# 'pvmodule': 'SPR E20-327',
# 'timezone': -8.0}

Yay?

Carousel Python Model Simulation Framework

Carousel - A Python Model Simulation Framework

I want to introduce Carousel, a project that my employer, SunPower has been supporting for use in prediction models. Carousel is an extensible framework for mathematical models that handles generic routines such as loading and saving data, generating reports, converting units, propagating uncertainty and running simulations so developers can focus on creating complex algorithms that are easy to share and maintain.

Introduction

Mathematical models consist of algorithms glued together with generic routines. While the algorithms may sometimes be unique and complex, the rest of the code is often simple and routine. Sometimes mathematical models developed by teams of developers over time become difficult to update because there is no framework for how new data, calculations and outputs are integrated into the existing models. Carousel allows developers to focus on creating complex mathematical models that are robust and easy to maintain by abstracting generic routines and establishing a simple but extensible framework.

The Framework

A Carousel basic model consists of 5 built in layers:

• Data
• Formulas
• Calculations
• Simulations
• Outputs

Layers

Carousel is extensible by creating more advanced models and layers. A Carousel model is a collection of layers. Carousel layers share a common base class. Each layer also has a corresponding object and a registry where objects are stored. All layers have a load method that loads all of the layer objects specified into the model. When a model is loaded it loads the objects specified for each layer.

Example

Consider a load shifting algorithm for residential or commercial rooftop solar power. The model might have a performance calculation, a load calculation, a cost calculation and an optimization algorithm that determines how home or business appliances are operated to minimize overall yearly cost of the system.

The performance calculation contains several formulas which require input data from an internal database of solar panel parameters and an online API of weather conditions, so the user creates a data source and reader for each of these. There are some data readers already included in Carousel and once a data reader is created it can be reused in many different projects. Maybe the user submits a pull request to Carousel to add the new API and database reader. The load calculation contains formulas for how the appliances are used. The input data for the appliances are entered into a generic worksheet so the user creates a data source for appliances and uses the XLRDReader to collect the data for each appliance from their worksheets.

The user organizes the formulas into 4 modules, that correspond to each of the calculations, but some formulas are reused since they are generic. For example, data frame summation formulas are used with different time series to create daily, monthly and annual outputs. The user maps out the calculations and specifies their interdependence to other formulas. For example, the cost calculation depends on the load and performance calculations and the optimization algorithm depends on the cost.

The user specifies each output name, initial value and other attributes. Specifications for each layer can be in a JSON parameter file or directly in the code as class attributes; Carousel will interpret either at runtime when it creates the model. Finally the user creates a simulation which in this case is unique because instead of marching through time or space, the simulation iterates over potential load shifting solutions from the algorithm. The user decides which data or outputs to log during the simulation and which to save in reports. Now that the model is created, the user loads the model and sends it the "start" command. After the simulation is complete, the user can examine the outputs and their variances. The outputs will have been automatically converted to the units specified in the model.

Data Layer

The data layer handles all inputs to the models. The data layer object is a data source. Each data source has a data reader. A data source is a document, API, database or other place from which input data for the model can be obtained. The data source and reader provide a framework for specifying how data is acquired. For example a data source for stock market prices might be a public API. An implementation of the stock market API data source specifies the names and attributes of each input data that will be read from the API and how the data reader should read them. The data source is similar to a deserializer because it describes how the data from the source should be interpreted by the model and creates an object in the data registry.

Formula Layer

The formula layer handles operations on input data that generate new outputs. It differs from the calculation layer which handles how formulas are combined together. The formula layer object is a formula, and each formula has a formula importer. For example the Python formula importer can import formulas that are written as Python functions.

Calculation Layer

Calculations are combinations of formulas. Each calculation also has a map of what data and output are used as arguments and what outputs the return values will be. Calculations also implement calculators. Currently there is a static calculator and a dynamic calculator, but new calculators can be implemented that can be reused in other models. The calculation also implements indexing into data and output registries in order retrieve items by index or at a specific time.

Output Layer

Outputs are just like data except they don't need a reader because they are only generated from calculations. Each output is like a serializer because it determines how output objects will be reported or displayed to the user.

Simulation Layer

The simulation layer determines the order of calculations based on the calculation dependencies. It first executes all of the static calculations and then loops over dynamic calculations, displaying logs and saving periodic reports as specified.

Model

The model is a low level class that can be extended to add new layers or implement new simulation commands. Currently only the basic model is implemented. A new model might contain a post processing layer that generates plots and reports from outputs.

Registry

Every layer has a dictionary called a registry that contains all of the layer objects and metadata corresponding to the layer attributes. The registry implements a register method that doesn't allow an item to registered more than once. Each layer registry is subclassed from the base registry so that specific layer attributes can be associated to each key. For example, data sources and outputs have a variance attribute while formulas have an args attribute.

Running Model Simulations

After a model has been described using the framework, it can be loaded. Then any or all of the model simulations can be executed from the model. The simulation specifies commands to the model that user sends using the models command method. Currently the basic model can execute the simulation start command.

Units and Uncertainty

Carousel uses the Pint units wrapper to convert units as specified. Uncertainty is propagated using the UncertaintyWrapper package which was developed for Carousel. It can wrap Python extensions and non-linear algorithms without changing any code. It propagates covariance and sensitivity across all formulas.

Future Work

A basic version of Carousel is ready now. There is an example of a photovoltaic module performance model in the documentation online at GitHub. Some ideas for new features are listed in the Carousel wiki on GitHub.

• Data validation
• Reuse 3rd party serializer/deserializer for data layer
• Model integrity check
• Database data reader
• REST API data reader
• Online repository to share data readers, simualtions, formulas and layers
• Automatic solver selection
• Post processing layer
• Testing tools
• Concurrency and speedups
• Remote process and Carousel client

Source, Docs, Issues and Wiki

• Carousel
• UncertaintyWrapper

Previous Presentations

Carousel was presented at the 5th Sandia PVPMC Workshop hosted by EPRI in Santa Clara in May 2016

Acknowledgement

Carousel and UncertaintyWrapper were developed with the support of SunPower Corp. They are distributed with a BSD 3-clause license.

Uncertainty Unwrapped

Please read my disclaimer.

I'm proud to announce UncertaintyWrapper at the Cheese Shop. This work was supported by my employer, SunPower Corp. and is currently offered with a standard 3-clause BSD license. The documentation, source code and releases are also available our SunPower org GitHub page.

So what does uncertainty_wrapper do? Let's say you have a Python function, to calculate solar position, and the function uses a C/C++ library via the Python ctypes library. Or maybe you just have a really complicated set of calculations that you repeat 8760 times, and you want it to run super fast, so you don't want it to calculate derivatives and uncertainty at every internal step, just the final output. Oh and by the way, you want all 8760 calculations vectorized, _ie_: done concurrently as much as possible.

Heres an example using PVLIB of just the first 24 hours.

#
import numpy as np  # v1.11.0
import pandas as pd  # v0.18.0
import pytz  # v2016.1
import pvlib  # v0.3.2
from uncertainty_wrapper import unc_wrapper_args  # v0.4.1

PST = pytz.timezone('US/Pacific')  # Pacific Standard Time
times = pd.DatetimeIndex(start='2015/1/1', end='2015/1/2', freq='1h', tz=PST)  # date range

# arguments for the number of observations
# new in UncertaintyWrapper==0.4.1, jagged arrays are okay
latitude, longitude, pressure, altitude, temperature = 37., -122., 101325., 0., 22.

# standard deviation of 1% assuming normal distribution
covariance = np.tile(np.diag([0.0001] * 5), (times.size, 1, 1))  # tile this for the number of observations


@unc_wrapper_args(1, 2, 3, 4, 5)
# indices specify positions of independent variables:
# 1: latitude, 2: longitude, 3: pressure, 4: altitude, 5: temperature
def spa(times, latitude, longitude, pressure, altitude, temperature):
    # location class only used prior to pvlib-0.3
    dataframe = pvlib.solarposition.spa_c(times, latitude, longitude, pressure, altitude, temperature)
    retvals = dataframe.to_records()
    zenith = retvals['apparent_zenith']
    zenith = np.where(zenith<90, zenith, np.nan)
    azimuth = retvals['azimuth']
    return zenith, azimuth


ze, az, cov, jac = spa(times, latitude, longitude, pressure, altitude, temperature, __covariance__=covariance)
df = pd.DataFrame({'zenith': ze, 'az': az}, index=times)  # easier to view as dataframe
print df
#                                    az     zenith
# 2015-01-01 00:00:00-08:00  349.297715        NaN
# 2015-01-01 01:00:00-08:00   40.210628        NaN
# 2015-01-01 02:00:00-08:00   66.719304        NaN
# 2015-01-01 03:00:00-08:00   80.930185        NaN
# 2015-01-01 04:00:00-08:00   90.852887        NaN
# 2015-01-01 05:00:00-08:00   99.212426        NaN
# 2015-01-01 06:00:00-08:00  107.181217        NaN
# 2015-01-01 07:00:00-08:00  115.450451        NaN
# 2015-01-01 08:00:00-08:00  124.564183  84.113440
# 2015-01-01 09:00:00-08:00  135.023137  74.984664
# 2015-01-01 10:00:00-08:00  147.247403  67.475783
# 2015-01-01 11:00:00-08:00  161.371578  62.273878
# 2015-01-01 12:00:00-08:00  176.922804  60.008978
# 2015-01-01 13:00:00-08:00  192.742327  61.017538
# 2015-01-01 14:00:00-08:00  207.519768  65.144340
# 2015-01-01 15:00:00-08:00  220.494108  71.839001
# 2015-01-01 16:00:00-08:00  231.600910  80.422988
# 2015-01-01 17:00:00-08:00  241.184075  89.948123
# 2015-01-01 18:00:00-08:00  249.726361        NaN
# 2015-01-01 19:00:00-08:00  257.751550        NaN
# 2015-01-01 20:00:00-08:00  265.873170        NaN
# 2015-01-01 21:00:00-08:00  275.014534        NaN
# 2015-01-01 22:00:00-08:00  287.078877        NaN
# 2015-01-01 23:00:00-08:00  307.283646        NaN
# 2015-01-02 00:00:00-08:00  348.921385        NaN

# covariance at 8AM
idx = 8
print times[idx]
# Timestamp('2015-01-01 08:00:00-0800', tz='US/Pacific', offset='H')
nf = 2  # number of dependent variables: [ze, az]
print cov[idx]
# [[ 0.6617299  -0.6152971 ]
#  [-0.6152971   0.62483904]]

# standard deviation
print np.sqrt(cov[8].diagonal())
# [ 0.81346782,  0.79046761]

# Jacobian at 8AM
nargs = 5  # number of independent args
print jac[idx]
# [[  5.56456716e-01  -6.45065654e-01  -1.37538277e-06   0.00000000e+00    4.72409055e-04]
#  [  8.29163154e-02   6.47436098e-01   0.00000000e+00   0.00000000e+00    0.00000000e+00]]
#

First this tells us that the standard deviation of the zenith is 1% if the input has a standard deviation of 1%. That's reasonable. This also tells that zenith is more sensitive to latitude and longitude than pressure or temperature and more sensitive to latitude than azimuth is.