Tuesday, July 19, 2016

Derived Django Database Field

The trick to this is creating a custom field and overloading pre_save. Pay special attention to the self.attname member that is set to the value. The source for DateField is a good example. Make sure that if you add any new attributes to the field in it's __init__ method you also add a corresponding deconstruct method.

Monday, July 18, 2016

Mocking Django App

I'm sure this is completely wrong. I needed a Django model for testing, but I don't have a Django app or even a Django project. I'm developing a Django model reader for Carousel, and so I needed a model to test it out with. Sure I could have created a quick django project, but that seemed silly, and my first instinct was to import django.db.models, make a model and use it, but this raised:

ImproperlyConfigured: Requested setting DEFAULT_INDEX_TABLESPACE, but settings are not
                      configured. You must either define the environment variable
                      DJANGO_SETTINGS_MODULE or call settings.configure() before accessing
                      settings.

Most normal people would turn back now, but instead I imported django.conf.settings and called settings.configure() just like it said to do. Now I got this error:

AppRegistryNotReady: Apps aren't loaded yet.

So now I felt like I was getting somewhere. But where? Googling told me to import django and run setup which I did and that raised:

RuntimeError: Model class __main__.MyModel doesn't declare an explicit app_label and isn't
              in an application in INSTALLED_APPS.

Wow! Normally RuntimeError is a scary warning, like you dumped your core, but this just said I needed to add the app to settings.INSTALLED_APPS, which makes perfect sense, and it also complained that my model wasn't actually part of an app and even explained how to explicitly declare it. Some more Googling and I discovered that app_label is a model option that can be set in class Meta. So I did as told, and it worked!

from django.db import models
from django.conf import settings
import django

MYAPP = 'myapp.MyApp'
settings.configure()
django.setup()
settings.INSTALLED_APPS.append(MYAPP)


class MyModel(models.Model):
    air_temp = models.FloatField()
    latitude = models.FloatField()
    longitude = models.FloatField()
    timezone = models.FloatField()
    pvmodule = models.CharField(max_length=20)

    class Meta:
        app_label = MYAPP


mymodel = MyModel(air_temp=25.0, latitude=38.0, longitude=-122.0,
                  timezone=-8.0, pvmodule='SPR E20-327')

mymodel.__dict__
#{'_state': <django.db.models.base.ModelState at 0x496b2b0>,
# 'air_temp': 25.0,
# 'id': None,
# 'latitude': 38.0,
# 'longitude': -122.0,
# 'pvmodule': 'SPR E20-327',
# 'timezone': -8.0}

Caveats

So I should stop here and point out that that evidently the order of these commands matters, because if I add the fake app to INSTALLED_APPS before calling django.setup() then I get this:

ImportError: No module named myapp

And unfortunately, I just figured this out now, in this post. But this isn't what I originally did. Yes, I'm completely crazy. First I added a fake module called 'myapp' to sys.modules setting it to a mock object, but that didn't work. I got back TypeError: 'Mock' object is not iterable because, as I found out later, there has to be an AppConfig subclass in the app module. But since I didn't know that yet, I did the only logical thing and put the module in a list. What? Yes, did I mention I'm an idiot? This nonsense yielded the following stern warning:

ImproperlyConfigured: The app module [] has no filesystem location, you
                      must configure this app with an AppConfig subclass with a 'path' class
                      attribute.

But this is where I found out about AppConfig in the Django docs which is covered quite nicely. Following the nice directions, I did as told and subclassed AppConfig, added path and also name which I learned from the docs, monkeypatched my mock module with it, and used the dotted name of the app myapp.MyApp now. I felt like I was getting closer, since I only got: AttributeError: __name__ which seemed like a problem with my pretend module. Another monkeypatch and we have my final ludicrously ridiculous hack.

from django.db import models
from django.conf import settings
import django
from django.apps import AppConfig
import sys
import mock

class MyApp(AppConfig):
    """
    Apps subclass ``AppConfig`` and define ``name`` and ``path``
    """
    path = '.'  # path to app
    name = 'myapp'  # name of app


# make a mock module with ``__name__`` and ``MyApp`` member
myapp_module = mock.Mock(__name__='myapp', MyApp=MyApp)
MYAPP = 'myapp.MyApp'  # full path to app
sys.modules['myapp'] = myapp_module  # register module
settings.configure()
settings.INSTALLED_APPS.append(MYAPP)
django.setup()


class MyModel(models.Model):
    air_temp = models.FloatField()
    latitude = models.FloatField()
    longitude = models.FloatField()
    timezone = models.FloatField()
    pvmodule = models.CharField(max_length=20)

    class Meta:
        app_label = MYAPP


mymodel = MyModel(air_temp=25.0, latitude=38.0, longitude=-122.0,
                  timezone=-8.0, pvmodule='SPR E20-327')

mymodel.__dict__
#{'_state': <django.db.models.base.ModelState at 0x496b2b0>,
# 'air_temp': 25.0,
# 'id': None,
# 'latitude': 38.0,
# 'longitude': -122.0,
# 'pvmodule': 'SPR E20-327',
# 'timezone': -8.0}

Yay?

Carousel Python Model Simulation Framework

Carousel - A Python Model Simulation Framework

I want to introduce Carousel, a project that my employer, SunPower has been supporting for use in prediction models. Carousel is an extensible framework for mathematical models that handles generic routines such as loading and saving data, generating reports, converting units, propagating uncertainty and running simulations so developers can focus on creating complex algorithms that are easy to share and maintain.

Introduction

Mathematical models consist of algorithms glued together with generic routines. While the algorithms may sometimes be unique and complex, the rest of the code is often simple and routine. Sometimes mathematical models developed by teams of developers over time become difficult to update because there is no framework for how new data, calculations and outputs are integrated into the existing models. Carousel allows developers to focus on creating complex mathematical models that are robust and easy to maintain by abstracting generic routines and establishing a simple but extensible framework.

The Framework

A Carousel basic model consists of 5 built in layers:

Data
Formulas
Calculations
Simulations
Outputs

Layers

Carousel is extensible by creating more advanced models and layers. A Carousel model is a collection of layers. Carousel layers share a common base class. Each layer also has a corresponding object and a registry where objects are stored. All layers have a load method that loads all of the layer objects specified into the model. When a model is loaded it loads the objects specified for each layer.

Example

Consider a load shifting algorithm for residential or commercial rooftop solar power. The model might have a performance calculation, a load calculation, a cost calculation and an optimization algorithm that determines how home or business appliances are operated to minimize overall yearly cost of the system.

The performance calculation contains several formulas which require input data from an internal database of solar panel parameters and an online API of weather conditions, so the user creates a data source and reader for each of these. There are some data readers already included in Carousel and once a data reader is created it can be reused in many different projects. Maybe the user submits a pull request to Carousel to add the new API and database reader. The load calculation contains formulas for how the appliances are used. The input data for the appliances are entered into a generic worksheet so the user creates a data source for appliances and uses the XLRDReader to collect the data for each appliance from their worksheets.

The user organizes the formulas into 4 modules, that correspond to each of the calculations, but some formulas are reused since they are generic. For example, data frame summation formulas are used with different time series to create daily, monthly and annual outputs. The user maps out the calculations and specifies their interdependence to other formulas. For example, the cost calculation depends on the load and performance calculations and the optimization algorithm depends on the cost.

The user specifies each output name, initial value and other attributes. Specifications for each layer can be in a JSON parameter file or directly in the code as class attributes; Carousel will interpret either at runtime when it creates the model. Finally the user creates a simulation which in this case is unique because instead of marching through time or space, the simulation iterates over potential load shifting solutions from the algorithm. The user decides which data or outputs to log during the simulation and which to save in reports. Now that the model is created, the user loads the model and sends it the "start" command. After the simulation is complete, the user can examine the outputs and their variances. The outputs will have been automatically converted to the units specified in the model.

Data Layer

The data layer handles all inputs to the models. The data layer object is a data source. Each data source has a data reader. A data source is a document, API, database or other place from which input data for the model can be obtained. The data source and reader provide a framework for specifying how data is acquired. For example a data source for stock market prices might be a public API. An implementation of the stock market API data source specifies the names and attributes of each input data that will be read from the API and how the data reader should read them. The data source is similar to a deserializer because it describes how the data from the source should be interpreted by the model and creates an object in the data registry.

Formula Layer

The formula layer handles operations on input data that generate new outputs. It differs from the calculation layer which handles how formulas are combined together. The formula layer object is a formula, and each formula has a formula importer. For example the Python formula importer can import formulas that are written as Python functions.

Calculation Layer

Calculations are combinations of formulas. Each calculation also has a map of what data and output are used as arguments and what outputs the return values will be. Calculations also implement calculators. Currently there is a static calculator and a dynamic calculator, but new calculators can be implemented that can be reused in other models. The calculation also implements indexing into data and output registries in order retrieve items by index or at a specific time.

Output Layer

Outputs are just like data except they don't need a reader because they are only generated from calculations. Each output is like a serializer because it determines how output objects will be reported or displayed to the user.

Simulation Layer

The simulation layer determines the order of calculations based on the calculation dependencies. It first executes all of the static calculations and then loops over dynamic calculations, displaying logs and saving periodic reports as specified.

Model

The model is a low level class that can be extended to add new layers or implement new simulation commands. Currently only the basic model is implemented. A new model might contain a post processing layer that generates plots and reports from outputs.

Registry

Every layer has a dictionary called a registry that contains all of the layer objects and metadata corresponding to the layer attributes. The registry implements a register method that doesn't allow an item to registered more than once. Each layer registry is subclassed from the base registry so that specific layer attributes can be associated to each key. For example, data sources and outputs have a variance attribute while formulas have an args attribute.

Running Model Simulations

After a model has been described using the framework, it can be loaded. Then any or all of the model simulations can be executed from the model. The simulation specifies commands to the model that user sends using the models command method. Currently the basic model can execute the simulation start command.

Units and Uncertainty

Carousel uses the Pint units wrapper to convert units as specified. Uncertainty is propagated using the UncertaintyWrapper package which was developed for Carousel. It can wrap Python extensions and non-linear algorithms without changing any code. It propagates covariance and sensitivity across all formulas.

Future Work

A basic version of Carousel is ready now. There is an example of a photovoltaic module performance model in the documentation online at GitHub. Some ideas for new features are listed in the Carousel wiki on GitHub.