Using model properties and raw SQL with django-chartit

django-chartit is a Django app that can be used to easily create charts from the data in your database. The charts are rendered using Highcharts and jQuery JavaScript libraries. Data in your database can be plotted as simple line charts, column charts, area charts, scatter plots, and many more chart types.

Few days ago we have released version 0.2.8 which includes support for model properties and raw SQL queries in your charts. This blog post will describe how to use them.

Using model properties

Sometimes it is convenient to calculate a value inside your models but not store it in the database. For example consider the following model

class City(models.Model):
    city = models.CharField(max_length=50)
    state = models.CharField(max_length=2)

    def region(self):
        return 'USA:%s' % self.city

You are not able to select the region "field" but you may use it as part of your chart terms as shown below

ds = DataPool(
        series=[{
            'options': {
                'source': SalesHistory.objects.only(
                            'bookstore__city', 'sale_qty'
                          )[:10],
            },
            'terms': [
                'bookstore__city__region',
                'sale_qty'
            ]
        }]
)

cht = Chart(
        datasource=ds,
        series_options=[{
            'options': {
                'type': 'column',
                'stacking': False,
                'stack': 0,
            },
            'terms': {
                'bookstore__city__region': [
                    'sale_qty'
                ]
            }},
        ],
        chart_options={
            'title': {
                'text': 'Sales reports'
            },
            'xAxis': {
                'title': {
                    'text': 'City'
                }
            }
        }
)

The full example, including source code and live charts is available at django-chartit/demoproject!

Using raw SQL queries

Django allows you to execute SQL queries directly. These will return objects in the form of RawQuerySet and can be used in the same way as any other QuerySet.

ds = DataPool(
        series=[{
            'options': {
                'source': MonthlyWeatherByCity.objects.raw(
                            "SELECT id, month, houston_temp, boston_temp "
                            "FROM demoproject_monthlyweatherbycity")
            },
            'terms': [
                'month',
                'houston_temp',
                'boston_temp'
            ]
        }]
)

You will have to select the primary key field and pay attention to field names as to avoid duplicates when performing JOIN operations. Otherwise there is no difference. Full examples, including source code and live charts are available at django-chartit/demoproject!

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

New release pelican-ab 0.2.3

pelican-ab is an A/B testing plugin for Pelican, the static site generator. It allows you to encode experiments in your templates and renders the experiment selected by the AB_EXPERIMENT environment variable.

Today we are releasing version 0.2.3 as an urgent bug fix update handling internal changes in the latest version of Pelican. In previous versions Pelican used the JINJA_EXTENSIONS setting to configure additional Jinja2 modules. In version 3.7.0, released yesterday, JINJA_EXTENSIONS has been replaced with the JINJA_ENVIRONMENT setting. Version 0.2.3 of pelican-ab handles both cases and is able to work with older and newer versions of Pelican!

This backward-incompatible change in Pelican has been automatically detected by our monitoring tool Strazar, which proves the value of automatic testing against the latest versions of upstream dependencies!

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

What I've worked on in November 2016

Hello everyone, during November I've been focusing on Cosmic Ray, pylint and django-chartit which are supported packages in our Python software stack.

The changes include

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Status Report for October 2016

Hello everyone, during October I've been focusing on Cosmic Ray which is a supported package of our Python software stack.

The changes include

All of my work this month was around visit_mutation_site() being able to generate multiple mutations at a single site. For example the code

a < b

can be mutated into:

a <= b
a is not b
a > b
... etc

Previously this was done using different mutation operators but now it is done using a single class and an index to the desired operator replacement. As a follow up other mutation operators had to be updated as well. As a result the code base is cleaner and easier to understand.

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Status Report for September 2016

Hello everyone, during September I've been focusing on Cosmic Ray which is a supported package of our Python software stack.

The changes include better error checking when running the test suite in Travis CI, fixes for some code smells and style updates, some Python 3.3 compatibility fixes, traceback fix for Python 3.4, new and-or replacement operators and internal refactoring.

During my work on and-or replacement operators it turned out that Cosmic Ray didn't support the notion of one operator producing multiple code mutations. This required refactoring of project internals and is now currently supported. As a follow up there are a couple more issues opened to clean up the existing code, mainly the comparisons replacement operators.

NOTE Cosmic Ray is still not compatible with Python 3.3 and will probably never be, despite my work. In cosmic_ray/importing.py we make use of importlib.machinery.ModuleSpec which was introduced in 3.4 and at this moment we don't want to backport and support this for Python 3.3. The rest of the code is 3.3 compatible though.

During September I have also worked on django-chartit v0.2.7 to fix a nasty recursion loop bug. The follow up of this bug is pylint #1109, which is now merged into master!

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Beware of recursion loop when using super()

When working with class inheritance in Python you often find yourself using super() to call the parent class methods. The format is

super([current class name], self).[base class method]([arguments])

When you have lots of similar code you may be tempted to short-cut the writing of current class name. The two possibilities are

super(type(self), self).some_method()
super(self.__class__, self).some_method()

I have seen this in production code. I'm guilty myself as well because I like to use the self.__class__ notation! This leads to problems, see django-chartit #41, when you inherit from the class which uses these shortcuts. For example

class Base(object):
    def method(self):
        print 'original', type(self), self.__class__

class Derived(Base):
    def method(self):
        print 'derived', type(self), self.__class__
        super(type(self), self).method()
        # super(self.__class__, self).method()

class Subclass(Derived):
    def method(self):
        print 'subclass of derived', type(self), self.__class__
        super(Subclass, self).method()

When you call Derived().method() the result is

derived <class '__main__.Derived'> <class '__main__.Derived'>
original <class '__main__.Derived'> <class '__main__.Derived'>

Here both shortcuts are evaluated correctly. However when you call Subclass().method() the result becomes

subclass of derived <class '__main__.Subclass'> <class '__main__.Subclass'>
derived <class '__main__.Subclass'> <class '__main__.Subclass'>
derived <class '__main__.Subclass'> <class '__main__.Subclass'>
derived <class '__main__.Subclass'> <class '__main__.Subclass'>
... skip ...
RuntimeError: maximum recursion depth exceeded while calling a Python object

In the example the call super(Subclass) works fine and invokes Derived.method() as expected. Then we call super(Subclass) inside Derived.method() which leads back to where we were hence the recursion loop. The problem is only visible when you have other classes that inherit from a class which uses incorrect invocation of super(). This is why it may lay hidden in production software!

I definitely expected a severe code smell like that to be discovered by pylint and Landscape.io. Indeed older versions (pylint-1.3.1-1.el7.noarch) will report error for super(type(self)) but not newer ones. As it turns out pylint didn't have a test for this condition and have introduced a regression in the master branch. I believe this is due to that fact that they didn't check for using type() directly but rather that was a side effect which ceased to exist once the code was updated. Pylint#1109 adds tests for the two code smells described above and updates the checkers to explicitly detect them! Happy testing!

I hope you like my work and please subscribe to Mr. Senko's support service should you need commercial support for this or other open source libraries!

There are comments.

New release django-chartit 0.2.7

django-chartit is a Django app that can be used to easily create charts from the data in your database. The charts are rendered using Highcharts and jQuery JavaScript libraries. Data in your database can be plotted as simple line charts, column charts, area charts, scatter plots, and many more chart types.

Today we are releasing version 0.2.7 as part of our regular update process. This is an urgent bug fix update because previous versions broke subclassing of Chart and PivotChart classes.

Changelog since version 0.2.6

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

August 2016 Status Report

Hello everyone, during August I've been focusing on django-chartit which is a supported package of our Python software stack.

Since I've taken over maintenance from Praveen Gollakota there were several bug-fix releases, 3 of them in August. The result is 6 closed issues and increased test coverage. I've spent time to refactor parts of the code and make it compatible with the latest Django version. Also cleaned up code smells identified by Landscape.io.

Future plans for django-chartit include working on the remaining open issues. I think I will have to refactor the code a lot more to make it compatible with the latest Highcharts.js API before being able to implement the requested features.

I have also spoken with Highcharts engineering on the topic of testing their JavaScript charting code in the context of django-chartit. My idea is to load the demo project using various versions of the JavaScript library and make sure everything works on the client side. This appears to be doable with Selenium.

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Loading initial data for Many-To-Many fields

Previously I've written about how to use JSON fixtures in Django migrations. This becomes a bit more complicated when you have ManyToMany fields in your models. A corner case is when you have a ManyToMany relation to self. The example below comes from django-chartit.

class Book(models.Model):
    title = models.CharField(max_length=50)
    rating = models.FloatField()
    rating_count = models.IntegerField()
    authors = models.ManyToManyField(Author)
    publisher = models.ForeignKey(Publisher, null=True, blank=True, on_delete=models.SET_NULL)
    published_at = models.DateTimeField(null=True, blank=True)
    related = models.ManyToManyField('self', blank=True)
    genre = models.ForeignKey(Genre, null=True, blank=True, on_delete=models.SET_NULL)

The fields authors and related are represented as separate tables and are computed when you access objects from this model. Django automatically handles these fields and creates classes for them. Before you can use them, you need a reference to their model classes.

Book = apps.get_model("demoproject", "Book")
BookRelated = None
BookAuthors = None
for relation in Book._meta.many_to_many:
    if relation.name == 'related':
        try:
            BookRelated = relation.remote_field.through
        except AttributeError:
            # available in Django 1.8
            BookRelated = relation.rel.through

    if relation.name == 'authors':
        try:
            BookAuthors = relation.remote_field.through
        except AttributeError:
            # available in Django 1.8
            BookAuthors = relation.rel.through

The JSON data looks like this

{ "fields" : {
    "authors" : [  ],
    "genre_id" : 4,
    "publisher_id" : 3,
    "rating" : 3.8999999999999999,
    "rating_count" : 1869,
    "related" : [ 10 ],
    "title" : "Freakonomics"
  },
"model" : "Book",
"pk" : 9
},
{ "fields" : {
    "authors" : [ 24 ],
    "genre_id" : 5,
    "publisher_id" : 9,
    "rating" : 4.4000000000000004,
    "rating_count" : 222,
    "related" : [ 23, 21 ],
    "title" : "Hyperspace"
     },
"model" : "Book",
"pk" : 24
},

Once we have our related model classes we proceed to store the data in the database like so

# create Book objects
for record in json_data:
    # skip everything which isn't a book
    if record['model'] != 'Book':
        continue

    # build a list of book authors using the intermediate BookAuthors model
    for author_id in record['fields']['authors']:
        author_obj = BookAuthors()
        author_obj.book_id = record['pk']
        author_obj.author_id = author_id
        author_obj.save()
    # you can't save the `authors` field directly in DB
    del record['fields']['authors']

    # build a list of related books using the intermediate BookRelated model
    for related_id in record['fields']['related']:
        related_obj = BookRelated()
        related_obj.from_book_id = record['pk']
        related_obj.to_book_id = related_id
        related_obj.save()
    # you can't save the `related` field directly in DB
    del record['fields']['related']

    # finally save the Book object
    model_class = apps.get_model("demoproject", record['model'])
    obj = model_class(**record['fields'])
    obj.pk = record['pk']
    obj.save()

This works well for django-chartit. You should take care to remove the ManyToMany fields from the JSON data because they don't actually exist in the Book class and Django will raise an exception if you try to assign to them.

I hope you like my work and please subscribe to Mr. Senko's support service should you need commercial support for this or other open source libraries!

There are comments.

New release django-chartit 0.2.6

django-chartit is a Django app that can be used to easily create charts from the data in your database. The charts are rendered using Highcharts and jQuery JavaScript libraries. Data in your database can be plotted as simple line charts, column charts, area charts, scatter plots, and many more chart types.

Today we are releasing version 0.2.6 as part of our regular update process.

Changelog since version 0.2.4

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

Converting JSON Fixtures to Django Migrations

Older Django apps like django-chartit and Nitrate used JSON fixtures to populate their databases with initial data. In this article I will show you an easy way to convert JSON fixtures into native Django migrations. The JSON fixture looks like this

{"fields": {"first_name": "Seth",
            "last_name": "Godin"
            },
 "model": "demoproject.Author",
 "pk": 1
 },
{"fields": {"first_name": "Guy",
            "last_name": "Kawasaki"
            },
 "model": "demoproject.Author",
 "pk": 2
 },
{"fields": {"first_name": "Geoffrey",
            "last_name": "Colvin"
            },
 "model": "demoproject.Author",
 "pk": 3
 }

Notice the pk and model fields which tell us where this data came from and what was the object PK when exported from the database. The fields dict is the actual data for this object.

In Python we can use json.loads and read the fixture data from disk or even better assign it directly to a variable inside our Python source file. Then iterate over all values and create the objects programmatically like this

from __future__ import unicode_literals
from django.db import migrations


def initialize_data(apps, schema_editor):
    data = [
        {"fields": {"first_name": "Seth",
                    "last_name": "Godin"
                    },
         "model": "demoproject.Author",
         "pk": 1
         },
        {"fields": {"first_name": "Guy",
                    "last_name": "Kawasaki"
                    },
         "model": "demoproject.Author",
         "pk": 2
         },
        {"fields": {"first_name": "Geoffrey",
                    "last_name": "Colvin"
                    },
         "model": "demoproject.Author",
         "pk": 3
         },
    ]

    for record in data:
        app_name, model_name = record['model'].split('.')
        ModelClass = apps.get_model(app_name, model_name)
        obj = ModelClass(**record['fields'])
        # this is required only if you have other models
        # with foreign keys referring to this object
        # obj.pk = record['pk']
        obj.save()

class Migration(migrations.Migration):

    dependencies = [
        ('demoproject', '0001_initial')
    ]

    operations = [
        migrations.RunPython(initialize_data),
    ]

This works well for most of the cases. You should take care to assign the same PKs in case there are other objects that hold references to them. If this isn't the case then you can drop these fields entirely to reduce your source code size.

I hope you like my work and please subscribe to Mr. Senko's support service should you need commercial support for this or other open source libraries!

There are comments.

Reviving django-chartit with version 0.2.4

django-chartit is a Django app that can be used to easily create charts from the data in your database. The charts are rendered using Highcharts and jQuery JavaScript libraries. Data in your database can be plotted as simple line charts, column charts, area charts, scatter plots, and many more chart types.

The project has been originally developed by Praveen Gollakota and had gained some popularity. Recently Mr. Senko was granted the necessary access rights and we've revived the project with two upstream releases.

Changelog

Version 0.2.3 was released a few days ago and merges the django-chartit2 fork by Grant McConnaughey which had a few releases of its own earlier this year. It also makes a few improvements and merged other pull requests.

Version 0.2.4 was released today and fixes the usage of deprecated Django APIs which were removed in Django 1.10.

Many thanks to Grant for his work on Python 3 and latest Django support! Many thanks to Praveen for letting us maintain the project.

Upgrade from django_chartit2

If you are using the django_chartit2 module by Grant then you have to

pip uninstall django_chartit2
pip install django_chartit

If you are not in a hurry to upgrade you may as well wait for the next release by Grant, which should automatically require django_chartit and transparently switch you back to using the original package and not the fork. However release of such an update is not under our control.

Feature plans

Mr. Senko is currently working actively to bring the rest of the project up to speed, including the demo site, which shows how to use the API. This is a mandatory step before we go ahead to work on various bug fixes and documentation improvements. We will also be looking into adding more tests for the project.

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

July 2016 Status Report

Hello everyone, July has been relatively busy in terms of pull requests. I've been working on Cosmic Ray the mutation testing tool for Python and in the last couple of days taken over maintenance of django-chartit.

I've made several improvements against Cosmic Ray. The most notable are:

I will be continuing to work on Cosmic Ray and also integrate mutation testing as part of the standard testing toolset used by Mr. Senko.

Then I've started working on django-chartit which is a very popular module that has been abandoned in the last couple of years. My immediate goal was to merge back the django-chartit2 fork by Grant McConnaughey, which adds Python 3 and latest Django support and merge some pending pull requests. I've been working on fixing quite a few errors and warnings reported by the test tools and getting the documentation up to speed. A much anticipated release on PyPI will be coming out very soon. Once the merger between django-chartit and django-chartit2 is complete I will continue working on the reported issues.

I hope you like my work and please subscribe to Mr. Senko if you need a faster response cycle for the open source libraries you use.

There are comments.

June 2016 Status Report

Hello everyone, last month has been a bit of a holiday and I don't have much to share. I've been playing around with mutation testing tools. The few patches I did are both related to Cosmic Ray:

Right now I'm experimenting with running mutation testing against a few popular open source libraries, written in both Python and Ruby. So far the tools seem to work fine, except minor issues. However the test suites I've been playing with aren't very robust and most of the mutants stay alive! I will write another blog post on the subject once I have more information to share. I hope you like my work and please subscribe to Mr. Senko! if need a faster response cycle for the open source libraries you use.

There are comments.

May 2016 Status Report

Here is a quick status report of my work for Mr. Senko. During May 2016 I've worked on:

Focusing on abandoned pull requests

While working on several issues I've noticed pull requests that were well written but they had minor issues, for example code formatting. The code review was done in the initial PR but the author didn't update their patches. As a result of this the pull request stayed open for a long time. I've also been focusing on issues and pull requests which were labeled for a particular release but were left open for quite a long time. Hopefully we'll see those releases coming out soon.

I hope you like my work and please subscribe to Mr. Senko! if need a faster response cycle for the open source libraries you use.

There are comments.

Reviving pelican-octopress and pelipress

pelican-octopress-theme and Pelipress are popular themes for Pelican but are not actively developed at the moment. Mr. Senko decided to fork and continue their active development. The new repository is at MrSenko/pelican-octopress-theme.

Why did we fork

Initially we've tried sending pull requests to the upstream repository but later noticed multiple issues and pull requests which were left open for more than a year, nearly 2 years in the worst cases. Pelipress on the other hand has not been rebased back to pelican-octopress-theme for almost 3 years!

We've emailed the original authors of both projects but didn't get a reply to our inquiries in a few weeks so decided to fork and continue forward.

Changelog

The following features have been merged from pelican-octopress-theme by resolving all conflicts which were present:

The following features have been inspired by Pelipress and cleanly re-implemented on top of the current fork:

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

Triggering Automatic Dependency Testing

Strazar (from Bulgarian for sentinel) helps you monitor upstream sources and once a new package version is found your .travis.yml environment is updated and the changes committed to GitHub which automatically triggers a new build! This approach relies on having a good test suite, but you already have that covered, right ?

Strazar was inspired by Forbes Lindesay's GitHub Automation talk at DEVit Conf 2015 and we're very excited to announce this release days before this year's DEVit Conf!

What is currently supported

In its initial release Strazar only supports PyPI, Travis-CI and GitHub. We already have written the code to detect new RubyGems and NPM packages and that will land in the next versions. However we also found the wonderful libraries.io service which is already aware of many more package repositories so we'll try to integrate Strazar with it in the future.

We've started with Travis-CI because this is what we use but support for other CI environments is very easy to add. Strazar calculates all possible combinations between package names and their versions and writes them to a text file. To add another CI environment we only need to parse and write the file in the correct format.

Installation and configuration

To install Strazar execute

pip install strazar

Then configure the GITHUB_TOKEN environment variable. This token needs the public_repo or repo permission in order to push commits to your repositories.

Prepare .travis.yml

Strazar uses the variable format _PACKAGE_NAME where the variable name starts with an under-score followed by the capitalized package name. All hyphens are converted to under-scores as well. We advise that your .travis.yml files follow the same convention. This is how Strazar's own .travis.yml looks like

language: python
python:
  - 2.7
  - 3.5
install:
  - pip install coverage flake8 mock PyYAML==$_PYYAML PyGithub==$_PYGITHUB
script:
  - ./test.sh
env:
  - _PYGITHUB=1.26.0 _PYYAML=3.11

How to monitor PyPI

PyPI doesn't provide web-hooks so we examine the RSS feed for packages of interest based on configuration settings. To start monitoring PyPI execute the following code from a cron job (here at Mr. Senko we do it every hour)

import os
import strazar

os.environ['GITHUB_TOKEN'] = 'xxxxxxxxx'
config = {
    "PyYAML" : [
        {
            'cb' : strazar.update_github,
            'args': {
                'GITHUB_REPO' : 'MrSenko/strazar',
                'GITHUB_BRANCH' : 'master',
                'GITHUB_FILE' : '.travis.yml'
            }
        },
    ],
}

strazar.monitor_pypi_rss(config)

The config dict uses package names as 1st level keys. If you are interested in a particular package add it here. All other packages detected from the RSS feed will be ignored. If your project depends on multiple packages you have to list all of them as 1st level keys in config and duplicate the key values.

The key value is a list of call-back methods and arguments to execute once a new package has been published online. If two or more repositories depend on the same package then add them as values to this list.

The strazar.update_github call-back knows how to commit to your source repo which will automatically trigger a new CI build.

Support

Strazar is provided as open source for everyone to use. At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

A/B Testing with Jinja2 and Pelican

A/B testing is a randomized experiment with two variants, which are the control and variation in the controlled experiment. In this article we're going to present a solution which works with Pelican and other Jinja2 based static site generators.

The markup

In statically generated websites you either have content or templates. We've been interested in encoding A/B experiments in templates and didn't find a solution so we made our own. jinja-ab is an A/B testing extension for Jinja2. It allows you to encode experiments in your templates and renders the experiment selected by the AB_EXPERIMENT environment variable. control is the default experiment name if AB_EXPERIMENT is not specified! The syntax looks like this

{% experiment control %}This is the control{% endexperiment %}
{% experiment v1 %}This is version 1{% endexperiment %}

Rendering and output control

jinja-ab deals with rendering the template string based on the value of AB_EXPERIMENT. It is up to you to decide what to do with the result. At Mr. Senko we use Pelican and created the complimentary pelican-ab plugin. To enable the plugin simply add it in your pelicanconf.py

PLUGIN_PATHS = ['path/to/pelican-ab']
PLUGINS = ['pelican_ab']

After encoding your experiments into the theme templates you can generate the resulting HTML files by running the command

AB_EXPERIMENT="v1" make html

When rendering experiments the resulting HTML files are saved under OUTPUT_PATH plus the experiment name. For example 'output/v1', 'output/v2', etc. The control experiments are rendered directly under OUTPUT_PATH. For example the control directory structure of this blog looks like this:

output/
|-- blog
|   |-- 2016
|   |   | ...
|   |-- archives
|   |-- atodorov
|   |   `-- ...
|   |-- mr-senko
|   |   `-- ...
|   `-- tags
|       | ...
|-- feeds
|-- images
|-- js
|-- support
`-- theme

After rendering an experiment called 'v1' the directory structure looks like this:

output/v1/
|-- blog
|   |-- 2016
|   |   | ...
|   |-- archives
|   |-- atodorov
|   |   `-- ...
|   |-- mr-senko
|   |   `-- ...
|   `-- tags
|       | ...
`-- support

Only content output is saved under the new directory because content it rendered using the templates which we want to A/B test. This is how pelican-ab works for the moment.

This plugin also automatically updates the Content.url and URLWrapper.url class properties from Pelican so that things like {{ article.url }} and {{ author.url }} will point to URLs from the same experiment.

In other words each experiment produces its own HTML and URL structure, using the experiment name as prefix. Once a user lands on a page from experiment "v1" all links to other content will also point to "v1", for example 'blog/about-me.html' becomes 'v1/blog/about-me.html', etc. This will help you gather more information from the experiment because all your internal URLs are under the same experiment, using the same HTML template variation.

NOTE: wherever you use the {{ SITEURL }} template tag without pointing to the content url property the values will not be changed. This means all your CSS, JavaScript and images will continue to work without being duplicated under the experiment directory.

Testing and publishing experiments

For local development use the command AB_EXPERIMENT="xy" make regenerate or AB_EXPERIMENT="xy" make html together with make serve to review the experiments. When you are ready to publish them online execute the following sequence of commands:

rm -rf output/
make github
AB_EXPERIMENT="01" make github
AB_EXPERIMENT="02" make github

By default publishconf.py contains DELETE_OUTPUT_DIRECTORY = True which causes pelican-ab to raise an exception. The problem is that you need to execute make publish or make github for each experiment you'd like to publish online. When DELETE_OUTPUT_DIRECTORY is True the previous contents will be deleted and ONLY that variation will be published! This will break your website because everything will be gone!

A/B testing

Now that we finally got our experiments encoded and rendered it is time to re-route some of the web traffic to them and analyze the results. pelican-ab is not designed to deal with this, you will have to use external tools to control how your web traffic is redirected to your experiments and what sort of results are collected. Our favorite ones are LuckyOrange and Optimizely.

Support

jinja-ab and pelican-ab are provided as open source for everyone to use. At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for these or other open source libraries please subscribe to Mr. Senko!

There are comments.

Conditional Include in Jinja2 and Pelican

How do you create Jinja templates which behave differently based on what Pelican plugins are loaded ? I've hit this problem while working on improving Flex PR #20. The straight forward solution looks like this

{% for name in PLUGINS if name == 'assets' %}
    {% assets "stylesheet/style.css", filters="cssmin", output="style.min.css" %}
        <link rel="stylesheet" type="text/css" href="{{ SITEURL }}/{{ ASSET_URL }}">
    {% endassets %}
{% else %}
    <link rel="stylesheet" type="text/css" href="{{ SITEURL }}/{{ THEME_STATIC_DIR }}/stylesheet/style.min.css">
{% endfor %}

It appears to work great except when the assets plugin isn't loaded. Then we get a TemplateSyntaxError:

TemplateSyntaxError: Encountered unknown tag 'assets'.
Jinja was looking for the following tags: 'endfor' or 'else'.
The innermost block that needs to be closed is 'for'.`

The {% assets %} tag is not defined because the assets plugin is missing. We may expect that Jinja will parse this tag only if the body of the for loop is executed but instead Jinja tries to parse all tags before rendering the template. The solution, as proposed by @ThiefMaster on GitHub, is to use a conditional {% include %} tag inside the for loop body like so

{% for name in PLUGINS if name == 'assets' %}
    {% include 'assets.html' %}
{% else %}
    <link rel="stylesheet" type="text/css" href="{{ SITEURL }}/{{ THEME_STATIC_DIR }}/stylesheet/style.min.css">
{% endfor %}

Where assets.html looks like this

{% assets "stylesheet/style.css", filters="cssmin", output="style.min.css" %}
    <link rel="stylesheet" type="text/css" href="{{ SITEURL }}/{{ ASSET_URL }}">
{% endassets %}

I don't like splitting out the HTML code in this way. Imagine that we later decide to add a second CSS file to the template. The risk of forgetting to add it in both places increases as the number of CSS files (or places where we use the same HTML pattern) increases. However this appears to be the only way to conditionally use the assets plugin only when it is loaded. The proposed changes are in Flex PR #40.

I hope you like my work and please subscribe to Mr. Senko if you need a faster release cycle for the open source libraries you use.

There are comments.

April 2016 Report

Here is a quick status report of my work for Mr. Senko. During April 2016 I've worked on:

Nitrate

As I've written previously Nitrate is a test case management system, which we use internally. Unfortunately it is using very old version of Django and it is not so easy to migrate forward.

I've decided to stall all of my previous pull requests and focus on getting the tests up and running in Travis-CI before going forward. I've managed to fix a few of them locally but then hit a road block. The fact that the Nitrate team is very limited in capacity (only 1 person at the moment) isn't helping either. I will continue working on this but with lower priority.

New features for i18n_viz

i18n_viz is a very cool Ruby gem which lets your browse your own Rails application and visualize and edit your translatable strings. Think of it as a reversed lookup for translations. i18n_viz highlights your app’s translatable text and adds a tooltip containing the translation key which links to the translation in your favorite online translation tool.

I have added the css_override option to allow for better styling of the highlights and tooltips provided by this gem. PR #20 has already been merged but not released into a new version yet.

I've also fixed a bug with nested HTML tags and changed the external_tool_url option to support code execution. With this the user is able to configure the URL to an online translation tool during runtime. My particular use-case is to include the currently selected locale in the URL string. These two pull requests are not merged yet and unfortunately the package author is busy at the moment so it will take some time.

New features for Pelican

I've managed to implement more granular control over tag, categories and author slugs in Pelican in PR #1926 and the code is already merged!

Under the hood the slugify() method now can skip replacing non-alphanumeric characters so you have more control over the generated URLs. I've experienced this problem when migrating my personal blog from Octopress to Pelican.

Then I've also added the possibility to configure author slugs, which is useful to blogs with multiple authors or if you want your author slug to match your GitHub username. In fact pelicanconf.py for Mr. Senko's blog looks like this:

ARTICLE_URL = 'blog/{author}/{date:%Y}/{date:%m}/{date:%d}/{slug}/'
ARTICLE_SAVE_AS = ARTICLE_URL + 'index.html'

AUTHOR_URL = 'blog/{slug}/'
AUTHOR_SAVE_AS  = AUTHOR_URL + 'index.html'

AUTHOR_SUBSTITUTIONS = [
    ('Alexander Todorov', 'atodorov'),
    ('Krasimir Tsonev',   'krasimir'),
]

NOTE: older URLs on this blog do not match above settings for compatibility reasons!

I hope you like my work and please consider subscribing to Mr. Senko! if need a faster release cycle for the open source libraries you use.

There are comments.