What I've worked on in December 2016

Hello everyone, during December I've been working on several different projects from our supported Python software stack and also ventured into Ruby land.

Versions 0.2.3 and 0.2.4 of pelican-ab

pelican-ab is an A/B testing plugin for Pelican, the static site generator for Python. It allows you to encode experiments in your templates and renders the experiment selected by the AB_EXPERIMENT environment variable.

There were two versions released in December 2016 making it compatible with the newly released Pelican 3.7.0, adding more tests and fixing some minor bugs.

Version 0.2.8 of django-chartit

I have worked on a new version for django-chartit which adds support for model properties and raw SQL queries in charts. I've also done internal work for further refactoring but without anything concrete yet!

New extensions for pylint

I've worked on the new compare-to-empty-string extension for pylint. Using this extension the following snippet

if S != "":
    pass

if S == '':
    pass

will be flagged as problematic because it can be written in a more natural way:

if S:
    pass

if not S:
    pass

The extension will be disabled by default because empty string may be a valid value depending on the behavior of your program. I have also created a similar extension to flag integer comparisons against zero, see pylint#1243. While these may seem a bit extreme they are very useful to identify code which can be refactored to reduce comparisons when coupled with mutation testing!

New Rspec formatters

I've worked on two new formatters for Rspec:

Both add simple syntactic sugar to how Rspec output can be formatted and make it easier for debugging or sending off for further machine processing.

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

What I've worked on in November 2016

Hello everyone, during November I've been focusing on Cosmic Ray, pylint and django-chartit which are supported packages in our Python software stack.

The changes include

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Status Report for October 2016

Hello everyone, during October I've been focusing on Cosmic Ray which is a supported package of our Python software stack.

The changes include

All of my work this month was around visit_mutation_site() being able to generate multiple mutations at a single site. For example the code

a < b

can be mutated into:

a <= b
a is not b
a > b
... etc

Previously this was done using different mutation operators but now it is done using a single class and an index to the desired operator replacement. As a follow up other mutation operators had to be updated as well. As a result the code base is cleaner and easier to understand.

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Status Report for September 2016

Hello everyone, during September I've been focusing on Cosmic Ray which is a supported package of our Python software stack.

The changes include better error checking when running the test suite in Travis CI, fixes for some code smells and style updates, some Python 3.3 compatibility fixes, traceback fix for Python 3.4, new and-or replacement operators and internal refactoring.

During my work on and-or replacement operators it turned out that Cosmic Ray didn't support the notion of one operator producing multiple code mutations. This required refactoring of project internals and is now currently supported. As a follow up there are a couple more issues opened to clean up the existing code, mainly the comparisons replacement operators.

NOTE Cosmic Ray is still not compatible with Python 3.3 and will probably never be, despite my work. In cosmic_ray/importing.py we make use of importlib.machinery.ModuleSpec which was introduced in 3.4 and at this moment we don't want to backport and support this for Python 3.3. The rest of the code is 3.3 compatible though.

During September I have also worked on django-chartit v0.2.7 to fix a nasty recursion loop bug. The follow up of this bug is pylint #1109, which is now merged into master!

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Beware of recursion loop when using super()

When working with class inheritance in Python you often find yourself using super() to call the parent class methods. The format is

super([current class name], self).[base class method]([arguments])

When you have lots of similar code you may be tempted to short-cut the writing of current class name. The two possibilities are

super(type(self), self).some_method()
super(self.__class__, self).some_method()

I have seen this in production code. I'm guilty myself as well because I like to use the self.__class__ notation! This leads to problems, see django-chartit #41, when you inherit from the class which uses these shortcuts. For example

class Base(object):
    def method(self):
        print 'original', type(self), self.__class__

class Derived(Base):
    def method(self):
        print 'derived', type(self), self.__class__
        super(type(self), self).method()
        # super(self.__class__, self).method()

class Subclass(Derived):
    def method(self):
        print 'subclass of derived', type(self), self.__class__
        super(Subclass, self).method()

When you call Derived().method() the result is

derived <class '__main__.Derived'> <class '__main__.Derived'>
original <class '__main__.Derived'> <class '__main__.Derived'>

Here both shortcuts are evaluated correctly. However when you call Subclass().method() the result becomes

subclass of derived <class '__main__.Subclass'> <class '__main__.Subclass'>
derived <class '__main__.Subclass'> <class '__main__.Subclass'>
derived <class '__main__.Subclass'> <class '__main__.Subclass'>
derived <class '__main__.Subclass'> <class '__main__.Subclass'>
... skip ...
RuntimeError: maximum recursion depth exceeded while calling a Python object

In the example the call super(Subclass) works fine and invokes Derived.method() as expected. Then we call super(Subclass) inside Derived.method() which leads back to where we were hence the recursion loop. The problem is only visible when you have other classes that inherit from a class which uses incorrect invocation of super(). This is why it may lay hidden in production software!

I definitely expected a severe code smell like that to be discovered by pylint and Landscape.io. Indeed older versions (pylint-1.3.1-1.el7.noarch) will report error for super(type(self)) but not newer ones. As it turns out pylint didn't have a test for this condition and have introduced a regression in the master branch. I believe this is due to that fact that they didn't check for using type() directly but rather that was a side effect which ceased to exist once the code was updated. Pylint#1109 adds tests for the two code smells described above and updates the checkers to explicitly detect them! Happy testing!

I hope you like my work and please subscribe to Mr. Senko's support service should you need commercial support for this or other open source libraries!

There are comments.

August 2016 Status Report

Hello everyone, during August I've been focusing on django-chartit which is a supported package of our Python software stack.

Since I've taken over maintenance from Praveen Gollakota there were several bug-fix releases, 3 of them in August. The result is 6 closed issues and increased test coverage. I've spent time to refactor parts of the code and make it compatible with the latest Django version. Also cleaned up code smells identified by Landscape.io.

Future plans for django-chartit include working on the remaining open issues. I think I will have to refactor the code a lot more to make it compatible with the latest Highcharts.js API before being able to implement the requested features.

I have also spoken with Highcharts engineering on the topic of testing their JavaScript charting code in the context of django-chartit. My idea is to load the demo project using various versions of the JavaScript library and make sure everything works on the client side. This appears to be doable with Selenium.

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Loading initial data for Many-To-Many fields

Previously I've written about how to use JSON fixtures in Django migrations. This becomes a bit more complicated when you have ManyToMany fields in your models. A corner case is when you have a ManyToMany relation to self. The example below comes from django-chartit.

class Book(models.Model):
    title = models.CharField(max_length=50)
    rating = models.FloatField()
    rating_count = models.IntegerField()
    authors = models.ManyToManyField(Author)
    publisher = models.ForeignKey(Publisher, null=True, blank=True, on_delete=models.SET_NULL)
    published_at = models.DateTimeField(null=True, blank=True)
    related = models.ManyToManyField('self', blank=True)
    genre = models.ForeignKey(Genre, null=True, blank=True, on_delete=models.SET_NULL)

The fields authors and related are represented as separate tables and are computed when you access objects from this model. Django automatically handles these fields and creates classes for them. Before you can use them, you need a reference to their model classes.

Book = apps.get_model("demoproject", "Book")
BookRelated = None
BookAuthors = None
for relation in Book._meta.many_to_many:
    if relation.name == 'related':
        try:
            BookRelated = relation.remote_field.through
        except AttributeError:
            # available in Django 1.8
            BookRelated = relation.rel.through

    if relation.name == 'authors':
        try:
            BookAuthors = relation.remote_field.through
        except AttributeError:
            # available in Django 1.8
            BookAuthors = relation.rel.through

The JSON data looks like this

{ "fields" : {
    "authors" : [  ],
    "genre_id" : 4,
    "publisher_id" : 3,
    "rating" : 3.8999999999999999,
    "rating_count" : 1869,
    "related" : [ 10 ],
    "title" : "Freakonomics"
  },
"model" : "Book",
"pk" : 9
},
{ "fields" : {
    "authors" : [ 24 ],
    "genre_id" : 5,
    "publisher_id" : 9,
    "rating" : 4.4000000000000004,
    "rating_count" : 222,
    "related" : [ 23, 21 ],
    "title" : "Hyperspace"
     },
"model" : "Book",
"pk" : 24
},

Once we have our related model classes we proceed to store the data in the database like so

# create Book objects
for record in json_data:
    # skip everything which isn't a book
    if record['model'] != 'Book':
        continue

    # build a list of book authors using the intermediate BookAuthors model
    for author_id in record['fields']['authors']:
        author_obj = BookAuthors()
        author_obj.book_id = record['pk']
        author_obj.author_id = author_id
        author_obj.save()
    # you can't save the `authors` field directly in DB
    del record['fields']['authors']

    # build a list of related books using the intermediate BookRelated model
    for related_id in record['fields']['related']:
        related_obj = BookRelated()
        related_obj.from_book_id = record['pk']
        related_obj.to_book_id = related_id
        related_obj.save()
    # you can't save the `related` field directly in DB
    del record['fields']['related']

    # finally save the Book object
    model_class = apps.get_model("demoproject", record['model'])
    obj = model_class(**record['fields'])
    obj.pk = record['pk']
    obj.save()

This works well for django-chartit. You should take care to remove the ManyToMany fields from the JSON data because they don't actually exist in the Book class and Django will raise an exception if you try to assign to them.

I hope you like my work and please subscribe to Mr. Senko's support service should you need commercial support for this or other open source libraries!

There are comments.

Converting JSON Fixtures to Django Migrations

Older Django apps like django-chartit and Nitrate used JSON fixtures to populate their databases with initial data. In this article I will show you an easy way to convert JSON fixtures into native Django migrations. The JSON fixture looks like this

{"fields": {"first_name": "Seth",
            "last_name": "Godin"
            },
 "model": "demoproject.Author",
 "pk": 1
 },
{"fields": {"first_name": "Guy",
            "last_name": "Kawasaki"
            },
 "model": "demoproject.Author",
 "pk": 2
 },
{"fields": {"first_name": "Geoffrey",
            "last_name": "Colvin"
            },
 "model": "demoproject.Author",
 "pk": 3
 }

Notice the pk and model fields which tell us where this data came from and what was the object PK when exported from the database. The fields dict is the actual data for this object.

In Python we can use json.loads and read the fixture data from disk or even better assign it directly to a variable inside our Python source file. Then iterate over all values and create the objects programmatically like this

from __future__ import unicode_literals
from django.db import migrations


def initialize_data(apps, schema_editor):
    data = [
        {"fields": {"first_name": "Seth",
                    "last_name": "Godin"
                    },
         "model": "demoproject.Author",
         "pk": 1
         },
        {"fields": {"first_name": "Guy",
                    "last_name": "Kawasaki"
                    },
         "model": "demoproject.Author",
         "pk": 2
         },
        {"fields": {"first_name": "Geoffrey",
                    "last_name": "Colvin"
                    },
         "model": "demoproject.Author",
         "pk": 3
         },
    ]

    for record in data:
        app_name, model_name = record['model'].split('.')
        ModelClass = apps.get_model(app_name, model_name)
        obj = ModelClass(**record['fields'])
        # this is required only if you have other models
        # with foreign keys referring to this object
        # obj.pk = record['pk']
        obj.save()

class Migration(migrations.Migration):

    dependencies = [
        ('demoproject', '0001_initial')
    ]

    operations = [
        migrations.RunPython(initialize_data),
    ]

This works well for most of the cases. You should take care to assign the same PKs in case there are other objects that hold references to them. If this isn't the case then you can drop these fields entirely to reduce your source code size.

I hope you like my work and please subscribe to Mr. Senko's support service should you need commercial support for this or other open source libraries!

There are comments.

July 2016 Status Report

Hello everyone, July has been relatively busy in terms of pull requests. I've been working on Cosmic Ray the mutation testing tool for Python and in the last couple of days taken over maintenance of django-chartit.

I've made several improvements against Cosmic Ray. The most notable are:

I will be continuing to work on Cosmic Ray and also integrate mutation testing as part of the standard testing toolset used by Mr. Senko.

Then I've started working on django-chartit which is a very popular module that has been abandoned in the last couple of years. My immediate goal was to merge back the django-chartit2 fork by Grant McConnaughey, which adds Python 3 and latest Django support and merge some pending pull requests. I've been working on fixing quite a few errors and warnings reported by the test tools and getting the documentation up to speed. A much anticipated release on PyPI will be coming out very soon. Once the merger between django-chartit and django-chartit2 is complete I will continue working on the reported issues.

I hope you like my work and please subscribe to Mr. Senko if you need a faster response cycle for the open source libraries you use.

There are comments.

June 2016 Status Report

Hello everyone, last month has been a bit of a holiday and I don't have much to share. I've been playing around with mutation testing tools. The few patches I did are both related to Cosmic Ray:

Right now I'm experimenting with running mutation testing against a few popular open source libraries, written in both Python and Ruby. So far the tools seem to work fine, except minor issues. However the test suites I've been playing with aren't very robust and most of the mutants stay alive! I will write another blog post on the subject once I have more information to share. I hope you like my work and please subscribe to Mr. Senko! if need a faster response cycle for the open source libraries you use.

There are comments.

May 2016 Status Report

Here is a quick status report of my work for Mr. Senko. During May 2016 I've worked on:

Focusing on abandoned pull requests

While working on several issues I've noticed pull requests that were well written but they had minor issues, for example code formatting. The code review was done in the initial PR but the author didn't update their patches. As a result of this the pull request stayed open for a long time. I've also been focusing on issues and pull requests which were labeled for a particular release but were left open for quite a long time. Hopefully we'll see those releases coming out soon.

I hope you like my work and please subscribe to Mr. Senko! if need a faster response cycle for the open source libraries you use.

There are comments.

Conditional Include in Jinja2 and Pelican

How do you create Jinja templates which behave differently based on what Pelican plugins are loaded ? I've hit this problem while working on improving Flex PR #20. The straight forward solution looks like this

{% for name in PLUGINS if name == 'assets' %}
    {% assets "stylesheet/style.css", filters="cssmin", output="style.min.css" %}
        <link rel="stylesheet" type="text/css" href="{{ SITEURL }}/{{ ASSET_URL }}">
    {% endassets %}
{% else %}
    <link rel="stylesheet" type="text/css" href="{{ SITEURL }}/{{ THEME_STATIC_DIR }}/stylesheet/style.min.css">
{% endfor %}

It appears to work great except when the assets plugin isn't loaded. Then we get a TemplateSyntaxError:

TemplateSyntaxError: Encountered unknown tag 'assets'.
Jinja was looking for the following tags: 'endfor' or 'else'.
The innermost block that needs to be closed is 'for'.`

The {% assets %} tag is not defined because the assets plugin is missing. We may expect that Jinja will parse this tag only if the body of the for loop is executed but instead Jinja tries to parse all tags before rendering the template. The solution, as proposed by @ThiefMaster on GitHub, is to use a conditional {% include %} tag inside the for loop body like so

{% for name in PLUGINS if name == 'assets' %}
    {% include 'assets.html' %}
{% else %}
    <link rel="stylesheet" type="text/css" href="{{ SITEURL }}/{{ THEME_STATIC_DIR }}/stylesheet/style.min.css">
{% endfor %}

Where assets.html looks like this

{% assets "stylesheet/style.css", filters="cssmin", output="style.min.css" %}
    <link rel="stylesheet" type="text/css" href="{{ SITEURL }}/{{ ASSET_URL }}">
{% endassets %}

I don't like splitting out the HTML code in this way. Imagine that we later decide to add a second CSS file to the template. The risk of forgetting to add it in both places increases as the number of CSS files (or places where we use the same HTML pattern) increases. However this appears to be the only way to conditionally use the assets plugin only when it is loaded. The proposed changes are in Flex PR #40.

I hope you like my work and please subscribe to Mr. Senko if you need a faster release cycle for the open source libraries you use.

There are comments.

April 2016 Report

Here is a quick status report of my work for Mr. Senko. During April 2016 I've worked on:

Nitrate

As I've written previously Nitrate is a test case management system, which we use internally. Unfortunately it is using very old version of Django and it is not so easy to migrate forward.

I've decided to stall all of my previous pull requests and focus on getting the tests up and running in Travis-CI before going forward. I've managed to fix a few of them locally but then hit a road block. The fact that the Nitrate team is very limited in capacity (only 1 person at the moment) isn't helping either. I will continue working on this but with lower priority.

New features for i18n_viz

i18n_viz is a very cool Ruby gem which lets your browse your own Rails application and visualize and edit your translatable strings. Think of it as a reversed lookup for translations. i18n_viz highlights your app’s translatable text and adds a tooltip containing the translation key which links to the translation in your favorite online translation tool.

I have added the css_override option to allow for better styling of the highlights and tooltips provided by this gem. PR #20 has already been merged but not released into a new version yet.

I've also fixed a bug with nested HTML tags and changed the external_tool_url option to support code execution. With this the user is able to configure the URL to an online translation tool during runtime. My particular use-case is to include the currently selected locale in the URL string. These two pull requests are not merged yet and unfortunately the package author is busy at the moment so it will take some time.

New features for Pelican

I've managed to implement more granular control over tag, categories and author slugs in Pelican in PR #1926 and the code is already merged!

Under the hood the slugify() method now can skip replacing non-alphanumeric characters so you have more control over the generated URLs. I've experienced this problem when migrating my personal blog from Octopress to Pelican.

Then I've also added the possibility to configure author slugs, which is useful to blogs with multiple authors or if you want your author slug to match your GitHub username. In fact pelicanconf.py for Mr. Senko's blog looks like this:

ARTICLE_URL = 'blog/{author}/{date:%Y}/{date:%m}/{date:%d}/{slug}/'
ARTICLE_SAVE_AS = ARTICLE_URL + 'index.html'

AUTHOR_URL = 'blog/{slug}/'
AUTHOR_SAVE_AS  = AUTHOR_URL + 'index.html'

AUTHOR_SUBSTITUTIONS = [
    ('Alexander Todorov', 'atodorov'),
    ('Krasimir Tsonev',   'krasimir'),
]

NOTE: older URLs on this blog do not match above settings for compatibility reasons!

I hope you like my work and please consider subscribing to Mr. Senko! if need a faster release cycle for the open source libraries you use.

There are comments.

March 2016 Report

Here is a quick status report for Mr. Senko. During March 2016 I've worked on:

Nitrate Test Case Management System

Nitrate is a test case management system written with Django. We use it internally to track testing activities. It is reasonably good at what it does but there are some issues with how the software has been developed in the past.

If you like my work and need a faster response cycle please consider subscribing to Mr. Senko!

There are comments.

January 2016 Report

This is my first status report for Mr. Senko. During January 2016 I've worked on:

Sphinx broke

The Sphinx changes are particularly interesting - the fix for issue #656 in commit 4c4450d changes the Graphviz's node['options'] type from list to dict which in turn breaks html_visit_inheritance_diagram() and the inheritance_diagram extension.

My previous work on this PR introduced some basic tests, which were missing and they caught the type change! Because the tests and my latest changes are not yet merged into master the inheritance_diagram.py extension is still broken! The tests fail in Travis-CI but probably due to dot not being installed. Locally everything seems to work.

Its also worth mentioning that it's been 2 weeks since I've updated my PR and a bit more since Sphinx introduced the regression. As far as I can see there's not been any updates with respect to this. If you'd like to get a faster response cycle please consider subscribing to Mr. Senko!

There are comments.

Founding of Mr. Senko

Hello everyone, welcome to Mr. Senko! We are a group of friends who have been working with open source for the last 10 years. During this time we've seen technologies come and go and hype projects becoming abandoned months later. We've seen the pain when companies had to rewrite their software because of this and the struggle to get your patches accepted upstream. Don't you wish you had somebody who could take care of all the hassle for you?

Mr. Senko is our attempt at providing commercial support for open source projects, specifically libraries which other companies use to build their products. As of now we have diverse experience in Java, Python and JavaScript as well as my personal experience with Quality Assurance.

In the next weeks and months we will be refining our value proposition and building the team so we can offer you great technical support and expertise. We plan to blog regularly about our progress and about technology and open source events in general! You can follow us at @Mr_Senko or tweet at the wizard using the #MrSenko hashtag!

Stay tuned for more news!

There are comments.