Beware of recursion loop when using super()

When working with class inheritance in Python you often find yourself using super() to call the parent class methods. The format is

super([current class name], self).[base class method]([arguments])

When you have lots of similar code you may be tempted to short-cut the writing of current class name. The two possibilities are

super(type(self), self).some_method()
super(self.__class__, self).some_method()

I have seen this in production code. I'm guilty myself as well because I like to use the self.__class__ notation! This leads to problems, see django-chartit #41, when you inherit from the class which uses these shortcuts. For example

class Base(object):
    def method(self):
        print 'original', type(self), self.__class__

class Derived(Base):
    def method(self):
        print 'derived', type(self), self.__class__
        super(type(self), self).method()
        # super(self.__class__, self).method()

class Subclass(Derived):
    def method(self):
        print 'subclass of derived', type(self), self.__class__
        super(Subclass, self).method()

When you call Derived().method() the result is

derived <class '__main__.Derived'> <class '__main__.Derived'>
original <class '__main__.Derived'> <class '__main__.Derived'>

Here both shortcuts are evaluated correctly. However when you call Subclass().method() the result becomes

subclass of derived <class '__main__.Subclass'> <class '__main__.Subclass'>
derived <class '__main__.Subclass'> <class '__main__.Subclass'>
derived <class '__main__.Subclass'> <class '__main__.Subclass'>
derived <class '__main__.Subclass'> <class '__main__.Subclass'>
... skip ...
RuntimeError: maximum recursion depth exceeded while calling a Python object

In the example the call super(Subclass) works fine and invokes Derived.method() as expected. Then we call super(Subclass) inside Derived.method() which leads back to where we were hence the recursion loop. The problem is only visible when you have other classes that inherit from a class which uses incorrect invocation of super(). This is why it may lay hidden in production software!

I definitely expected a severe code smell like that to be discovered by pylint and Landscape.io. Indeed older versions (pylint-1.3.1-1.el7.noarch) will report error for super(type(self)) but not newer ones. As it turns out pylint didn't have a test for this condition and have introduced a regression in the master branch. I believe this is due to that fact that they didn't check for using type() directly but rather that was a side effect which ceased to exist once the code was updated. Pylint#1109 adds tests for the two code smells described above and updates the checkers to explicitly detect them! Happy testing!

I hope you like my work and please subscribe to Mr. Senko's support service should you need commercial support for this or other open source libraries!

There are comments.

New release django-chartit 0.2.7

django-chartit is a Django app that can be used to easily create charts from the data in your database. The charts are rendered using Highcharts and jQuery JavaScript libraries. Data in your database can be plotted as simple line charts, column charts, area charts, scatter plots, and many more chart types.

Today we are releasing version 0.2.7 as part of our regular update process. This is an urgent bug fix update because previous versions broke subclassing of Chart and PivotChart classes.

Changelog since version 0.2.6

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

August 2016 Status Report

Hello everyone, during August I've been focusing on django-chartit which is a supported package of our Python software stack.

Since I've taken over maintenance from Praveen Gollakota there were several bug-fix releases, 3 of them in August. The result is 6 closed issues and increased test coverage. I've spent time to refactor parts of the code and make it compatible with the latest Django version. Also cleaned up code smells identified by Landscape.io.

Future plans for django-chartit include working on the remaining open issues. I think I will have to refactor the code a lot more to make it compatible with the latest Highcharts.js API before being able to implement the requested features.

I have also spoken with Highcharts engineering on the topic of testing their JavaScript charting code in the context of django-chartit. My idea is to load the demo project using various versions of the JavaScript library and make sure everything works on the client side. This appears to be doable with Selenium.

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Loading initial data for Many-To-Many fields

Previously I've written about how to use JSON fixtures in Django migrations. This becomes a bit more complicated when you have ManyToMany fields in your models. A corner case is when you have a ManyToMany relation to self. The example below comes from django-chartit.

class Book(models.Model):
    title = models.CharField(max_length=50)
    rating = models.FloatField()
    rating_count = models.IntegerField()
    authors = models.ManyToManyField(Author)
    publisher = models.ForeignKey(Publisher, null=True, blank=True, on_delete=models.SET_NULL)
    published_at = models.DateTimeField(null=True, blank=True)
    related = models.ManyToManyField('self', blank=True)
    genre = models.ForeignKey(Genre, null=True, blank=True, on_delete=models.SET_NULL)

The fields authors and related are represented as separate tables and are computed when you access objects from this model. Django automatically handles these fields and creates classes for them. Before you can use them, you need a reference to their model classes.

Book = apps.get_model("demoproject", "Book")
BookRelated = None
BookAuthors = None
for relation in Book._meta.many_to_many:
    if relation.name == 'related':
        try:
            BookRelated = relation.remote_field.through
        except AttributeError:
            # available in Django 1.8
            BookRelated = relation.rel.through

    if relation.name == 'authors':
        try:
            BookAuthors = relation.remote_field.through
        except AttributeError:
            # available in Django 1.8
            BookAuthors = relation.rel.through

The JSON data looks like this

{ "fields" : {
    "authors" : [  ],
    "genre_id" : 4,
    "publisher_id" : 3,
    "rating" : 3.8999999999999999,
    "rating_count" : 1869,
    "related" : [ 10 ],
    "title" : "Freakonomics"
  },
"model" : "Book",
"pk" : 9
},
{ "fields" : {
    "authors" : [ 24 ],
    "genre_id" : 5,
    "publisher_id" : 9,
    "rating" : 4.4000000000000004,
    "rating_count" : 222,
    "related" : [ 23, 21 ],
    "title" : "Hyperspace"
     },
"model" : "Book",
"pk" : 24
},

Once we have our related model classes we proceed to store the data in the database like so

# create Book objects
for record in json_data:
    # skip everything which isn't a book
    if record['model'] != 'Book':
        continue

    # build a list of book authors using the intermediate BookAuthors model
    for author_id in record['fields']['authors']:
        author_obj = BookAuthors()
        author_obj.book_id = record['pk']
        author_obj.author_id = author_id
        author_obj.save()
    # you can't save the `authors` field directly in DB
    del record['fields']['authors']

    # build a list of related books using the intermediate BookRelated model
    for related_id in record['fields']['related']:
        related_obj = BookRelated()
        related_obj.from_book_id = record['pk']
        related_obj.to_book_id = related_id
        related_obj.save()
    # you can't save the `related` field directly in DB
    del record['fields']['related']

    # finally save the Book object
    model_class = apps.get_model("demoproject", record['model'])
    obj = model_class(**record['fields'])
    obj.pk = record['pk']
    obj.save()

This works well for django-chartit. You should take care to remove the ManyToMany fields from the JSON data because they don't actually exist in the Book class and Django will raise an exception if you try to assign to them.

I hope you like my work and please subscribe to Mr. Senko's support service should you need commercial support for this or other open source libraries!

There are comments.

New release django-chartit 0.2.6

django-chartit is a Django app that can be used to easily create charts from the data in your database. The charts are rendered using Highcharts and jQuery JavaScript libraries. Data in your database can be plotted as simple line charts, column charts, area charts, scatter plots, and many more chart types.

Today we are releasing version 0.2.6 as part of our regular update process.

Changelog since version 0.2.4

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

Converting JSON Fixtures to Django Migrations

Older Django apps like django-chartit and Nitrate used JSON fixtures to populate their databases with initial data. In this article I will show you an easy way to convert JSON fixtures into native Django migrations. The JSON fixture looks like this

{"fields": {"first_name": "Seth",
            "last_name": "Godin"
            },
 "model": "demoproject.Author",
 "pk": 1
 },
{"fields": {"first_name": "Guy",
            "last_name": "Kawasaki"
            },
 "model": "demoproject.Author",
 "pk": 2
 },
{"fields": {"first_name": "Geoffrey",
            "last_name": "Colvin"
            },
 "model": "demoproject.Author",
 "pk": 3
 }

Notice the pk and model fields which tell us where this data came from and what was the object PK when exported from the database. The fields dict is the actual data for this object.

In Python we can use json.loads and read the fixture data from disk or even better assign it directly to a variable inside our Python source file. Then iterate over all values and create the objects programmatically like this

from __future__ import unicode_literals
from django.db import migrations


def initialize_data(apps, schema_editor):
    data = [
        {"fields": {"first_name": "Seth",
                    "last_name": "Godin"
                    },
         "model": "demoproject.Author",
         "pk": 1
         },
        {"fields": {"first_name": "Guy",
                    "last_name": "Kawasaki"
                    },
         "model": "demoproject.Author",
         "pk": 2
         },
        {"fields": {"first_name": "Geoffrey",
                    "last_name": "Colvin"
                    },
         "model": "demoproject.Author",
         "pk": 3
         },
    ]

    for record in data:
        app_name, model_name = record['model'].split('.')
        ModelClass = apps.get_model(app_name, model_name)
        obj = ModelClass(**record['fields'])
        # this is required only if you have other models
        # with foreign keys referring to this object
        # obj.pk = record['pk']
        obj.save()

class Migration(migrations.Migration):

    dependencies = [
        ('demoproject', '0001_initial')
    ]

    operations = [
        migrations.RunPython(initialize_data),
    ]

This works well for most of the cases. You should take care to assign the same PKs in case there are other objects that hold references to them. If this isn't the case then you can drop these fields entirely to reduce your source code size.

I hope you like my work and please subscribe to Mr. Senko's support service should you need commercial support for this or other open source libraries!

There are comments.

Reviving django-chartit with version 0.2.4

django-chartit is a Django app that can be used to easily create charts from the data in your database. The charts are rendered using Highcharts and jQuery JavaScript libraries. Data in your database can be plotted as simple line charts, column charts, area charts, scatter plots, and many more chart types.

The project has been originally developed by Praveen Gollakota and had gained some popularity. Recently Mr. Senko was granted the necessary access rights and we've revived the project with two upstream releases.

Changelog

Version 0.2.3 was released a few days ago and merges the django-chartit2 fork by Grant McConnaughey which had a few releases of its own earlier this year. It also makes a few improvements and merged other pull requests.

Version 0.2.4 was released today and fixes the usage of deprecated Django APIs which were removed in Django 1.10.

Many thanks to Grant for his work on Python 3 and latest Django support! Many thanks to Praveen for letting us maintain the project.

Upgrade from django_chartit2

If you are using the django_chartit2 module by Grant then you have to

pip uninstall django_chartit2
pip install django_chartit

If you are not in a hurry to upgrade you may as well wait for the next release by Grant, which should automatically require django_chartit and transparently switch you back to using the original package and not the fork. However release of such an update is not under our control.

Feature plans

Mr. Senko is currently working actively to bring the rest of the project up to speed, including the demo site, which shows how to use the API. This is a mandatory step before we go ahead to work on various bug fixes and documentation improvements. We will also be looking into adding more tests for the project.

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

July 2016 Status Report

Hello everyone, July has been relatively busy in terms of pull requests. I've been working on Cosmic Ray the mutation testing tool for Python and in the last couple of days taken over maintenance of django-chartit.

I've made several improvements against Cosmic Ray. The most notable are:

I will be continuing to work on Cosmic Ray and also integrate mutation testing as part of the standard testing toolset used by Mr. Senko.

Then I've started working on django-chartit which is a very popular module that has been abandoned in the last couple of years. My immediate goal was to merge back the django-chartit2 fork by Grant McConnaughey, which adds Python 3 and latest Django support and merge some pending pull requests. I've been working on fixing quite a few errors and warnings reported by the test tools and getting the documentation up to speed. A much anticipated release on PyPI will be coming out very soon. Once the merger between django-chartit and django-chartit2 is complete I will continue working on the reported issues.

I hope you like my work and please subscribe to Mr. Senko if you need a faster response cycle for the open source libraries you use.

There are comments.

June 2016 Status Report

Hello everyone, last month has been a bit of a holiday and I don't have much to share. I've been playing around with mutation testing tools. The few patches I did are both related to Cosmic Ray:

Right now I'm experimenting with running mutation testing against a few popular open source libraries, written in both Python and Ruby. So far the tools seem to work fine, except minor issues. However the test suites I've been playing with aren't very robust and most of the mutants stay alive! I will write another blog post on the subject once I have more information to share. I hope you like my work and please subscribe to Mr. Senko! if need a faster response cycle for the open source libraries you use.

There are comments.

May 2016 Status Report

Here is a quick status report of my work for Mr. Senko. During May 2016 I've worked on:

Focusing on abandoned pull requests

While working on several issues I've noticed pull requests that were well written but they had minor issues, for example code formatting. The code review was done in the initial PR but the author didn't update their patches. As a result of this the pull request stayed open for a long time. I've also been focusing on issues and pull requests which were labeled for a particular release but were left open for quite a long time. Hopefully we'll see those releases coming out soon.

I hope you like my work and please subscribe to Mr. Senko! if need a faster response cycle for the open source libraries you use.

There are comments.

Reviving pelican-octopress and pelipress

pelican-octopress-theme and Pelipress are popular themes for Pelican but are not actively developed at the moment. Mr. Senko decided to fork and continue their active development. The new repository is at MrSenko/pelican-octopress-theme.

Why did we fork

Initially we've tried sending pull requests to the upstream repository but later noticed multiple issues and pull requests which were left open for more than a year, nearly 2 years in the worst cases. Pelipress on the other hand has not been rebased back to pelican-octopress-theme for almost 3 years!

We've emailed the original authors of both projects but didn't get a reply to our inquiries in a few weeks so decided to fork and continue forward.

Changelog

The following features have been merged from pelican-octopress-theme by resolving all conflicts which were present:

The following features have been inspired by Pelipress and cleanly re-implemented on top of the current fork:

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

Triggering Automatic Dependency Testing

Strazar (from Bulgarian for sentinel) helps you monitor upstream sources and once a new package version is found your .travis.yml environment is updated and the changes committed to GitHub which automatically triggers a new build! This approach relies on having a good test suite, but you already have that covered, right ?

Strazar was inspired by Forbes Lindesay's GitHub Automation talk at DEVit Conf 2015 and we're very excited to announce this release days before this year's DEVit Conf!

What is currently supported

In its initial release Strazar only supports PyPI, Travis-CI and GitHub. We already have written the code to detect new RubyGems and NPM packages and that will land in the next versions. However we also found the wonderful libraries.io service which is already aware of many more package repositories so we'll try to integrate Strazar with it in the future.

We've started with Travis-CI because this is what we use but support for other CI environments is very easy to add. Strazar calculates all possible combinations between package names and their versions and writes them to a text file. To add another CI environment we only need to parse and write the file in the correct format.

Installation and configuration

To install Strazar execute

pip install strazar

Then configure the GITHUB_TOKEN environment variable. This token needs the public_repo or repo permission in order to push commits to your repositories.

Prepare .travis.yml

Strazar uses the variable format _PACKAGE_NAME where the variable name starts with an under-score followed by the capitalized package name. All hyphens are converted to under-scores as well. We advise that your .travis.yml files follow the same convention. This is how Strazar's own .travis.yml looks like

language: python
python:
  - 2.7
  - 3.5
install:
  - pip install coverage flake8 mock PyYAML==$_PYYAML PyGithub==$_PYGITHUB
script:
  - ./test.sh
env:
  - _PYGITHUB=1.26.0 _PYYAML=3.11

How to monitor PyPI

PyPI doesn't provide web-hooks so we examine the RSS feed for packages of interest based on configuration settings. To start monitoring PyPI execute the following code from a cron job (here at Mr. Senko we do it every hour)

import os
import strazar

os.environ['GITHUB_TOKEN'] = 'xxxxxxxxx'
config = {
    "PyYAML" : [
        {
            'cb' : strazar.update_github,
            'args': {
                'GITHUB_REPO' : 'MrSenko/strazar',
                'GITHUB_BRANCH' : 'master',
                'GITHUB_FILE' : '.travis.yml'
            }
        },
    ],
}

strazar.monitor_pypi_rss(config)

The config dict uses package names as 1st level keys. If you are interested in a particular package add it here. All other packages detected from the RSS feed will be ignored. If your project depends on multiple packages you have to list all of them as 1st level keys in config and duplicate the key values.

The key value is a list of call-back methods and arguments to execute once a new package has been published online. If two or more repositories depend on the same package then add them as values to this list.

The strazar.update_github call-back knows how to commit to your source repo which will automatically trigger a new CI build.

Support

Strazar is provided as open source for everyone to use. At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

A/B Testing with Jinja2 and Pelican

A/B testing is a randomized experiment with two variants, which are the control and variation in the controlled experiment. In this article we're going to present a solution which works with Pelican and other Jinja2 based static site generators.

The markup

In statically generated websites you either have content or templates. We've been interested in encoding A/B experiments in templates and didn't find a solution so we made our own. jinja-ab is an A/B testing extension for Jinja2. It allows you to encode experiments in your templates and renders the experiment selected by the AB_EXPERIMENT environment variable. control is the default experiment name if AB_EXPERIMENT is not specified! The syntax looks like this

{% experiment control %}This is the control{% endexperiment %}
{% experiment v1 %}This is version 1{% endexperiment %}

Rendering and output control

jinja-ab deals with rendering the template string based on the value of AB_EXPERIMENT. It is up to you to decide what to do with the result. At Mr. Senko we use Pelican and created the complimentary pelican-ab plugin. To enable the plugin simply add it in your pelicanconf.py

PLUGIN_PATHS = ['path/to/pelican-ab']
PLUGINS = ['pelican_ab']

After encoding your experiments into the theme templates you can generate the resulting HTML files by running the command

AB_EXPERIMENT="v1" make html

When rendering experiments the resulting HTML files are saved under OUTPUT_PATH plus the experiment name. For example 'output/v1', 'output/v2', etc. The control experiments are rendered directly under OUTPUT_PATH. For example the control directory structure of this blog looks like this:

output/
|-- blog
|   |-- 2016
|   |   | ...
|   |-- archives
|   |-- atodorov
|   |   `-- ...
|   |-- mr-senko
|   |   `-- ...
|   `-- tags
|       | ...
|-- feeds
|-- images
|-- js
|-- support
`-- theme

After rendering an experiment called 'v1' the directory structure looks like this:

output/v1/
|-- blog
|   |-- 2016
|   |   | ...
|   |-- archives
|   |-- atodorov
|   |   `-- ...
|   |-- mr-senko
|   |   `-- ...
|   `-- tags
|       | ...
`-- support

Only content output is saved under the new directory because content it rendered using the templates which we want to A/B test. This is how pelican-ab works for the moment.

This plugin also automatically updates the Content.url and URLWrapper.url class properties from Pelican so that things like {{ article.url }} and {{ author.url }} will point to URLs from the same experiment.

In other words each experiment produces its own HTML and URL structure, using the experiment name as prefix. Once a user lands on a page from experiment "v1" all links to other content will also point to "v1", for example 'blog/about-me.html' becomes 'v1/blog/about-me.html', etc. This will help you gather more information from the experiment because all your internal URLs are under the same experiment, using the same HTML template variation.

NOTE: wherever you use the {{ SITEURL }} template tag without pointing to the content url property the values will not be changed. This means all your CSS, JavaScript and images will continue to work without being duplicated under the experiment directory.

Testing and publishing experiments

For local development use the command AB_EXPERIMENT="xy" make regenerate or AB_EXPERIMENT="xy" make html together with make serve to review the experiments. When you are ready to publish them online execute the following sequence of commands:

rm -rf output/
make github
AB_EXPERIMENT="01" make github
AB_EXPERIMENT="02" make github

By default publishconf.py contains DELETE_OUTPUT_DIRECTORY = True which causes pelican-ab to raise an exception. The problem is that you need to execute make publish or make github for each experiment you'd like to publish online. When DELETE_OUTPUT_DIRECTORY is True the previous contents will be deleted and ONLY that variation will be published! This will break your website because everything will be gone!

A/B testing

Now that we finally got our experiments encoded and rendered it is time to re-route some of the web traffic to them and analyze the results. pelican-ab is not designed to deal with this, you will have to use external tools to control how your web traffic is redirected to your experiments and what sort of results are collected. Our favorite ones are LuckyOrange and Optimizely.

Support

jinja-ab and pelican-ab are provided as open source for everyone to use. At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for these or other open source libraries please subscribe to Mr. Senko!

There are comments.

Conditional Include in Jinja2 and Pelican

How do you create Jinja templates which behave differently based on what Pelican plugins are loaded ? I've hit this problem while working on improving Flex PR #20. The straight forward solution looks like this

{% for name in PLUGINS if name == 'assets' %}
    {% assets "stylesheet/style.css", filters="cssmin", output="style.min.css" %}
        <link rel="stylesheet" type="text/css" href="{{ SITEURL }}/{{ ASSET_URL }}">
    {% endassets %}
{% else %}
    <link rel="stylesheet" type="text/css" href="{{ SITEURL }}/{{ THEME_STATIC_DIR }}/stylesheet/style.min.css">
{% endfor %}

It appears to work great except when the assets plugin isn't loaded. Then we get a TemplateSyntaxError:

TemplateSyntaxError: Encountered unknown tag 'assets'.
Jinja was looking for the following tags: 'endfor' or 'else'.
The innermost block that needs to be closed is 'for'.`

The {% assets %} tag is not defined because the assets plugin is missing. We may expect that Jinja will parse this tag only if the body of the for loop is executed but instead Jinja tries to parse all tags before rendering the template. The solution, as proposed by @ThiefMaster on GitHub, is to use a conditional {% include %} tag inside the for loop body like so

{% for name in PLUGINS if name == 'assets' %}
    {% include 'assets.html' %}
{% else %}
    <link rel="stylesheet" type="text/css" href="{{ SITEURL }}/{{ THEME_STATIC_DIR }}/stylesheet/style.min.css">
{% endfor %}

Where assets.html looks like this

{% assets "stylesheet/style.css", filters="cssmin", output="style.min.css" %}
    <link rel="stylesheet" type="text/css" href="{{ SITEURL }}/{{ ASSET_URL }}">
{% endassets %}

I don't like splitting out the HTML code in this way. Imagine that we later decide to add a second CSS file to the template. The risk of forgetting to add it in both places increases as the number of CSS files (or places where we use the same HTML pattern) increases. However this appears to be the only way to conditionally use the assets plugin only when it is loaded. The proposed changes are in Flex PR #40.

I hope you like my work and please subscribe to Mr. Senko if you need a faster release cycle for the open source libraries you use.

There are comments.

April 2016 Report

Here is a quick status report of my work for Mr. Senko. During April 2016 I've worked on:

Nitrate

As I've written previously Nitrate is a test case management system, which we use internally. Unfortunately it is using very old version of Django and it is not so easy to migrate forward.

I've decided to stall all of my previous pull requests and focus on getting the tests up and running in Travis-CI before going forward. I've managed to fix a few of them locally but then hit a road block. The fact that the Nitrate team is very limited in capacity (only 1 person at the moment) isn't helping either. I will continue working on this but with lower priority.

New features for i18n_viz

i18n_viz is a very cool Ruby gem which lets your browse your own Rails application and visualize and edit your translatable strings. Think of it as a reversed lookup for translations. i18n_viz highlights your app’s translatable text and adds a tooltip containing the translation key which links to the translation in your favorite online translation tool.

I have added the css_override option to allow for better styling of the highlights and tooltips provided by this gem. PR #20 has already been merged but not released into a new version yet.

I've also fixed a bug with nested HTML tags and changed the external_tool_url option to support code execution. With this the user is able to configure the URL to an online translation tool during runtime. My particular use-case is to include the currently selected locale in the URL string. These two pull requests are not merged yet and unfortunately the package author is busy at the moment so it will take some time.

New features for Pelican

I've managed to implement more granular control over tag, categories and author slugs in Pelican in PR #1926 and the code is already merged!

Under the hood the slugify() method now can skip replacing non-alphanumeric characters so you have more control over the generated URLs. I've experienced this problem when migrating my personal blog from Octopress to Pelican.

Then I've also added the possibility to configure author slugs, which is useful to blogs with multiple authors or if you want your author slug to match your GitHub username. In fact pelicanconf.py for Mr. Senko's blog looks like this:

ARTICLE_URL = 'blog/{author}/{date:%Y}/{date:%m}/{date:%d}/{slug}/'
ARTICLE_SAVE_AS = ARTICLE_URL + 'index.html'

AUTHOR_URL = 'blog/{slug}/'
AUTHOR_SAVE_AS  = AUTHOR_URL + 'index.html'

AUTHOR_SUBSTITUTIONS = [
    ('Alexander Todorov', 'atodorov'),
    ('Krasimir Tsonev',   'krasimir'),
]

NOTE: older URLs on this blog do not match above settings for compatibility reasons!

I hope you like my work and please consider subscribing to Mr. Senko! if need a faster release cycle for the open source libraries you use.

There are comments.

March 2016 Report

Here is a quick status report for Mr. Senko. During March 2016 I've worked on:

Nitrate Test Case Management System

Nitrate is a test case management system written with Django. We use it internally to track testing activities. It is reasonably good at what it does but there are some issues with how the software has been developed in the past.

If you like my work and need a faster response cycle please consider subscribing to Mr. Senko!

There are comments.

Why Your Open Source Bugs Will Not Be Fixed

Here at Mr. Senko we've been researching why many bugs reported against open source projects don't get fixed. This is not surprising given that many contributors to open source use the "scratch your own itch" principle. We've been talking to various people why some bugs are left open. We've reached both open source maintainers and users and here are the results.

Time is Code

The core of the problem revolves around time constraints. Open source developers are limited by the time they have to work on the project, even more so if they are volunteers. Their focus is on critical and high priority issues and driving the project through its roadmap and they don't have enough time for you. The side effect of this is that many smaller issues are dealt with only if there is free time left. Some factors to consider here:

You see where this is going - the more problems there are, the less time there is to fix them and the problems pile up even more, shrinking the available time as we go along.

Larger and more popular projects tend to exhibit more severe problems arising from the above factors. Of the top 1000 Python, Ruby and Node.js packages the average time an issue spends between open and closed is around 30 days, some projects going to the extremes of 100s of days. All of this because the developer to user ratio in these projects is too low. On the other hand smaller communities are usually more responsive and easier to work with. From our experience single developer projects tend to respond within 2 days on average.

How to Help

Leadership, Management and Politics

Less frequently cited problems are in the fields of leadership, project management and community politics. On the extreme end here we see projects which are not under active development anymore but are still widely adopted by the general public. This is the case with some of Node.js most downloaded packages. In the same category are also projects which do have some development community(possibly with write access) but it doesn't quite understand what the source code is doing and is not able to promptly fix bugs or is unwilling to do so.

How to Help

Management and politics are often a problem in large and popular projects where every developer has their own vision of how the project should evolve and all of these visions need to be brought together and steered into a single direction. In projects which are corporately sponsored there are trusted developers who perform these tasks. In bigger projects there could even be a management board who decides on where the project is going next and how developer resources will be best utilized. For projects backed by a foundation this is certainly the case.

How to Help

Provide a management capacity who will serve on the project's board. That person will help raise visibility to your particular issues but their general task will be to help the project move forward. If they fail to do so the relationship will simply not work. Some projects have a fixed term for board members to prevent a single entity overtaking control of the project.

Lorem Ipsum

Being humans developers are not prone to simple facts that make their work non-effective or slow.

We've been told that some developers don't understand the bug-fix or more likely the software development lifecycle. It sounds like we're talking about inexperienced developers who managed to create a popular open source project alone. Or maybe we're talking about developers who come from the start-up scene while users are coming from the enterprise world and are used to working in a different manner ? Either-way both parties have to learn from each other and not allow tempers to rise too much.

Others have claimed that some developers don't know how to ask for outside help or are reluctant to give up control of their projects to external contributors. We don't have data about how often this happens but apparently often enough to be visible.

How to Help

As a user (or developer) clearly communicate your intentions and expectations. If you feel the other party doesn't quite understand you explain where are you coming from. It is possible that you working environment and mental processes are quite alien to them and they don't see the point in your words. If there is a local technology user group or a near-by event then go meet the other guy and buy them a beer.

Why Should You Care

If you are still reading then chances are you've experienced some of these problems yourself. We certainly have seen quite a few from the list above. The bottom line is that whenever you have problems with a particular open source project don't expect somebody to help you. Open source comes at a price and it is being able to help yourself or fix your own problems. Kind of like buying a new car without a warranty and a mechanic.

Practical example - Sphinx

Sphinx PR #1902 introduces new features to the inheritance diagram plugin but also adds new tests for the existing code base as well as new tests for the proposed modifications. 6 months later and the PR is still open, not even undergone code review. Later in commit 4c4450 the core devel team changes a parameter type in one of the functions used by the inheritance diagram plugin. Not having any tests for the caller code they make a backward incompatible change which is still broken in the master branch. For the fix see commit d67fee.

Practical example - dnf-plugins-core

The dnf download command in Fedora had a test suite which used badly designed test stubs. They have been redesigned in PR #118 but developers requested the refactoring be merged together with PR #113 which also adds new functionality. 2 months afterwards comes commit fe1306 which changes behavior in the same plugin. The badly designed test stubs ignored the change silently and it landed in the master branch. The newly redesigned tests broke immediately but they were still under review and not merged until 1 month later.

Many developers and companies consuming open source software are OK (or OK-ish) with this state of art. They've accepted that they have to fix their own problems or work around them when the time comes. This is how the open source is expected to work, isn't it ?

If you happen to be from those folks who don't have enough technical knowledge to work on a particular project or don't have the time to do so please consider subscribing to Mr. Senko! We're happy to provide open source support to those who need it.

There are comments.

January 2016 Report

This is my first status report for Mr. Senko. During January 2016 I've worked on:

Sphinx broke

The Sphinx changes are particularly interesting - the fix for issue #656 in commit 4c4450d changes the Graphviz's node['options'] type from list to dict which in turn breaks html_visit_inheritance_diagram() and the inheritance_diagram extension.

My previous work on this PR introduced some basic tests, which were missing and they caught the type change! Because the tests and my latest changes are not yet merged into master the inheritance_diagram.py extension is still broken! The tests fail in Travis-CI but probably due to dot not being installed. Locally everything seems to work.

Its also worth mentioning that it's been 2 weeks since I've updated my PR and a bit more since Sphinx introduced the regression. As far as I can see there's not been any updates with respect to this. If you'd like to get a faster response cycle please consider subscribing to Mr. Senko!

There are comments.

FOSDEM 2016 Report

Hello everyone, Mr. Senko flew to Brussels for his first public appearances and we are very excited about this. He visited FOSDEM to get an update on the latest news in open source and scored the highest ping-pong score for BBC Open Source on Sunday! He also learned a bit about Elixir with the Belgian Elixir Study group, attended a panel on open source and start-ups organized by The Startup Bus and made lots of new friends. Here are some of the details and links for further reading.

Ping-pong with computers

Friday - Day 0

Friday was a fruitful day for Mr. Senko. It started with a meeting at BetaCowork - one of the co-working spaces in Brussels and continued with a visit to Brussels Erlang Factory Lite where we learned a bit about Erlang and Elixir and even managed to create a small pull request as part of the exercises. Funny enough, one of the organizers, Yuri Leikind was spot on to recognize our Bulgarian accent.

BetaCowork was bursting with people who've come for FOSDEM. There was a GNOME team and the Libre Office Italian team! The place is huge and is definitely an alternative for the entrepreneurs who are tired of working from home! It is also very close to ULB so next time you visit FOSDEM definitely check their events schedule as well.

Saturday - Day 1

Saturday was the Testing and Automation day. The morning track showed us Frida - a tool which injects JavaScript into existing processes and could be used for testing interoperability with closed source applications. Another interesting use case is overriding the standard read and write functions in order to automate testing of interactive text based applications on Linux. Mr. Senko had the pleasure talking to Frida's authors on both days. These are extremely tallented and knowledgeable folks so definitely keep an eye on their work.

In the afternoon the room got packed when Marcin and Lukasz presented their experience with Jenkins Job DSL plugin. The plugin is very flexible and powerfull, using Groovy as its programming language. The only drawback is that it is sometimes too complex to use and requires a steep learning curve. Regardless of this the only tools we know about related to Jenkins as Code are the Jenkins Job DSL and Jenkins Job Builder plugins.

In the evening Mr. Senko visited a panel talk about the impact of open source on the tech industry at the second big co-workign space in Brussels - Co.Station. The place is in the city center so definitely worth to visit at any time. Mr. Senko made quite a few new friends there and possibly found potential pilot customers. We've also made contact with the organizers of Startup Bus Europe and hopefully we'll see the bus coming to Sofia as well. And we also got invited to the first Angular.js conference in Belgium and with a bit of luck you'll see a talk from us in the summer.

Sunday - Day 2

Sunday was the lazy day. The only talk of interest to Mr. Senko was Tweaking Ruby GC parameters for speed and profit . The rest of the time was spent networking, collecting t-shirts, reporting a bug against Fedora Account System and meeting with old friends.

hammer down

In the early afternoon Mr. Senko played a few geeky games at the BBC Open Source stand. BBC is developing the micro:bit - a small computer which is intended to teach programming to small children. With its help they has built a ping-pong device which counts how many times you hit the ball.

In the second game you had to smash a red circle with a foam hammer as soon as it lighted up. Mr. Senko managed to win the first game and then lost the second one. He even got to test the device and tried to crash it by scoring penalty points when the counter was still zero. BBC definitely did their job right because the device didn't crash.

Thank you for taking the time to read this report. See you at the next event!

There are comments.

Founding of Mr. Senko

Hello everyone, welcome to Mr. Senko! We are a group of friends who have been working with open source for the last 10 years. During this time we've seen technologies come and go and hype projects becoming abandoned months later. We've seen the pain when companies had to rewrite their software because of this and the struggle to get your patches accepted upstream. Don't you wish you had somebody who could take care of all the hassle for you?

Mr. Senko is our attempt at providing commercial support for open source projects, specifically libraries which other companies use to build their products. As of now we have diverse experience in Java, Python and JavaScript as well as my personal experience with Quality Assurance.

In the next weeks and months we will be refining our value proposition and building the team so we can offer you great technical support and expertise. We plan to blog regularly about our progress and about technology and open source events in general! You can follow us at @Mr_Senko or tweet at the wizard using the #MrSenko hashtag!

Stay tuned for more news!

There are comments.