Nitrate enhancement update

Version: nitrate-3.8.18.08 Released on: 2017-02-21 Severity: High

Details

Nitrate is an open source test plan, test run and test case management system. It has a lot of great features, such as Bugzilla and JIRA integration, QPID integration, fast test plan and runs search, powerful access control for each plan, run and case, and XMLRPC APIs.

Updated Nitrate package is now available that adds various enhancements.

Changelog since 3.8.18.07

How to upgrade

Subscribe to Mr. Senko and configure your private PyPI repositories before upgrading!

There are comments.

Nitrate enhancement update

Version: nitrate-3.8.18.07 Released on: 2017-02-14 Severity: Medium

Details

Nitrate is an open source test plan, test run and test case management system. It has a lot of great features, such as Bugzilla and JIRA integration, QPID integration, fast test plan and runs search, powerful access control for each plan, run and case, and XMLRPC APIs.

Updated Nitrate package is now available that adds various enhancements.

Changelog since 3.8.18.05

How to upgrade

Subscribe to Mr. Senko and configure your private PyPI repositories before upgrading!

There are comments.

Nitrate enhancement update

Version: nitrate-3.8.18.05 Released on: 2017-01-31 Severity: Low

Details

Nitrate is an open source test plan, test run and test case management system. It has a lot of great features, such as Bugzilla and JIRA integration, QPID integration, fast test plan and runs search, powerful access control for each plan, run and case, and XMLRPC APIs.

Updated Nitrate package is now available that adds various enhancements.

Changelog since 3.8.18.04

How to upgrade

Subscribe to Mr. Senko and configure your private PyPI repositories before upgrading!

There are comments.

Nitrate bug fix and enhancement update

Version: nitrate-3.8.18.04 Released on: 2017-01-24

Details

Nitrate is an open source test plan, test run and test case management system. It has a lot of great features, such as Bugzilla and JIRA integration, QPID integration, fast test plan and runs search, powerful access control for each plan, run and case, and XMLRPC APIs.

Updated Nitrate package is now available that adds various enhancements and bug fixes.

Changelog

How to upgrade

Subscribe to Mr. Senko and configure your private PyPI repositories before upgrading!

There are comments.

Using private PyPI repositories

Once you have subscribed to Mr. Senko you will receive a repository URL from which to install package versions supported by Mr. Senko. The URL is of the form https://d1v12p11qcbqzw.cloudfront.net/YOUR-CUSTOMER-ID

You must keep this URL secret! You can install supported packaged by either command line:

pip install nitrate --extra-index-url https://d1v12p11qcbqzw.cloudfront.net/YOUR-CUSTOMER-ID

or inside of your project by adding the following statement to requirements.txt:

--extra-index-url https://d1v12p11qcbqzw.cloudfront.net/YOUR-CUSTOMER-ID

You can also configure your ~/.pip/pip.conf file to look like this

[global]
; Extra index from Mr. Senko
extra-index-url = https://d1v12p11qcbqzw.cloudfront.net/YOUR-CUSTOMER-ID

Another possibility is to configure the private repository URL directly into setup.py:

setup(
    name='YourAppName',
    install_requires=[
        'strazar',
    ],
    dependency_links=[
        'https://d1v12p11qcbqzw.cloudfront.net/YOUR-CUSTOMER-ID',
    ],
)

And you're ready to seamlessly pip install private packages from Mr. Senko.

What is Mr. Senko?

Mr. Senko provides long term support for various open source libraries. We do bug fixes and feature development, improve testing and write documentation so you don't have to! Subscribe to Mr. Senko if you need faster response cycle for the open source components which you use.

There are comments.

What I've worked on in December 2016

Hello everyone, during December I've been working on several different projects from our supported Python software stack and also ventured into Ruby land.

Versions 0.2.3 and 0.2.4 of pelican-ab

pelican-ab is an A/B testing plugin for Pelican, the static site generator for Python. It allows you to encode experiments in your templates and renders the experiment selected by the AB_EXPERIMENT environment variable.

There were two versions released in December 2016 making it compatible with the newly released Pelican 3.7.0, adding more tests and fixing some minor bugs.

Version 0.2.8 of django-chartit

I have worked on a new version for django-chartit which adds support for model properties and raw SQL queries in charts. I've also done internal work for further refactoring but without anything concrete yet!

New extensions for pylint

I've worked on the new compare-to-empty-string extension for pylint. Using this extension the following snippet

if S != "":
    pass

if S == '':
    pass

will be flagged as problematic because it can be written in a more natural way:

if S:
    pass

if not S:
    pass

The extension will be disabled by default because empty string may be a valid value depending on the behavior of your program. I have also created a similar extension to flag integer comparisons against zero, see pylint#1243. While these may seem a bit extreme they are very useful to identify code which can be refactored to reduce comparisons when coupled with mutation testing!

New Rspec formatters

I've worked on two new formatters for Rspec:

Both add simple syntactic sugar to how Rspec output can be formatted and make it easier for debugging or sending off for further machine processing.

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Using model properties and raw SQL with django-chartit

django-chartit is a Django app that can be used to easily create charts from the data in your database. The charts are rendered using Highcharts and jQuery JavaScript libraries. Data in your database can be plotted as simple line charts, column charts, area charts, scatter plots, and many more chart types.

Few days ago we have released version 0.2.8 which includes support for model properties and raw SQL queries in your charts. This blog post will describe how to use them.

Using model properties

Sometimes it is convenient to calculate a value inside your models but not store it in the database. For example consider the following model

class City(models.Model):
    city = models.CharField(max_length=50)
    state = models.CharField(max_length=2)

    def region(self):
        return 'USA:%s' % self.city

You are not able to select the region "field" but you may use it as part of your chart terms as shown below

ds = DataPool(
        series=[{
            'options': {
                'source': SalesHistory.objects.only(
                            'bookstore__city', 'sale_qty'
                          )[:10],
            },
            'terms': [
                'bookstore__city__region',
                'sale_qty'
            ]
        }]
)

cht = Chart(
        datasource=ds,
        series_options=[{
            'options': {
                'type': 'column',
                'stacking': False,
                'stack': 0,
            },
            'terms': {
                'bookstore__city__region': [
                    'sale_qty'
                ]
            }},
        ],
        chart_options={
            'title': {
                'text': 'Sales reports'
            },
            'xAxis': {
                'title': {
                    'text': 'City'
                }
            }
        }
)

The full example, including source code and live charts is available at django-chartit/demoproject!

Using raw SQL queries

Django allows you to execute SQL queries directly. These will return objects in the form of RawQuerySet and can be used in the same way as any other QuerySet.

ds = DataPool(
        series=[{
            'options': {
                'source': MonthlyWeatherByCity.objects.raw(
                            "SELECT id, month, houston_temp, boston_temp "
                            "FROM demoproject_monthlyweatherbycity")
            },
            'terms': [
                'month',
                'houston_temp',
                'boston_temp'
            ]
        }]
)

You will have to select the primary key field and pay attention to field names as to avoid duplicates when performing JOIN operations. Otherwise there is no difference. Full examples, including source code and live charts are available at django-chartit/demoproject!

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

New release pelican-ab 0.2.3

pelican-ab is an A/B testing plugin for Pelican, the static site generator. It allows you to encode experiments in your templates and renders the experiment selected by the AB_EXPERIMENT environment variable.

Today we are releasing version 0.2.3 as an urgent bug fix update handling internal changes in the latest version of Pelican. In previous versions Pelican used the JINJA_EXTENSIONS setting to configure additional Jinja2 modules. In version 3.7.0, released yesterday, JINJA_EXTENSIONS has been replaced with the JINJA_ENVIRONMENT setting. Version 0.2.3 of pelican-ab handles both cases and is able to work with older and newer versions of Pelican!

This backward-incompatible change in Pelican has been automatically detected by our monitoring tool Strazar, which proves the value of automatic testing against the latest versions of upstream dependencies!

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

What I've worked on in November 2016

Hello everyone, during November I've been focusing on Cosmic Ray, pylint and django-chartit which are supported packages in our Python software stack.

The changes include

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Status Report for October 2016

Hello everyone, during October I've been focusing on Cosmic Ray which is a supported package of our Python software stack.

The changes include

All of my work this month was around visit_mutation_site() being able to generate multiple mutations at a single site. For example the code

a < b

can be mutated into:

a <= b
a is not b
a > b
... etc

Previously this was done using different mutation operators but now it is done using a single class and an index to the desired operator replacement. As a follow up other mutation operators had to be updated as well. As a result the code base is cleaner and easier to understand.

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Status Report for September 2016

Hello everyone, during September I've been focusing on Cosmic Ray which is a supported package of our Python software stack.

The changes include better error checking when running the test suite in Travis CI, fixes for some code smells and style updates, some Python 3.3 compatibility fixes, traceback fix for Python 3.4, new and-or replacement operators and internal refactoring.

During my work on and-or replacement operators it turned out that Cosmic Ray didn't support the notion of one operator producing multiple code mutations. This required refactoring of project internals and is now currently supported. As a follow up there are a couple more issues opened to clean up the existing code, mainly the comparisons replacement operators.

NOTE Cosmic Ray is still not compatible with Python 3.3 and will probably never be, despite my work. In cosmic_ray/importing.py we make use of importlib.machinery.ModuleSpec which was introduced in 3.4 and at this moment we don't want to backport and support this for Python 3.3. The rest of the code is 3.3 compatible though.

During September I have also worked on django-chartit v0.2.7 to fix a nasty recursion loop bug. The follow up of this bug is pylint #1109, which is now merged into master!

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Beware of recursion loop when using super()

When working with class inheritance in Python you often find yourself using super() to call the parent class methods. The format is

super([current class name], self).[base class method]([arguments])

When you have lots of similar code you may be tempted to short-cut the writing of current class name. The two possibilities are

super(type(self), self).some_method()
super(self.__class__, self).some_method()

I have seen this in production code. I'm guilty myself as well because I like to use the self.__class__ notation! This leads to problems, see django-chartit #41, when you inherit from the class which uses these shortcuts. For example

class Base(object):
    def method(self):
        print 'original', type(self), self.__class__

class Derived(Base):
    def method(self):
        print 'derived', type(self), self.__class__
        super(type(self), self).method()
        # super(self.__class__, self).method()

class Subclass(Derived):
    def method(self):
        print 'subclass of derived', type(self), self.__class__
        super(Subclass, self).method()

When you call Derived().method() the result is

derived <class '__main__.Derived'> <class '__main__.Derived'>
original <class '__main__.Derived'> <class '__main__.Derived'>

Here both shortcuts are evaluated correctly. However when you call Subclass().method() the result becomes

subclass of derived <class '__main__.Subclass'> <class '__main__.Subclass'>
derived <class '__main__.Subclass'> <class '__main__.Subclass'>
derived <class '__main__.Subclass'> <class '__main__.Subclass'>
derived <class '__main__.Subclass'> <class '__main__.Subclass'>
... skip ...
RuntimeError: maximum recursion depth exceeded while calling a Python object

In the example the call super(Subclass) works fine and invokes Derived.method() as expected. Then we call super(Subclass) inside Derived.method() which leads back to where we were hence the recursion loop. The problem is only visible when you have other classes that inherit from a class which uses incorrect invocation of super(). This is why it may lay hidden in production software!

I definitely expected a severe code smell like that to be discovered by pylint and Landscape.io. Indeed older versions (pylint-1.3.1-1.el7.noarch) will report error for super(type(self)) but not newer ones. As it turns out pylint didn't have a test for this condition and have introduced a regression in the master branch. I believe this is due to that fact that they didn't check for using type() directly but rather that was a side effect which ceased to exist once the code was updated. Pylint#1109 adds tests for the two code smells described above and updates the checkers to explicitly detect them! Happy testing!

I hope you like my work and please subscribe to Mr. Senko's support service should you need commercial support for this or other open source libraries!

There are comments.

New release django-chartit 0.2.7

django-chartit is a Django app that can be used to easily create charts from the data in your database. The charts are rendered using Highcharts and jQuery JavaScript libraries. Data in your database can be plotted as simple line charts, column charts, area charts, scatter plots, and many more chart types.

Today we are releasing version 0.2.7 as part of our regular update process. This is an urgent bug fix update because previous versions broke subclassing of Chart and PivotChart classes.

Changelog since version 0.2.6

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

August 2016 Status Report

Hello everyone, during August I've been focusing on django-chartit which is a supported package of our Python software stack.

Since I've taken over maintenance from Praveen Gollakota there were several bug-fix releases, 3 of them in August. The result is 6 closed issues and increased test coverage. I've spent time to refactor parts of the code and make it compatible with the latest Django version. Also cleaned up code smells identified by Landscape.io.

Future plans for django-chartit include working on the remaining open issues. I think I will have to refactor the code a lot more to make it compatible with the latest Highcharts.js API before being able to implement the requested features.

I have also spoken with Highcharts engineering on the topic of testing their JavaScript charting code in the context of django-chartit. My idea is to load the demo project using various versions of the JavaScript library and make sure everything works on the client side. This appears to be doable with Selenium.

I hope you like my work and please subscribe to Mr. Senko if you need faster response cycle for the open source libraries you use.

There are comments.

Loading initial data for Many-To-Many fields

Previously I've written about how to use JSON fixtures in Django migrations. This becomes a bit more complicated when you have ManyToMany fields in your models. A corner case is when you have a ManyToMany relation to self. The example below comes from django-chartit.

class Book(models.Model):
    title = models.CharField(max_length=50)
    rating = models.FloatField()
    rating_count = models.IntegerField()
    authors = models.ManyToManyField(Author)
    publisher = models.ForeignKey(Publisher, null=True, blank=True, on_delete=models.SET_NULL)
    published_at = models.DateTimeField(null=True, blank=True)
    related = models.ManyToManyField('self', blank=True)
    genre = models.ForeignKey(Genre, null=True, blank=True, on_delete=models.SET_NULL)

The fields authors and related are represented as separate tables and are computed when you access objects from this model. Django automatically handles these fields and creates classes for them. Before you can use them, you need a reference to their model classes.

Book = apps.get_model("demoproject", "Book")
BookRelated = None
BookAuthors = None
for relation in Book._meta.many_to_many:
    if relation.name == 'related':
        try:
            BookRelated = relation.remote_field.through
        except AttributeError:
            # available in Django 1.8
            BookRelated = relation.rel.through

    if relation.name == 'authors':
        try:
            BookAuthors = relation.remote_field.through
        except AttributeError:
            # available in Django 1.8
            BookAuthors = relation.rel.through

The JSON data looks like this

{ "fields" : {
    "authors" : [  ],
    "genre_id" : 4,
    "publisher_id" : 3,
    "rating" : 3.8999999999999999,
    "rating_count" : 1869,
    "related" : [ 10 ],
    "title" : "Freakonomics"
  },
"model" : "Book",
"pk" : 9
},
{ "fields" : {
    "authors" : [ 24 ],
    "genre_id" : 5,
    "publisher_id" : 9,
    "rating" : 4.4000000000000004,
    "rating_count" : 222,
    "related" : [ 23, 21 ],
    "title" : "Hyperspace"
     },
"model" : "Book",
"pk" : 24
},

Once we have our related model classes we proceed to store the data in the database like so

# create Book objects
for record in json_data:
    # skip everything which isn't a book
    if record['model'] != 'Book':
        continue

    # build a list of book authors using the intermediate BookAuthors model
    for author_id in record['fields']['authors']:
        author_obj = BookAuthors()
        author_obj.book_id = record['pk']
        author_obj.author_id = author_id
        author_obj.save()
    # you can't save the `authors` field directly in DB
    del record['fields']['authors']

    # build a list of related books using the intermediate BookRelated model
    for related_id in record['fields']['related']:
        related_obj = BookRelated()
        related_obj.from_book_id = record['pk']
        related_obj.to_book_id = related_id
        related_obj.save()
    # you can't save the `related` field directly in DB
    del record['fields']['related']

    # finally save the Book object
    model_class = apps.get_model("demoproject", record['model'])
    obj = model_class(**record['fields'])
    obj.pk = record['pk']
    obj.save()

This works well for django-chartit. You should take care to remove the ManyToMany fields from the JSON data because they don't actually exist in the Book class and Django will raise an exception if you try to assign to them.

I hope you like my work and please subscribe to Mr. Senko's support service should you need commercial support for this or other open source libraries!

There are comments.

New release django-chartit 0.2.6

django-chartit is a Django app that can be used to easily create charts from the data in your database. The charts are rendered using Highcharts and jQuery JavaScript libraries. Data in your database can be plotted as simple line charts, column charts, area charts, scatter plots, and many more chart types.

Today we are releasing version 0.2.6 as part of our regular update process.

Changelog since version 0.2.4

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

Converting JSON Fixtures to Django Migrations

Older Django apps like django-chartit and Nitrate used JSON fixtures to populate their databases with initial data. In this article I will show you an easy way to convert JSON fixtures into native Django migrations. The JSON fixture looks like this

{"fields": {"first_name": "Seth",
            "last_name": "Godin"
            },
 "model": "demoproject.Author",
 "pk": 1
 },
{"fields": {"first_name": "Guy",
            "last_name": "Kawasaki"
            },
 "model": "demoproject.Author",
 "pk": 2
 },
{"fields": {"first_name": "Geoffrey",
            "last_name": "Colvin"
            },
 "model": "demoproject.Author",
 "pk": 3
 }

Notice the pk and model fields which tell us where this data came from and what was the object PK when exported from the database. The fields dict is the actual data for this object.

In Python we can use json.loads and read the fixture data from disk or even better assign it directly to a variable inside our Python source file. Then iterate over all values and create the objects programmatically like this

from __future__ import unicode_literals
from django.db import migrations


def initialize_data(apps, schema_editor):
    data = [
        {"fields": {"first_name": "Seth",
                    "last_name": "Godin"
                    },
         "model": "demoproject.Author",
         "pk": 1
         },
        {"fields": {"first_name": "Guy",
                    "last_name": "Kawasaki"
                    },
         "model": "demoproject.Author",
         "pk": 2
         },
        {"fields": {"first_name": "Geoffrey",
                    "last_name": "Colvin"
                    },
         "model": "demoproject.Author",
         "pk": 3
         },
    ]

    for record in data:
        app_name, model_name = record['model'].split('.')
        ModelClass = apps.get_model(app_name, model_name)
        obj = ModelClass(**record['fields'])
        # this is required only if you have other models
        # with foreign keys referring to this object
        # obj.pk = record['pk']
        obj.save()

class Migration(migrations.Migration):

    dependencies = [
        ('demoproject', '0001_initial')
    ]

    operations = [
        migrations.RunPython(initialize_data),
    ]

This works well for most of the cases. You should take care to assign the same PKs in case there are other objects that hold references to them. If this isn't the case then you can drop these fields entirely to reduce your source code size.

I hope you like my work and please subscribe to Mr. Senko's support service should you need commercial support for this or other open source libraries!

There are comments.

Reviving django-chartit with version 0.2.4

django-chartit is a Django app that can be used to easily create charts from the data in your database. The charts are rendered using Highcharts and jQuery JavaScript libraries. Data in your database can be plotted as simple line charts, column charts, area charts, scatter plots, and many more chart types.

The project has been originally developed by Praveen Gollakota and had gained some popularity. Recently Mr. Senko was granted the necessary access rights and we've revived the project with two upstream releases.

Changelog

Version 0.2.3 was released a few days ago and merges the django-chartit2 fork by Grant McConnaughey which had a few releases of its own earlier this year. It also makes a few improvements and merged other pull requests.

Version 0.2.4 was released today and fixes the usage of deprecated Django APIs which were removed in Django 1.10.

Many thanks to Grant for his work on Python 3 and latest Django support! Many thanks to Praveen for letting us maintain the project.

Upgrade from django_chartit2

If you are using the django_chartit2 module by Grant then you have to

pip uninstall django_chartit2
pip install django_chartit

If you are not in a hurry to upgrade you may as well wait for the next release by Grant, which should automatically require django_chartit and transparently switch you back to using the original package and not the fork. However release of such an update is not under our control.

Feature plans

Mr. Senko is currently working actively to bring the rest of the project up to speed, including the demo site, which shows how to use the API. This is a mandatory step before we go ahead to work on various bug fixes and documentation improvements. We will also be looking into adding more tests for the project.

Support

At Mr. Senko we will do our best to accommodate every need and merge patches and feature requests as they come in. Should you need commercial support for this or other open source libraries please subscribe to Mr. Senko's support service!

There are comments.

July 2016 Status Report

Hello everyone, July has been relatively busy in terms of pull requests. I've been working on Cosmic Ray the mutation testing tool for Python and in the last couple of days taken over maintenance of django-chartit.

I've made several improvements against Cosmic Ray. The most notable are:

I will be continuing to work on Cosmic Ray and also integrate mutation testing as part of the standard testing toolset used by Mr. Senko.

Then I've started working on django-chartit which is a very popular module that has been abandoned in the last couple of years. My immediate goal was to merge back the django-chartit2 fork by Grant McConnaughey, which adds Python 3 and latest Django support and merge some pending pull requests. I've been working on fixing quite a few errors and warnings reported by the test tools and getting the documentation up to speed. A much anticipated release on PyPI will be coming out very soon. Once the merger between django-chartit and django-chartit2 is complete I will continue working on the reported issues.

I hope you like my work and please subscribe to Mr. Senko if you need a faster response cycle for the open source libraries you use.

There are comments.

June 2016 Status Report

Hello everyone, last month has been a bit of a holiday and I don't have much to share. I've been playing around with mutation testing tools. The few patches I did are both related to Cosmic Ray:

Right now I'm experimenting with running mutation testing against a few popular open source libraries, written in both Python and Ruby. So far the tools seem to work fine, except minor issues. However the test suites I've been playing with aren't very robust and most of the mutants stay alive! I will write another blog post on the subject once I have more information to share. I hope you like my work and please subscribe to Mr. Senko! if need a faster response cycle for the open source libraries you use.

There are comments.