Andrey Khavryuchenko himself

Python transactional memory?

Ok, nosetests whitewashed in one of accused misactions - it didn’t actually messed up with the metadata of my django database backend. It was me, myself.

Looks like my test wrappers db_mock and twill_mock where not quite up to date to latest django db backend changes and forgot to save their environment cleanly and then - even restore it properly.

Now the question - is there some tool or module for a python that allows to handle global variable state in a transactional manner? A kind of lightweight STM?..

I understand that it could hardly be possible for C modules, but I’m asking just for native python variables.

Any ideas?

Posted in Microentry, Django, Python | No Comments »

satchmo multi-shop

Good to see that satchmo multishop branch is merged into the trunk. Bad is that this is not reflected on the MultiShop wiki page - had to waste some time and even try to start a fork on multishop branch to update it to django 1.0…

Posted in Microentry, Django | No Comments »

nosetests smashing module content

Image via Wikipedia Spent several hours yesterday untangling yet another mysterious test crash:

Warning: Field 'id' doesn't have a default value

As appears, it happened during usual INSERT to djangocontenttype… WTF!?

mysql> describe django_content_type;
+-----------+--------------+------+-----+---------+-------+
| Field     | Type         | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+-------+
| id        | int(11)      | NO   | PRI | NULL    |       |
| name      | varchar(100) | NO   |     | NULL    |       |
| app_label | varchar(100) | NO   | MUL | NULL    |       |
| model     | varchar(100) | NO   |     | NULL    |       |
+-----------+--------------+------+-----+---------+-------+
4 rows in set (0.01 sec)

Where the #$^ AUTOINCREMENT is?

To make long story short, it appeared that for some strange reason django.db.connection.creation.data__types contains the values for sqlite that we use in previous tests and not mysql backend that is reported to be used.

And, yes, isolation plugin of nosetests doesn’t help.

Posted in Microentry, Python | No Comments »

How to add svn tracking to existing git project

Image via Wikipedia Tracking svn repository with a git is a pleasure, thanks to git-svn:

git svn clone (svn url)

and go.

But until today it wasn’t clear to me how to add an svn for already existing git project. For example, if a client requested to push the progress to his svn repo to be able to contribute himself.

It appears that it’s dead easy too:

git svn init (svn url)

works lovely in already existing git repository without bothering any of your git objects.

Repeat with me: git is good

PS Have an excellent and detailed introduction for git-svn integration.

Posted in Microentry, Software Development | 2 Comments »

nosetest mangles sys.path?

Struggling for a second hour in a row. Feeling dumber with every minute spent.

Code:

import pprint
import sys
import django.conf
assert django.conf.__file__.startswith(
    '/home/akhavr'
    ), pprint.pformat((sys.path, django.conf.__file__))

Test run:

PYTHONPATH=/home/akhavr/src/django-hotclub:/home/akhavr/django/:/home/akhavr/src/django-hotclub/apps:/home/akhavr/src/django-hotclub/site-packages/misc:/home/akhavr/src/django-hotclub/site-packages/python-openid-2.1.1:/home/akhavr/src/django-hotclub/site-packages/python-yadis-1.1.0:/home/akhavr/src/django-hotclub/site-packages/python-twitter-0.5/:/home/akhavr/src/django-hotclub/site-packages/pownce-api-0.2:.. DJANGO_SETTINGS_MODULE=pinax.settings nosetests --with-coverage --cover-package=projects --exe -x  projects
F
======================================================================
FAIL: Failure: AssertionError ((['/home/akhavr/src/django-hotclub/pinax/local_apps/projects/tests',
  '/usr/bin',
  '/usr/lib/python2.5/site-packages/coverage-2.78-py2.5.egg',
  '/usr/lib/python2.5/site-packages/twill-0.9-py2.5.egg',
  '/usr/lib/python2.5/site-packages/opml-0.5-py2.5.egg',
  '/usr/lib/python2.5/site-packages/geopy-0.93-py2.5.egg',
  '/usr/lib/python2.5/site-packages/PyCAPTCHA-0.4-py2.5.egg',
  '/usr/lib/python2.5/site-packages/flickrapi-1.1-py2.5.egg',
  '/usr/lib/python2.5/site-packages/simplejson-1.9.1-py2.5-linux-i686.egg',
  '/usr/lib/python2.5/site-packages/httplib2-0.4.0-py2.5.egg',
  '/usr/lib/python2.5/site-packages/icalendar-1.2-py2.5.egg',
  '/usr/lib/python2.5/site-packages/CouchDB-0.5dev_r88-py2.5.egg',
  '/usr/lib/python2.5/site-packages/django_mptt-0.2.1-py2.5.egg',
  '/home/akhavr/src/django-hotclub',
  '/home/akhavr/django',
  '/home/akhavr/src/django-hotclub/apps',
  '/home/akhavr/src/django-hotclub/site-packages/misc',
  '/home/akhavr/src/django-hotclub/site-packages/python-openid-2.1.1',
  '/home/akhavr/src/django-hotclub/site-packages/python-yadis-1.1.0',
  '/home/akhavr/src/django-hotclub/site-packages/python-twitter-0.5',
  '/home/akhavr/src/django-hotclub/site-packages/pownce-api-0.2',
  '/home/akhavr/src/django-hotclub/pinax/local_apps',
  '/usr/lib/python25.zip',
  '/usr/lib/python2.5',
  '/usr/lib/python2.5/plat-linux2',
  '/usr/lib/python2.5/lib-tk',
  '/usr/lib/python2.5/lib-dynload',
  '/usr/lib/python2.5/site-packages',
  '/usr/lib/python2.5/site-packages/Numeric',
  '/usr/lib/python2.5/site-packages/PIL',
  '/usr/lib/python2.5/site-packages/gtk-2.0',
  '/usr/lib/python2.5/site-packages/wx-2.8-gtk2-unicode'],
 '/usr/lib/python2.5/site-packages/django/conf/__init__.pyc'))

Why sys.path is ignored and django is loaded from the site-packages instead of my ~/src snapshot is beyond my understanding now.

Posted in Microentry, Python | 2 Comments »

django-hotclub AKA pinax svn externals

Have yet to think out how to handle those django-hotclub numerous svn:externals. Git advise given here is nice (I’m following the same route), but it doesn’t answer the issue of managing the regular single-command deployment of those dependencies.

On the other hand, packaging each of these packagelets into tiny RPMs (or debs) feels like an overkill. Even if ALT gear would follow and build them automatically.

Food for thoughts..

Posted in Microentry, Software Development | No Comments »

Integration using git and buildbot

Another daily task here is somewhat opposite to autofollowing an open source project.

As I’ve mentioned there, we develop our modifications in git branches. This also stands for commercial projects. And, of course, our customers expect us to integrate our modifications back into their codebase.

Since I’m the integration point, I spend most of my time watching how tests pass:

• full test suite on the feature branch, run by buildbot
• git merge --no-commit with the feature branch from the main branch; if we’re following the svn project, we use also --squash to not mess the git-svn logic
• run the full test suite again, this time run manually
• commit with a reasonable message and push the change upstream

If both tests pass ok, the only point where manual intervention may be actually necessary is the step when the commit message gets written.

Definitely the next thing to be scripted.

Posted in Software Development | No Comments »

Autofollowing an open source project

Looks like the next biggest time eater (NBTE? ;)) during our development are the efforts to follow public projects repositories.

See, we develop web software mostly by integrating already available open source components. But what if a component is not yet stable and is actively developed?

Also, we often have to branch these projects to implement our own modifications: bug fixes or new features. Thanks to git that’s easy, but what about keeping track on the development of the original project?

Let the ‘master’ branch follow an svn and local modifications be integrated in the branch ‘local’. Then, the workflow is easy:

• git svn rebase on the ‘master’ branch
• optionally run the tests
• git merge --no-commit master on the ‘local’ branch
• optionally run the tests
• commit if tests are ok
• move ‘last good’ tag on a ‘master’ and ‘local’ branches

Will spend couple of hours in next few days to implement and get a feeling how that works in real life.

Wanna lend me an hours per day extra?

Posted in Software Development | No Comments »

Interchangeable hardware or “home cloud”

Yesterday my border router in the home network crashed.

It was a P4 PC under Linux with several network cards routing a traffic between three Internet connections (yes, I’m paranoid), my home network and my house network.

That morning either CPU or motherboard fried and…

Now I’m thinking about something more interchangeable than stock PC. Well, actually, this is more about plug-n-play software than a hardware.

You may imagine that I have quite sophisticated routing setup here. Also this server ran squid, postfix, imap, traffic accounting software and was also a file server.

Most of the software is em… box-agnostic, at least until you move the config and data files. Routing is somewhat more difficult, since on a different PC box, ether mac addresses most probably would be different.

Ideal usage scenario would be something like that:

1. Bring a replacement server, install a bare-bone system there and connect it to the network.
2. Power the old system up on a temporary hardware (or just attach old drives to a new box)
3. Issue something like vzmigrate of OpenVZ and, voila - new system has all old services up and running.

Obviously, that won’t be that easy with the routing, but if OS on both system have similar configs, that would save much time.

Interesting that the easiest implementation of this approach would be to do it verbatim: just move each service into its own OpenVZ VE - service appliance.

Unfortunately, the old system was set up in late 90th when the virtualization state of art was VmWare. So, I’m going to have a long-long-long evening

Posted in Sysadm | No Comments »

bug tracking flavours

Yesterday found two interesting instances of distributed bug tracking systems:

• Bugs Everywhere
• git-issues

Both are command-line, both support git.

Would be interesting to have try a web frontend for them on a live project..

Posted in Software Development | No Comments »