GSM 03.38 Encoding for Python 2.4

Vim screenshot of gsm0338 encoding

These days I’ve been working with SMS messages. Due to the fact that I work in a Greek company, the character set of these messages should be Greek (or able to contain greek letters). As far as I know there are 3 options to accomplish this:

  • Use unicode (UCS2, it’s like UTF-16). But you’ll get only 70 characters per SMS.
  • Use GSM 03.38 encoding. You get 160 characters per SMS, but a reduced set of characters.
  • 8 bit encoding. I wasn’t able to figure how this work with the Greek charset (yet).
So for our new SMS delivery system we decided to go with the almost universal GSM 03.38 encoding. Also we changed our hacked java SMPP client to pythomnic and it’s SMPP library.
That works great but the library doesn’t know anything about encoding. Using iso-8859-1 works, but for Greek we need some kind of “automatic conversion”. This was done using a python encoding module (for python 2.4).
From Greek to GSM 03.38 you have to understand that gsm supports only a few greek letters, they’re in uppercase and without tones. Therefore:
  • All lowercase greek letters are transformed to uppercase.
  • Tones and diaeresises are removed.
  • For some upper case letters, latin ones are used instead. Example: rho: ρΡ -> P (Even in a web browser is difficult to see the subtle difference).
Example:

>>> import gsm0338
>>> msg = u”Αναταράξεις στο πολιτικό σκηνικό προκαλεί η υπόθεση Siemens”
>>> msg.encode(”gsm0338″)
‘ANATAPA\x1aEI\x18 \x18TO \x16O\x14ITIKO \x18KHNIKO \x16POKA\x14EI H Y\x16O\x19E\x18H Siemens’
ANATAPAEI TO OITIKO KHNIKO POKAEI H YOEH Siemens
>>> print msg.encode(”gsm0338″).decode(”gsm0338″)
ΑΝΑΤΑΡΑΞΕΙΣ ΣΤΟ ΠΟΛΙΤΙΚΟ ΣΚΗΝΙΚΟ ΠΡΟΚΑΛΕΙ Η YΠΟΘΕΣΗ Siemens

If what you see is useful for you, here is the gsm0338.py encoding module for python 2.4.

Tags:, , , »
No comment »

Computers, friends and sea in Markopoulo

My macbook outside waiting for me

After long Friday and Saturday finishing some java and xhtml jobs, Sunday was the perfect day to take things easy and spend the day with friends. We (Sascha and me) decided to go to Markopoulo, Panos base camp, and try to relax.

Panos was repairing his iBook. The harddrive of that beautiful piece of technology broke, and he got a new one. But disassembling a laptop is not easy: lots of screws around, manuals, drafts and of course the uncertainty of “will it work? or I’ll break it?”. So that was the reason we decided to give him some company and support.

But panos is very capable of doing that job. What will we do meanwhile? Well, I wanted to learn about Zope, and he works with Zope: perfect combination (no, not a “secret combination“). So Sascha prepared a tutorial on how to install Zope, and some examples. Also he introduced me to Darcs (code management system), and we were playing a bit with that and the local area.

Also I introduced him to Git (another code management system). But I’m not so expert on git, and also Git installation on Mac OS requieres patience and be able to stand MacPorts installing everything on your system (even if you had it). Anyway, you should check this video about Git with Linus Torvalds talking in Google. It’s about git, but it can be about any distributed source control system.

We did a stop for lunch, and perhaps because the planet was bored (it happens from time to time), an earthquake happened. I didn’t feel it, perhaps the two pizzas we ordered left me a bit numb (I fixed that with a fresh frape).

After the iBook surgery was complete, we went to the nearest beach (Markopoulo has no sea, but it’s near) on “Porto Rafti” for a quick and fresh swim. The water was a bit cold, but at 6 in the afternoon, there was no one there :) We were talking about how would be life doing the same job we do, but doing it in a small island in the Aegean sea. A VPN to the main office, no traffic jams, living near the beach, good food and cheap coffees.

Nice things to note from today.

  • Zope seems a powerful web application server for python. I have to give it some time.
  • Darcs also seems good. I’ll continue my experiments with Git, but now I know that I can work with Sascha really easily using Darcs.
  • MacPorts behaves in a silly way when he wants to install software that it’s in your system. (The next step is to compile the Darwin kernel in order to install bash).
  • I do not feel earthquakes when I’m sleepy.
  • Love to have a swim late in the afternoon.

Now we continue slacking off while Panos installs Tiger in his iBook. I guess it’s time to return I still have to clean the dishes from yesterday and prepare dinner :D (Well, breaking news, we will have dinner here). Tomorrow more boring office job not doing much, just a bit here and there and waiting for the typical “We have to fix this for today” at 5:30 in the afternoon.

Tags:, , , , , , , »
Comments (1) »

Hostmonster + python 2.5 + subversion 1.4 + django-svn + mysqldb + fcgid

I’ve chosen Hostmonster as a shared hosting provider for one my projects using Django. I’ve been using hostmonster for this blog so I wanted to give it a try for more professional matters. The problem is that hostmonster is not django/python friendly. I’ve found the following problems:

As you can see:

$ python -V
Python 2.3.4
$ svn
-bash: svn: command not found

But not supported doesn’t mean impossible. You can have your python 2.5 and Django from subversion (And everything you want to install). You only need to install the software we need from scratch. If you want to know how, just read the rest of this entry.

Read the rest of this entry »

Tags:, , , , , , , »
Comments (16) »

MySQL-python in Leopard.

Python and mysqldb

Installing the python library for MySQL in Leopard can be a bit tricky. If you’re an easy_install lover, the first thing to try would be:

$ sudo easy_install MySQL-python

But the results you’ll obtain will be not very encouraging.

Searching for MySQL-python
Reading http://pypi.python.org/simple/MySQL-python/
Reading http://sourceforge.net/projects/mysql-python
Reading http://sourceforge.net/projects/mysql-python/
Best match: MySQL-python 1.2.2
Downloading http://downloads.sourceforge.net/mysql-python/MySQL-python-1.2.2.tar.gz
Processing MySQL-python-1.2.2.tar.gz
Running MySQL-python-1.2.2/setup.py -q bdist_egg –dist-dir /tmp/easy_install-07GTpt/MySQL-python-1.2.2/egg-dist-tmp-t78bko
In file included from /opt/local/include/mysql5/mysql/mysql.h:47,
from _mysql.c:40:
/usr/include/sys/types.h:92: error: duplicate ‘unsigned’
/usr/include/sys/types.h:92: error: two or more data types in declaration specifiers
In file included from /opt/local/include/mysql5/mysql/mysql.h:47,
from _mysql.c:40:
/usr/include/sys/types.h:92: error: duplicate ‘unsigned’
/usr/include/sys/types.h:92: error: two or more data types in declaration specifiers
lipo: can’t open input file: /var/tmp//cczla7iC.out (No such file or directory)
error: Setup script exited with error: command ‘gcc’ failed with exit status 1

But the solution is there. As you can see, the command talks about an error in line 40 for some file called _mysql.c. We’re going to get to that file and remove one line.

First we need to download the package. We will use easy_install, but later. So now grab your wget or your browser to here. And when you have the file, uncompress it (terminal or through finder).

$ wget http://downloads.sourceforge.net/mysql-python/MySQL-python-1.2.2.tar.gz

$ tar xzf MySQL-python-1.2.2.tar.gz

Now go to the directory that has just been created and open the file _mysql.c. Go to the line number 38.

35 #include “my_config.h”
36 #endif
37 #ifndef uint
38 #define uint unsigned int
39 #endif
40 #include “mysql.h”

Delete line 38, save and install your modified egg from the parent directory with:

$ easy_install MySQL-python-1.2.2
Processing MySQL-python-1.2.2
Running setup.py -q bdist_egg –dist-dir /Users/alff/tmp/MySQL-python-1.2.2/egg-dist-tmp-S4uIM7
ld: warning in /opt/local/lib/mysql5/mysql/libmysqlclient_r.dylib, file is not of required architecture
ld: warning in /opt/local/lib/libz.dylib, file is not of required architecture
ld: warning in /opt/local/lib/libssl.dylib, file is not of required architecture
ld: warning in /opt/local/lib/libcrypto.dylib, file is not of required architecture
zip_safe flag not set; analyzing archive contents…
Adding MySQL-python 1.2.2 to easy-install.pth file
.
Installed /Library/Python/2.5/site-packages/MySQL_python-1.2.2-py2.5-macosx-10.5-i386.egg
Processing dependencies for MySQL-python==1.2.2
Finished processing dependencies for MySQL-python==1.2.2

The problem was that _mysql.c defined the type unit and that type is defined again when it includes the file mysql.h. We remove the duplicate, the compiler is happy :)

Edit: If you installed mysql5 with MacPorts, and you get the error EnvironmentError: mysql_config not found when installing the egg. Just add a link from mysql_config to mysql_config5.

# which mysql_config5
/opt/local/bin/mysql_config5
# cd /opt/local/bin/
# ln -s mysql_config5 mysql_config

Tags:, , , , , »
Comments (2) »

Installing pycurl in MacOS Leopard

Lib Curl

Leopard comes with Python 2.5.1. And it’s nice to be able to use easy_install to add python packages, but it can happen that some of those libraries depend in their C/C++ counterparts, and they are a bit outdated in Leopard.

My case was with pycurl. Leopard python doesn’t come with pycurl. However it includes cURL and libcurl version 7.16.3.

$ curl –version
curl 7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IPv6 Largefile NTLM SSL libz

If I try to install pycurl from easy_install, I get this message regarding the version of the C version of the library

$ sudo easy_install-2.5 pycurl

Using curl-config (libcurl 7.16.3)
src/pycurl.c:55:4: error: #error “Need libcurl version 7.16.4 or greater to compile pycurl.”

I have also macports, so I tried to install the version requiered through macports. It´s not an upgrade, because you continue having the original Leopard version plus the new one in the /opt directory.

$ port search curl

curl net/curl 7.18.0 Tool for transferring files with URL syntax
$ sudo port install curl
—> Fetching curl
—> Verifying checksum(s) for curl
—> Extracting curl
—> Configuring curl
—> Building curl with target all
—> Staging curl into destroot
—> Installing curl 7.18.0_0
—> Activating curl 7.18.0_0
—> Cleaning curl

Seems that everything is ok. But before you run to run easy_install again, you have to realize that easy_install doesn’t know anything about your new curl installation of curl in /opt. easy_install will use curl-config to determine the location of the curl library. So, if you update your PATH (if you don’t ave it already) adding the /opt entry in the beginning, everything will be solved.

$ curl-config –version
libcurl 7.16.3
$ export PATH=/opt/local/bin/:$PATH
$ curl-config –version
libcurl 7.18.0

Now the installation will proceed:

$ sudo easy_install pycurl
Searching for pycurl
Reading http://pypi.python.org/simple/pycurl/
Reading http://pycurl.sourceforge.net/
Reading http://pycurl.sourceforge.net/download/
Best match: pycurl 7.16.4
Downloading http://pycurl.sourceforge.net/download/pycurl-7.16.4.tar.gz
Processing pycurl-7.16.4.tar.gz
Running pycurl-7.16.4/setup.py -q bdist_egg –dist-dir /tmp/easy_install-f4LJrU/pycurl-7.16.4/egg-dist-tmp-9l6i1M
Using curl-config (libcurl 7.18.0)
ld: warning in /opt/local/lib/libcurl.dylib, file is not of required architecture
ld: warning in /opt/local/lib/libz.dylib, file is not of required architecture
zip_safe flag not set; analyzing archive contents…
Adding pycurl 7.16.4 to easy-install.pth file
.
Installed /Library/Python/2.5/site-packages/pycurl-7.16.4-py2.5-macosx-10.5-i386.egg
Processing dependencies for pycurl
Finished processing dependencies for pycurl

And we’re ready to rock’n'roll:

>>> import pycurl
>>> pycurl.version
‘libcurl/7.18.0 zlib/1.2.3′
>>> c = pycurl.Curl()
>>> c.setopt(pycurl.URL, “http://www.google.com”)
>>> c.setopt(pycurl.FOLLOWLOCATION, 1)
>>> c.perform()
… (dumped html) …
>>> c.getinfo(pycurl.HTTP_CODE)
200

:)

Tags:, , , , »
Comments (1) »