Testing POP over SSL the old way

In the old days it was common to test SMTP/POP servers using a simple telnet connection to the corresponding port (25/110) and interacting directly with the server via text commands. This was simple, clean and easy to do without need of configuring any mail clients just for testing.

These days any basic SMTP/POP server is configured with TLS or SSL to improve security. This is a necessity rather than a luxury these days as more commerce and business are dependent of Internet services like Mail.

For a time I thought that the old school of direct text interaction with these servers was impossible until I found the openssl command utility. From the man page it is easy to see that this openssl utility has several uses and one of them is. taken from the man page itself: SSL/TLS Client and Server Tests!!.

For example to connect to a POP3 server that uses SSL we call the following command:

openssl s_client -connect localhost:995

Make sure you replace localhost for the server address and if you use a different port (995) change it as well. This command will connect to the POP3 server and handle all the SSL stuff for you. You will see a lot of messages about certificates and SSL handshakes etc... if you are not interested in SSL simply ignore them.

After all the SSL stuff has passed you will be greeted by the POP3 server message (+OK Hello There for courier-pop) and then you can start interacting with the server as you would using telnet.

If you don't know how to interact with a POP server using text commands check my previous post about the topic.

Faking POP

As part of a networking class at my lab we teach students protocols by asking them to implement a simple POP3 client in Java. The client must connect to the POP3 server and interact with the server as a normal POP client.

As soon as the class started we realized the university POP server uses SSL to encrypt the connection (for obvious security reasons) and asking the students to implement the SSL part of the client was not an option.

We needed a POP server set up quickly before the students could present their programs or at least something that behaves like a POP server. A quick search resulted in FakePOP and the great Ubuntu has it in their repositories.

I simply issued the following command:

sudo aptitude install xinetd fakepop

Then create a xinetd configuration file "/etc/xinet.d/fakepop" that contains the following:

service pop3
{
server = /usr/sbin/in.fakepop
disable = no
socket_type = stream
protocol = tcp
user = nobody
wait = no
}

and restart the xinetd daemon:

sudo /etc/init.d/xinetd restart

and in less than a minute I had a POP3 server (or at least something that behaves like one) on my machine and the class was saved.

Also you can create a custom message to be delivered to anyone connecting to the fakepop server. You can create as many messages as you want by putting them as text files inside the "/etc/fakepop" directory. Some examples can be found in the "/usr/share/doc/fakepop/examples" directory like the one below:

From: System Administrator
Subject: example message - text only
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

This is a simple message.
It contains only text.

Long Vacations in Panama

I was in a long vacation (over two months) in a tropical paradise known as Panama. Most people (specially in Japan) do not know Panama is actually a country and only know about it because of the Panama Canal.

If you like beaches you would like to go to Isla Grande, Isla Colon and San Blas were you will find beautiful caribbean beaches or travel the Via Panamericana visiting all the resorts at the Pacific Sea. If you like mountain hiking you can go to El Valle, Boquete or Volcan to find a rich tropical fauna and flora only seen in Panama.


City people can simply stay in Panama City and visit the malls (Multicentro, Multiplaza and Albrook Malls) and all the night clubs around the city (The Gallery, Oz, Mistyk) and of course the Coast Way to eat on the finest restaurants in the country.

Panama has it all in a relative small country so make sure you visit it in your next vacation. Here are some links with information about Panama:




Chiriqui Chatter: a blog by James Dean where he talks about his like in Panama. Here you will find a lot about the culture and cousine and also how to retire in Panama if interested.

Blogs Panama The name says it all. You will find here a lot of blogs about Panama and it's people.

Tavel Blogs from Panama Here you can get the view of Panama from foreign people in Panama. People that have traveled to Panama and wanted to share their experiences, photos and even videos with the world.

Dealante.com Finally if you want to know about the night life in Panama (Wild on E! style stuff) then check this page.

After this long vacations under the sun I got a little rusty in the technology camp. This is the principal reason I haven't learn anything new or interesting to blog about. I will try to resume my studies/work activities and start blogging soon.

Differences in SQL Implementations

If you need to migrate your SQL database from say, MySQL to MSSQL, you should know that most SQL implementation differ in some trivial and some core ways that may make the migration very difficult and anoying.

Before you choose to migrate SQL implementations I recomend you to check this link with information about the differences in the most popular SQL implementations (MySQL, Postgres, MMSQL, Oracle and DB2).

Web 2.0 Network Management

Before working as a programmer I was a network administrator for a metropolitan ATM network. I had to deal with more than 500 network devices all over the city including servers, routers, ATM switches and cable modem CPE's.

To handle the daunting task of monitoring all the interfaces and services of all these equipments I tried a lot of tools from commercial solutions like IBM Tivoli, open source solutions like NetSaint, BigBrother and my own custom solutions with perl, mrtg and net snmp. In this blog I will tell you about my experiences in the field of network management.

The company I worked bought the IBM Tivoli Framework for the network management task. After months in trainings, setup, IBM consultants and technical staff coming and going and expending thousands of dollars in equipment,training and support we ended with a complete, ultra powerful and insanely complex system for network management. Too bad no one ever used it fully, not because we did not know how to use it but because we had no actual use for it.

All we wanted was to monitor a set of network devices via SNMP. The network monitor had to be configured with IP address and auto detect via snmp the interfaces, add them to the system, monitor them (traffic, errors, etc). The Tivoli Netview was more than sufficient for this but the sales guys at IBM would insist that we would also need the Tivoli Access Manager, Tivoli Identity Manager, Tivoli Directory Integrator, Tivoli Directory Server, Tivoli Storage Manager and lot more stuff. Each of these products came with a server, lost's of manuals, training, etc.

We tried to use these Tivoli things, since our company payed huge amounts of money for it, but no matter how hard we tried these systems had no use for us. Even the Tivoli NetView was not that useful. It was unable to autodetect correctly the interfaces of our CISCO routers and gave us ambiguous names and descriptions for out ATM Switches interfaces.

This really was not a NetView problem, the fault was on the ATM Switches manufacturer that put the wrong descriptions in their SNMP MIB. We had to manually edit the descriptions of each device interface and there were a LOT. The NetView GUI was far from fluid and each interface had to be edited one by one using like 5 different dialog boxes for each one.

The more we used Tivoli's NetView (the other Tivoli servers were accumulating dust and consuming power only) we were more convinced we needed to throw it away and so we did. As linux enthusiasts we started looking in the open source camp.

We played with open source monitoring solutions like NetSaint (now Nagios) and BigBrother and they are good but we must face it, web interfaces (at that time) were not the best in terms of usability and speed.

In the end we simply developed our own perl scripts with snmp and MRTG in linux. The linux console interface (with the GNU tools) is very powerful if you know how to use it, perl was perfect to handle large amounts of text data (mrtg cfg files) and mrtg combined with snmp command tools was simple and more that enough to our needs.

With our scripts we got a complete solution and we were happy but as time passed and people started finishing graduate school and moving to other companies (including me) the inners of out custom solution became taboo and black magic for those that came after us. Also there are now better tools like RRD Tool and Ruby that could be used to improve or replace the system.

A few days ago I read an article about Nagios and started thinking that these web based monitoring applications in the back end are really using the same tools I used like perl, mrtg, snmp. If only the interface was more fluid/flexible as to give the impression of a desktop application and real time monitoring it could become the next big application. Suddenly it hit my head: Wait! Web 2.0 is about exactly that!! make web interfaces more fluid and flexible as desktop applications!! I felt as if it was the discovery of the century and started to get ideas on how to develop such applications using Ruby/Rails/GoogleMaps/RRDTool.

Well my excitement only lasted a few minutes. After some thinking I realized that first: I have no time to do such thing, and second: Google showed me that this is not a discovery and that there are already implmentations of Web2.0 monitoring systems. See here, here, here and here for some examples.

Well these applications are still in diapers and there is a lot of space for improvement. Maybe when I finish school I will do something like this. With my experience as network administrator and now as professional programmer I may get something good...

Scrapping japanese web pages with Ruby and Mechanize

With WWW::Mechanize it is possible to scrap all kinds of web pages no matter if they require sessions, complicated forms, or have difficult/bad html syntax. So why the emphasis of "japanese" in the title you ask??. Do we need a Japanese version of Mechanize to scrap Japanese characters??. Well the answer to the second question is NO, but we still need to be carefull when scrapping Japanese html pages.

If you have to scrap several Japanese html pages from different places you will find that in Japan there are three standard encodings used for the text in html pages, namely EUC-JP, Shift-JIS and UTF-8. At first I was using the content-type of the html header to determine the page encoding and then convert the page to Shift-JIS encoding that is the encoding used by the MSSQL database where I store the scrapped data.

Unfortunately as more pages I scrapped it was clear that the validity of the content-type field depends on how serious is the html page developer. Some pages were missing the charset parameter in the content-type and others even had it wrong (charset=sjis in a UTF encoded page). Suddenly I was testing every page to find out the encoding before scrapping it and had three ruby scripts, one for each type of encoding.

Later I learned that ruby actually has ways to find out automatically between the three encodings by checking the text itself (no wonder since Ruby was developed in Japan). This capability combined with the flexibility of Mechanize allowed me to write a single script to extract any japanese page without concerns in encodings : ).

The magic is done by Kconv (Ruby Kanji Converter) that has methods to convert any string to an specific encoding. Kconv makes some guesses on the current encoding of the string and then makes the necessary steps to convert it to the desired one.

To integrate Kconv with Mechanize we take advantage of the pluggable parsers:

require 'rubygems'
require 'mechanize'
require 'kconv'

# Create a pluggable parser
class SjisParser < WWW::Mechanize::Page
 def initialize(uri = nil, response = nil, body = nil, code = nil)
       body = Kconv.tosjis(body)     # Magic Line
       super(uri, response, body, code)
 end
end

# Create the WWW::Mechanize object
agent = WWW::Mechanize.new

# Register our parser
agent.pluggable_parser.html = SjisParser

# Load the UTF or EUCJP encoded page
page = agent.get("http://utf-or-eucjp/encoded/page.html")

# Print the result
puts page.body

There is nothing new in this code with respect to Mechanize but if you do not understand what this code does the you should read the GUIDE file that comes with Mechanize. The important line here is the one marked Magic Line. This simple line of code uses Kconv to convert whatever it receives (the html page) to Shift-Jis encoding. The encoded html page is then used to create a Page object as Mechanize would normaly do. No matter if the page is in UTF, EUC or SJIS encoding the result will be a Page object with the body in Shift-JIS encoding. Pretty cool isn't it?

The code above works well for Mechanize 0.5.1 and 0.6.0 that support pluggable parsers but in Mechanize 0.4.7 there are no pluggable parsers.In Mechanize 0.4.7 we have a less flexible "body_filter" that can be used to encode the html pages before they are parsed by Mechanize.

require 'rubygems'
require_gem 'mechanize', '=0.4.7'   # Make sure we are using version 0.4.7
require 'mechanize'
require 'kconv'

# Create the WWW::Mechanize object
agent = WWW::Mechanize.new

page = agent.get("http://utf-or-eucjp/encoded/page.html")
page.body_filter = lambda { |body| Kconv.tosjis(body) } # Magic Line

# Print the links text
puts page.links.each { |link| puts link.text }

Again the magic is marked by the Magic Line line. From the Mechanize documentation:

The body filter sends the page body to the code block, and parses what the code block returns. The filter on WWW::Page#body_filter is a "per-page" filter, meaning that it is only applied to one page object.

In short we convert the html to Shift-Jis before it is parsed by Mechanize.

Warning!!: There is a small difference on how body_filter works in comparison to the pluggable parser presented above. The body_filter will encode the html text to sjis and the result will be passed to the parser for further processing. The body_filter WILL NOT modify the html text itself!!. If you print out the html text using "puts page.body" you will get the html text in the original encoding. Only the parsed html will be in Shift-JIS and that is why I only print the links text the example above.

In the case of the pluggable parser we convert the html text and pass the encoded version to the Page class initialize method. This way the Page object is created with the Shift-JIS encoded version from the start.

If you need to access the encoded html text it is possible to exploit the openness of Ruby classes and modify the Page class to do it. For this we simply redefine the "body" method from the Page class as follows:

require 'rubygems'
require_gem 'mechanize', '=0.4.7'   # Make sure we are using version 0.4.7
require 'mechanize'
require 'kconv'

# Redefine the body method of the Page class
module WWW
 class Page
  def body
   Kconv.tosjis(@body)
  end
 end
end 

# Create the WWW::Mechanize object
agent = WWW::Mechanize.new

page = agent.get("http://utf-or-eucjp/encoded/page.html")
page.body_filter = lambda { |body| Kconv.tosjis(body) } # Magic Line

# Print the Shift-JIS encoded body
puts page.body

There is no limit to the flexibility that Ruby offers us. With the definition of the Page class in the above code we are modifiying the way Mechanize works without need to change anything in the original Mechanize source. Now we can use the "body" method to obtaing the Shift-JIS encoded version of the html text. Moreover we could go rampant and add methods to get the body in different encodings like:

require 'rubygems'
require_gem 'mechanize', '=0.4.7'   # Make sure we are using version 0.4.7
require 'mechanize'
require 'kconv'

# Redefine the Page class
module WWW
 class Page
                # Return the body Shift-JIS encoded
  def sjis_body
   Kconv.tosjis(@body)
  end

                # Return the body EUC-JP encoded
  def euc_body
   Kconv.toeuc(@body)
  end

                # Return the body UTF-8 encoded
  def utf8_body
   Kconv.toutf8(@body)
  end
 end
end 

We see here that Ruby and everything Ruby related is powerfull and flexible without being too complicated. I am still learning and the more I learn the more I am stick with it. If only there were MIDP/CLDC implementations in Ruby... Have fun scrapping japanese pages!

Linux can do almost anything!

I like to browse linux application pages like sourceforge, kde-apps and freshmeat just to see what cool applications are there to play with. There are always little gems not known to the general public and who knows? there may be the next killer application waiting to be discovered.

After years of browsing these pages you will see that you can do almost anything with linux and on more than one way. Granted that most of the applications you see are not active, are pet projects of someone or were student projects that are no longer being developed. But this does not mean that all of the projects are of no use, lots of good projects are there to be discovered that can be usefull and maybe become the next big thing. Did you know that most popular applications today started as small pet projects or student projects of someone?.

Some of the little known applications I use are the Japanese dictionary software for kde, Kiten, the bible study software, Sword, the cool quake like console Yakuake and little eye candy apps like xsnow, xpenguins and amor.

One of my all favorite tools is x2vnc/win2vnc to share a single mouse/keyboard between different computers running different OS.

Another way to find applications is to browse your distro software repository. This way you can also install/unintall the software using your distro management tools with a few clicks/commands. For example in gentoo simply browse the /usr/portage directory to find out what softwares are avaliable and in kubuntu you can browse the list of available apps using adept.

With time browsing applications you will agree with me that almost anything can be done using linux. I mean you can even make coffee and brew beer with linux!

Better Bash prompt

More colorfull and useful bash prompt

Here is what I did to give a little more color and style to my bash prompt following this blog post:

Edit the .bashrc file in your home directory and define your PS1 env variable in the section that looks like this:

# Comment in the above and uncomment this below for a color prompt
PS1='\[\e[32m\][\u@\h]\[\e[0m\]\[\e[37m\][\t]\[\e[0m\]\n\[\e[34m\][\w]\[\e[0m\]> '

Make sure to replace the PS1 value to anything you prefer. For more information on the meaning of the PS1 values check the Bash-Prompt How-To.

To make the change take effect in the current bash session simply source the .bashrc file like:

[~]> source ~/.bashrc

X11 for Windows

As a normal geek I have access to a large number of computers at work, school and home. To avoid duplication of information (i.e. emails, bookmarks and files spread among several computers) I usually have all my data and applications on my computer at home and use X-Forwarding to access it remotely from work and school.

This works great when I have access to Linux computers at work and school (I usually do) but in some rare situations I end surrounded by Windows only computers. Fortunately there are ways to get X-Forwarding to work in a Windows environment and here I explain the method I use.

X11 server for Windows

For X11 based applications (i.e. all Linux applications with GUI) to be displayed on Windows we need an implementation of X11 for Windows. I chose to use XMing that is small, easy to install and just works out of the box.

For installation simply follow these instruction and you are good to go.

SSH Client for Windows

To connect to the remote machine (i.e. home machine) with X-Forwarding support we need a good ssh client for Windows. I have used PuTTY and OpenSSH with success. If you prefer command line use OpenSSH or if you like a GUI ssh client then use PuTTY as both are equally functional and feature rich.

For instruction on how to get and install PuTTY look at this link.

X-Forwarding with OpenSSH on Windows

If you prefer to use OpenSSH command line instead of PuTTY GUI you need to install the package from http://sshwindows.sourceforge.net/.

Simply download and install the most recent self installation package and from the command prompt you will be able to run ssh with all it's command line options and features. For X-Forwarding make sure that XMing is running and then run the ssh command with the -C and -X command switches to connect to the remote machine.

SET DISPLAY=127.0.0.1:0.0
ssh -CX -l username remote_ip_address

As you see it is important to set the DISPLAY environment variable before running the ssh command. Without this environment variable all X11 applications will be unable to find the XMing display and fail to start. Normally the DISPLAY used by XMing is 127.0.0.1:0.0 but in case it is not you can find the value looking at the XMing log files.

The -CX switch simply means to enable compression (-C) and X-Forwarding (-X). Read the ssh documentation for more command switches and options.

Mandatory Screenshots

First we see my favorite mail client (Kmail) in conjunction with my favorite music jukebox (Amarok).



This second screenshot you can see Konqueror, Firefox and IE in the same machine with the KDE page loaded.



Final Remarks

Remember that this is NOT running Linux applications on a Windows PC. The KDE applications you see in the screenshots are running at a remote computer (my home computer) that has Linux (Kubuntu) installed. The X11 client (XMing) only displays the applications on the Windows PC and allows you to interact with them via mouse and keyboard.

Also note that you need to enable X11 forwarding at the server side and have Xorg or Xfree server installed. In Kubuntu these are the default settings so no change is needed at the server side except for installing the openssh-server.

sudo aptitude install openssh-server

Little Drawback

Using XMing to run remote X11 applications in a Windows PC is very useful and it works perfectly except for one little drawback: the system tray . Applications like Amarok, KMail, Akregator, Kopete when configured to use the system tray will try to dock on the system tray when minimized but on XMing they will simply disappear with no way (that I can find) to get them back. So if you use XMing try to avoid using the system tray.

Tips on RubyGems: Installation and Versioning

Installing RubyGems in Kubuntu

Kubuntu has it's own packaging system for software (deb files) that can be conveniently installed/uninstalled from your installation with a few clicks using adept or a few commands using apt-get/aptitude.

Unfortunately most of the ruby libraries I use at work are not yet in the Kubuntu repositories so I usually resort to the RubyGems packaging system to manage those libraries (gems). It seems there are conflicts in the way these two packaging systems manage the ruby libraries so RubyGems seems to be banned from he Kubuntu repositories requiring us to install it manually. Fortunately it is very easy to do and here is how I do it:

# Make sure everything is up to date
sudo aptitude update
sudo aptitude dist-upgrade

# Install ri and rdoc needed for RubyGems to work
sudo aptitude install ri rdoc

# Install RubyGems
wget http://rubyforge.org/frs/download.php/11289/rubygems-0.9.0.tgz
tar zxvf rubygems-0.9.0.tgz
cd rubygems-0.9.0
sudo ruby setup.rb

Now you can use the gems command to search, install and uninstall your favorite Ruby libraries like Mechanize and rMagick.

Specifying Gem Versions in Ruby scripts

There seems to be some misleading (or erroneous?) documentation on the RubyGems User Guide about using specific versions of a particular gem in a ruby script. In the section "Using Explicit Versions" the example indicates that a single call to require_gem with the version as second argument is enough to load that particular version of the gem in our script. For example to use Mechanize version 0.4.7 that is incompatible with 0.6.0 the user guide indicates the following steps:

require 'rubygems'
require_gem 'mechanize', '= 0.4.7'

If you try this and create a WWW::Mechanize object ruby will complain with a NameError like this one:

NameError: unitialized constant WWW

The problem seems to be that the require_gem does sets the gem version you want but it does not load the gem. So after the require_gem directive you need to explicitly load the mechanize gem as follows:

require 'rubygems'
require_gem 'mechanize', '= 0.4.7'
require 'mechanize'

As you see I explicitly load the mechanize library after specifying the version with require_gem. After this you can now easily use Mechanize version 0.4.7 in your script.

In windows installations using the One-Click Ruby Installer it is also necessary to load the gem after specifying the version with require_gem.

Install the Mechanize Gem in Kubuntu

Note that to use Mechanize with ruby in Kubuntu you need the libopenssl-ruby deb installed or you will get errors like "LoadError: no such file to load -- net/https" when loading mechanize. To install the libopenssl-ruby deb simply run:

sudo aptitude install libopenssl-ruby

Working in Japan

Danny Choo at dannychoo.com has an interesting post about his experience in job hunting in Japan and I just wanted to add some points of my own experience.

The best way to get a job here is to come to Japan as a student. A tourist visa works but they are at most for a couple of months which is not enough time to get a full view of Japan and search for a good job.

As a student you get minimum one or two years student visa. This is time enough to learn the language and search without rushing the best options for job. Moreover if you obtain a Masters or PhD degree you are almost guaranteed a job offer after graduation because every year companies in Japan go to universities in search for new meat from within the graduates.

The only problem of this method is how to pay the school fee and living expenses in Japan for so long?. Well what all the people I know do is apply for the Monbukagakusho scholarship. This is especially good if you live far away from Japan (America, Europe, Africa, etc.). Asian people (China, Korea, Vietnam, etc.) come to Japan as japanese language students. They first pay all expenses from their own pockets and once in Japan they apply for local scholarships within Japan.

The Monbukagakusho scholarship (see here) is one of the best scholarships to come to Japan. It will cover your full school expenses, your roundtrip plane ticket and give you 175.000 Yen monthy for living expenses. This scholarship at first is for two years research program but if you apply and enter the Masters and PhD programs at your university then the scholarship can be extended up to seven years. This means you can live in Japan up to seven years as an student (as long as you comply with your university requirements like attaining credits, writing papers, thesis, etc.). During this long time you get to know Japan very well, learn the language (at least conversation skills) and look for a part time job. I came as a Monbusho student in 2002 and now I can talk the language (but not read it), got a Master degree, I am now running for a PhD and have a nice part time job as programmer. I also worked as intern in several companies as part of universtity projects.

There is also a JAIST scholarship (see here) but I don't have much details except that the scholarship is for three years max and after the three years you are forced to return to your own country and work for two years there. At least this is the situation of all my Bangladesh friends that have the scholarship.

If you are interested in getting the Monbusho scholarship the best source of information is the Japanese Embassy in your own country. Simply go there and ask about the Monbusho.

About the job well once you are in Japan it would not take much effort to get a job. Especially in the IT/ICT camps. I started scanning books at an IT company for 900 Yen/Hour and now I am a part time programmer for 1500 Yen/Hour. This and the scholarship is enough for living in Japan with my family and even take nice vacation trips around Asia (Beijin, Seoul, Okinawa, Kyoto, Osaka, etc.).

That's it... remember the best option is to come as a student, once here you can see your options for job.

About what university?? well any university will do but the famous ones are Tokyo, Meiji, Keio and Waseda. All these universities are high quality and studing hard is imperative to keep your scholarship. Remember that the scholarship is not for free! you must earn credits, write papers, participate on international conferences and if you are a Masters or PhD student you must pass the screening process and write your thesis. If you don't fulfill the University requirements your scholarship won't be extended and you will have to return to your country.

If you like to study extremly hard then go to Tokyo University but you will get less time for jobs, even part time jobs. Waseda is more relaxed on the Masters but the PhD is very, VERY difficult (experience). The other two are more or less the same as Waseda but the PhD is not as difficult as Waseda.

MPlayer on Kubuntu Edgy Eft

No matter how many media player for linux I try I always end up using MPlayer. In my opinion is the best player ever.

To install it in Edgy Eft we must add the multiverse repositories to the source.list file. Simply add these lines to the /etc/apt/sources.list file:

deb http://archive.ubuntu.com/ubuntu edgy multiverse
deb-src http://archive.ubuntu.com/ubuntu edgy multiverse

Then simply update and install mplayer repository with:

sudo aptitude update
sudo aptitude install mplayer

To add the win32codecs (for wmv playback) first download the lastest version from here and install it using dpkg:

sudo dpkg -i w32codecs_xxxxxxxxi386.deb

Replace the xxxxxxxx with the version you downloaded and enjoy!.

update: In Feisty Fawn there is no need to enable the multiverse repositories. Mplayer seems to be included in the default repositories.

Japanese Input in Kubuntu Edgy with Scim and Anthy.

After installing Kubuntu Edgy Eft in my laptop I was surprised to see that scim with the anthy (Japanese IM Engine) was not installed even when I selected Japanese as primary language during the GUI installation.

Well not to despair because enabling japanese input support is very easy.

1) Install the scim-anthy package

# sudo aptitude install scim-anthy

This package gives us the anthy IM Engine that is my prefered one for Japanese input. You are free to install other IM Engines for japanese or other languages like Chinese and Korean.

Aptitude will take care of the dependencies and will automatically install scim if it is not already installed apart from other packages.

2) Set up environment variables on X startup.

Create a file called "74custom-scim_startup" inside the /etc/X11/Xsession.d directory and write the following inside it:

export XMODIFIERS="@im=SCIM"
export GTK_IM_MODULE="scim"
export XIM_PROGRAM="scim -d"
export QT_IM_MODULE="scim"

The QT_IM_MODULE environment variable causes Skype to crash on startup with a glibc invalid pointer error. Read my previous blog for a work around to this problem.

3) Running scim on KDE startup.

Other blogs and posts about this recommend to create a startscim script in the Autostart directory of KDE to run scim when KDE starts. I have found that this is not required if you use skim (scim for KDE). Simply run skim and you are done. Skim always gets started at KDE startup (at least on my machine).


Some links with more information:

Scim in Ubuntu guide: I used this guide to set up my laptop and to write this blog post.
Scim Home Page: With a lot of more information about scim and other IM engines.

*** glibc detected *** skype: free(): invalid pointer:

Since I installed Kubuntu (Dapper) on my laptop I was unable to run Skype on it. Every time I tried it glibc would complain about an invalid pointer error. The error I was getting was something like:

*** glibc detected *** skype: free(): invalid pointer: 0x08be74b8 ***

For weeks I searched in forums, mailing lists and even send a support request to Skype with no avail but today I found a work around. With Kubuntu Edgy released I wanted to try it so I made a fresh install on my laptop. Soon after the install I used the Skype repository to install Skype and to my delight I was able to run it and make calls.

Happy with this result I started installing scim and anthy for japanese input support and when done I found that Skype would not start and was giving me exactly the same invalid pointer error as before...

After installing/desinstalling packages and setting/unsetting environment variables I found that Skype fails if the environment variable QT_IM_MODULE=scim is set. This is one of the environment variables needed to enable scim in Qt applications and since Skype is written in Qt this environment variable seems to be confusing it.

So the solution is in fact very simply... unset QT_IM_MODULE before running Skype:

$ unset QT_IM_MODULE; skype

or

$ QT_IM_MODULE="" skype

I find it easier to create an alias at the end of the .bash_profile file so I can simply call the alias:

$ alias skype='QT_IM_MODULE="" skype'

Running skype now will set the QT_IM_MODULE environment variable to "" and call the skype binary.

The only problem I find with this is that I cannot input japanese or any other scim input method in Skype. But since I use Skype only for phone calls is not a big issue for me. Now I only have to wait for the video support to be included in the linux version...

Developing for Japanese Phones in Netbeans (Part 3)

IAppli (NttDocomo) Development in Netbeans

In parts one and two of this large blog post I explained how to use Netbeans with the Mobility pack to develop mobile applications for two of the major carriers in Japan (au/KDDI and Softbank). In this last part I will talk about developing IAppli's that are how the mobile applications for NttDocomo phones are called.

There is a big difference in development for NttDocomo phones when compared with the other two carriers and any other J2ME platform. NttDocomo uses a propietary profile called Doja instead of the standard MIDP profile.

There are core differences between MIDP and Doja that make them incompatible. For example in MIDP the main application class inherits from the MIDLet class while in Doja the main application class inherits from IApplication class. This kind of core differences and others (see here) makes it necessary to have two complete different versions of the source code (Doja and MIDP versions) if we want the application to run on as most platforms as possible.

If we ignore this incompatibility details then Doja is a great platform for developing applications for phones. I really found Doja development much easier and entertaining than the MIDP counterparts. Maybe because at the time Doja had much more functionality than MIDP. Current versions of MIDP may be by now as good as Doja but I bet Doja is even more powerful than it was before.

Doja Versions

There are several versions of Doja here in Japan and I now of at least two international versions that are used in other parts of the world. In Japan we have versions: 1.0, 1.50E, 2.0, 2.1, 2.50E, 3.0, 3.5, 4.0 and 4.1 and internationally we have versions: 1.5 and 2.5.

I will talk here how to use Japanese versions of Doja versions between and including version 3.0 to 4.1 since the procedure is exactly the same. I have not used the international versions and really don't know if they are excactly the same as the Japanese ones. For information (in english) of the international versions of Doja please refer to the Doja Development Network. All I can tell is that the information in this page is applicable to the Japanese versions of Doja as well.

Getting the IAppli Development Toolkits

All the necessary files for IAppli development can be downloaded from this page. The firsts links are development guides (pdf files) and java api documentation (zip files) for each of the Doja versions available. In the section that says something like アプリ開発ツールのダウンロード is were we can download the development toolkits. You can choose the version of Doja you want to develop but for this guide I have only tested versions 3.0 to 4.1.

Click in the version you want and a page with the requirements and a download link will appear. As usual you need a Windows2000/XP machine with Java2 SDK 1.4.X and for this guide you also need Netbeans with the Mobility Pack version 4.2 or 5.0.

Click in the download link (is in english) and a terms of use page appears. If you agree with the terms of use click in the link below that says 同意する and save the zip file on disk. The file is called emufordoja4_1_201.zip for Doja4.1. If you choose a different Doja version the file name may be different.

Unzip, click the install exe file and accept all the defaults. If you have doubts you can follow the instructions in this page since the steps are very similar to the Japanese versions of Doja.

If everything went OK you will have the tool installed in C:\iDKDoJaX.Y where X and Y are the version number of the Doja toolkit. In my case it is C:\iDKDoJa4.1. I must note that Doja4.1 comes with a plugin for Netbeans already but it does not use the Mobility Pack. This means that all the features that the mobility pack offers cannot be used with this plugin.

The idea of using Netbeans is the ability of the Mobility Pack to handle device partitions and the automatic application build process so I will not be using the plugin that comes with Doja. Also previous versions of Doja do not have this plugin making it difficult to develop applications that support several versions of Doja.

You can develop IAppli's with the toolkit alone following these instructions. In fact all the tutorials that you can find in the Doja Development Network should work in any version of Doja so if you are not interested in using Netbeans for Doja development stop reading this blog and study all the tutorials in the Doja Development Network page.

IAppli's with Netbeans

As with EzAppli's the Doja development toolkits are not UEI compatible so some manual work is needed in order to develop Doja applications with Netbeans. For Netbeans to detect the Doja toolkit as a J2ME platform we must create a platform definition file:

<?xml version='1.0'?>
<!DOCTYPE platform PUBLIC '-//NetBeans//DTD J2ME PlatformDefinition 1.0//EN' 'http://www.netbeans.org/dtds/j2me-platformdefinition-1_0.dtd'>
<platform name="iDKDoJa4.1" home="C:\iDKDoJa4.1" type="custom" displayname="NTT Doja 4.1" srcpath="${platform.home}/doja_stub_src.zip:${platform.home}/stub_src.zip:" docpath="${platform.home}/doc," preverifycmd=""{platformhome}/bin/preverify.exe" -classpath "{classpath}" -d "{destdir}" "{srcdir}"" runcmd="">
<device name="FOMA" securitydomains="trusted,untrusted" description="FOMA">
<configuration name="CLDC" version="1.0" displayname="CLDC1.0" classpath="${platform.home}/lib/classes.zip" dependencies="" default="true"/>
<profile name="DOJA" version="4.1" displayname="Doja4.1" classpath="${platform.home}/lib/doja_classes.zip" dependencies="" default="true"/>
</device>
</platform>

This platform definition file works for all Doja versions (3.0~4.1). Only make sure to change the version number (i.e. 4.1 with your version) and the path where the toolkit is installed in you machine.

Save this file as "Doja4.1.xml" and put it in the platform directory (\config\Services\Platforms\org-netbeans-api-java-Platform) in your user home directory (the Windows default is C:\Documents and Settings\{username}\.netbeans. If you installed more than one Doja toolkit create different files for each one (i.e. Doja3.0.xml, Doja3.5.xml and so on) making sure to change the version names and the installation path. Copy all these platform definitions files in the directory above.

Now when entering the Netbeans Java Platform Manager (Tools menu) you will see the Doja platform for all the toolkits you created xml files for.

The description file (JAM)

When creating mobile applications using the MIDP profile we need to create a description file (JAD file) that contains information about the application. In Netbeans this JAD file is generated for us and if we need to add attributes to the file (i.e. HTTP support) we can do it easily in the projects properties of Netbeans.

For Doja we need to create a similar file but is called JAM instead of JAD and has different attributes that a JAD file does not support. The best way I have found to create this file when building the project is to use the Eclipse Doja plugin. This plugin contain some ant tasks that make it easy to create the JAM file and since Netbeans build system is based on ANT tasks we can easily add support for the JAM file creation.

First we must download the Eclipse Doja Plugin from this page. Look for a file called jp.gr.ttj.Doja_0.0.1.zip and download it. Uncompress the file and look for a file called anttask.jar and copy it in a directory where the Netbeans ant can find it. I simply copy it in the projects directory but this means I must copy this file in each of the Doja projects I create.

HelloNtt example IAppli

Now the main part. To develop an IAppli using Netbeans we proceed as any normal MIDLet. We create a new "Mobile Application" project and follow the Netbeans dialogs selecting the projects name and folder. For this example create a project called HelloNtt. You can check the Set as main project box but uncheck the Create Hello Midlet box.

In the next dialog select in the Emulator Platform the NTT Doja 4.1 platform (or any other Doja platform you have installed) and finish the project creation process.

If the Doja emulators do not appear in the list box of emulator platforms then you forgot to create the platform definition xml file and copy them in the Netbeans platforms directory in your userhome as indicated above.

As I mentioned in the previous section we need to modify the building process of Netbeans to create a JAM descriptor file. For this we need the anttasks.jar file that comes with the Eclipse Doja plugin (see previous section). Copy the anttasks.jar file inside the newly created project directory (i.e. HelloNtt) and open the build.xml file inside there.

You must add the following targets in between the <project> and </project> tags in the build.xml file:

        <target name="post-jar">
        <taskdef name="file" classname="jp.gr.ttj.ant.FileTask">
            <classpath>
                <pathelement location="anttask.jar"/>
            </classpath>
        </taskdef>
        <taskdef name="date" classname="jp.gr.ttj.ant.DateTask">
            <classpath>
                <pathelement location="anttask.jar"/>
            </classpath>
        </taskdef>
        <file property="jarsize"  type="size" file="${basedir}/${dist.dir}/${dist.jar}"  />
  <date property="TODAY_i" pattern="E, dd MMM yyyy HH:mm:ss" />
        <echo file="${dist.dir}/${name}.jam">
LastModified = ${TODAY_i}
AppClass = ${name}
AppName = HelloNtt
PackageURL = http://www.mypage.com/${name}.jar
AppSize = ${jarsize}
UseNetwork = http
SPsize = 102400
 </echo>
    </target>
    
    <target name="run" depends="init,jar">
 <exec executable="${platform.home}/bin/doja.exe" dir="${dist.dir}">
  <arg value="-i" />
  <arg value="${name}.jam" />
 </exec>  
    </target>    
    
    <target name="post-clean">
         <delete file="${dist.dir}/${name}.jam"/>
    </target>

Only look where it says AppName and PackageURL. These are the most important attributes of the JAM file and MUST be setup to reflect your application. The AppName is simply the name of the application you are developing. The one that will appear in the phone when the application appears in the list of IAppli's installed in the phone. The PackageURL is the URL address from where the IAppli is to be downloaded. This URL is not important when running from an emulator but must be the real URL from where the IAppli is to be downloaded when deploying to real phones.

The UseNetwork and SPsize attributes tell that the IAppli can use http connections and the Scratch Pad (Persistent Storage) respectively. They are not required for our little HelloNtt example but you may need them for more interesting applications.

Please make sure that all attributes that are to be written to the JAM file have no white spaces to the left in the build.xml file. This way there will be no spaces to the left in the JAM file. All the attributes you add must be in between the <echo> and </echo> tags.

If you understand how ANT works you may see that there is also a run task there. This task simply invokes the Doja.exe emulator with the JAM file as parameter. You may be also define a debug task to use the Dojag.exe emulator that supports debugging functions like limited phone memory and storage.

HelloNtt Source code

The little example here is gonna be very different from that used for EzAppli and S!-Appli mostly because Doja is different. Within the project's context menu (right click) select New->MIDLet and in the dialog that appears input the IAppli name (HelloNtt). A new java source will be created in the src directory of the project. Delete everything in that file and put the following code:

import com.nttdocomo.ui.*;

public class HelloNtt extends IApplication {   // --- (1)

    public void start() {
        Display.setCurrent(new MyPanel());      // --- (2)
    }
    
    class MyPanel extends Panel implements ComponentListener{
    // --- (3)
        /**--- (4) ---**/
        private Label lbl = new Label("Your Name");
        private TextBox tbx =
        new TextBox("",16,1,TextBox.DISPLAY_ANY);
        private Button btn = new Button("Press");
        
        public  MyPanel(){
            /**--- (5) ---**/
            add(lbl);
            add(tbx);
            add(btn);
            
            setComponentListener(this);         // --- (6)
        }
        
        public void componentAction
        (Component source, int type, int param){
            /**--- (7) ---**/
            if(source == btn){
                Dialog d =
                new Dialog(Dialog.BUTTON_OK,"Message");
                d.setText("Hello " + tbx.getText());
                d.show();
            }
        }
    }
}

This example was taken from the beginners section of the Doja Development Network page without any modification.

Now simply build and run the project as any other Netbeans project and you should see the Doja emulator being launched with your newly HelloNtt IAppli on it.

Now with this part 3 of the blog you should be able to develop mobile applications for all the three major cellular phone carriers in Japan. EzAppli's for au/KDDI, S!-Applis for Softbank and IAppli's for NTTDoCoMo.

Local DNS cache in Kubuntu using pdnsd

I have a little wireless network (802.11g) in my house to connect to the Internet. The hub I use provides me with NAT addressing, DHCP server and DNS server all in a little package. The NAT and DHCP servers work flawlessly but I cannot say the same of the DNS.

The DNS is very slow (100ms) to give me a query response using dig and I have found that every three requests if fails to give me a response at all. This is very annoying when using a web search engine as every third click the browser would complain that could not resolve the hostname.

To solve this I decided to install a local DNS cache and googling I found this blog . In this blog the author recommends Dnsmasq as a local cache server. I tried it and works perfectly but I found out that the cache is not persistent which means that every time I restart the Dnsmasq service (i.e. reboot the pc) the cache is erased. Also Dnsmasq is not only a DNS cache server but also a full DNS and DHCP server that feels a little overkill for what I needed.

In between the comments to the blog post above the commenter's provided two alternatives to Dnsmasq. One was nscd and the other was pdnsd. I only overlooked nscd that seemed too simple and decided to try pdnsd that has persistent cache and is only a DNS Proxy and nothing more.

The steps are simple in Kubuntu. First we install the pdnsd package:

sudo aptitude install pdnsd

If the installation cannot find the pdnsd package make sure you have enabled the universe repository in your sources.list file. Next we edit the pdnsd configuration file:

/etc/pdnsd.conf

make sure the listen-address is set to the loop interface:

global {
        perm_cache=512;
        cache_dir="/var/cache/pdnsd";
        max_ttl=604800;
        run_as="pdnsd";
        paranoid=on;
        status_ctl=on;
#       server_port=53;
        server_ip="127.0.0.1";
}


server {
        ip="your_isp_dns_ip_address";
        timeout=30;
        interval=30;
        uptest=ping;
        ping_timeout=50;
        purge_cache=off;
}


# if you installed resolvconf, and status_ctl=on
server {
    label="resolvconf";
}

source {
        ttl=86400;
        owner="localhost.";
#       serve_aliases=on;
        file="/etc/hosts";
}

I simply modified the default configuration that came with the pdnsd package. Simply pay attention to the blue parts above. Make sure the server is listening to the loop interface (i.e. 127.0.0.1) and that you configure the DNS server ip address of your ISP.

Now we must tell the DHCP client in our machine to add the loop interface as a nameserver. Without this we would require to edit the resolv.conf file every time we get a new lease from the DHCP server.

/etc/dhcp3/dhclient.conf

make sure this file looks like this:

#supersede domain-name "fugue.com home.vix.com";
prepend domain-name-servers 127.0.0.1;
request subnet-mask, broadcast-address, time-offset, routers,
domain-name, domain-name-servers, host-name,
netbios-name-servers, netbios-scope;

Now make sure you get a new dhcp lease by restarting the network and start the pdnsd service (i.e. /etc/init.d/pdnsd start).

In my machine now a dns query using dig takes ~4ms.

[~]> dig www.kde.org

; <<>> DiG 9.3.2 <<>> www.kde.org
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44676
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.kde.org. IN A

;; ANSWER SECTION:
www.kde.org. 8801 IN A 62.70.27.118

;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Oct 9 14:45:16 2006
;; MSG SIZE rcvd: 45


I must also note that the performance problems I had due to my DNS server failing were not present in Firefox. This is because Firefox implements it's own DNS caching. Since I use KDE applications mostly (Kmail, Akregator, Konqueror, Kopete) the bad performance of my DNS server was causing a pronounced degradation on my Internet experience (i.e. Akregator would fail to fetch every third feed or Kmail would fail to fetch emails from my gmail account if I had three or more pages loading in Konqueror at the time.). With the local DNS cache now everything works just perfect.

Developing for Japanese Phones in Netbeans (Part 2)

S!-Appli (Softbank) Development in Netbeans

In the previous post I explained a little about development of mobile applications for Japanese cellular phones. In that post I only touched development for Ez-Applis that are applications for Au(KDDI) phones.

In this post I will explain how to develop S!-Applis that are simply mobile applications for Softbank phones.

Some history

Before there was Vodaphone and the applications that ran on Vodaphone phones were called V-Applis. Developing V-Applis was exactly the same as any MIDLet using the Wireless Toolkit with the addition of the JSCL profile (J-Phone Specific Class Library). So in short a V-Appli was a MIDLet using the MIDP-1.0 plus a JSCL profile.

There were several JSCL profiles and different phones that supported those profiles. There was also an JSCL in steroids called MEXA (Mobile Entertainment eXtension API) that added some multimedia, sound and graphic capabilities to the phones. Here is a list of the phone models and the profiles they supported.

• C4 Model Phones -> MIDP-1.0


• P4 Model Phones -> MIDP-1.0 + JSCL-1.1.0


• P5 Model Phones -> MIDP-1.0 + JSCL-1.2.0


• P6 Model Phones -> MIDP-1.0 + JSCL-1.2.2


• P7 Model Phones -> MIDP-1.0 + JSCL-1.3.2


• S Model Phones -> MIDP-2.0


• X Model Phones -> MIDP-2.0 + MEXA/JSCL

Each profile had it's own toolkit that you needed to download in order to develop V-Applis. These toolkits only contained the emulator binary and the JSCL jar file. To compile, preverify and pack the V-Appli you needed to use the Wireless Toolkit and Java Development SDK (> 1.3).

Vodaphone Japan was then adquired by Softbank and V-Applis were replaced by S!-Applis. All the V-Appli toolkits disappeared from the web and were replaced by the S!-Appli toolkit. Fortunatelly the only thing that changed was the name. Developing an S!-Appli is exactly the same as an V-Appli. The only differences is that the S!-Appli toolkit supports all JSCL profiles from JSCL-1.0.x to JSCL-1.3.2 profile.

I am unable to find the V-Appli toolkits in the web so this post will be about S!-Appli toolkit. In any case this toolkit includes all the previous profiles so the other toolkits are unecessary.

Getting the S!-Appli toolkit

The download page is here. Simply go were it says S!アプリ開発ツール and click were it says something like エミュレータ.

You will be asked to accept the copyright and term of use. Simply click where it says　同意する　twice. You will have to fill a little form. The first two fields put your name, next your email address and finally select your development environment settings. In the next page click in the download link and you will get a file called S_Appli_Emulator_JSCL-1.3.2.zip.

Uncompress, run the installation exe and accept all the defaults. This will install the toolkit in the directory C:\S_appli_SDK_132. You can change this directory but make sure the path has no spaces in blank.

S!-Appli with Netbeans

All Vodaphone toolkits and the Softbank toolkit are UIE compliant so using them with Netbeans is a breeze. In Tools->Java Platform Manager in the menu of Netbeans press the Add Platform button. In the next panel select Java Micro Edition Platform Emulator and press next. In the next dialog all toolkits that Netbeans sees will appear in a list. If the Softbank toolkit is not in the list (most probably won't be) press the Find More Java ME Platform Folders... button and select the directory were you installed the toolkit (i.e. c:\). You will get the list of all toolkits installed in that directory. Simply check the toolkits you want added to Netbeans and press next.

Now in the Java Platform Manager in the J2ME section should appear the newly installed toolkit and that's it!! we are ready to develop S!-Applis.

Hello Softbank

Developing an S!-Appli is exactly as an Ez-Appli or a normal MIDLet. In the Netbeans File menu we select new project and select a Mobile Application as usual. Create the MIDlet you want and build/run from the project's context menu (right click).

Here is a little test MIDLet for a S!-Appli:

/*
 * HelloSoftbank.java
 *
 * Created on 2006/10/03, 13:10
 */

import javax.microedition.midlet.*;
import javax.microedition.lcdui.*;

/**
 *
 * @author  panama
 * @version
 */
public class HelloSoftbank extends MIDlet {
    public void startApp() {
        Display.getDisplay(this).setCurrent(new HelloCanvas());
    }
    
    public void pauseApp() {
    }
    
    public void destroyApp(boolean unconditional) {
    }
    
    
    public class HelloCanvas extends Canvas {
     
        private final Font sFont = Font.getFont(Font.FACE_MONOSPACE ,Font.STYLE_PLAIN ,Font.SIZE_MEDIUM );
        String s = "Hello World";
        
 public void paint(Graphics g) {
            g.setColor(0,222,100);  
            g.setFont(sFont);
            g.drawString(s, (getWidth() - sFont.stringWidth(s)) /2,getHeight()/2,Graphics.TOP|Graphics.HCENTER);
 }
    }
}

There is a little problem that does not allow the excecution of the Emulator from within Netbeans. I am not sure if it is a problem of Netbeans or the Softbank toolkit . The problem seems to be that Netbeans gives the wrong path of the jad file to the emulator and the emulator complains with a "File not found" error.

The work around this problem is to change the "Running" properties of the project to give the correct path. To do this simply enter the project properties (right click) and select the "Running" section.

In the "Emulator_Command Line Options" text box write the following line. Making sure to change the paths to reflect your configuration:

-Xdevice:"S!_Appli_Universal_JSCL1_3_2" -Xdescriptor:"C:\path\to\project\dist\project.jad"

This way we can call the emulator from within Netbeans. The only disadvantage is that we must modify this configuration for each new project we create. Maybe future versions of Netbeans will fix this problem.

Final Notes

As we can see developing S!-Applis is not different than a normal MIDLet until we start using the JSCL classes. These classes allow us to manage the phone's hardware for example and with MEXA we get 3D graphics, 3D sound, etc.

Since the applications I develop must run in as many devices as possible I do not use the JSCL/MEXA classes nor the KDDIP classes that come with Ez-Applis. This way my applicatications can run with only minor modifications in au phones and Softbank phones.

If you are interested in using MEXA in your applications you need to download the MEXA SDK Ver1.1.0 from here. This SDK seems to come with a complete development environment and it is not recognized by Netbeans as UIE compliant. Maybe if someday I require to use MEXA for a project I will try to integrate it with Netbeans but is very unlikely to be any time soon.

Doja IAppli's

In the next part of this series of post I will talk about NTTDocomo's IAppli. An IAppli is different from Ez-Applis and S!-Applis because they do not use MIDP. IApplis use Doja and as MIDP there are several versions of Doja(1.0 to 4.1).

I will explain how to create Doja Platform Profiles for Netbeans and create a little test IAppli. If time permits me I will also try to point out the most important differences between Doja and MIDP that have caused me development headaches.

Developing for Japanese Phones in Netbeans (Part 1)

Netbeans Mobility Pack

Developing for J2ME applications for the primary carriers in Japan can be a living nightmare. There are three main mobile phone carriers in Japan: au/KDDI, Voda Phone (now Softbank) and NttDocomo. Each of these providers have a midraid of phones of all sizes and capabilities and each week there are new ones. Here in Japan people change phones as often as their underwear...

For a J2ME application to be successful it must be compatible with the current phones used at the moment which means that development never stops and evolves very, very fast. As new phones come with new MIDP profiles and propietary extensions programming this little MIDLet's is a new experience each time.

I am a VIM guy and I am used to develop huge enterprise projects and web applications in a console with screen+vim. But for J2ME development I really recomend an IDE especially Netbeans with it's preprocessor to handle device fragmentation.

Developing a J2ME application requires several steps like class preverification, ofuscation, signing, packing, deployment. These steps can vary from carrier to carrier and even from phone to phone of the same carrier. A single change on the source code would require to excecute all these steps for each phone of each carrier and doing by hand is tedious and error prone. This gets worse if we have a different source code for each phone (Device Fragmentation Problem).

The Mobility Pack of Netbeans is a lifesaver in this case with support for several configurations in a single project. But life is not perfect and the only carrier that provide UIC compliant development environments is Voda Phone. To develop au and NttDocomo applications in Netbeans we must do some magic.

Installing Ez Appli's (au/KDDI) Toolkit

To develop an Ez Appli we need to download the tool kit Ez Appli. Just click where it says 同意する that means you accept the agreement. In the next page click where it says ダウンロード to download the tool kit. You will end with a file called "ezplus_emulator.exe".

You can also download a programming guide and the KDDIP javadoc files from this page but they are useless if you cannot read Japanese.

To install the tool kit simply double click the exe file and accept any defaults you get except for the install directory part. When the installation programs asks for an installation path make sure the path has no blank spaces. I have found the emulator cannot understand spaces on the path names.

From now on I assume the toolkit is installed in c:\ezplus. Inside that directory must be an structure like:

-- Doc               -> PDF files with manual and guide
-- Tools
   -- CmdTool        -> Development Tools
   -- Emulator       -> Contains the phone emulator
   -- KDDIP          -> Contains the KDDIP library

Steps for Ez Appli Development

There are several steps involved in developing an Ez Appli. The tool kit contains a CmdTool that sorts of eases the job but is not that feature rich. Ez Appli development is very similar to J2ME MIDLet development. We need to create a MIDLet (jar file) and a Description file (jad file). This includes preverification and ofuscation of the classes and source code etc.

When developing MIDLets the jar and jad files are enough to run the application. In the case of Ez Appli we must create a KJX file using the KJXArchiver.jar that comes with the toolkit. All au phones need this KJX file to excecute the application.

All these steps can be done manually or by using the CmdTool that comes with the tool kit. As I said this can be error prone and tedious when there are several versions of the source code to support several devices.

Ez Appli configurations and profiles

Ez Appli's use the CLCD1.0 configuration and the MIDP1.0 profile combined with the au custom profile KDDIP. The KDDIP profile simply adds classes to control the phones hardware (backlight, vibration, etc) and to access services specific to au (c-mail, smarf, e-mail, etc.). If you can read a little japanese then the documentation that comes with the tool kit can give you an idea of what you can do.

Fortunatelly since we want to create applications that run equally in all carrier phones we must avoid all custom classes. This completely eliminates KDDIP classes from the picture and we end with MIDP1.0. This means that we can write a MIDLet using the J2ME english documentation available on the net and it will run without too much problems on au phones and hopefully also run on Vodaphone phones.

Even without using the KDDIP profiles we must understand some physical limitations present on the phones that depend on the profile they can run. There are several versions of KDDIP profiles listed below:

Version	KXJ Size	RecordStore Size
Phase1	50KB	10KB
Phase2	50KB	10KB
Phase2.5	50KB	10KB
Phase3	150KB	10KB + 100KB extended

The KJX size refers to the maximum size the Ez Appli can have. Any KJX file larger than the limit will not download to the real phones even if it runs well on the emulator. The RecordStore limit refers to the max amount of persitent storage the phone can handle.

Ez Appli Development With Netbeans

With Netbeans is easy to add new J2ME development platforms in the Tool->Java Platform Manager->Add Platform dialog. With this dialog we can simply point to the directory where the tool kit is installed and all the black magick is done for us.

Unfortunatelly this only works for tool kits that comply with the UEI (Universal Emulator Interface) standard and the Ez Appli toolkit is not compatible. But don't despair! we can add custom tool kits with some work. Essentially what we need to do is to tell Netbeans the steps it must follow to create an Ez Appli.

For au I took the time to generate the platform definitions for the KDDIP Phase1 and Phase2 profiles.

Phase1.xml file:

<!DOCTYPE platform PUBLIC '-//NetBeans//DTD J2ME PlatformDefinition 1.0//EN'
             'http://www.netbeans.org/dtds/j2me-platformdefinition-1_0.dtd'>

<platform name="ezplusTool" home="C:\ezplus" type="custom"
      displayname="KDDI Phase1" srcpath="" docpath="${platform.home}/doc"   
      preverifycmd=""C:/WTK22/bin/preverify.exe"
      -classpath "{classpath}" -d "{destdir}"
      "{srcdir}""
      runcmd=""
>
 <device name="C452CA/C451H" securitydomains="trusted,untrusted" description="Phase 2 Devices">
     <configuration name="CLDC" version="1.0" displayname="CLDC1.0"
               classpath="C:/WTK22/lib/cldcapi10.jar" dependencies="" default="true"/>
    
     <profile name="KDDIP" version="1.0" displayname="KDDIP1.0"
                   classpath="C:/WTK22/lib/midpapi10.jar;${platform.home}/Tools/KDDI-P/kddip.jar"
                   dependencies="" default="true"/>
       </device>

</platform>

Phase2.xml file:

<!DOCTYPE platform PUBLIC '-//NetBeans//DTD J2ME PlatformDefinition 1.0//EN'
'http://www.netbeans.org/dtds/j2me-platformdefinition-1_0.dtd'>>
<platform name="ezplusTool" home="C:\ezplus" type="custom" displayname="KDDI
 Phase2" srcpath="" docpath="${platform.home}/doc"
 preverifycmd=""C:/WTK22/bin/preverify.exe"
 -classpath "{classpath}" -d "{destdir}"
 "{srcdir}""
 runcmd="">
 
 <device name="C3001H/C3002K/C5001T/A3011SA/A3012CA/A3013T/A3014S"
  securitydomains="trusted,untrusted" description="Phase 2 Devices">
    
  <configuration name="CLDC" version="1.0" displayname="CLDC1.0"
   classpath="C:/WTK22/lib/cldcapi10.jar" dependencies="" default="true"/>
 
  <profile name="KDDIP" version="2.0" displayname="KDDIP2.0"
   classpath="C:/WTK22/lib/midpapi10.jar;${platform.home}/Tools/KDDI-P/kddip.jar"
   dependencies="" default="true"/>
       </device>

</platform>

These are simply XML files that defines the environment needed to build Ez Appli's. You do not need to understand the syntax and simply save them inside the "Services\Platforms\org-netbeans-api-java-Platform" directory inside your .netbeans config directory.

From the definition file you can see that the Ezplus toolkit makes use of the preverify command and the midpapi.zip library from the Wireless Toolkit. Make sure you set the path to these two files to reflect the installation path of the Wireless Toolkit in your machine. If you do not have the Wireless Toolkit installed you can use the one installed with Netbeans. I have tested these platform definitions files with Netbeans 4.1 and 5.0 without any visible problems.

Also make sure the path to the kddip.jar file coincides to the path where you installed the Ezplus toolkit.

After saving this files in the appropiate place when you restart Netbeans you will see the two new platforms added to the list of J2ME Platforms in the Java Platform Manager of Netbeans. They will be called KDDI Phase1 and KDDI Phase2 respectively.

Ez Appli Hello World

Now we can try to develop a simple Hello World Ez Appli. With Netbeans create a new Mobile Application Project (File->New Project->Mobile->Mobile Application) and proceed with the project creation dialog as normal until you get the the Default Platform Selection dialog. Here make sure to select any of the new platforms as default platform. The CLCD1.0 and KDDIP1.0/KDDIP2.0 will be selected automatically depending on the profile you chose.

Next select other platforms templates if you want and finish the creation process. At this point we should have a new Mobile Project ready to start development. Before we compile we need to modify the build.xml script a little to handle the KJX file creation and to invoke the Ez plus emulator when running the project. To do this simply open the build.xml file of the project and add these tags inside the <project> and </project> tags.

 <target description="Create AU KJX file" name="post-jar">
    <java jar="${platform.home}/Tools/CmdTool/KJXArchiver.jar" fork="true">
       <arg line="-c ${dist.dir}/${dist.jad} ${dist.dir}/${dist.jar} {dist.dir}/${name}.kjx">
    </java>
 </target>
  
 <target name="run" depends="init,post-jar">
    <exec executable="${platform.home}/Tools/Emulator/run.bat" dir="${dist.dir}">
        <arg value="${basedir}/${dist.dir}/${name}.kjx">
    </exec>  
 </target>   

The code above defines two ant tasks in the build.xml file. One is to generate the KXJ file using the KJXArchiver that comes with the Ezplus toolkit. The other task is to call the emulator with the created KJX file. This task requires a run.bat file I generated to call the emulator. Simply copy the next file a run.bat inside the Emulator subdirectory of the Ezplus toolkit:.

@echo off
cd C:\j2me\ToolKits\ezplusTools\Tools\Emulator
javaw -jar jbmidp.jar %1

Of course make sure you change the path to reflect that of your machine. With the build.xml file in order we can now create a simple MIDLet as usual. With the project select create a ne MIDLet (Right Click->New->MIDLet) and edit the source as a normal MIDLet. Here is my example source code:

import javax.microedition.midlet.*;
import javax.microedition.lcdui.*;

public class HelloKddi extends MIDlet {
    public void startApp() {
        Display.getDisplay(this).setCurrent(new HelloCanvas());
    }
    
    public void pauseApp() {
    }
    
    public void destroyApp(boolean unconditional) {
    }
    
    public class HelloCanvas extends Canvas {
 public void paint(Graphics g) {
            g.setColor(0,222,100);                  
            g.drawString("Hello World",40,20,Graphics.TOP|Graphics.HCENTER);
 }
    }
}

If you have ever programed a MIDLet for any other mobile device based on the MIDP profile you will see that these one is identical.

Now we can build and run this application as any other Netbeans project with the context menu (right click). Not only that we can now add new configurations to the project and use all features of the Mobility Pack to develop for au (KDDI) phones.

Notes

I was able to create platform configurations for Phase1 and Phase2 of the Ezplus toolkit. For Phase2.5 and Phase3 there is a different toolkit but I am unable to make it work with Netbeans. In fact I am not able to develop with that toolkit even with the CmdTool that comes with it.

Since au gives priority to the Brew platform and not to J2ME the kind of applications we can develop for au phones is very limited. If you want to develop rich applications for au Phones you better start learning Brew.

In subsequent posts I will talk about Vodaphone (JSCL/MEXA) and NttDocomo (Doja) development using Netbeans.

Sparklines What??

While looking information about embedding images to display in eruby scripts I was directed to a link that showed how to use the data: URI (RFC2397). This URI schema allows inclusion of small data items as "immediate" data, as if it had been included externally. This means that we can create an IMG html tag from image data directly instead of a URL to a external file like "/path/to/file/img.png".

But what caught my attention was not the data: URI but what it was being used for. It was being used to display Sparklines?. Reading some of the online pages of the paper that describe them I found them very interesting and usefull. The link with the original description can be found here.

See all those small word-sized graphs all around? well those are sparklines and the paper defines them as Intense, Simple, Word-Sized Graphics.

How can we create such pretty little graphs? In this link Joe Gregorio explains how to generate this Sparklines in Python and display them using the data: URI for emdebbing in HTML pages.

This work by Gregorio was then adapted for Ruby by the people at RedHanded to generate BMP and PNG Sparklines. And there is already a Sparklines library for Ruby with some Rails helpers so we can create cool word-sized graphs in Rails applications.

Of course I could not resist to create some Sparklines and even now that I just installed a testing server with eruby so here is my little test taken from the RedHanded page:

<%

require 'base64'
require 'cgi'
require 'zlib'

def build_png_chunk(type,data)
to_check = type + data
return [data.length].pack("N") + to_check + [Zlib.crc32(to_check)].pack("N")
end

def build_png(image_rows)
header = [137, 80, 78, 71, 13, 10, 26, 10].pack("C*")
raw_data = image_rows.map { |row| [0] + row }.flatten.pack("C*")
ihdr_data = [image_rows.first.length,image_rows.length,8,2,0,0,0].pack("NNCCCCC")
ihdr = build_png_chunk("IHDR", ihdr_data)
idat = build_png_chunk("IDAT", Zlib::Deflate.deflate(raw_data))
iend = build_png_chunk("IEND", "")

return header + ihdr + idat + iend
end

def bumpspark( results )
white, red, grey = [0xFF,0xFF,0xFF], [0,0,0xFF], [0x99,0x99,0x99]

rows = results.inject([]) do |ary, r|
ary << [white]*15 << [white]*15
ary.last[r/9,4] = [(r > 50 and red or grey)]*4
ary
end.transpose

return build_png(rows)
end

def build_data_url(type,data)
data = Base64.encode64(data).delete("\n")
return "data:#{type};base64,#{CGI.escape(data)}"
end

# Here I generate an array with 100 random elements.
data1 = []

100.times do |i|
data1[i] = rand(100)
end

%>

<img src="<%=build_data_url("image/png",bumpspark(data1))%>"></img> Random Graph

A little about data:URI

Since I was actually looking for data:URI instead of Sparklines on the first place I think I will talk a little about this URI format. Usually when we want to display a dynamically generate image in a web page we generate the image and write it somewhere in the HTTP server. Then we use the path of the saved file as the src attribute in a IMG tag to display it on the client's browser.

With the data:URI now we can put the image data directly in the src attribute of the IMG tag and it will be displayed. This way there is no need to save any file on the server that sometimes can be a tedious task.

Here is a little example:

<%

require 'RMagick'
require 'base64'
require 'cgi'

include Magick

# Function to generate the data:URI
def build_url(data)
data = Base64.encode64(data).delete("\n")
return "data:image/jpeg;base64,#{CGI.escape(data)}"
end

# Function to generate the PNG Image
def build_img
grad = GradientFill.new(0, 0, 170, 0, "#99eb24", "#3c7f05}")
green_grad = Image.new(170, 170, grad)
green_grad.format = "PNG"

gc = Draw.new
gc.annotate(green_grad, 0, 0, 0, 0, "Ruby\n+\nMagick!\n+\ndata:URI") do
gc.font = "/usr/share/fonts/ttf-bitstream-vera/Vera.ttf"
gc.gravity = CenterGravity
gc.stroke = 'none'
gc.fill = 'yellow'
gc.pointsize = 24
gc.font_weight = BoldWeight
end

green_grad.rotate!(-20)

return green_grad
end


%>

<img src="<%=build_url(build_img.to_blob)%>"></img>

The function to generate the image with rMagick was taken from here and the the function to generate the data:URI from the RedHanded example.

The data:URL has the following format:

data:[<mediatype>][;base64],<data>

Where data: is is the protocol scheme (like http: for HTTP pages or ftp: for FTP etc.). The mediatype is the mime type of the data (i.e. image/png, text/plain, etc.). The base64 indicates that the data is base64 encoded and if not present then the data is asumed to be ASCII encoded. Finally the data is the base64 encoded raw data to be send to the client. In our case the mediatype is "image/png" and the data is the image data we get from the to_blob method of the rMagick Image object.

Little Details

For the examples here to run the rMagick library must be installed on the system. I installed the rMagick library using Gentoo portage and to use it in the source code it is enough to call require RMagick and the subsequent include Magick. If rMagick is installed via gems then it may be necessary to also require the rubygems with require "rubygems" before RMagick.

Also to note is that Internet Explorer does not support the data:URI so these examples will not run on IE. I have tested these examples on Firefox only.

For more information on ImageMagick and example see Anthony's example page and the ImageMagick page. For more information on rMagick look at this link.

Using eruby with Lighttpd

Some times I am asked to develop some web application clients for a particular web service. The clients I have developed include for example J2ME/Doja clients for cellular phones that download information form a special server and fat clients developed on C++/J2SE that access some special service in a corporate server.

Usually the server I am supposed to interact with is developed in parallel with the client I am developing and there are always situations were I finish a client feature but I cannot test it because that particular feature is not implemented yet in the server.

To solve this problem and to avoid discussions with the server developers and the manager I decided to implement a fake server that would behave like the real server to the client but in the back is a simple program that receives the client requests and returns the expected response (or a bogus response if I want to test the client robustness to server failures).

So my dilema was to choose the best server and dynamic language that would allow me to develop small server side Unit Tests to validate my client implementation. From all the possible options Lighttpd combined with eruby offered the most simplicity with the power of Ruby.

Apache/PHP?? since I started to learn Rails it has been very difficult for me to go back to PHP. IIS/ASP?? well from my previous posts you can see I am not a Microsoft fan and I don't want to install a Windows Server just for testing. Rails?? it very good and flexible I don't want to be creating complete Rails applications everytime I need to test a little feature in my application.

Lighttpd in Gentoo Linux

My linux server is running the Gentoo distribution and fortunately Lighttpd is in the portage tree. Installing lighttpd can be done with one command:

# emerge lighttpd

At this time the version of lighttpd is 1.4.11. One of the reasons I choose lighttpd is that configuring Lighttpd is a breeze. With little effort I can have a working HTTP server. The configuration file I use is below. Pay attention to the blue lines that are required to serve eruby pages.

# Load the modules I need
server.modules = (
"mod_rewrite", # For rewriting URLs
"mod_cgi" # For dispaching eruby
)

# Server configuration
server.port = 80
server.username = "lighttpd"
server.groupname = "lighttpd"
server.pid-file = "/var/run/lighttpd.pid"
server.indexfiles = ( "index.html", "index.rhtml" )
server.document-root = "/var/www/localhost/htdocs"
server.errorlog = "/var/www/localhost/htdocs/log/server.log"


mimetype.assign = (
".css" => "text/css",
".gif" => "image/gif",
".htm" => "text/html",
".html" => "text/html",
".jpeg" => "image/jpeg",
".jpg" => "image/jpeg",
".js" => "text/javascript",
".pdf" => "application/pdf",
".png" => "image/png",
".txt" => "text/plain",
)

cgi.assign = ( ".rhtml" => "/usr/bin/eruby" )

This configuration usually goes in the /etc/lighttpd/lighttpd.conf file. Also we need to install eruby that is also in the portage tree:

# emerge eruby

We can now restart the lighttpd server:

# /etc/init.d/lighttpd restart

eruby Hello World

How this all works?? is very simple in fact. When a client requests a rhtml file the server (lighttpd) calls the eruby interpreter with the rhtml file as input. The eruby parses the rhtml file and replaces all the ruby code with their respective outputs (Usually HTML) and returns it to the server. Finally the server sends the generated HTML to the client.

The point here is that the ruby code that is contained in the rhtml file can be anything ruby has to offer and with the power of ruby it can be practically anything!. Let's see a simple example:

Create a file hello.rhtml with the following code and save it in the htdocs directory you configured on the lighttpd.conf file.

<HTML>
<HEAD>
<TITLE><%= "Hello World Page" %></TITLE>
</HEAD>
<BODY>
<%= "Hello World" %>
</BODY>
</HTML>

Anyone familiar with Rails/PHP/ASP/JSP can see easily what this code does. What eruby does is to replace the ruby code in between the <% and %> tags with it's return value. In our case we simply created a String object in the tags.

There is far more to eruby than this and when eruby is combined with other Ruby libraries like CGI, rMagick, Base64, etc with get a rich and powerfull set of tools to create from small test web applications to full blow web applications a la Rails.

Some external links with a little more information.

eRuby: Getting Started with Ruby on the Web

Ruby and the Web