私のTech記憶

Google contacts with sup email client.

2011-02-22T23:56:00.010+09:00

Introduction

Recently I found about sup [1] and gave it a try to replace mutt. One of my requirements is access to my Gmail contacts from within the client so I can auto-complete the To, CC and BCC fields with my already registered contacts.

Fortunately I already had a ruby script to integrate Gmail contacts with mutt [2] and since sup can be extended via ruby scripts it was extremely easy to get my script working with sup.

SUP email client with Gmail contacts

Sup hooks system [3] is very powerful and allows us to add all kinds of functionality to sup using simple ruby scripts. One such hooks is called "extra-contact-addresses.rb" and from the hooks documentation we can read:

What this means is that we must create an script inside our $HOME/.sup/hooks directory called extra-contact-addresses.rb that returns an array of email addresses in plain format (e.g. myemail@gmail.com) or full format (e.g. User Name <username@gmail.com>).

Here is the script I implemented for this purpose:

Simply copy this script to "~/.sup/hooks/extra-contact-addresses.rb" and your are ready to go. When composing or replying and email you can press Tab to autocomplete email addresses.

Resources

[1] http://sup.rubyforge.org/
[2] http://piao-tech.blogspot.com/2010/01/google-contacts-in-mutt.html
[3] http://sup.rubyforge.org/wiki/wiki.pl?Hooks

Get KDE 4 .4.2 working with xrdp

2010-12-29T23:08:00.009+09:00

Problem Statement

KDE4 [1] looks horrible when accessed remotely via vnc [2] and since xrdp [3] is simply a proxy that depends on vnc we get the same horrble look when accessing KDE using Windows RDP client.

The usual response from KDE developers is that vnc is bad and you should change your vnc server or use another remote desktop technology so I tested all vnc server implementations I found and I can tell that x11vnc is the only one that is simple and works correctly with KDE4 so the solution to my problem was simple: get xrdp to work with x11vnc.

Configure KDE4 with x11vnc server

This cannot be more easy. In Ubuntu simply install the package:

then in a terminal (konsole) run this command:

The first time you run this command it will ask you to input a password. Choose any one you like and remember it because you will need it to connect remotely to x11vnc. The -noxdamage option is only needed if you have desktop effects enabled in plasma. If you have these plasma effects enabled and you do not use the -noxdamage option you will see some XDamage errors in the x11vnc output. You can also ask x11vnc to turn off the monitor when a remote client connects using the -clientdpms option and to lock the screen when the client disconnects using the -gone option. Be warned that the -clientdpms and -gone options were not working very well last time I tried. For example is difficult to recover the session from the desktop unless the remote client
disconnects.

Configure KDE4 with xrdp server

Install the xrdp packages:

Edit the /etc/xrdp/xrdp.ini file and add an entry like:

Finally restart xrdp service:

Accessing remotely you KDE session via Windows RDP client

In a windows machine open the remote desktop client (usually found in accessories) and input your KDE machine IP address. After the connection is established you will get a popup with some options including the one we created above (MyHOME) so select it, input the password you set to x11vnc and enjoy.

Troubleshooting

If you use Gnome instead of KDE you be encounter a bug that messes up the keyboard [6]. To fix it you must disable the keyboard plugin of the gnome-settings-daemon using hte gconf-editor. To do this open the gconf-editor and navigate all the way to the folder:

and set the active option to False.

Enabling the Kana Henkan key (Japanese Keyboards)

To get the japanese kana key (top left in japanese keyboards) to work you need to apply a patch
[7] and install the patched deb:

Resources

[1] www.kde.org
[2] http://en.wikipedia.org/wiki/Virtual_Network_Computing
[3] http://xrdp.sourceforge.net
[4] http://forum.xbmc.org/showthread.php?t=74791
[5] http://adrianpopagh.blogspot.com/2010/06/connecting-with-remote-desktop-rdp-to.html
[6] https://bugs.launchpad.net/ubuntu/+source/xrdp/+bug/320393
[7] http://blogs.sun.com/thaniwa/entry/ja_xrdp

2010 Vim Configuration

2010-11-06T12:13:00.004+09:00

Japanese Input in Lucid Lynx (Kubuntu 10.04)

2010-05-10T23:40:00.018+09:00

Introduction

Since Kubuntu 9.10 there is a movement to replace skim [1] with ibus [2] that seems to be a better alternative. Unfortunately after my latest upgrade to Kubuntu 10.04 this movement caused my Japanese input to stop working.

To get Japanese working again using ibus was actually very easy and here Iexplain the steps to do it.

Uninstall SCIM/SKIM packages

First we uninstall all SCIM and SKIM related packages to avoid any kinds of conflicts with the new iBus system.

Install iBus required packages

Second we install the iBus packages needed to support different language input methods.

Configure the system to auto-select the input method.

The im-switch utility is used to select input methods. For better results make sure you select auto.

Trouble Shooting

Japanese/Chinese/Korean/etc does not work in any applications.
Japanese/Chinese/Korean/etc input work in KDE apps but no in GTK (Firefox, Chrome, Skype) apps!
Japanese/Chinese/Korean/etc input work in GTK (Firefox, Chrome, Skype) apps but no in KDE apps!

Resources

[1] http://www.scim-im.org scim/skim
[2] http://code.google.com/p/ibus

Get offlineimap working with non ASCII characters.

2010-03-21T17:15:00.011+09:00

Introduction

I recently started using offlineimap [1] to manage my GMail accounts locally on my machine.

With offlineimap all my email is local so the navigation is very fast and also I am a little more relieved that I have a local copy of all my email.

The only problem is that offlineimap does not work well with non english characters. This makes it difficult to use for those with labels in Russian, Chinese, Japanese etc [2].

Fortunately offlineimap offers some nice options that allows us to modify it's behavior. Using the "pythonfile" and "nametrans" options I was able to get Japanese working flawlessly.

Problem Description

IMAP4 uses a modified UTF-7 coding for all folder names but offlineimap does not convert the folder names to UTF-8 or something readable before creating local repositories that results in very cryptic folder names like "&MMYwuTDI-".

Some research (googling) on the topic resulted in a very nice code to add IMAP4 UTF-7 encoding capabilities to Python [3]. So with a little time and some copy/paste skills I added this code to offlineimap and ban! I got Japanese working correctly in my folders.

Fixing unicode issues in offlineimap

First we must add the code to convert from IMAP4 UTF7 encoding support to offlineimap. Thanks to Dominic LoBue (offlineimap developer) I learned that we can add our own python code to offlineimap. To do this we created a simple python script (e.g. ~/.utf7.py) that contains the following code (copy/paste from [3]:

Then in our offlineimap configuration file, in the "[general]" section we add a line to load this python script like:

Finally in our remote repository configuration add a nametrans option to convert all foldernames from "imap4-utf-7" encoding to your encoding of preference. My Ubuntu installation is all UTF-8 so I convert all folder names to this encoding:

The "imap4-utf-7" encoding is added by our "utf7.py" script. With this configuration I can now see the correct Japanese names for all the folders (labels) in my GMail accounts inside Mutt [4].

Resources

[1] http://software.complete.org/software/wiki/offlineimap
[2] http://software.complete.org/software/issues/show/102
[3] IMAP4 UTF-7 Encoding/Decoding
[4] Best email client

Google Contacts in Mutt

2010-01-26T22:35:00.039+09:00

Introduction

For some time now I have managing all my contact information (a.k.a address book) in GMail because this allows me for easy synchronization of contacts with my cell phone that is a Google Phone HTC Hero.

What I was missing was the ability to query the GMail contacts from within my favorite email client: Mutt and the solutions I found in Google were not working for me.

The first solution I found [1] was to export all my GMail contacts to vcards and then import these to abook [2] that is an address book compatible with Mutt. Not really a solution should I say.

The second solution [3] I found is actually pretty good. Is a small python script that can be used within mutt using the "query_command" [4] and actually queries GMail contacts web service API. Also supports caching so you do not need to connect every time to GMail service speeding up the query time.

Unfortunately this solution did not work very well for me. Some contacts would not be auto completed when composing an email and some other would not even appear. For example contacts with non ASCII characters (e.g. Japanese Kanji) would never get autocompleted, even thought they appeared when listing all contacts.

So after some time I implemented my own solution, in Ruby, that works perfectly for me and also has some additional features for groups and mailing lists.

Query GMail Contacts within Mutt

Using the gdata gem [5] and some knowledge of the Google contacts web API [6] it was relatively easy to implement a Ruby script that could be used with Mutt to query my contacts directly from GMail:


#!/usr/bin/env ruby1.8
##
# File:   contactos.rb
# Author: Horacio Sanson (hsanson at gmail)
# Date:   2010/01/22
#
# Descr:
#    Small script to facilitate use of GMail contacts within mutt email client.
#
# Features:
#    - Generates a list of group and subscribe commands generated from your
#      GMail contacts groups that can be sourced from your muttrc file.
#    - Can be used to search for contacts within mutt using the query_command
#      option in the same way abook or ldbd are used.
#    - Results are cached for a period of time (configurable) to speed up
#      queries.
#
# Installation:
#    To use this script you need ruby and rubygems 1.8 and the gdata gem. To
#    install these in Ubuntu/Kubuntu use the following commands.
#
#      sudo aptitude install ruby1.8 rubygems-1.8 libopenssl-ruby1.8
#      sudo gem1.8 install gdata 
#
#    Make sure to read the comments below to learn how to configure this script
#    to work with your GMail account.
#
#    Then in your muttrc file add these lines:
#
#       source '/PATH_TO_SCRIPT/contactos.rb --groups|'
#       source '/PATH_TO_SCRIPT/contactos.rb --subscribes|'
#       set query_command ="/PATH_TO_SCRIPT/contactos.rb --mutt %s"
#       set query_format="%4c %t %-40.40a %-40.40n %?e?(%e)?"
#       bind editor <Tab> complete-query
#       bind editor ^T complete
#
#    with this configuration you will be able to use your GMail groups in all
#    regexp's and create hooks for them. Pressing <tab> when filling a To: or
#    From: header will give you a list of options to choose or auto complete the
#    address if only one matches.
#
# Resources:
#    - http://antonyjepson.wordpress.com/2009/06/27/using-your-gmail-contacts-in-mutt/
#    - http://www.chizang.net/alex/blog/code/mutt-query-gmail/
#    - http://castrojo.wordpress.com/2009/01/28/gmail-contacts-with-mutt/
#    - http://wiki.mutt.org/?QueryCommand
#    - http://code.google.com/apis/contacts/docs/3.0/reference.html#Feeds
#    - http://code.google.com/apis/gdata/articles/gdata_on_rails.html#GettingStarted
#
# TODO:
#   - Mutt only sources the group list once on startup. If we add new groups or
#     add contacts to a group these won't be reflected in mutt until we restart
#     it. Not really a problem because mutt has the fastest start up time.
#   - Create a script to add contacts from within mutt.
#      - Find if create-alias can accept a script as parameter.

require "rubygems"
require "gdata"
require "fileutils"
require "pp"

## Set up here your GMail account username and password
USERNAME="user"
PASSWORD="secret"

# You may create a group in you GMail contacts and add all the mailing lists
# you are subscribed into that group. This script will generate a bunch of
# subscribe commands for all addresses in that group.
MAILISTGROUP="Mailing Lists"

# Make sure this value is larger than the total number of contacts you have.
MAXRESULTS=999

# How much time before the local cache expires in seconds.
UPDATE_INTERVAL=3600

# Where to store the local cache for faster query times.
CACHEFILE="~/.mutt/cache/gcontacts"

###############################################################################
## DON'T CHANGE ANYTHING BELOW THIS POINT
##   unless you know what your are doing
###############################################################################

class Contact
  attr_accessor :emails, :groups
  attr_reader :name, :id
  def initialize(id, name="")
    @id = id
    @name = name
    @emails = []
    @groups = []
  end

  def mutt_match(pattern)
    return mutt_fmt if @name and @name =~ Regexp.new(pattern,Regexp::IGNORECASE)

    str = ""
    @emails.each { |email|
      str << "#{email}\t#{@name||email}\t#{@groups.first}\n" if email =~ Regexp.new(pattern,Regexp::IGNORECASE)
    }
    str
  end

  def mutt_fmt
    str = ""
    @emails.each { |email|
       str << "#{email}\t#{@name||email}\t#{@groups.first}\n"
    }
    str
  end
end

# This method updates the local cache if the cache file is not present or if the
# UPDATE_INTERVAL has expired.
def update_cache
  user_hash = nil
  if ! File.exists?(File.expand_path(CACHEFILE)) or Time.now - File.stat(File.expand_path(CACHEFILE)).mtime > UPDATE_INTERVAL
    #STDERR << "Updating from gmail\n"
    user_hash = {}
    client = GData::Client::Contacts.new

    begin
      client.clientlogin("#{USERNAME}@gmail.com", PASSWORD)
    rescue GData::Client::AuthorizationError
      STDERR << "Failed to authenticate\n"
      return nil
    rescue => e
      STDERR << "Failed to log into Gmail: #{e}\n"
    end

    # Create a hash list of all groups
    group_hash = {}
    groups = client.get("http://www.google.com/m8/feeds/groups/#{USERNAME}%40gmail.com/full?max-results=#{MAXRESULTS}").to_xml
    groups.elements.each('entry') { |entry|
      name = entry.elements['title'].text.gsub("System Group: ","")
      id = entry.elements['id'].text
      group_hash[id] = name
    }

    # Create a hash list of all users
    feeds = client.get("http://www.google.com/m8/feeds/contacts/#{USERNAME}%40gmail.com/full?max-results=#{MAXRESULTS}").to_xml
    feeds.elements.each('entry') { |entry|
      name = entry.elements['title'].text
      id = entry.elements['id'].text
      new_contact = Contact.new(id, name)

      entry.elements.each('gd:email') { |email|
        new_contact.emails << email.attribute('address').value
      }
      entry.elements.each('gContact:groupMembershipInfo') { |group|
        new_contact.groups << group_hash[group.attribute('href').to_s]
      }
      user_hash[id] = new_contact
    }

    File.open(File.expand_path(CACHEFILE),"wb") { |fd|
      fd << Marshal.dump(user_hash)
    }
  end
end

def load_cache
  #STDERR << "Updating from local cache\n"
  user_hash = {}
  if File.exists?(File.expand_path(CACHEFILE))
    File.open(File.expand_path(CACHEFILE),"rb") { |fd|
      user_hash = Marshal.load(fd.read)
    }
  end

  return user_hash
end

def print_help
 puts "usage: "
 puts "  contactos.rb --mutt [pattern]"
 puts "  contactos.rb --groups"
 puts "  contactos.rb --subscribes"
 puts "  contactos.rb --aliases"
end

if ARGV.empty?
  print_help
else
  update_cache
  contacts = load_cache
  case ARGV[0]
    when "--mutt"
      puts ""   # Mutt ignores the first line 
      if ARGV[1]
        contacts.each { |k, contact|
          STDOUT << contact.mutt_match(ARGV[1])
        }
      else
        contacts.each { |k, contact|
          STDOUT << contact.mutt_fmt
        }
      end
    when "--groups"
      contacts.each { |k,contact|
        contact.groups.each { |g|
          puts "group -group \"#{g}\" -addr #{contact.emails.join(', ')}" if ! contact.emails.empty?
        }
      }
    when "--subscribes"
      contacts.each { |k,contact|
        contact.groups.each { |g|
          if g == MAILISTGROUP and contact.emails.size > 0
            puts "subscribe #{contact.emails.join(' ')}"
          end
        }
      }
    when "--aliases"
      contacts.each { |k,contact|
        contact.emails.each { |e|
          if contact.name
            puts "alias \"#{contact.name}\" #{contact.name} <#{e}>"
          else
            puts "alias \"#{e}\" #{e}"
          end
        }
      }
    when "-v"
    when "--version"
      puts "Contactos.rb version 0.2.0, Copyright 2010 Horacio Sanson"
    when "-h"
    when "--help"
      print_help
    else
      print_help
  end
end

Simply copy the above script somewhere in your machine (e.g. ~/.mutt/contactos.rb) and change the USERNAME and PASSWORD variables. Give the script execution property and some secure permissions:

chmod 700 ~/.mutt/contactos.rb

You may also change the MAXRESULTS, UPDATE_INTERVAL and CACHEFILE variables to fit your needs. Just make sure MAXRESULTS is larger than the total number of contacts you have and that you have write access to the CACHEFILE you set. If you set UPDATE_INTERVAL to zero then the scripts queries GMail directly every time that can be slow depending on you network connection.

Also make sure you have installed Ruby and the gdata gem. In Ubuntu/Kubuntu this can be accomplished with the following commands:

sudo aptitude install ruby1.8 rubygems-1.8 libopenssl-ruby1.8
sudo gem1.8 install gdata

Finally configure Mutt to use this script to query contacts when you are filling the To: and From: headers of emails. To do this add these commands in your muttrc file:

set query_command ="~/mutt/contactos.rb --mutt %s"
set query_format="%4c %t %-40.40a %-40.40n %?e?(%e)?"
bind editor complete-query
bind editor ^T complete

After this you can press to autocomplete contacts using you GMail contact list and all without leaving the comfort of Mutt.

Integrating GMail contact groups within Mutt

After finishing my contacts script I realized that I could make even more things thanks to the rich Google contacts API and the scripting power and simplicity of ruby.

To integrate Mutt with you Google contact groups simply add this to your muttrc file:

source '~/.mutt/contactos.rb --groups|'

This command generates a list of "group" commands that map all GMail contact groups with Mut groups. Once this command is inserted in your muttrc file you may search, tag, delete and create hooks based on your GMail groups and manage the groups within GMail itself.

Managing Mutt mailing lists within GMail

One anoying thing about Mutt mailing lists is that you have to add a subscribe command for each mailing list you read in order to get all the features Mutt offers to handle mailing lists.

With my script you can create a special group for mailing lists (e.g. Mailing Lists) and add all the addresses of all mailing lists you are subscribed into that group. Then you can add this command to your muttrc file:

source './~mutt/contactos.rb --subscribes|'

and magically you get all the subscribe commands automagically generated for you from you GMail contacts. If you prefer to use a different name for you mailing list group you have to change the MAILISTGROUP variable in the ruby script to match the name you have in you GMail contacts.

Resources

Brother DCP-595CN Printer In Ubuntu/Kubuntu

2010-01-03T11:30:00.010+09:00

This little machine is a printer, scanner and copy machine all in one. It has USB and ethernet ports but even better a wireless 802.11a/b/g port. This means I can have the printer anywhere in the house and print from my desktop or lapton freely. In Japan it costs around ￥15,000 (USD 150) that is pretty cheap for all the features packaged in.

And the most important thing for me is that is 100% compatible with linux (Kubuntu). Here I will show the steps to get this little beast working with my Kubuntu machine.

Prerequisites

I use the wireless interface to connect to the printer/scanner so before you follow these instructions make sure the printer has an IP configured. The wireless configuration is done using the printer control panel and is already covered in the printer manual so I will skip it.

You may ping the printer IP address to check that you can reach it from your computer. If you cannot ping it then it is not well configured or you have problems with your LAN network.

Installation

These instructions are based on the Brother solution page instructions.

First we need to download both the LPR driver and the CUPS wrapper driver by going to the following URLs. Inside you will find RPM and DEB packages for the different printer models.

If you have the same printer as me (DCP-595CN) then look for the DEB packages of this printer and click the link. You will be directed to a GPL license page where you must agree to the GPL terms (Left Button) after which the download will start.

Once the DEB packages are downloaded we install them using the following commands:

sudo mkdir /usr/share/cups/model
sudo mkdir /var/spool/lpd
sudo mkdir -p /usr/lib/cups/filter
sudo dpkg -i dcp595cnlpr-1.1.2-1.i386.deb
sudo dpkg -i dcp595cncupswrapper-1.1.2-2.i386.deb

note that there are only packages for i386 architectures but if you see the Q&A link they explain how to install on amd64 architectures. All you have to do is add the --force-arquitecture switch to the dpkg command like:

sudo dpkg --force-arquitecture -i dcp595cnlpr-1.1.2-1.i386.deb
sudo dpkg --force-arquitecture -i dcp595cncupswrapper-1.1.2-2.i386.deb

Now with the drivers installed we can use the CUPS administration page to add the printer. To do this we open our favorite Internet browser and access the following URL: http://localhost:631.

Enter the printer administration tab and add a new printer. If your printer is already configured with an IP address in your LAN it should appear in the list of detected printers (see screenshot below):

Select the printer and in the next window you will be asked to add some details about the printer like the place, description and name. You may add anything you like or leave the defaults and go to the next screen.

In the next screen (see screenshot below) we select the printer driver to use. If you installed the DEB packages correctly then the correct driver should be selected here by default:

Simply press the "add printer" button to finish. Now your printer should be ready for printing. Go ahead the print something.

Brother DCP-595CN Scanner In Ubuntu/Kubuntu

Getting the scanner working in Kubuntu is even easier. First we need to download the SANE driver from Brothers home page. Which driver to download depends on the scanner model you have. In my case (DCP-595CN) I need the brscan3 driver. There are 32 and 64bit versions and RPM and DEB packages. For my configuration I need the brscan3 32bit DEB pacakge. Download the package for your configuration and install it using this command:

sudo aptitude install xsane sane-utils brscan3-0.2.8-1.i386.deb

In the command above I am installing some SANE packages and the brscan3 driver. Next we need to configure the scanner with the following command:

sudo brsaneconfig3 -a name=DCP-595CN model=DCP-595CN ip=192.168.11.4

Depending on the driver you installed the configuration command changes. For brscan3 we use the "brsaneconfig3" command. The name can be anything you want as long as it has no white spaces but the model and the ip must be set to the correct values. The ip of course is the IP address the printer has configured and the model is the model name of the printer. If you are not sure about the model or IP you can use the following command to query the network:

sudo brsaneconfig3 -q

This command gives a list of supported models and at the end a list of detected scanners. In my configuration at the end I get:

Devices on network
0 DCP-595CN           "DCP-595CN"         I:192.168.11.4

If you do not see this list then your machine is not able to reach the scanner on the IP level so make sure it has the IP address configured and that you can ping it.

After you have configured the scanner with the brscanconfig3 tool you are ready to scan. To test it you can run xsane and the scanner should be listed like in the screenshot below:

Select the scanner and scan something. Now you have a fully functional printer, scanner and copy machine in you Kubuntu installation.

Synchronizing two machines

2009-12-27T13:37:00.004+09:00

Synchronizing two machines                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                               
  Introduction                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                               
    The method I document here shows how I use unison in combination with                                                                                                                                                                        
    incrontab to keep some folders shared between my work and home machines in                                                                                                                                                                   
    sync almost in real time.                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                               
  Background                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                               
    This is only a rant so you can skip this if you are in a hurry.                                                                                                                                                                              
                                                                                                                                                                                                                                                                               
    Before this setup I had a cron job with a rsync task to sync my shared                                                                                                                                                                       
    folders. The problem is that rsync is one-way sync tool and works well                                                                                                                                                                       
    when you have a master machine where you add/modify/delete files and then                                                                                                                                                                    
    rsync one or more slave machines to the master. In my case I have two                                                                                                                                                                        
    machines that are both master and changes on any of them have to be                                                                                                                                                                          
    propagated to the other.  Of course this master-master configuration can                                                                                                                                                                     
    be done with rsync but you really MUST be very careful on what order you                                                                                                                                                                     
    sync (e.g.  from A to B or from B to A) when you use the --delete switch                                                                                                                                                                     
    (see [1] and [2] fpr details). Not using the --delete switch means that no                                                                                                                                                                   
    file in those machines will ever be deleted unless you manually delete the                                                                                                                                                                   
    file from both sides.  I have been using rsync with a crob job for years                                                                                                                                                                     
    but a few months ago I was victim of the --delete switch. Looking for                                                                                                                                                                        
    alternatives I found unison that does the master-master configuration                                                                                                                                                                        
    amazingly well and at the same time I found about incrontab. Both unison                                                                                                                                                                     
    and incrontab together resulted in a very good setup that I will now                                                                                                                                                                         
    document in the rest of this post.                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                               
  Two way sync of two machines (master-master backup)                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
    Unison will keep two machines in sync taking care of the --delete flag for                                                                                                                                                                   
    us and even has an option to keep several days of backups of any file it                                                                                                                                                                     
    deletes so you can check once a week if it deleted a file you did not want                                                                                                                                                                   
    deleted.                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                               
    To install unison in (K)Ubuntu we simply do:                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                               
    ; sudo aptitude install unison                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                               
    Note that unison MUST be installed in the two machines you want to keep in                                                                                                                                                                   
    sync. As additional bonus this tool works in all major operating systems                                                                                                                                                                     
    so you can sync your work and home machines even if you use Windows at                                                                                                                                                                       
    work and Linux at home.                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                               
    Now to sync some folders between machines you issue a simple command:                                                                                                                                                                        
                                                                                                                                                                                                                                                                               
    ; unison /home/user ssh://<ipaddress>//home/user      \                                                                                                                                                                                 
    ; -path .vim -path .vimrc -path .muttrc -path .mutt   \                                                                                                                                                                                       
    ; -path Music -path Photos -path Docs                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                               
    In the above example I sync my vim, mutt configuration files and my Music,                                                                                                                                                                   
    Photos and Docs folders in my home directory. Replace the ipaddress with                                                                                                                                                                     
    the address of the remote machine and note that the double slash (//)                                                                                                                                                                        
    after the ipaddress is not a typo, you MUST have two slashes there when                                                                                                                                                                      
    using absolute paths in the remote side.                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                               
    This command can be executed in any of the two machines and the result                                                                                                                                                                       
    will be the same. Both of them will have the directories selected (-path)                                                                                                                                                                    
    syncrhonized to the last detail.                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                               
    Since I use ssh as transport I also set up my machines to authenticate                                                                                                                                                                       
    using ssh keys and configure Kwallet [3] as my ssh passphrase manager.                                                                                                                                                                       
    This is very important if we want to automate the task via cron or incron                                                                                                                                                                    
    jobs as I show later in this post.                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                               
    To simplify the synchronization task I created a small bash script that                                                                                                                                                                      
    loads my ssh key manager (keychain), executes unison to synchronize my                                                                                                                                                                       
    machines while logging everything to /tmp/unison.log. 
    <SCRIPT                                                                                                                                                                                                                                 
    < #!/usr/bin/env bash                                                                                                                                                                                                                   
    < LOGGER=/usr/bin/logger                                                                                                                                                                                                                
    < UNISON=/usr/bin/unison                                                                                                                                                                                                                
    < KEYCHAIN=/usr/bin/keychain                                                                                                                                                                                                            
    < $LOGGER -t unison -p cron.info "Unison started for user $HOME"                                                                                                                                                              
    < if [ -e /tmp/unison.lock ]                                                                                                                                                                                                            
    < then                                                                                                                                                                                                                                  
    <   $LOGGER -t unison -p cron.info "Unison already running... abort"                                                                                                                                                          
    < else                                                                                                                                                                                                                                  
    <   echo "running" > /tmp/unison.lock                                                                                                                                                                                      
    <   $KEYCHAIN id_rsa                                                                                                                                                                                                                    
    <   source $HOME/.keychain/$HOSTNAME-sh                                                                                                                                                                                                 
    <   $UNISON $HOME ssh://192.168.1.10/$HOME -batch -log -logfile /tmp/unison.log \                                                                                                                                                       
    <                           -path .vim -path .vimrc -path .muttrc -path .mutt   \                                                                                                                                                       
    <                           -path Music -path Photos -path Docs                                                                                                                                                                         
    <   [ $? -eq 0 ] && $LOGGER -t unison -p cron.info "Unison finished" || $LOGGER -t unison -p cron.info "Unison failed with error $?"                                                                        
    <   rm -f /tmp/unison.lock                                                                                                                                                                                                              
    < fi                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                               
  Using incrontab for almost real time two way file sync                                                                                                                                                                                       
                                                                                                                                                                                                                                                                               
    Now that I can sync my two machines with a simple script I wanted this to be                                                                                                                                                                 
    automated. The normal way would be using a cron job but a problem I always                                                                                                                                                                   
    had with cron was how to set the time period. Once an hour or a day or a                                                                                                                                                                     
    minute? While researching I found incrontab that is like cron but it reacts                                                                                                                                                                  
    to file system events. With it I could invoke my unison script every time a                                                                                                                                                                  
    file is created or edited that would result in almost real time sync between                                                                                                                                                                 
    my machines.                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                               
    To install incrontab in (K)Ubuntu simply do:                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                               
    ; sudo aptitude install incron                                                                                                                                                                                                                
    ; echo "username" >> /etc/incron.allow                                                                                                                                                                                        
                                                                                                                                                                                                                                                                               
    The first command installs incron and the second one adds your user name                                                                                                                                                                     
    to the list of allowed incron users. If your username is not in the                                                                                                                                                                          
    "incron.allow" file then you wont be able to use incrontab.                                                                                                                                                                        
                                                                                                                                                                                                                                                                               
    A common problem that happens when using rsync or unison from within cron                                                                                                                                                                    
    and incron is how to handle the authentication of the ssh session.                                                                                                                                                                           
    Remember that the cron and incron processes are running in a different                                                                                                                                                                       
    environment as your user so setting a key manager like ssh-agent or                                                                                                                                                                          
    keychain in the current user termial wont work. If you have ever had rsync                                                                                                                                                                   
    work perfectly from a terminal but fail with an error like: unexplained                                                                                                                                                                      
    error (code 255) then you know what I am talking about.                                                                                                                                                                                      
                                                                                                                                                                                                                                                                               
    To solve this problem you first need to set your machines to use                                                                                                                                                                             
    password-less ssh key authentication. This way you can execute remote                                                                                                                                                                        
    commands via ssh without need to input a password.                                                                                                                                                                                           
                                                                                                                                                                                                                                                                               
    Second you need to setup a keymanager like ssh-agent or keychain. I found                                                                                                                                                                    
    that keychain does a better job in keeping a single instance of itself to                                                                                                                                                                    
    handle the keys on multiple processes. So I setup keychain and configure                                                                                                                                                                     
    Kwallet [3] as the interface that asks and stores the ssh pashphrases.                                                                                                                                                                       
    With this setup Kwallet asks me to unlock it at KDE startup and from that                                                                                                                                                                    
    point on it takes care of providing authentication to all processes that                                                                                                                                                                     
    require ssh loging (incrontab included).                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                               
    Once you have setup the key manager you can create an incrontab rule to                                                                                                                                                                      
    invoke the unison script (from the previous section) every time a file is                                                                                                                                                                    
    changed:                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                               
    ; incrontab -e                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                               
    This command will open you text editor (vim) with your user's incrontab                                                                                                                                                                      
    configuration file. All you have to do is to input the following line and                                                                                                                                                                    
    save/close the editor:                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                               
    ; /home/user IN_ATTRIB,IN_DONT_FOLLOW,IN_NO_LOOP /home/user/unison.sh                                                                                                                                                                         
                                                                                                                                                                                                                                                                               
    This incrontab rule invokes "unison.sh" every time an attribute changes in                                                                                                                                                         
    a file or folder inside my home directory. The IN_ATTRIB switch may be                                                                                                                                                                       
    overkill and you may prefer to use IN_CREATE or IN_CLOSE_WRITE. For                                                                                                                                                                          
    details on incrontab switches you may read the man page:                                                                                                                                                                                     
                                                                                                                                                                                                                                                                               
    ; man 5 incrontab 

    This command will open you text editor (vim) with your user's incrontab                                                                                                                                                                      
    configuration file. All you have to do is to input the following line and                                                                                                                                                                    
    save/close the editor:                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                               
    ; /home/user IN_ATTRIB,IN_DONT_FOLLOW,IN_NO_LOOP /home/user/unison.sh                                                                                                                                                                         
                                                                                                                                                                                                                                                                               
    This incrontab rule invokes "unison.sh" every time an attribute changes in                                                                                                                                                         
    a file or folder inside my home directory. The IN_ATTRIB switch may be                                                                                                                                                                       
    overkill and you may prefer to use IN_CREATE or IN_CLOSE_WRITE. For                                                                                                                                                                          
    details on incrontab switches you may read the man page:                                                                                                                                                                                     
                                                                                                                                                                                                                                                                               
    ; man 5 incrontab                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                               
    The IN_NO_LOOP avoids the script from being invoked several times when                                                                                                                                                                       
    more than one file changes at a time. Also the script I provided uses a                                                                                                                                                                      
    simple lock mechanism to avoid it from running several times in parallel.                                                                                                                                                                    
                                                                                                                                                                                                                                                                               
    As a final note you can open /var/log/syslog and search for all entries                                                                                                                                                                      
    with the unison tag to see if the script is running correctly or failing.                                                                                                                                                                    
    If it is failing you can further chech the /tmp/unison.log file to see why                                                                                                                                                                   
    it is failing. Note that an abort is not an error, it means that then                                                                                                                                                                        
    unison script was invoked several times in succession and we only allow                                                                                                                                                                      
    one instance at any given time.                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                               
  Resources                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                               
    [1] http://www.darcynorman.net/2006/10/03/be-careful-with-rsync-delete/                                                                                    
    [2] http://www.davidgrant.ca/rsync_delete_dangerous                                                                                                                            
    [3] http://piao-tech.blogspot.com/2009/12/manage-ssh-and-gpg-keys-efficiently-in.html

Manage SSH and GPG keys efficiently in KDE

2009-12-17T17:02:00.010+09:00

Manage SSH and GPG keys efficiently in KDE                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                               
  Introduction                                                                                                                                                                                                                                 
    Here I present a simple way to handle SSH and GPG keys easily in KDE4.                                                                                                                                                                       
    These method presents several advantages:                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                               
      * Works great with KDE4 that ships with Kubuntu Maverick Meerkat and may                                                                                                                                                                    
        work in other KDE distributions without too much trouble.                                                                                                                                                                                 
      * Use Kwallet as the passphrase manager so you unlock Kwallet once on                                                                                                                                                                       
        login and from there it will handle all passphrase requests.                                                                                                                                                                              
      * Works great with automated tasks (via cron or incron) that use SSH key                                                                                                                                                                    
        authentication (e.g. famous rsync unexplained error 255).                                                                                                                                                                                 
                                                                                                                                                                                                                                                                               
  Pre-requisities                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                               
    Considering that you are reading this post because you need a better way                                                                                                                                                                     
    to manage you ssh and gpg keys then it is safe to assume you have already                                                                                                                                                                    
    generated your ssh/gpg keys and that you know how to use them.                                                                                                                                                                               
                                                                                                                                                                                                                                                                               
  Disable KDE4 from starting ssh-agent and gpp-agent                                                                                                                                                                                           
                                                                                                                                                                                                                                                                               
    Kubuntu 10.10 by default starts the ssh-agent and gpg-agent causing some                                                                                                                                                                     
    conflicts with this setup based on keychain. Using the default                                                                                                                                                                               
    configuration does not seem to use kwallet and certainly does not work                                                                                                                                                                       
    with kmail/mutt so I prefer to disable these and enable keychain instead.                                                                                                                                                                    
                                                                                                                                                                                                                                                                               
    To disable the default ssh-agent edit the "/etc/X11/Xsession.options" file                                                                                                                                                         
    and comment out the line that says use-ssh-agent.                                                                                                                                                                                            
                                                                                                                                                                                                                                                 
    To disable the default gpg-agent edit the "~/.gnupg/gpg.conf" file and                                                                                                                                                             
    comment out the line that says use-agent.                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                               
    With these default agent's disabled now we can configure KDE to use                                                                                                                                                                          
    keychain that I consider a superior tool to handle ssh/gpg keys.                                                                                                                                                                             
                                                                                                                                                                                                                                                                               
  SSH/GPG Key management with KWallet                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                               
    First we install the needed software packages:                                                                                                                                                                                               
                                                                                                                                                                                                                                                                               
    ; sudo aptitude install keychain ksshaskpass kwalletcli                                                                                                                                                                                       
                                                                                                                                                                                                                                                                               
    configure the "~/.gnupg/gpg-agent.conf" file so it uses the kwallet                                                                                                                                                                
    pinentry program to manage gpg keys. Simply add the pinentry-program line                                                                                                                                                                    
    or replace it if it already exists with:                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                               
    ; pinentry-program /usr/bin/pinentry-kwallet                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                               
    Now we need to load keychain and all the environment variables it sets                                                                                                                                                                       
    when KDE starts. To do this we simply create a small script, say                                                                                                                                                                             
    "keychain.sh" and put it inside out ".kde/env" directory. The script                                                                                                                                                     
    contains these lines:                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                               
    <SCRIPT                                                                                                                                                                                                                                 
    < #!/bin/sh                                                                                                                                                                                                                             
    < ################################################################################                                                                                                                                                      
    < # Load keychain to handle ssh and gpg keys                                                                                                                                                                                            
    < ################################################################################                                                                                                                                                      
    < if [ -f /usr/bin/keychain ]; then                                                                                                                                                                                                     
    <   if [ -f /usr/bin/ksshaskpass ]; then                                                                                                                                                                                                
    <     export SSH_ASKPASS=/usr/bin/ksshaskpass                                                                                                                                                                                           
    <   else                                                                                                                                                                                                                                
    <     export SSH_ASKPASS=/usr/bin/askpass                                                                                                                                                                                               
    <   fi                                                                                                                                                                                                                                  
    <   /usr/bin/keychain                                                                                                                                                                                                                   
    <   $HOME/.keychain/`hostname`-sh                                                                                                                                                                                                       
    <   $HOME/.keychain/`hostname`-sh-gpg                                                                                                                                                                                                   
    < fi                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                               
    what this does is to setup the SSH_ASKPASS environment variable to use the                                                                                                                                                                   
    ksshaskpass program that handles ssh keys inside KWallet. Then invokes                                                                                                                                                                       
    keychain which starts the ssh-agent and gpg-agent daemons and sets some                                                                                                                                                                      
    environment variables so all KDE applications can see them.                                                                                                                                                                                  
                                                                                                                                                                                                                                                                               
    Finally we must load our ssh/gpg keys into keychain. The best place to do                                                                                                                                                                    
    this is with the KDE Autostart scripts. Simply create a script, say                                                                                                                                                                          
    add_keys.sh, into you ".kde/Autostart" folder that contains something                                                                                                                                                              
    like: 
                                                                                                                                                                                                                                                                               
    <SCRIPT                                                                                                                                                                                                                                 
    < #!/bin/sh                                                                                                                                                                                                                             
    < ################################################################################                                                                                                                                                      
    < # Load keychain to handle ssh and gpg keys                                                                                                                                                                                            
    < ################################################################################                                                                                                                                                      
    < if [ -f /usr/bin/keychain ]; then                                                                                                                                                                                                     
    <   if [ -f /usr/bin/ksshaskpass ]; then                                                                                                                                                                                                
    <     export SSH_ASKPASS=/usr/bin/ksshaskpass                                                                                                                                                                                           
    <   else                                                                                                                                                                                                                                
    <     export SSH_ASKPASS=/usr/bin/askpass                                                                                                                                                                                               
    <   fi                                                                                                                                                                                                                                  
    <   /usr/bin/keychain                                                                                                                                                                                                                   
    <   $HOME/.keychain/`hostname`-sh                                                                                                                                                                                                       
    <   $HOME/.keychain/`hostname`-sh-gpg                                                                                                                                                                                                   
    < fi                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                               
    what this does is to setup the SSH_ASKPASS environment variable to use the                                                                                                                                                                   
    ksshaskpass program that handles ssh keys inside KWallet. Then invokes                                                                                                                                                                       
    keychain which starts the ssh-agent and gpg-agent daemons and sets some                                                                                                                                                                      
    environment variables so all KDE applications can see them.                                                                                                                                                                                  
                                                                                                                                                                                                                                                                               
    Finally we must load our ssh/gpg keys into keychain. The best place to do                                                                                                                                                                    
    this is with the KDE Autostart scripts. Simply create a script, say                                                                                                                                                                          
    add_keys.sh, into you ".kde/Autostart" folder that contains something                                                                                                                                                              
    like:                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                               
    <SCRIPT                                                                                                                                                                                                                                 
    < #!/bin/sh                                                                                                                                                                                                                             
    < ################################################################################                                                                                                                                                      
    < # Load keychain to handle ssh and gpg keys                                                                                                                                                                                            
    < ################################################################################                                                                                                                                                      
    < if [ -f /usr/bin/keychain ]; then                                                                                                                                                                                                     
    <  /usr/bin/keychain id_rsa 0x12345 0x23456                                                                                                                                                                                             
    <  $HOME/.keychain/`hostname`-sh                                                                                                                                                                                                        
    <  $HOME/.keychain/`hostname`-sh-gpg                                                                                                                                                                                                    
    < fi                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                               
    what this Autostart script does is to load your ssh key (id_rsa) and gpg                                                                                                                                                                     
    keys (0x12345, 0x23456) into keychain. The next time you log into a KDE                                                                                                                                                                      
    session this script will ask you if you want to give keychain access to                                                                                                                                                                      
    Kwallet and then ask all the registered key passphrases. Once registered                                                                                                                                                                     
    with kwallet all your applications will be able to use these keys without                                                                                                                                                                    
    asking you for the passphrase each time.                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                               
    Make sure the env and Autostart scripts have exec privileges:                                                                                                                                                                                
                                                                                                                                                                                                                                                                               
    ; chmod +x ~/.kde/env/keychain.sh                                                                                                                                                                                                             
    ; chmod +x ~/.kde/Autostart/add_key.sh

How to create Netem distribution tables

2009-10-08T16:51:00.006+09:00

The Netem and Dummynet emulators are widely used in networking research to achieve emulated network environments that can be used to evaluate new protocols and applications under controlled and repeateable conditions. In the case of Netem it is possible to use custom ditribution tables to better approximate the round-trip-time variations in the nework.

Unfortunately there is little information on how to generate this distribution tables based on real measured data. The only reference available is in the iproute2 source code itself and is a little confusing. After some reading and re-reading I think I finally got the magic formula and here I present it so others can benefit of my work.

First download the iproute2 source code:

git clone \
git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git

Then enter the netem subdirectory and compile the maketable and stats utilities:

cd iproute2/netem
gcc -lm -o maketable maketable.c
gcc -lm -o stats stats.c

Next you need to extract some delay statistics of the channel you want to emulate. Suppose you want to emulate the network between a server in Japan and a client PC in Chile. Then simply run a ping between both machines for some period of time. A whole day of ping data would be enough:

ping 192.168.4.111 > mydata.txt

The command above does a ping to the address 192.168.4.111 and stores the output to a file named mydata.txt. The file will contain the round-trip-times between the local machine and the machine with address 192.168.4.111 like shown below:

PING 192.168.4.111 (192.168.4.111) 56(84) bytes of data.
64 bytes from 192.168.4.111: icmp_seq=1 ttl=43 time=306 ms
64 bytes from 192.168.4.111: icmp_seq=2 ttl=43 time=273 ms
64 bytes from 192.168.4.111: icmp_seq=3 ttl=43 time=310 ms
64 bytes from 192.168.4.111: icmp_seq=4 ttl=43 time=300 ms
64 bytes from 192.168.4.111: icmp_seq=5 ttl=43 time=300 ms
64 bytes from 192.168.4.111: icmp_seq=6 ttl=43 time=304 ms
...

The more data you have the better so leave the ping running for some long period of time. Next we need to extract the round-trip-time data only:


cat mydata.txt | grep icmp_seq | cut -d'=' -f4 | cut -d' ' -f1 > myrtt.txt

The above command will extract all the round-trip-times from mydata.txt and put them in a single column in a file named myrtt.txt. Finally we can create a distribution table that Netem understands using the maketable utility:

./maketable myrtt.txt > mydist.dist

This will create a distribution table and store it in the file mydist.dist that must be copied in a special folder (/usr/lib/tc) so Netem can find it:

sudo cp mydist.dist /usr/lib/tc

Additionally we can obtain the mean and standard deviation of the round-trip-time using the stats utility:

./stats myrtt.txt
> mu = 296.636364
> sigma = 12.808378
> rho = -0.103472

This command gives use the mean (mu) the deviation (sigma) and the correlation (rho) of the round-trip-time. And at last we can now use Netem to emulate the channel's round-trip-time using the following command:


sudo tc  qdisc add dev eth0 root netem delay 296.636364ms 12.808378ms distribution mydist

Note the the distribution name is the same as that of the file we copied inside the /usr/lib/tc folder but without the .dist extension. Now Netem will provide you with a channel that has similar round-trip-time characteristics as those of the real channel.

Embbed cover art in MP3 files with command line

2009-09-03T11:51:00.010+09:00

Introduction

I needed a command line (cli) application that would allow me to embed cover art to my MP3 collection in order to get the covers diplay correctly in my wife's iPhone and my own gPhone (HTC-Hero). After looking around I finally found the ideal tool: eyed3.

Embedding cover art in MP3 files

In Ubuntu/Kubuntu install ImageMagick and Eyed3 with the following command:

then you can use these tools to embed the image in the MP3 files:

The first command uses ImageMagick to resize the image if any dimension is larger than 300px and the second command adds the cover art to all the MP3 files present in the current directory.

Note that the cover image is called AlbumArt.jpg and that the resize operation overwrites it. If you prefer to keep the original change the name of the output to something else. Also you can embed more than one image to each MP3 and assign it different types like FRONT_COVER, BACK_COVER, MEDIA, etc. Check eyeD3 help for more information.

The embedded cover art will display correctly in iTunes, iPhone and Android without problems.

Android cover art for Ogg/Vorbis and MP3 files

All my Japanese music is in Ogg/Vorbis format mostly because MP3 players cannot agree in the metadata encoding for Japanese characters. Each MP3 player I have tried (including Amarok) use a custom (i.e. no standard) way to save non ASCII characters making it impossible to display Japanese correctly in all devices/players I own.

In the other hand Ogg/Vorbis displays the Japanese correctly in all players I use (Amarok/gPhone/ffmpeg/mplayer) but it does not support embedded cover art. Fortunately in Android phones if there is a file named AlbumArt.jpg the same folder the Ogg/Vorbis file is located then it will use that file as cover art.

With this layout both Amarok and my Android Phone display the AlbumArt.jpg for all the ogg/mp3 files contained inside the same folder.

Handling multiple encodings in Vim

2009-06-12T13:27:00.003+09:00

Most people edit, load and save files in a single character encoding (i.e. en_US.UTF-8) but for many this is not the case. For me I need to write documents and emails in japanese UTF-8 (ja_JP.UTF-8), Latex in EUC-JP (ja_JP.EUC-JP) and source code comments in Shift-JIS (ja_JP.SJIS).

To handle a certain character encoding there are three things you must consider:

Your editor character encoding

Your terminal character encoding

Your font character encoding support

In the prehistoric era editors and terminal could support a single character encoding so you needed a different pair for each encoding you needed. Believe me when I tell you this was not fun at all.

These days most editors and terminals support a large array of character encodings and a lot of free fonts are available that support all the character sets I need. Still it is necessary to reconfigure each part (editor/terminal) or create different profiles for each character encoding you needed to edit (i.e. link) for the editor and the console.

Today I took the time to understand how vim character encoding support works and I found that it has everything I need and a lot more. By carefully manipulating the the fenc, fencs, enc and tenc configuration parameters I can edit any file in any character encoding with little effort. Here are my .vimrc configuration parameters:

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

"" Character encoding settings

"" By manipulating this variables it is possible to edit all files in one

"" encoding while using the terminal in a different encoding and writing/reading

"" the file in another encoding. Here we set all three variables to UTF-8.

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

" Default file encoding for new files

setglobal fenc=utf-8

" Auto detect file encoding when opening a file. To check what file encoding was

" selected run ":set fenc" and if you know the auto detection failed and want to

" force another one run ":edit ++enc=<your_enc>".

set fencs=utf-8,euc-jp,sjis

" Internal encoding used by vim buffers, help and commands

set enc=utf-8

" Terminal encoding used for input and terminal display

" Make sure your terminal is configured with the same encoding.

set tenc=utf-8

tenc:
This is the character encoding used to display and input text to the terminal. I configure my terminal (Konsole) always in UTF-8 and as far as I know my input method for Japanese (scim/anthy) is also UTF-8 so to avoid visual/input problems I leave this in UTF-8.
enc:
The encoding used internally by vim buffers, help and commands. This does not need to be the same as tenc as vim will convert from one encoding to the other if they differ. This way you can use your native language encoding in your terminal and input method and let vim handle everything internally using UTF-8.

fenc:
Is the character encoding used for reading/writing files. Again this can differ from enc and tenc because vim will convert between them if they differ. This way you can have your terminal configured with your native language (i.e. Japanese, Russian, Chinese...), let vim work internally in UTF-8 and finally save your files in any coding you want by setting fenc.

fencs:
This is used by vim to try to auto detect the character encoding when opening an already existing file. The order in which you put the options is important so read the help ":h fencs" to learn how to set this correctly. For example if I put euc-jp first in the list all my English documents will be detected as euc-jp instead of utf-8 because all English characters are a subset of euc-jp, the same goes for latin1 and cp1250 encodings so make sure to put these at the end of the list. If the auto detection fails and your document is not displayed correctly you can always reload it forcing an encoding using ":edit ++enc=euc-jp" of course replace euc-jp with your desired encoding.

In my example above I set everything to UTF-8 that is recommended because converting from other encodings may cause loss of information. The only parameter I change is fenc when I need edit/save a file in a different encoding.

For example if I want to create a new file in euc-jp encoding:


  - Open new file as normal using vim< 
  - Set file encoding using :set fenc=euc-jp
  - Edit/Save as much as you like and rest assured that your file is euc-jp.

To edit an existing file simply open it and let vim auto detect the encoding using the options available in fencs. To check what encoding was set by vim you can use the command ":set fenc" and it will display the auto detected encoding. If it is not the correct one you can force the encoding by reloading the file using the command ":edit ++enc=euc-jp" replacing "euc-jp" with the encoding you desire.

Vim tips for Latex editing

2009-06-12T13:26:00.008+09:00

Simple Latex Makefile

With this simple Makefile you can compile your latex projects from within vim and use the quickfix window in case of errors to traverse and fix them one by one.

#

# This makefile generates a PDF of the MAINTEX file.

# The output can be found inside the build subdirectory.

#

# Makefile based on  http://www.wlug.org.nz/LatexMakefiles

#

#

# This makefile has been tested in Kubuntu 9.04

#

# Prerequisites:

#

# sudo aptitude install ptex-bin ptex-base okumura-clsfiles ptex-jisfonts \

#     xdvik-ja dvipsk-ja dvi2ps gv jbibtex-bin jmpost mendexk okumura-clsfiles \

#     vfdata-morisawa5 dvi2ps-fontdesc-morisawa5 texlive-latex-extra latexmk \

#     dvipng xpdf gs-cjk-resource vfdata-morisawa5 dvi2ps-fontdesc-morisawa5 \

#     cmap-adobe-japan1 cmap-adobe-japan2 cmap-adobe-cns1 cmap-adobe-gb1

# sudo jisftconfig add

#

# When writing Japanese make sure your editor is saving the tex files

# in euc-jp encoding. In VIM this can be accomplished by setting the fenc

# variable:

#

#  - Open file as normal using vim <filename>

#  - Type  :edit ++enc=euc-jp

#  - Type  :set fenc=euc-jp

#  - Type  :set enc=utf-8

## TODO Add rule to convert images

## TODO Add makeindex to create .toc files

###########################################################################

## Put here the file name of the main tex file.

MAINTEX      = main

############################################################################

## Change the followin only if you know what you are doing

LATEXCMD     = platex     # [latex | platex]

BIBCMD       = jbibtex    # [bibtex | jbibtex]

DVIPDFCMD    = dvipdf     # [dvipdf | dvipdfm | dvipdfmx]

PDFVIEWER    = xpdf       # [okular | xpdf | evince ]

DVIVIEWER    = xdvi-ja    # [xdvi | xdvi-ja | kdvi ]

## No need to change anything below this line

TEXFILES     = $(wildcard *.tex)

TEXINPUTS=:$(PWD)//:    # Path to search for .tex, .cls and .sty files

BSTINPUTS=:$(PWD)//:    # Path to search for .bst files

BIBINPUTS=:$(PWD)//:    # Path to search for .bib files

TEXMFOUTPUT=$(PWD)/build  # Output dir for bibtex and jbibtex

LATEXOPTS= -output-directory=build -file-line-error -interaction=nonstopmode

export TEXFILES TEXINPUTS BSTINPUTS BIBINPUTS TEXMFOUPUT

.PHONY: all wordcount charcount pdf dvi

all: build/$(MAINTEX).pdf

build :

        @echo "Creating build directory"

        @mkdir build

build/$(MAINTEX).aux: $(MAINTEX).tex $(TEXFILES) build

        $(LATEXCMD) $(LATEXOPTS) $(MAINTEX)

build/$(MAINTEX).bbl: build/$(MAINTEX).aux

        $(BIBCMD) build/$(MAINTEX)

build/$(MAINTEX).dvi: build/$(MAINTEX).bbl

        $(LATEXCMD) $(LATEXOPTS) $(MAINTEX)

        $(LATEXCMD) $(LATEXOPTS) $(MAINTEX)

build/$(MAINTEX).pdf: build/$(MAINTEX).dvi

        $(DVIPDFCMD) build/$(MAINTEX).dvi build/$(MAINTEX).pdf

dvi: build/$(MAINTEX).dvi

        $(DVIVIEWER) build/$(MAINTEX).dvi

pdf: build/$(MAINTEX).pdf

        $(PDFVIEWER) build/$(MAINTEX).pdf

# Word counting can be done in VIM usng Ctrl-g g but this command also includes

# latex commands. This make rule strips the latex and counts what is left

wordcount:

        @echo Approximate word count: `grep -v '^\\\\' $(TEXFILES)|grep -v '^%'|wc -w`

charcount:

        @echo Approximate char count: `grep -v '^\\\\' $(TEXFILES)|grep -v '^%'|wc -c`

Put all your tex files, images, sty, cls, bib, bst files inside a directory with any subfolder tree structure you like. This Makefile sets some environment variables that allow it to find all these files as long as they are below the current directory tree.

Make sure your main tex file is called main.tex or if you prefer another one replace the MAINTEX variable in the Makefile. Take note that it has no extension!.

Also note that this Makefile is to compile Japanese EUC-JP tex files (IEICE Trans). If you do not need Japanese replace platex with simple latex and jbibtex with bibtex.

All output files generated by latex/platex, bibtex/jbibtex are stored inside a build subdirectory.

Now in your .vimrc you can add the following parameter to compile your latex project:

""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

" Working with Makefiles

"

" Press F5 to compile and open the error window if there

" are errors. If there are errors you can use :cn and :cN to

" jump foward and backward thru the error list. Press :ccl to

" close the error list or F5 again to recompile

map <F5> <ESC>:make<CR><ESC>:botright cwindow<CR> " Compile and open quick fix list

map <F6> <ESC>:cN<CR> " Jump to prev error/warn

map <F7> <ESC>:cn<CR> " Jump to next error/warn

Now when editing you latex files press <F5> to compile them and if errors occur you will be presented with a quickfix window where you can use <F6> and <F7> to quickly jump to the previous and next error message.

Additionally this Makefile also has a few useful commands you can invoke from within vim:

:!make pdf to display the pdf file using xpdf or if you prefer a different viewer replace the PDFVIEWER variable in the Makefile.

:!make dvi to display the dvi file using xdvi or if you prefer a different viewer replace the DVIVIEWER variable in the Makefile.

:!make wordcount to display the number of words in all the tex documents. In vim you can also use g<ctrl-g> in normal mode but this also counts latex commands.

:!make charcount same as wordcount but for characters instead of words.

Latex with Vim TagList plugin

We can enable TagList to display important keywords like section, labels and references for easy navigation (link).

First create a ~/.ctags file that contains the following:

--langdef=tex
--langmap=tex:.tex
--regex-tex=/\\subsubsection[ \t]*\*?\{[ \t]*([^}]*)\}/- \1/s,subsubsection/
--regex-tex=/\\subsection[ \t]*\*?\{[ \t]*([^}]*)\}/+\1/s,subsection/
--regex-tex=/\\section[ \t]*\*?\{[ \t]*([^}]*)\}/\1/s,section/
--regex-tex=/\\chapter[ \t]*\*?\{[ \t]*([^}]*)\}/\1/c,chapter/
--regex-tex=/\\label[ \t]*\*?\{[ \t]*([^}]*)\}/\1/l,label/
--regex-tex=/\\ref[ \t]*\*?\{[ \t]*([^}]*)\}/\1/r,ref/

Then modify the taflist.vim file and add a new language. To do this search for "yacc language" and add these line before it:

" tex language
let s:tlist_def_tex_settings = 'tex;s:section;c:chapter;l:label;r:ref'

" yacc language

Now when you open the taglist window you will get a nice list of section, subsections, chapters, references, labels, etc... that you can navigate and use to jump quickly to each of them.

If you do not have the taglist plugin installed you can follow my instructions here.

Reference autocompletion

If you write your labels as \label{fig:something}, then put this in your ~/.vimrc file:

set iskeyword+=: " type /ref{fig: and prec <C-n> to autocomplete references

With this now you can type \ref{fig: and press to get a list of labels you can cycle as shown in the image above.

Spell checking

This is not only for latex files. Simply put the following in your ~/.vimrc file to enable automatic spell checking. Learn how to use the spell checker to correct, add and/or ignore words.

""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

" Enable spell checking in latex, bib and txt files

" Commands:

"            [s      ->  jump to next bad word

"            ]s      ->  jump to prev bad word

"            z=      ->  suggest word

"            zg      ->  mark word as good (add to dictionary)

"            zw      ->  mark word as bad  (remove from dictionary)

"            :h spell -> get more details about spelling

""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

au BufRead,BufNewFile *.txt,*.tex,*.bib  setlocal spell spelllang=en_us

Mounting Windows Shares Using CIFS on Ubuntu

2008-11-12T10:53:00.002+09:00

So you have a cool Buffalo LinkStation Pro Duo or a Windows machine with some shared folders you want to access from your super Linux machine. No problems we have CIFS to the rescue!!

These instructions were tested in Kubuntu 8.10 (Ibex) but shall work with previous versions and future versions of Kubuntu and/or Ubuntu. Also I assume you want the share to be available to all users in the system so you need root access (i.e. sudo).

First install smbfs package:

sudo aptitude install smbfs

Create a directory were you want to mount the shared folder, any place is ok:

sudo mkdir /media/share

If the share requires username and password to access create a password file in you home directory:

sudo echo -e "username=my_username\npassword=my_password" > /root/.share_pass
sudo chown root:root /root
sudo chmod 700 /root/.share_pass

Replace "my_username" and "my_password" with the corresponding user and pass to access the remote share.

Finally create an entry in the "/etc/fstab" file to mount the shared folder in the desired mount point:

//ip-address/share /media/share credentials=/root/.share_pass,iocharset=utf8,file_mode=0777,dir_mode=0777 0 0

Here you must replace //ip-address/share with the ip address of the machine with the shared folder and the complete path to the respective folder.

Finally mount all partitions:

sudo mount -a

Now you must be able to access the shared folder at the remote machine by accessing the /media/share directory.

References

http://ubuntuforums.org/showthread.php?t=288534

NVidia InitialPixmapPlacement=2 really improves KDE speed

2008-10-19T13:10:00.002+09:00

Some people say KDE4 rendering speed is slow when using Nvidia cards. I never felt this so I never bothered in applying the proposed solutions. Using firefox (note: this test does not run in konqueror) simply access the penroseTitling test page and run both tests. My first tests are in the image above and as you can see not bad as I have read in the Internet that for some people the Canvas test can take up to 30 seconds while it took 3 seconds on my machine.

Anyway I tried one of the solutions that is simply running this command in the konsole:

nvidia-settings -a InitialPixmapPlacement=2 -a GlyphCache=1

And here are the new times:

Wow this is a real improvement so now I have a kde startup script that calls this command when KDE starts:

# ~/.kde4/Autostart/nvidia.sh
!#/bin/bash
nvidia-settings -a InitialPixmapPlacement=2 -a GlyphCache=1 &

Resources:
http://movingparts.net/2008/10/03/kde-42-trunk-now-rocking-on-my-thinkpad-t61/
http://techbase.kde.org/User:Lemma/KDE4-NVIDIA

Use vimdiff as Subversion diff command

2008-10-09T22:19:00.000+09:00

First create a script svbdiff inside the ~/.vim/scripts/ directory that contains the following

#!/bin/bash
shift 5;/usr/bin/vimdiff -f "$@"

and make it executable:

chmod +x ~/.vim/scripts/svndiff

Then create an alias in your ~/.bashrc file like:

alias svndiff="svn diff --diff-cmd ~/.vim/scripts/svndiff"

Now you can go to any Subversion repository and display file differences with

svndiff h264.c

The screenshot above shows the diff output of the h264.c of libavcodec. Make sure you have a color scheme that highlights the file differences. The screenshot above uses the inkpot color scheme.

Open new files as tabs in current Vim Console

2008-10-09T21:22:00.004+09:00

When programming I like to have a single Vim editor open with all my files as tabs. Until now I was using the ":tabnew" command to open files in the current Vim window as I knew no other way.

Some research led me to the --remote-tab switch that allows to open files as tabs in currently open Vim processes but it seemed to work only with the graphical interface (gvim) and not with the console (vim). But as I made some tests I found this can work with the vim in console mode and more over can be nicely integrated with my favorite file managers (Konqueror/Dolphin).

All I needed to do was configure my bash so it will always start vim in server mode if it is not already and to always use the --remote-tab switch when opening files. To do this I simply added these functions to my ~/.bashrc file:

function v {

   if [ "$#" -eq "0" ]; then

      /usr/bin/vim --servername VIMLOCAL

   else

      if echo "$*" | grep -q -- "--servername" ; then

        # echo "Command already has --servername use it"

        /usr/bin/vim $*

      else

          #echo "Use default VIM server VIMLOCAL"

          /usr/bin/vim --servername VIMLOCAL --remote-tab-silent $*

      fi

   fi

}

function gv {

   if [ "$#" -eq "0" ]; then

      /usr/bin/gvim --servername GVIMLOCAL

   else

      if echo "$*" | grep -q -- "--servername" ; then

        # echo "Command already has --servername use it"

        /usr/bin/gvim $*

      else

          #echo "Use default VIM server VIMLOCAL"

          /usr/bin/gvim --servername GVIMLOCAL --remote-tab-silent $*

      fi

   fi

}

alias vi=v

alias vim=v

alias gvim=gv

With these functions loaded (i.e. source ~/.bashrc) the first time you execute vim/gvim it will start a new process in server mode.

# this command will open vim in the current console
# and display first_file.txt for editing.

vim first_file.txt

The second and all subsequent times you execute vim/gvim the files you set as parameters will be opened as tabs in the already opened vim/gvim instance.

# This command in other console will open the files
# second_file.txt and third_file.txt as tabs in the
# other console with first_file.txt open

vim second_file.txt third_file.txt

The above examples work also with gvim instead of vim. The only drawback is that now it is difficult to open files in a separate console. The best solution I found is to call vim with a different server name:

# This command will create a new instance of vim in the current console window
# and display the other_file.txt file for editing.

vim --servername OTHERNAME other_file.txt

If you prefer not to use tabs and like files to be opened as buffers inside a current running Vim instance then simply replace "--remote-tab-silent" with "--remote-silent" in the above scripts.

Make Vim and Ctag play nice together

2008-10-09T19:48:00.017+09:00

The classic tutorial of Ctags + Vim will go something like:

- Install exuberant ctags

- Generate tags for your project

- Also install the tag list plugin

- Use ^] and ^T to jump around the code

In this post I will try to go a little further and show how I manage tags in vim for real and large C projects.

Note that I mostly program in C (not C++), Ruby and Bash so I cannot guarantee my method works for all languages. In special Ctags has problems supporting C++. On the other side I have worked in small projects in Java and Python and I had no problems navigating them with my method.

Pre requisites

The following instructions were tested on a Linux machine installed using Kubuntu 8.10, 9.04 and 9.10. The first step is of course to make sure you have installed Vim and exuberant-ctags in you machine:

sudo aptitude install vim-ruby vim-scripts vim-common exuberant-ctags

If you are new to Vim then you should take some time to set your minimal configuration. You may follow mine if you want link.

Generate system wide ctags

All large projects will use at some point external libraries. My C projects usually use the Linux/FreeBSD system calls, SDL, OpenGL, libavcodec and libavformat among others.

To be able to jump to these libraries we must generate tags for each of them. I usually generate them inside my vim directory but you may create them in a system wide folder.

This is extremely useful to easily find out the members of libavcodec large structs or to quickly find out what are the arguments of SDL functions or system calls.

Ctags for Linux System Calls

mkdir -p ~/.vim/tags
sudo aptitude install linux-headers-`uname -r`
ctags -R -f ~/.vim/tags/kerneltags /usr/src/linux-headers-`uname -r`

Ctags for Ruby core and gem libraries

mkdir -p ~/.vim/tags
ctags -R -f ~/.vim/tags/rbtags /usr/lib/ruby

Ctags Java libraries

# For the java tags to work you need to install java source package.
# In Kubuntu/Ubuntu follow these instructions:
mkdir -p ~/.vim/tags
sudo aptitude install sun-java6-source
sudo mkdir -p /usr/lib/jvm/java-6-sun/src
sudo unzip -d /usr/lib/jvm/java-6-sun/ /usr/lib/jvm/java-6-sun/src.zip
ctags -R -f ~/.vim/tags/javatags /usr/lib/jvm/java-6-sun/src

At this you should be able to create tags for all the libraries you use in your projects. Simply get the source code or development packages of the library and run ctags on the folder that contains the source code and save the tags in a file you can access.

TODO: I really need to research what are the best ctags flags for each language in order to generate the most information possible per tag.

Updating your system wide ctags

Here we have three options:

Update your system/library tags manually each time you upgrade them

Use a cron job to update them every certain period of time (e.g. once a week).
Use incrontab to monitor changes in the system/library folders and automatically update the tags every time these folders change.

Personally I use the cron job method and for some libraries that I update often (e.g. libavcodec from subversion) I manually generate the tags soon after I update the library.

I found about incrontab just recently and I think will be the best way to update system wide tags. It behaves mostly like crontab but instead of triggering actions based on time periods it triggers by file system events like file change, move, save, etc. It should be easy to monitor the folders containing the libraries of interest and regenerate the tags for each library based on change events in the folder contents. The tricky part would be to determine what file system event we must listen to decide if we should regenerate or not the tags.

Generating ctags for your own libraries/projects

Now we need to generate tags for projects/libraries we are developing at the moment. These projects are constantly changing so we need to update them as soon as possible.

This is a common problem that lot's of people solve in different ways. See for example these links:

Vim auto-generate ctags

Vim+ctags tips and tricks

TagsParser : Automatic tagfile updating and tag viewer

vim with ctags for multi-level directory hierarchy

http://vim.wikia.com/wiki/Tags_generation_for_current_file

Auto tags by project.vim

Most methods I found do the same more or less: they set some auto commands to autogenerate tag files when a file is edited and set the search path in a way it can find the generated tags.

I used to have some autocommands in vim to generate tags every time I modified a file in a project. Unfortunately I never figured out how to generate a single tags file per project/library. I always ended with a global tag file for all projects or a single tag file per folder.

Another methods worked by adding tags generation commands to the build system so the tags get generated when we recompile the project (e.g. Makefile, Ant, CMake) and others simply created scripts per project and executed them manually from the project root directory or within vim using a key map.

Since I use the Project.vim plugin it is natural for me to use it to generate the tags files as it is done here. My method is a lot different in that I do not use a Makefile to generate the tags and I only re-generate them when a file has been edited, not every time we open a file.

Auto-generating Ctags Using the Project.vim plugin

Anyone that has used the Project.vim plugin is familiar with the snippet shown below:


 myproj=<src-path> in=in.vim out=out.vim CD=<src-path> {
   CMakeLists.txt
   in.vim
   out.vim
   include----------------- 
   src---------------------
   test--------------------
   tools-------------------
 }

This corresponds to a project entry called "myproj" with root directory set to "src-path". Here the important parts are the "in", "out" and "CD" parameters.

The "CD" parameter makes sure that you are at the project root directory when editing a file from that project. This is useful to have a single tags file per project.

The "in" and "out" parameters are simple vim scripts. The "in" script is executed when opening a file from the project and the "out" script when leaving the file. In these scripts is where I set/unset the autocommands that take care of generating the tags files per project.

Every time I create a new project using the Project.vim plugin I also create these in.vim and out.vim scripts:

in.vim

" let ctags_cmd='/usr/local/bin/exctags' " Use this one in FreeBSD

let ctags_cmd='/usr/bin/ctags' " Use this one in Linux

let proj_path = escape(getcwd(), ' ')

let _ctagargs_ = " --fields=+iaS --extra=+q -R "

let _ctag_ = ctags_cmd . _ctagargs_ . " -f " . proj_path . "/.tags " . proj_path

au BufWritePost <buffer> call system(_ctag_)

out.vim

au! * <buffer>

Now every time you open a file from a project an autocommand will be created for that file that will regenerate tags for the whole project every time the file is saved. The generated tags file is always in the root directory of the project. You may prefer to use the "--append" switch of ctags to generate tags for the current file only but I personally prefer regenerating the tags of the whole project every time.

When you leave the file the autocommand is removed in the out.vim script. If we do not remove the autocommand, it will be created every time we enter the file that results in a lot of autocommands around. I am not really sure if this can cause problems but is better go on the safe side.

Now the last piece of the puzzle is how to tell Vim where to search for the tags.

Tell Vim where to search for tags

To find the per project tags we can simply add this to our vimrc file:

set tags=./.tags;${HOME}

This simple command tells vim to search tag files from the current directory backwards up to our $HOME directory. Since we are sure that we are always at the projects root directory where the tags file is automatically generated we can be sure that vim will always find the tag we are looking for.

There are some system wide tags that we want to always load depending on the language we are developing. For example when I develop in C I would like to have the kernel system calls tag file (that we generated before) or when developing in Ruby/Java I expect to have these languages core tags loaded too.

To do this I load all system wide libraries based on filetype using these commands in my vimrc file:

au BufRead,BufNewFile *.rb setlocal tags+=~/.vim/tags/rbtags

au BufRead,BufNewFile *.cpp,*.h,*.c setlocal tags+=~/.vim/tags/kerneltags

au BufRead,BufNewFile *.rl,*.def setlocal tags+=~/.vim/tags/kerneltags

au BufRead,BufNewFile *.py setlocal tags+=~/.vim/tags/pytags

au BufRead,BufNewFile *.java setlocal tags+=~/.vim/tags/javatags

What these commands do is to load the system wide tag files we generated before depending on the file type. For example when editing a ruby file it loads the Ruby tags file.

We must note that the commands only apply to the current file being edited (e.g. setlocal), not all the files in the project and that we are adding the tags file not replacing the current one (e.g. += ).

Finally how can we add other tag files to the project that are not from the project itself and not system wide?. For example my project requires external libraries like libavcodec and SDL so it would be nice to query the function and struct definitions of these libraries.

To do this I use the "in.vim" script of the Project.vim plugin like:

" let ctags_cmd='/usr/local/bin/exctags' " Use this one in FreeBSD

let ctags_cmd='/usr/bin/ctags' " Use this one in Linux

let proj_path = escape(getcwd(), ' ')

let _ctagargs_ = " --c++-kinds=+p --fields=+iaS --extra=+q -R "

let _ctag_ = ctags_cmd . _ctagargs_ . " -f " . proj_path . "/.tags " . proj_path

au BufWritePost <buffer> call system(_ctag_)

" Add external library tag files

setlocal tags+=~/.vim/tags/sdltags

setlocal tags+=~/.vim/tags/ffmpegtags

This is exactly the same "in.vim" script as above with two added lines that include the libavcodec (FFMpeg) and SDL library tags files to the search path of vim for all files in the project. These files were of course generated before the same way we created the system wide tag files and updated via a cron job.

Once you setup this once you don't need to worry about this anymore until you reinstall your computer. The system wide libraries will be always updated if you configured the corresponding cron jobs and once a project is created it will always have the tags files updated and ready to query.

Setting up the TagList Plugin

No blog post about Vim+Ctags is complete without the mighty TagList plugin. To install this plugin go to http://vim-taglist.sourceforge.net/ and download the taglist_45.zip file. Then add it to your vim installation using the command:

unzip -d ~/.vim taglist_45.zip

Then add these commands to your vimrc file:

" let Tlist_Ctags_Cmd='/usr/local/bin/exctags' " Use this one in FreeBSD

let Tlist_Ctags_Cmd='/usr/bin/ctags' " Use this one in Linux

let Tlist_Auto_Open=0

let Tlist_Auto_Update=1

let Tlist_Use_Horiz_Window = 0

"let Tlist_Inc_Winwidth=0

"let Tlist_Show_One_File=1

let Tlist_Exist_OnlyWindow=1

let Tlist_Use_Right_Window = 1

let Tlist_Sort_Type="name"

let Tlist_Display_Prototype=0

let Tlist_Compact_Format=1 " Compact?

let Tlist_GainFocus_On_ToggleOpen=1

let Tlist_Display_Tag_Scope=1

let Tlist_Close_On_Select=1

let Tlist_Enable_Fold_Column=1

"let TList_WinWidth=25

" Map a F8 to the TlistToggle command for easy tags list access

nnoremap <silent> <F8> <ESC>:TlistToggle<CR>

Of course I recommend you to read the help file and set the configuration as it fits you better. With my configuration you can press to open the tags list window and when you press on a tag the windows automatically closes. This is specially useful on small screen laptops to allow more code visible all the time.

Basic .vimrc file

2008-10-08T14:43:00.009+09:00

My first Vim configuration files: .vimrc, plugins, color schemes, etc were very small and modest. It's been eight years since I started using Vim and these configuration files have morphed, evolved, grown and recently shrink in too many ways, more than I can remember, and every time I check www.vim.org I find new ways I would like my Vim to evolve even further.

Here is the simplest version of my .vimrc file with no ctags or desktop integration as these topics require a separate post to explain. Most of the options here can be understood directly from Vim help files and by the comments in the file itself.

If you would like to use this file make sure to copy it in your home directory and read the comments as to get full functionality.

For comparison see the default install and my older configuration screenshot at the end of the post and compare it with my current configuration screenshot at the top of this post.

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

"" General Vim Settings

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

"   In Kubuntu/Ubuntu make sure you have a complete vim install

"   i.e.  sudo aptitude install vim-ruby vim-scripts vim-common

"

" Disable compatibility with vi

set nocompatible

" Use the Unix file format

set fileformats=unix,dos

" Display line number in front of each line in the left margin.

set number

" I keep my own backups and all those files ending in '~' are

" anoying.

set nobackup

" Display commands in the bottom right corner as they are typed.

set showcmd

" Smoother redraws

set ttyfast

" Avoid vim complains about not written file when jumping

" between buffers using ctags.

set autowrite

" Vim Tip #1160 - Auto Save files when focus is lost.

au FocusLost * :wa

" Vim Tip #1279 - Highlight current line in Insert Mode

autocmd InsertLeave * se nocul

autocmd InsertEnter * se cul

" Enable syntax highlighting with 256 color support so we

" can use nicer color schemes. For more information and to

" get some nice color schemes check:

"   http://www.frexx.de/xterm-256-notes/

" Tested with Konsole-2.1 and yakuake 2.9.4

set t_Co=256

syntax on

" Set this background depending on your console and color scheme.

set background=light

"set background=dark

" My favorite color schemes. You must download the color themes

" and put them inside the color folder of your vim installation.

"colors default     " Good fro transparent consoles

"colors zenburn     " Vim Tip #415

colors 256_asu1dark " http://www.frexx.de/xterm-256-notes/

"colors pyte        " Vim Tip #1492

" If in diff mode (vimdiff) use the inkpot color scheme

" that better highlights file differences

if &diff

  colors inkpot    " Vim Tip #1143

endif

" Additional filetypes

au BufRead,BufNewFile sconstruct setlocal filetype=python

au BufRead,BufNewFile *.rl       setlocal filetype=ragel

au BufRead,BufNewFile *.rb       setlocal filetype=ruby

" Additional Syntax Files. The ragel.vim file can be downloaded

" from Ragel's home page http://www.complang.org/ragel/.

au! Syntax ragel source ragel.vim

" Set path to search (i.e. gf) files recursively

set path=/usr/include,/usr/local/include,**;$HOME

" Vim Tip #1274 - Highlight trailing whitespace characters

set list

set listchars=tab:->,trail:·

" Number of spaces used for (auto)indenting

"set shiftwidth=2

" Number of spaces to insert for a tab

"set softtabstop=2

set tabstop=2

" Insert spaces when the tab key is pressed

" If you want a real tab use "ctrl-v, tab" in insert mode. This

" is usefull for Makefiles that require real tabs to work.

set expandtab

" Word wrap with line breaks if they are longer than 80 characters

" long. If you prefer to wrap lines visually only use the wrap and

" lbr commands below instead. If you have a document that has lines

" longer than 80 characters you can use "gq}" to format it as a

" paragraph. This is more easier format for navigating the text.

set textwidth=80

set formatoptions=tcq

" Vim Tip #989: Word wrap without line breaks

"set wrap

"set lbr

" Enable specific indenting

set autoindent

set smartindent

" Enable code folding

set foldmethod=syntax

" Show briefly matching bracket when closing it.

set showmatch

" Show cursor position

set ruler

" Additonal key mappings

map <F12> <ESC>ggVGg?                             " ROT13 fun

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

"" Mini buffer explorer features

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

let g:miniBufExplSplitBelow=0

let g:miniBufExplSplitToEdge = 0

let g:miniBufExplVSplit = 20

let g:miniBufExplMapCTabSwitchBufs = 1

let g:miniBufExplUseSingleClick = 1

let g:miniBufExplorerMoreThanOne=0

let g:miniBufExplModSelTarget = 1

" Map Tab and Shift-Tab for buffer navigation

" Note that S-TAB does not work in certain consoles (i.e. KDE Konsole)

map <TAB> <ESC><C-TAB>

map <S-TAB> <ESC><C-S-TAB>

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

"" Improve VIM autocomplete features

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

" Enable search as you type:

set incsearch

"improve autocomplete menu color

highlight Pmenu ctermbg=238 gui=bold

" http://www.cuberick.com/2008/10/ruby-autocomplete-in-vim.html

autocmd FileType ruby,eruby set omnifunc=rubycomplete#Complete

autocmd FileType ruby,eruby let g:rubycomplete_buffer_loading = 1

autocmd FileType ruby,eruby let g:rubycomplete_rails = 1

autocmd FileType ruby,eruby let g:rubycomplete_classes_in_global = 1

""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

" Vim tip #90 Enable VCS integration in vim

""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

"  Get vcscommand.zip from http://www.vim.org/scripts/script.php?script_id=90

"  Uncompress to your vim directory:  unzip -d ~/.vim vcscommand.zip

"

"  Now when editing a file that is under revision control (SVN, CVS, Git,...)

"  we can access the versioning commands:

"

"  VCSAdd

"  VCSAnnotate

"  VCSCommit

"  VCSDelete

"  VCSDiff

"  VCSGotoOriginal

"  VCSGotoOriginal!

"  VCSInfo

"  VCSLog

"  VCSLock

"  VCSReview

"  VCSStatus

"  VCSUpdate

"  VCSUnlock

"  VCSVimDiff

" Map F4 to VimDiff to check differences

map <F4> <ESC>:VCSVimDiff<CR>

""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

" Latex Configuration

" http://vim-latex.sourceforge.net/documentation/latex-suite/recommended-settings.html

""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

au BufRead,BufNewFile *.cls      setlocal filetype=tex

"let g:tex_flavor='latex'   " loads latexsuite with empty tex files.

set iskeyword+=:           " type /ref{fig: and press <C-n> to autocomplete references

""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

" Working with Makefiles

"

" Press F5 to compile and open the error window if there

" are errors. If there are errors you can use :cn and :cN to

" jump foward and backward thru the error list. Press :ccl to

" close the error list or F5 again to recompile

map <F5> <ESC>:make<CR><ESC>:botright cwindow<CR> " Compile and open quick fix list

map <F6> <ESC>:cN<CR>                             " Jump to prev error/warn

map <F7> <ESC>:cn<CR>                             " Jump to next error/warn

""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

" Enable spell checking in latex, bib and txt files

" Commands:

"            [s      ->  jump to next bad word

"            ]s      ->  jump to prev bad word

"            z=      ->  suggest word

"            zg      ->  mark word as good (add to dictionary)

"            zw      ->  mark word as bad  (remove from dictionary)

"            :h spell -> get more details about spelling

""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

au BufRead,BufNewFile *.txt,*.tex,*.bib  setlocal spell spelllang=en_us

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

" Other Good Vim tips not used here

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

" Vim Tip #64   - Set working dir to the file you are editing

"                 Why not used: Makes difficult the management of sessions and

"                   makefiles.

" Vim Tip #305  - List of editing vim tips

"                 Why not used: I can remember all the commands I use.

" Vim Tip #1203 - mapping to set up vtreeexplorer and taglist in left window

"                 Why not used: I use MiniBufExpl instead of vtreeexplorer

" Vim Tip #1318 - An attempt to emulate TextMate's snippet expansion

"                 Why not used: I am unable to get this working correctly.

" Vim Tip #???? - Force gf to open in new tabs

"                 Command: nnoremap gf <C-W>gf

"                 Why not used: I use MiniBufExpl so no more tabs for me.

" Vim Tip #565  - Never see ^M and pesky trailing spaces again

"                 Command: autocmd BufRead * silent! %s/[\r \t]\+$//

"                 Why not used: Not a good idea when working with others source

"                  code. All the automatically deleted chars will be seen as

"                  changes in the diff files making it difficult to review the

"                  changes. Never use this if you plan to submit patches to ffmpeg

"                  or the Linux kernel or they will spam filter you forever!

Canon Satera LBP5910 Linux Driver

2008-09-25T10:47:00.001+09:00

If you have a Canon printer based on the LIPSLX driver (i.e. LBP family) then you are lucky because Canon provides the linux drivers for these printers.

Simply go to this page Canon Linux Driver check for the supported printers and the respective drivers. For my case I have the LBP5910F printer that is supported by the LIPS LX Printer Driver so I enter this page and download the two corresponding .deb packages for Ubuntu/Debian. If you use Redhat or any other distribution that uses RPMs then download the corresponding RPMs.

To install in Ubuntu/Debian 32bit:

sudo dpkg -i cndrvcups-common_2.20-1_i386.deb
sudo dpkg -i cndrvcups-lipslx_2.20-1_i386.deb
sudo /etc/init.d/cupsys restart

Now enter CUPS configuration page (i.e. http://localhost:631) and follow the steps. In my case the printer was already listed in the select box of printers with IP address and everithing. In Make/Manufacturer select Canon and finally in Model/Driver select the corresponding driver:

And that is all now you can try to print a test page.

To install in Ubuntu/Debian 64bit:

For some reason there are no deb packages for 64bit architecture but fortunately is easy to convert the 64bit rpms to deb packages using alien.

First install alien:

sudo aptitude install alien

then using this tool convert the rpm packages to deb packages:

sudo alien -k cndrvcups-lipslx-2.20-1.x86_64.rpm
sudo alien -k cndrvcups-common-2.20-1.x86_64.rpm

this will generate two 64bit deb packages that work perfectly with Ubuntu/Kubuntu and can be installed as the 32 bit versions above:

sudo dpkg -i cndrvcups-lipslx_2.20-1_amd64.deb cndrvcups-common_2.20-1_amd64.deb

after installing the packages you can proceed with the installation the same way as with the 32bit packages.

Use PuTTY generated keys with SftpDrive

2008-06-25T12:38:00.000+09:00

So you created a public/private key pair using PuTTYgen and now you want to use these keys in other applications like OpenSSH and/or SftpDrive but you realise that the private key generated by PuTTYgen is not compatible with these programs??

No worries... all you have to do is convert your PuTTYgen key (.ppk) to and OpenSSH compatible key and it happens to be that PuTTYgen can do this conversion.

Open PuTTYgen and load your private key using the "load" button and opening your private key .ppk file. Once loaded select from the top menu "Conversions -> Export OpenSSH key" and select a file name to save the exported key. Now this saved key can be used in other applications that are OpenSSH compatible.

To use this key with SftpDrive open SftpDrive Manager and on "New Driver" change the Authentication method (see image above) to "Use a public key to log in...". This will open a dialog box (see image below).

Here you can generate a new public/private key pair but we want to use the one we generated with PuTTYgen so select "Import Existing Key Pair" that will open yet another dialog box (see image below):

Here simply browse the location of the private and the corresponding public keys and press "OK". Now SftpDrive will use these keys to connect to your SSH server.

If you get errors make sure you have selected the exported OpenSSH compatible key and not the PuTTYgen generated one (i.e. .ppk). Also if you saved the public key using the PuTTYgen "save public key" button then this key has a format SftpDrive does not understands so it will be unable to read the key. To solve this copy and paste the public key as it appears in PuTTYgen text box (i.e. the large string starting with ssh-rsa or ssh-dsa) in a text file (i.e. mykey.pub) and then import that file as the public key. This format is OpenSSH compatible so should work with no problems.

Subversion + OpenSSH Server + PuTTY Client Setup

2008-06-24T09:10:00.002+09:00

This setup is of a subversion server running on Linux (Kubuntu 8.04) and subversion clients (Tortoise SVN) running in Windows (WinXP).

Pre-requisites

I assume you already have a subversion repository working and now you want to access it using ssh tunnelling for security reasons. Also the server must have remote ssh access and be configured to do Private Key authentication.

Setting up Putty

The best SSH client and related tools for Windows are the PuTTY family of programs that can be found at http://www.chiark.greenend.org.uk/~sgtatham/putty/. You can download the individual ".exe" files an put them in your path but I recommend you to use the installer that will install all the tools and set them nicely in your start menu.

Create public/private key pair

Objective

To access the subversion server via ssh you need a public/private key pair. To generate these key pair you can use the PuTTY Key Generator (PuTTYgen.exe) that was installed along with PuTTY installer.

Simply execute the PuTTYgen.exe program and press the "generate" button. After that keep moving your mouse, to generate some randomness, until the progress bar reaches the end. At this point you will get your public and private keys generated.

Make sure you add a "Key passphrase" that will be like a password to access the keys. The comment part is not necessary but is customary to put your email address.

Also make sure you are generating a SSH-2 key pair as SSH-1 is not secure. Choosing between RSA or DSA keys is a matter of application. It is known that DSA is fast at key generation and signing while RSA is faster at verification. So if your application requires a lot of signing (i.e. SSL Web Application) then DSA is good but if your application requires a lot of verifying (i.e. Subversion) then RSA is best. Since we are going to use this key for Subversion I recommend subversion but you are free to use the one you prefer.

Now we need to save your generated keys somewhere in your hard disk. The public key is visible in the PuTTYgen application widget (i.e. the long sequence of letters in the top text box). This key starts usually with "ssh-rsa" or "ssh-dsa" depending on what type of key you generated (RSA or DSA).

To save your public key on disk do not use the "save public key" button as this will generate a key in a format that is not compatible with OpenSSH. Simply copy and paste the key from the text box to a file on disk. You can save this file with whatever name you like but I recommend putting a ".pub" extension just as OpenSSH does.

The private key you must save it using the "save private key" button as this generates a file with ".ppk" extension that is understood only by PuTTY programs. Make sure you put the same name as the public key but with the ".ppk" extension instead of ".pub".

As an additional step if you would like to use your private key with other applications like SftpDrive so these can also connect to the OpenSSH server without need for a password, then you must convert your private key (.ppk file) to the OpenSSH key format. To do this simply select "Conversions" in the PuTTYgen.exe menu and then "Export OpenSSH key" to generate a key that can be used with OpenSSH compatible applications.

Note that with PuTTYgen.exe you can convert your private keys between PuTTY (.ppk) format and OpenSSH format so if you generate your public/private keys using OpenSSH ssh-keygen program you can use them in PuTTY by importing the private key using PuTTYgen.

Setting your public keys

The way the public/private key pair are used is simple. You keep the private key in your local machine usually in a key manager like ssh-agent or pageant and send your public key to all the servers you want to access via ssh.

To set up your public key in the SSH server (assumed OpenSSH server) simply log to the server shell using your account username and password (if you don't have one ask you administrator or if you are the administrator create one yourself). Then copy the contents of your public key file inside the file "~/.ssh/authorized_keys2".

To do this edit the "~/.ssh/authorized_keys2" file with any text editor (Vi, Nano, Kwrite, gEdit) and make sure it looks like in the next figure:

The key must start with ssh-rsa or ssh-dsa depending on the key you generated followed by the key all in one line (i.e. no new lines and spaces) and finally the comment again all in one line.

Finally make sure the authorized_keys2 file has correct permissions by executing "chmod 600 authorized_keys2".

You can set the public key in as many SSH servers as you like so you can access them all using the same private key.

Setting your private key

Your private key must be your most guarded secret and shall not be passed to anyone. Only you should have access to this key because anyone who gets his/her hands on that private key can access all the servers where you installed your public key and we don't want anyone to have that access power do we?

To use a your private key we usually use a SSH authentication agent like ssh-agent in Linux/Unix or pageant in Windows. The PuTTY installer comes with pageant.exe that is the one I use and recommend.

Simply execute pageant.exe from the application menu and a small icon will appear in your system tray. Right click the icon and select "Add key" on the context menu, then browse the directory where you saved your private key (i.e. .ppk) and select it. And that is it....

Testing SSH access

To test open the PuTTY.exe SSH client and create a new session to connect to your SSH server (make sure you save the session). When you open the session it should log you in without asking for any username and passwords. If it asks you for username and password then you did something wrong. Make sure the publickey format is correct on the server and that the private key is set in pageant. Of course make sure the OpenSSH server is running on the remote machine and that pageant is running on the local machine.

Once you can connect to the SSH server without need for a username and password then we can proceed to set up TortoiseSVN to access your Subversion repositories via SSH tunnels.

Setting TortoiseSVN with SSH tunnelling

Once you have SSH set to work with public/private key authentication and have pageant configured configuring TortoiseSVN is a breeze. All you have to do is make sure you use the correct URL format to access the repository.

If you have TortoiseSVN installed try to make a checkout and in the URL put something like:

svn+ssh://username@hostname/dir/to/repo

And you should be able to ckeck out that project over an SSH tunnel. You don't even need to have svnserve running on the server as the svn+ssh scheme tells the TortoiseSVN to start it for you and tunnel it over SSH.

Of course replace username with the username you use to access the SSH server and hostname with the IP address or FQDN of your SSH server. The path to the repository must be absolute path starting from the root "/" directory and have correct access privileges (read/write) for the username you use.

Resources

http://neubia.com/archives/000191.html
http://www.unixwiz.net/techtips/putty-openssh.html

Best gift ever??

2008-06-14T11:47:00.000+09:00

Now I can reaffirm that I married the right person, just look what I got for fathers day!!!

I know I do not blog that often but be assured that now I will be blogging none at all for at least a few weeks, or months....

Plotting SVN history??

2008-04-17T10:18:00.000+09:00

Found a blog post by Samuel Jansen were he plots his subversion repository history so out of curiosity I plotted mine too...

actually nothing interesting, steady development almost linear and as you can see but it made me notice that I am almost near my 1000 mark!! when I reach it then I will have an excuse to celebrate and give a nice "1000 SVN passed mark present".

Linux Cafe in Akihabara

2008-04-13T17:00:00.000+09:00

Have been in Akihabara (Electric Town) several times but never saw this shop before:

If you read the details below the logo you will see this shop is been there since 2001!! and I never saw it... I must go check my eyes. Anyway if you are an Open Source person walking around Akihabara you may well pass a get yourself a nice cup of open source coffee.

Simple Mail Server (Kubuntu)

2008-04-09T09:05:00.003+09:00

If you need a small personal mail server to send log reports and system alarms from your Kubuntu machines to a local and/or external mail address then you may use Exim4 that is the default mail server (MTA) that comes with (K)Ubuntu.

To install is very easy:

1 sudo aptitude install exim4 exim4-base

and to configure simply run:

1 sudo dpkg-reconfigure exim4-config

and in the first dialog (image below) select "internet site". For all next options I used the defaults that would allow local users and services to send emails locally and to remote addresses (i.e. gmail accounts).

Now to test the new server you can send a simple email to your Gmail account:

1 mail -s test mygmail@gmail.com

2 test mail

3 .

4 Cc:

The "-s" switch is the mail subject. After the command simply type whatever you want in the mail body. To finish the email press followed by a dot "." and then again. After this you will be prompted with "Cc:" so press and then check your Gmail account. You should have received an email with the message and subject you used above.

Now say you want to receive LogWatch reports or any other mail directed to root in your Gmail account. Then you can simply create an alias in the "/etc/aliases" file like:

1 # Added by installer for initial user

2 root: mygmail@gmail.com

then rebuild the aliases database

1 sudo newaliases

If you require more complex mail configurations you may check my previous posts about Sendmail and Postfix setup in (K)Ubuntu here.

Get latest ffmpeg (svn) in FreeBSD-7.0

2008-04-04T15:46:00.006+09:00

If you want to compile the latest ffmpeg from subversion with some external libraries support like x264, faac, Theora and Vorbis in the latest FreeBSD release (7.0) then follow this simple steps:

If you do not have the ports tree installed (i.e. /usr/ports is empty or non existent) then you can use portsnap to download and update it (these commands must be done as root):

1 pkg_add -r portsnap

2 mkdir /usr/ports

3 portsnap fetch

4 portsnap extract

5 portsnap update

Install some ports to enable more funcionality in ffmpeg (these commands also must be done as root):

1 # faac support

2 cd /usr/ports/audio/faac

3 make install clean

4

5 # vorbis support

6 cd /usr/ports/audio/vorbis-tools

7 make install clean

8

9 # x264 support

10 cd /usr/ports/multimedia/x264

11 make install clean

12

13 # theora support

14 cd /usr/ports/multimedia/libtheora

15 make install clean

16

17 # ffplay support

18 cd /usr/ports/deve/sdl12

19 make install clean

Checkout ffmpeg from subversion, configure, compile and install (only the last command needs root priviledges);

1 # checkout ffmpeg from subversion

2 svn co svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg

3 cd ffmpeg

4

5 # export library and include paths

6 # so configure can find the external

7 # libraries (i.e. x264)

8 export LIBRARY_PATH=/usr/local/lib

9 export CPATH=/usr/local/include

10

11 # run configure with desired options

12 ./configure --enable-gpl --enable-pthreads \

13 --enable-shared --enable-swscale \

14 --enable-libx264 --enable-libfaac \

15 --enable-libtheora --enable-libvorbis \

16 --enable-encoder=g726 --enable-encoder=ljpeg \

17 --enable-encoder=mjpeg

18

19 # compile using GNU Make (gmake), not BSD Make

20 gmake

21

22 # install

23 su root -c 'gmake install'

If you want to enable other external codec libraries supported by ffmpeg make sure to install their respective ports (if available) and then run the configure script with the desired libraries enabled. You can see a list of supported codec libraries with ".configure --help".

Note that this instructions were tested with revision 12684 of ffmpeg subverion but ffmpeg is a very active project and this instructions are not guarantee to work with the latest subversion revision.

Japanese Input in KDE 4.0.3 (Anthy)

2008-04-03T10:57:00.000+09:00

So you want to try the newest and greatest KDE 4.0.3 release but you can't because you need to be able to write other languages other than English like in my case Japanese, Chinese and/or Korean??

After googling I found here that there is a scim input method that allows input of these languages in KDE4 and also fixes some problems with proprietary software like Skype, Adobe Reader, etc.

This input method is called scim-bridge and can be easily installed in Kubuntu with the command:

sudo aptitude install scim-bridge-agent scim-bridge-client-qt4 scim-bridge-client-qt scim-bridge-client-gtk

Then to enable scim-bridge as your input method you can use im-switch:

im-switch -s scim-bridge

or if you do not have im-switch then you can do it manually with:

sudo ln -sf /etc/alternatives/xinput-all_ALL /etc/X11/xinit/xinput.d/scim-bridge

Seems that scim-bridge is now the default input method in Ubuntu/Kubuntu but not sure, see resources below to get more detailed explanation on how to setup SCIM in Ubuntu.

https://help.ubuntu.com/community/SCIM
https://help.ubuntu.com/community/SCIM/Kubuntu
https://wiki.ubuntu.com/InputMethods/SCIM/Setup

Developing Android in Netbeans

2008-03-03T23:05:00.001+09:00

Anyone who reads this blog would know that I like Netbeans a lot for mobile development (see here, here, here and here) and when I found that Google's mobile platform (Android) does not have a Netbeans plugin I was very disappointed.

Of course Netbeans enthusiasts developed a plugin (Undroid) but it is not supported by Google, I could not get it to launch the emulator directly from the IDE and currently it does not work with the latest Android SDK so still we do not have a full solution for Android development in Netbeans.

Now if you don't mind working in a mixed setting using Android tools from command line and using Netbeans to code, compile and install your Android applications then here is how you can do it.

Prerequisites

I assume you have the latest Netbeans 6.0.1 already installed and the latest Android SDK m5-rc14 also installed in your computer. I have tested this configuration in my Linux (Kubuntu) machine but should also work in Windows and other Linux distributions.

Creating a new project

The first thing is to create a new project using the activityCreator.py script that comes with the Android SDK:

activityCreator.py --out HelloAndroid org.piaotech.android

this script will create the project skeleton and more important the build.xml file that can be used by Netbeans to build/install your Android applications.

Start Netbeans and create a new "Java Project with Existing Ant Script" as shown in the figure above. In the next screen browse the location of the project directory created by the activityCreator.py script (HelloAndroid above) and the rest of the fields will be filled automatically by Netbeans. You can change the project name if you wish but the default is ok.

In the next screen (figure above) select the Ant tasks that each of the Netbeans commands should execute. I simply put compile for Build Project and install for Run Project.

In the last screen make sure to add the src and res of the project to the Source Package Folders as shown above.

finally select Java Sources Classpath in categories and add the doc directory that is inside the Android SDK as shown in the picture above and press finish.

Running you project

Now you should have your new Android project in Netbeans project view and now you can code and build it as you would using any other Netbeans project. Now if you try to run the application you will get an error indicating that there is no device. This is because I still have not found a way to launch the emulator from within Netbeans so we must launch it manually. From the command line run the emulator:

android-sdk_m5-rc14_linux-x86/tools/emulator

now with the emulator running if you run your project from within Netbeans the Android application will be automatically installed in the emulator.

It takes some effort to setup an Android project in Netbeans and actually there is no benefit from doing this as any java editor with support for Ant scripts can do the same. In my case is simply a matter of preference as all my Java projects are in Netbeans I did not want to have a separate project format only for Android applications.

Note that there is no need to close the emulator each time you rebuild your application so leave it running during the time you are developing. I also recommend you set up a console running the adb logcat command so you can see all the debugging messages (Log.v()) that Android throws.

Dr. Randy Pausch

2008-02-23T17:56:00.000+09:00

A very inspirational talk by Dr. Randy Paush.

Reduce GIF size using RMagick

2008-02-22T01:25:00.005+09:00

If you are creating GIF images using RMagick and your application has very stringent size restrictions (i.e. mobile devices) then this post may help you get the most of RMagick.

While it is possible to control the compression of JPEG and PNG files using the quality parameter for GIF files we are left with almost no options. Fortunately a little understanding of the default, and only, compression scheme used in GIF can help you improve the compression performance of your images.

The LZW compression algorithm used in GIF images and developed by Abraham Lempel, Jakob Ziv and Terry Welch, contructs a color table for an image wherein each color value is matched to a pixel. Thus, images with large areas of one color will be compressed far better than those that those that do not have such color blocks.

So to get better compression (smaller images) we must ensure we have large contiguous blocks of the same color in our image. When creating images in ImageMagick we can use these tips to ensure small size results:

First: Make use simple graphics with flat colors (i.e. avoid using Magick::GradientFill).

Second: when using an RMagick::Draw object always make sure you call the stroke_antialias(false) method on it:

1 require "RMagick"

2

3 gc = Magick::Draw.new

4 gc.stroke_antialias(false)

5

6 # your drawing code below

Third: When writing text using the annotate method from the Magick::Draw or Magick::Image classes do not use text antialiasing:

1 require "RMagick"

2

3 gc = Magick::Draw.new

4

5 gc.annotate(img,...) {

6 self.text_antialias = false

7 }

A very simple method to reduce the image size is using the despeckle filter on the image. This will reduce the color dither making the color distribution more planar so we have larger portions of the image with the same color. By doing this the LZW algorithm can reduce the final image size up to a 20% of the original size depending on the image.

1 require "RMagick"
2
3 small_image = big_image.despeckle
4 small_image.write("small_image.gif"){
5     self.compression = Magick::LZWCompression
6     self.dither = false
7 }

Actually calling self.compression is not needed as it is set by default by RMagick but is good just to make sure.

Resources
http://rmagick.rubyforge.org/
http://www.faqs.org/faqs/jpeg-faq/part1/section-9.html
http://rmagick.rubyforge.org/src_over.html
http://www.webdevelopersnotes.com/graphics/gifs_compression_algorithm.php3

Using ActiveRecord with Microsoft SQL Server (MSSQL)

2008-02-18T18:43:00.007+09:00

There seems to be a lot of confusion on how to use ActiveRecord from Ruby on Rails to query a MSSQL (Microsoft SQL) Database. During my last two projects I had to research this topic and found that this confusion stems from the fact that the documentation is sparced all over the web and most of the time it is incomplete and misleading.

After some tests and real projects I found there are mainly two ways to get ActiveRecord working with a MSSQL Database server: (1) Using the ADO driver and (2) using an ODBC driver. How these methods are used differ if we are in a Windows client or in a Linux client so we get a total of four different ways to connect ActiveRecord to MSSQL, in fact I only got three as we will see later, but in theory there can be four if I get ADO to work in Linux.

The confusion most people new to this topic confront is that the documentation found in the web usually mention these methods separately and without much detail on the configuration. For example some pages in the web explain how to connect to MSSQL using the ADO driver but they don't mention that this works only if you are using a Windows client where win32ole can be installed. Some others talk about success using ODBC and present the configuration but do not explain how to create the DSN needed for connection because they assume this is common knowledge (it was not for me).

To avoid the pain I had to suffer when getting my work done to others, here I present my experiences in this topic in four subsections. Each one explains how to use the ADO or ODBC methods from a Windows or Linux client.

Windows Client using ADO driver

I have tested this configuration using a WinXP Home client with the ruby one click installer (184-20 and 186-26) connecting to a Windows Server 2003 running Microsoft SQL Enterprise 2000 SP3.

To start download an extract the ruby-dbi package (dbi-0.1.1.tar.gz) somewhere in your Windows machine.

The following instructions are to be executed from within a command shell (cmd.exe).

1 gem update --system

2 gem install activerecord-sqlserver-adapter --source=http://gems.rubyonrails.org

3 mkdir c:\ruby\lib\ruby\site_ruby\1.8\DBD\ADO

4 copy ruby-dbi\lib\dbd\ADO.rb c:\ruby\lib\ruby\site_ruby\1.8\DBD\ADO

These commands simply update the RubyGems system and install the Active Record gem with all dependencies needed to connect to MSSQL. We then copy the ADO driver from ruby-dbi inside our ruby install directory. If you have ruby installed in a place other than "c:\ruby" make the appropiate changes to the commands above.

Now in your code you can connect to the MSSQL database using the following:

1 ActiveRecord::Base.establish_connection(

2     :adapter => "sqlserver",

3     :mode => "ADO",

4     :username => "youruser",

5     :password => "yourpass",

6     :host => "serverhostname_or_ipaddress",

7     :database => "YourDbName"

8 )

I have tested this configuration with ActiveRecord version 1.14.4 and 2.0.2 but I assume this should work with all version in between too.

Windows Client using ODBC

I have only tested this with the latest ActiveRecord 2.0.2 so I don't know if this works with previous versions and using a WinXP Home client with the ruby one click installer (184-20 and 186-26) connecting to a Windows Server 2003 running Microsoft SQL Enterprise 2000 SP3

The following instructions are to be executed from within a command shell (cmd.exe).

1 gem update --system

2 gem install activerecord-sqlserver-adapter --source=http://gems.rubyonrails.org

As with the ADO driver these commands simply update the RubyGems system and install the ActiveRecord gem with all dependencies needed to connect to MSSQL.

The next step is where most documentation I have seen fails. They say you must specify a DSN to connect via ODBC but they do not mention what this DSN is or where to get it. I guess these gurus assume we must know this things or we would not be looking for this info but in my case it took me a while to figure out where this DSN thing is supposed to come from.

The DSN (Data Source Name) must be created and in Windows we do it by using a tool called "odbcad32.exe" or choosing "Data Sources (ODBC)" from the "Administrative Tools" inside Windows "Control Panel". For instructions on how to use this tool refer to the detailed explanation at www.truthsolutions.com.

When creating the new DSN make note of the name you set on it (see figure above) because this is the name you must use to connect to the MSSQL server when using ActiveRecord. Also make sure you select the "SQL Server" driver as instructed at www.truthsolutions.com. The authentication method depends on how you have your MSSQL server configured but I always use the using Login and Password entered by the user option so I don't know about the network login ID option. At the end make sure you test the DSN to make sure it is valid and is working.

Once the DSN is created you can connect to the MSSQL server using ActiveRecord like:

1 ActiveRecord::Base.establish_connection(

2     :adapter => "sqlserver",

3     :mode => "odbc",

4     :username => "youruser",

5     :password => "yourpass",

6     :dsn => "SQLServer"

7 )

and make sure to replace "SQLServer" with the DSN name you set up when creating it using the "odbcad32.exe" tool.

Linux Client using ODBC

The following instructions were tested in Kubuntu Gusty (7.10) using the latest ActiveRecord 2.0.2 as client connecting to a Windows Server 2003 running Microsoft SQL Enterprise 2000 SP3.

From console install the following packages:

1 sudo aptitude install ruby1.8 unixodbc tdsodbc libodbc-ruby1.8 rubygems

Next you can choose to install ActiveRecord using Ubuntu repositories or using RubyGems repositories. I chose to use RubyGems as this provides the latest versions and is updated more often.

1 sudo gem install activerecord-sqlserver-adapter --source=http://gems.rubyonrails.org

Now as in Windows we must create a DSN for unixODBC. The first step is to add the freeTDS driver to unixODBC and to to this we create a configuration file (simple text file) called for example "myodbcdriver.ini" that contains:

1 [FreeTDS]

2 Description     = TDS driver (Sybase/MS SQL)

3 Driver          = /usr/lib/odbc/libtdsodbc.so

4 Setup           = /usr/lib/odbc/libtdsS.so

5 CPTimeout       =

6 CPReuse         =

If you are not using Kubuntu/Ubuntu then make sure you have installed unixODBC and the freeTDS ODBC driver and change the "Driver" and "Setup" paths to reflect your system installation. In Kubuntu the "tdsodbc" package we installed above provides us with this configuration file in "/usr/share/tdsodbc/odbcinst.ini" so we do not need to create it.

With this configuration we can add the freeTDS driver to unixODBC by issuing the following command:

1 sudo odbcinst -i -d -f myodbcdriver.ini

or in Kubuntu that provides the configuration file

1 sudo odbcinst -i -d -f /usr/share/tdsodbc/odbcinst.ini

Next we must create a DSN configuration file, say mydsn.ini, that looks like:

1 [SQLServer]

2 Driver      = FreeTDS

3 Description = My First SQLServer Database

4 Trace       = No

5 Server      = hostname_or_ipaddress

6 Port        = 1433

7 Database    = SQLServerDB

As in Windows take note of the name you set to the DSN ([SQLServer]) as this is used for the connection. Finally we add the DSN to unixODBC with the following command:

1 odbcinst -i -s -f mydsn.ini -h

Note that we execute this command without the "sudo" because we are installing a User DSN that will be accessible only to the system user that installed the DSN. To install a system wide DSN we use a similar command but with the "-l" switch instead of the "-h".

1 odbcinst -i -s -f mydsn.ini -l

You can run the odbcinst command without any arguments to get a list of possible options. To test the newly created DSN we can use the isql command tool that comes with unixODBC:

1 isql SQLServer username password

replacing SQLServer with the name you used when creating the DSN and if everything is working you should get a message similar to:

1 +---------------------------------------+

2 | Connected!                            |

3 |                                       |

4 | sql-statement                         |

5 | help [tablename]                      |

6 | quit                                  |

7 |                                       |

8 +---------------------------------------+

9 SQL>

input "quit" to exit the SQL prompt. As in Windows, once the DSN is created you can connect to the MSSQL server using ActiveRecord using:

1 ActiveRecord::Base.establish_connection(

2     :adapter => "sqlserver",

3     :mode => "odbc",

4     :username => "youruser",

5     :password => "yourpass",

6     :dsn => "SQLServer"

7 )

again make sure to replace "SQLServer" with the DSN name you set up when creating it.

Some notes about the Linux to MSSQL connection:

There are GUI based tools similar to "odbcad32.exe" in Windows that allows you to manage DSN's and ODBC drivers more easily. In Kubuntu simply install the "unixodbc-bin" package and then execute "OBDCConfig". As you can se from the image above this tool is quite similar to the Windows counterpart.

The freeTDS driver is free and even thought it is rather stable I have found some small issues specially when working with non-ascii character sets like SJIS and ISO-2022-JP used in Japanese OSes. There is a commercial alternative driver produced by EasySoft in case you need commercial support and reliability.

There is another implementation of ODBC in Linux called iODBC but have never used it. It may be interesting to test the Windows ODBC, unixODBC and iODBC in performance.

Linux Client using ADO driver

So far I have been unable to make this configuration work. It seems that the ADO driver uses some components from the win32ole library that is only available in Windows machines. This means the only way to connect to a MSSQL server from a Linux client is using the unixODBC/freeTDS driver method described before.

If someone knows of other ways to make Linux communicate with a MSSQL database please let me know.

Resources

http://softiesonrails.com/2006/6/28/activerecord-with-sqlserver-without-rails
http://www.imarichardson.com/
http://www.themolehill.com/
http://www.freetds.org/
http://www.unixodbc.org/
http://www.unixodbc.org/doc/FreeTDS2.html
http://www.truthsolutions.com/sql/odbc/creating_a_new_odbc_dsn.htm
http://josiah.ritchietribe.net/blog/archive/2006/01/1176/

RubyGems Explicit Versioning

2008-02-18T11:12:00.004+09:00

One of the most useful features of RubyGems as a package manager is the ability to force explicit versions of the packages in your ruby code. Unfortunately worst than not being documented this feature is wrongly documented even in RubyGems own manual pages.

So for the record here is how to tell your ruby code to use a specific gem version:

For RubyGems version < 0.9.4 we use:

1 require 'rubygems'

2 require_gem 'mechanize','=0.4.7'

3 require 'mechanize'

What we do is call "require_gem" to set up the version we want and then load the gem using "require".

After RubyGems 0.9.4 onwards the "Kernel#require_gem" method was deprecated and replaced with "Kernel#gem". So if you have the latest RubyGems the code above becomes:

1 require 'rubygems'

2 gem 'mechanize','=0.4.7'

3 require 'mechanize'

Instead of using "=0.4.7" it is also possible to use "<0.4.7" or ">0.4.7" if the specific version is not required.

We can be more professional and make our scripts handle both cases above using the Gem::RubyGemsVersion constant:

1 require 'rubygems'

2 if Gem::RubyGemsVersion < "0.9.4"

3 require_gem 'mechanize','=0.4.7'

4 else

5 gem 'mechanize','=0.4.7'

6 end

7 require 'mechanize'

This way our code will (hopefully) work with all versions of RubyGems.

References:
http://redhanded.hobix.com/inspect/autorequireIsBasicallyGoneEveryone.html
http://piao-tech.blogspot.com/2006/11/tips-on-rubygems-installation-and.html

Per user Latex style and bst files

2008-02-12T13:21:00.001+09:00

Easy way to tell latex where to look for custom sty, cls and bst files:

1 # The double slash at the end means recursive.

2 export BSTINPUTS=.:~/.latex//:

3 export BIBINPUTS=.:~/.latex//:

4 export TEXINPUTS=.:~/.latex//:

Change the path to reflect your system and make sure to include the current dir "." in the list of paths or latex won't be able to find the tex files in the current directory.

Reset Master Password in Thunderbird 2.0.0.9

2008-01-07T23:03:00.000+09:00

For some mysterious reason the master password of Mozilla Thunderbird got set in my wife's computer (Windows XP) and every minutes a dialog box would pop up asking for it. Of course no one knew what the master password was so using Thunderbird in that machine was starting to get annoying.

After searching in Google I found two solutions: one consisted in create a new profile and transfer the mail data from the problematic profile to the new one.This solution required too much effort for my liking. The second solution that everybody said would not fail was running the following command:

"C:\path to thunderbird\thunderbird.exe" -chrome chrome://pippki/content/resetpassword.xul

Unfortunately for me this command never worked no matter what I did and as this forum thread shows I am not alone so I assume is a Thunderbird problem and not mine. I even tested the command in two different machines one with a clean Windows installation and I can assure this command does not work!.

So a little more research showed me that the passwords are stored in a file called key3.db that resides inside the mail profile folder. If you don't know where is your profile folder check this link.

So I simply closed Thunderbird and renamed the "key3.db" file to "key3.db.bak" and started Thunderbird. And that was it... all passwords were gone but also the annoying master password was gone too.

If you need to reset your master password for any reason and you do not care loosing the already stored passwords (web account passwords) then this is the simplest method to do it.

Shade windows in KDE using double click in the title bar

2007-12-27T00:07:00.000+09:00

In KDE using compiz-fusion to shade the current active window we need to press Ctrl+Alt+s that is not a very easy thing to do specially in small keyboards (i.e. laptops).

Another way to shade windows is by double clicking in the window title bar but for this to work we must first enable it in kwin by editing the file: ~/.kde/share/config/kwinrc. Inside this file after the section [Windows] simply add the following line:

TitlebarDoubleClickCommand=Shade

Now after you have restarted you KDE session you will be able to shade windows by double clicking their title bars.

How to fix the "Warning: Unable to load the OpenGL" error in Matlab

2007-10-21T13:42:00.000+09:00

If you use Matlab in K/Ubuntu and you get a warning message like "Warning: Unable to load the OpenGL" everytime you start Matlab it means you are not using the powerful OpenGL rendering engine to draw your plots.

This may not be a big problem as Matlab will work as expected but if you have a 3D accelated graphics card (ATI or NVidia) then you could greatly benefit from having Matlab use the OpenGL renderer. Even without a 3D accelerated graphics card I would say that using OpenGL for rendering is a plus.

So solve this problem you simply need to remove a file (libgcc_s.so.1) from the Matlab install directory in order for it to use the one in the system that has correct links to the OpenGL libraries.

Find the file inside $MATHROOT/sys/os/glnx86 and move it to another place like:


mv libgcc_s.so.1 libgcc_s.so.1.old

Restart Matlab and the warning message should have dissapeared and the command "opengl info" should give and output like:


Version        = 2.0.6473 (8.37.6)
Vendor         = ATI Technologies Inc.
Renderer       = ATI MOBILITY FireGL V3200
MaxTextureSize = 2048
Visual         = 0x29 (TrueColor, depth 24, RGB mask 0xff0000 0xff00 0x00ff)
...

The output may vary depending on your graphics card or if you have accelerated or non-accelerated (i.e. Mesa) OpenGL graphics.

Dealing with Errno::EBADF in ruby net/http

2007-08-01T09:08:00.000+09:00

In August 2006 I posted a question in a forum about an error (Bad File Descriptor) I was getting when using the net/http library of Ruby. Today, almost one year latter, I received an email from someone with that same problem asking for my advice and a few hours latter the same person wrote again saying he found a satisfactory solution and wanted to share it with me.

As part of this wonderful online community I am obliged to share my new knowledge in hopes it is useful to others as it was useful for me.

The Bad File Descriptor error (Errno::EBADF) occurred sporadically while using the net/http library to connect to a lot of pages (web spider) in a short time span. The main problem was that I could never catch that error (i.e. rescue it) and the script would not finish leaving a lot of pages without processing. To solve this problem at the time I split my script is several smaller ones and added a small delay between web pages.

My solution works but that small delay for a thousand pages add up and the scripts take not minutes but hours to finish.

The email I got explained that the cause of this error is that the operating system (Win XP) is running out of TCP ports for new connections. Many of the sockets open are put in TIME_WAIT state (meaning that the client has closed but the server has yet to close from its side).

The first approach to solve this is to force Ruby to close the connection but there is no such facility, at least in Ruby 1.8.x, for doing it. The second solution and the one I received by mail was to increase the upper range of dynamically allocated to client TCP/IP connections to a value.

Here are the instruction on how to do it:
http://msdn2.microsoft.com/en-us/library/Aa560610.aspx

If you expect to make a lot of http connections fast using Ruby in a Windows XP machine then you better increase that number of ports or you will be around asking yourself what this random EBADF error is all about.

Doja Development in Netbeans

2007-06-26T16:40:00.000+09:00

Back in October 2006 I wrote a three part post (one, two and three) on how to develop mobile applications for Japanese phones using Netbeans as IDE. When I say Japanese phones I refer to phones provided by the three major cellular network providers in Japan: KDDI, Softbank and NTTDoCoMo.

Recently I was assigned a new project that requires the implementation of a Doja IApplication for the 903i release of NTTDoCoMo phones. Since these phones support Doja-5.0 profile I started to add support for it into my Netbeans installation and in the way I decided also to upgrade to Java-6u1 and Netbeans 5.51.

After some time setting all up I can say it was worth the effort and this new set up is cleaner and more easy than my previous one. For those interested here is my set up.

Pre-requisites

If you want to upgrade to Java-6u1 I strongly recommend you to uninstall all other instances of Java on your system. I had a lot of rare problems until I removed the previous Java 1.4.2 from my system. Uninstall Netbeans and the mobility pack if you have it.

Also uninstall all previous Doja toolkits you may have as they are all included in the Doja-5.0 toolkit.

Download and install Java SDK, Netbeans, Netbeans Mobility Pack and the Doja Toolkit-5.0 in that order. Here are the links:

Java Development Platform 6u1. link

Netbeans and Mobility pack link

Doja Toolkit 5.0. link

Doja Netbeans Pack

Once you have all installed you can download my Doja Netbeans pack (doja-netbeans.zip) that contains some needed configuration files and a single ant task that we will use to generate the JAM file and a nice download html file that are needed for deploying IApplications.

To use the Doja Netbeans Pack you need to do some manual work. First uncompress the zip file and you will see three files:

Doja.xml - Contains Doja profile configurations.

build.xml - Build script that creates JAM and HTML files for IAppli deployment

doja-ant-tasks.jar - My custom ant task used by the build script to create the JAM and HTML files

Adding the Doja Emulator Platform to Netbeans

Make sure Netbeans is not running and copy the Doja.xml configuration file inside your users .netbeans platform configuration directory. This directory is usually:

c:\Document and Settings\username\.netbeans\5.5.1\config\Services\Platforms\org-netbeans-api-java-Platform\

After you have copied the Doja.xml file to the correct directory you must open it and edit the platform home element. Simply look at the top of the file the xml tag and change the home="C:\install_path\iDKDoja5.0" with the directory were you installed the Doja toolkit. If you choose default installation when installing the Doja toolkit then you should put home="C:\iDKDoja5.0".

If everything is ok then when you start Netbeans and create a new Mobile Application you should be able to select the NTT Doja Emulator Platform and one of the FOMA devices. Each device uses an specific Doja Profile version:

FOMA904i - Doja-5.0

FOMA903i - Doja-5.0

FOMA902i - Doja-4.1

FOMA901i - Doja-4.0

FOMA900i - Doja-3.5

FOMA506i - Doja-3.0

FOMA505i - Doja-3.0

You must choose the device depending on what Doja profile the device you are developing for supports. A list of devices and the Doja version they support is available at NTTDoCoMo home page here .

If you don't see the Mobile Application option when creating a new project make sure you have installed the Netbeans Mobility Pack correctly. If you are able to create Mobile Application projects but you don't get the NTT Doja emulator platform make sure you have copied the Doja.xml file in the correct place and edited the home property correctly.

Setting up the Doja ant task

Create a new Mobile Application using Netbeans and once created replace the project's build.xml file with the one that came with the Doja Netbeans pack. This modified build.xml script should look like:

<?xml version="1.0" encoding="UTF-8"?>
<project name="" default="jar" basedir=".">
<description>Builds, tests, and runs the project .</description>
<import file="nbproject/build-impl.xml"/>

<property name="jam.AppClass" value="HelloNtt" />
<property name="jam.AppName" value="HelloNtt" />
<property name="jam.PackageURL" value="http://test.mydomain.org" />
<property name="jam.Network" value="http" />
<property name="jam.SPSize" value="1024" />

<taskdef name="jamwriter" classname="jp.doja.ant.JamTask">
<classpath>
<pathelement location="C:\path_to_jar\doja-ant-tasks.jar"/>
</classpath>
</taskdef>

<target name="post-jar">
<jamwriter />
</target>

<target name="run" depends="init,jar">
<exec executable="${platform.home}/bin/doja.exe" dir="${dist.dir}">
<arg value="-i" />
<arg value="${name}.jam" />
</exec>
</target>

<target name="post-clean">
<delete file="${dist.dir}/${name}.jam"/>
</target>
</project>

For this build script to work you must search for the tag inside the jamwriter taskdef and modify the location to point to the doja-ant-tasks.jar file that came with the Doja Netbeans Pack. Simply said modify the "C:\path_to_jar\doja-ant-tasks.jar" line to point to the place that file is located in your computer.

This custom task will create the needed JAM descriptor file and an HTML file you can use to deploy your IApplis. The most relevant entries that go inside the JAM file are already on the build.xml script:

You can add as many entries as you like by simply creating a new tag with the name starting with "jam." followed by the actual property name and the value set to the entry value. If you do not understand what the JAM descriptor file is or what properties are available refer to this tutorial at the Doja Developers Network. Again the most used ones are the ones presented above.

Final Remarks

Well we are finished. Now we can create Doja IAppli's easily in Netbeans taking advantage of all the tools Netbeans has to offer for that purpose. We get automatic preverification, several levels of ofuscation, packaging, JAM and download HTML files creation (via custom ant task) and most important device fragmentation control. Actually the only reason I like to use Netbeans for J2ME/Doja development rather than using the native Doja Toolkit or Eclipse is because Netbeans has a nice preprocessor that makes it easy to deal with the device fragmentation problem.

To test this setup you can refer to my previous blog post on the topic (see links at the beginning of this post).

Skype Linux is comming fast!!

2007-06-16T14:13:00.000+09:00

When the new 1.4.0.74 release of Skype was released with Feisty Fawn binary packages and a repository I simply jumped in and installed Skype on my system using those repositories.

Happy with the new release I fired up Skype to be greeted with my old friend, the ***glibc detected *** bug.

Sad for the step back I started to read the FAQ on how to report a bug and when I was about to submit the report the little apt-get update icon appeared on my systray. I clicked and saw that a new version of Skype was available to install... that was fast!! . A couple of clicks and I had the new 1.4.0.74-1 release installed and the what is better the bug was gone!. Skype runs just perfect and looks great on my laptop now. At this speed I expect video support to be here very soon.

Komodo - Japanese Encodings

2007-05-31T18:15:00.000+09:00

If you are considering Komodo Edit as editor and you work for a Japanese company you should be aware that the default installation of Komodo does not support Japanese text encodings like cp932 (Windows), EUC-JP and Shift-JIS.

Fortunately by modifying some configuration files we can make Komodo Edit to recognize and render these text encodings correctly (ref1 and ref2). The configuration file you need to modify is called "koEncodingServices.py" and the location depends on the OS you are using:

In Mac the file is located in

Komodo IDE.app/Contents/MacOS/components/koEncodingServices.py

In Linux and Windows the file is located in

${INSTALL_DIR}/lib/mozilla/components/koEncodingServices.py

Change ${INSTALL_DIR} with the root path were you installed Komodo Edit in you machine.

Close Komodo Edit and using you next favorite text editor open the file and in line 81 add the following three lines (see screenshot):

('EUC-JP', 'EUC-JP', 'EUC-JP', '', 1, 'EUC-JP'),
('Shift_JIS', 'Shift_JIS', 'Shift_JIS', '', 1, 'Shift_JIS'),
('cp932', 'Japanese (cp932)', 'CP932', '', 1,'cp932'),

Impressions

The only thing I liked about it is the ability to make it feel like Vim using some key bindings with project management enhancements. This only reason was not good enough to make me switch and soon I was back to my Command Prompt + Vim + muCommander in Windows and yakuake + Vim + Konqueror in Linux.

Komodo felt too cluttered and was way too slow to start and edit with specially on Linux. Finally if you decide to use Komodo Edit for any reason you may as well change the default fonts as they are not too good for programming tasks. You can find the best fonts for programming here. From the list my favorites are the Bistream Vera Sans Mono and Dina Fonts that look rather good on Linux and on Windows as well.

Synergy Japanese Keyboard Support (Kana key)

2007-04-25T17:33:00.000+09:00

For the impatient:

I patched and compiled Synergy to enable the Kana key used in Japanese keyboards to switch between input methods. I also created the Windows installer for the patched version for those Windows users that do not know what a patch is or what compiling means. You can find the installer here.

Unfortunately for Linux users I have no idea on how to create packages for any distribution so you will have to compile from sources. Fear not as it is not complicated at all and I explain how to do it below.

A dual screen hack (revisited):

In one of my previous post I talked about how I used X2VNC/Win2VNC in combination with RealVNC/X11VNC to share a single keyboard and mouse between several computers even if these computers had different operating systems.

Using the above programs works well but I have been experiencing some problems like not being able to use the Japanese keys on my keyboard and some responsiveness degradation (late mouse reaction) when the machine running the VNC server is under heavy load.

Due to these small problems I started looking for an alternative and found Synergy. This application is small and does not require the additional VNC server. Runs in Windows, Linux and Mac and the transition between screens is smooth even if the computers are under heavy load. Another thing I like a lot is the centralized configuration (only one server) that allows to easily configure more than two computers. Creating a setup with more than three computers using X2VNC/WIN2VNC could be very frustrating. Also there are GUI configuration tools for Windows, Linux and Mac that make even easier to setup (here).

There are a lot of resources in the Internet that explain how to install and use Synergy to create dual screen systems (here, here and here) so I wont explain it here. What I will explain is how to get the Japanese keys to work with synergy.

Some videos of Synergy in action can be found here

Japanese Key's support in Synergy

The current version of Synergy does not handle the Japanese keys used to change the input method (i.e. ローマ字/漢字). There is a Japanese keyboard (Hankaku/Zenkaku-Kanji and Kana key) patch in the patches section of the Synergy sourceforge page but no binary releases are available yet. I took the time to patch and compile the source code to enable these keys on my Japanese keyboard in Linux and Windows and here are the instructions on how I did it.

Instructions for Linux Kubuntu Desktop

1. Download the source code from here. The last version was 1.3.1 at the time of this writing.

2. Make sure you have installed the development tools and the XTest development package

sudo aptitude install build-essential
sudo aptitude install libxtst-dev

3. Unpack and edit the source code

tar xvfz synergy-1.3.1.tar.gz
cd synergy-1.3.1

4. Edit the source code to add Hankaku/Zenkaku-Kanji and Kana key support

Open the lib/platform/CMSWindowsKeyState.cpp file and search the following three lines:

/* 0x0f2 */ { kKeyNone }, // OEM specific
/* 0x0f3 */ { kKeyNone }, // OEM specific
/* 0x0f4 */ { kKeyNone }, // OEM specific

and replace them with these corresponding lines

/* 0x0f2 */ { kKeyOEMCopy }, // VK_OEM_COPY
/* 0x0f3 */ { kKeyZenkaku }, // VK_OEM_AUTO
/* 0x0f4 */ { kKeyZenkaku }, // VK_OEM_ENLW

Open the lib/synergy/KeyTypes.h file and search for a line like:

static const KeyID kKeyDelete = 0xEFFF; /* Delete, rubout */

and add the following below:

static const KeyID kKeyOEMCopy = 0xEF27; /* OEMCopy(Kana Key) */

Actually you can add this line of code anywhere you want inside the KeyTypes.h file but to keep things in order I add it after the kKeyDelete const declaration.

5. Make the configure/make/make install dance.

./configure -x-includes /usr/include -x-libraries /usr/lib --prefix=/usr
make
sudo make install

6. Synergy is now installed on your Kubuntu system and you can proceed to configure it. Refer to the links in the previous section for instructions.

Instruction for Windows

1. To compile Synergy in Windows you need Visual C++ (Express Edition) that can be downloaded for free from here.

2. To create the installer we need NSIS. Simply download the installable from here and install it.

3. Download the source code (synergy.1.3.1.zip), unpack it somewhere. Edit the file "dist/nullsoft/installer.mak" inside the source code to make sure it points to the correct NSIS installation path. Simply open the installer.mak file with any text editor and make sure the path for makensis is correct:


NSIS="C:\Program Files\NSIS\makensis"

This example is if you installed NSIS in the default path.

3. Open the synergy.dsw VC++ project. Say yes to all the requests to convert the project format and edit the source code as directed below

Open the platform->Source Files->CMSWindowsKeyState.cpp file and search these six lines:

/* 0x0f2 */ { kKeyNone }, // OEM specific
/* 0x0f3 */ { kKeyNone }, // OEM specific
/* 0x0f4 */ { kKeyNone }, // OEM specific

and replace them with these corresponding lines

/* 0x0f2 */ { kKeyOEMCopy }, // VK_OEM_COPY
/* 0x0f3 */ { kKeyZenkaku }, // VK_OEM_AUTO
/* 0x0f4 */ { kKeyZenkaku }, // VK_OEM_ENLW

Open the libsynergy->Header File->KeyTypes.h file and add the following line:

static const KeyID kKeyOEMCopy = 0xEF27; /* OEMCopy(Kana Key) */

just after this line:

static const KeyID kKeyDelete = 0xEFFF; /* Delete, rubout */

If you followed the Linux instructions you will note that these changes are exactly the same as does we did in Linux.

4. Make sure the compiler is set to Release configuration (i.e. select release in the project properties) and build the "all" and "installer" projects. To compile right click on the project names (all and installer) and select build from the context menu.

5. After both builds finish there will be a "SynergyInstaller.exe" file inside the "build" directory inside the source code directory. Execute this file to install Synergy as normal.

6. Proceed to configure (see references in previous section) and now you will be able to change from romaji to kana using the Kana key available on all japanese keyboards.

Mac Instructions

Unfortunately I do not have a Mac to try this patch but there are instructions on how to compile Synergy on a Mac inside the doc/compiling.html file. If someone is successful applying this patch on the Mac I would like to hear about it.

私のTech記憶

Google contacts with sup email client.

Introduction

SUP email client with Gmail contacts

Resources

Get KDE 4 .4.2 working with xrdp

Problem Statement

Configure KDE4 with x11vnc server

Configure KDE4 with xrdp server

Accessing remotely you KDE session via Windows RDP client

Troubleshooting

Enabling the Kana Henkan key (Japanese Keyboards)

Resources

2010 Vim Configuration

Japanese Input in Lucid Lynx (Kubuntu 10.04)

Introduction

Uninstall SCIM/SKIM packages

Install iBus required packages

Configure the system to auto-select the input method.

Trouble Shooting

Resources

Get offlineimap working with non ASCII characters.

Introduction

Problem Description

Fixing unicode issues in offlineimap

Resources

Google Contacts in Mutt

Introduction

Query GMail Contacts within Mutt

Integrating GMail contact groups within Mutt

Managing Mutt mailing lists within GMail

Resources

Brother DCP-595CN Printer In Ubuntu/Kubuntu

Prerequisites

Installation

Brother DCP-595CN Scanner In Ubuntu/Kubuntu

Synchronizing two machines

Manage SSH and GPG keys efficiently in KDE

How to create Netem distribution tables

Embbed cover art in MP3 files with command line

Introduction

Embedding cover art in MP3 files

Android cover art for Ogg/Vorbis and MP3 files

Handling multiple encodings in Vim

tenc:

enc:

fenc:

fencs:

Vim tips for Latex editing

Simple Latex Makefile

Latex with Vim TagList plugin

Reference autocompletion

Spell checking

Mounting Windows Shares Using CIFS on Ubuntu

References

NVidia InitialPixmapPlacement=2 really improves KDE speed

Use vimdiff as Subversion diff command

Open new files as tabs in current Vim Console

Make Vim and Ctag play nice together

Pre requisites

Generate system wide ctags

Ctags for Linux System Calls

Ctags for Ruby core and gem libraries

Ctags Java libraries

Updating your system wide ctags

Generating ctags for your own libraries/projects

Auto-generating Ctags Using the Project.vim plugin

in.vim

out.vim

Tell Vim where to search for tags

Setting up the TagList Plugin

Basic .vimrc file

Canon Satera LBP5910 Linux Driver

Use PuTTY generated keys with SftpDrive

Subversion + OpenSSH Server + PuTTY Client Setup

Best gift ever??

Plotting SVN history??

Linux Cafe in Akihabara

Simple Mail Server (Kubuntu)

Get latest ffmpeg (svn) in FreeBSD-7.0