Skip to content

The Definitive Analysis of Palm Pre App Install Limits and the Palm App Catalog Hanging

After the Preware 0.9.4 release, which included Applications, Themes, and Patches, and offers over 670 homebrew packages for installation, we (webos-internals.org)  started getting reports of the Palm App Catalog “hanging” the user interface for 30 seconds or more when the installation of a new application is initiated, but only when the Package Manager Service (the service which does all the Linux-level work for the Preware front-end) was installed.

After some analysis, I found that disabling all the feeds in Preware reduced the “hang” from more than 30 seconds to less than a second.

Looking through the output of ‘dbus-util –capture’ showed that the “hang” was during the call to the queryInstallCapacity method of the com.palm.appinstaller service, the routine that the Palm App Catalog application uses to determine whether there is enough space to install a new application from the Palm App Catalog.  Yes, this is the method which is at the heart of the dreaded “Too many applications” errors that people are seeing when they have a number of homebrew applications installed and try to install a new application from the Palm App Catalog.

Watching the output of ‘ps -ef’ during calls to this method (you can call it manually using luna-send) showed that palm was calling “ipkg -o /var list”.  Curious.  Why would you want to know the *complete* list of all *available* applications when determining whether there is room to install one known new application.  I suspect that Palm should be calling “ipkg -o /var list_installed” instead (which just lists the installed applications).  Note that Palm doesn’t use feeds the way that Preware does, so for Palm’s official use of ipkg, list and list_installed would return the same thing in their testing, but list_installed is definitely what they should be using to determine the size of installed applications.

The plot thickens when you use strace (which Palm conveniently includes in the official firmware image) on the running LunaSysMgr process.

It seems that LunaSysMgr calls “ipkg -o /var list” to get a list of installed packages (the probably intend to just get the list of installed packages, but when you have Preware installed and have feed configuration files in /var/etc/ipkg/*.conf, it actually returns a list of all *available packages).

LunaSysMgr then does an execve of “/usr/bin/du -s /var/usr/palm/applications/package” for each package in that list.  (BTW Palm, you seem to have a bug in the logic of that code, cause it’s running du on random garbage strings after the end of the real package list)

Yes, that’s right.  A call to queryInstallCapacity spawns a new program (“du”) once for each package returned by “ipkg -o /var list”.  No wonder the UI hangs for 30 seconds or more!

A single “du -s /var/usr/palm/applications/*” would be a far more efficient way to get exactly the same information, but again, Palm would not see this difference in testing because they do not support the third-party Preware application usage of ipkg feeds.

You can imagine that this behaviour is probably related to the app install limit that many people are experiencing too.  Unfortunately, I’ll have to reduce my /var partition size down from it’s current 2GB size (courtesy of the WebOS Internals Meta-Doctor) to be able to investigate this one.

Now the webos-internals.org developers need to develop a new method of installing homebrew applications so that this bug in Palm’s appInstaller service is not triggered.

In the meantime, the work-around is to go into the Preware Preferences screen, hit the “Feeds” button in the top-right corner, and disable all the feeds while you use the Palm App Catalog application in another card.  No need to exit the Feeds screen, just toggle all the button to “Off”, and then toggle them back to “On” when you’re finished with the App Catalog.

For the solution to this problem, see Update #2, below.

I’ve created a thread on PreCentral where this issue can be discussed.  As I uncover more information, I’ll publish my finding here.

Update #1: I’ve now webOS Doctored my Pre in the name of science, and have tested the limits of installing applications.

If you run “du -s /var/usr/palm/applications/*”, and add up all the numbers in the first column, then as soon as you hit the 62367 1K blocks limit of the addition of the sizes reported by that “du” command and the size of the app you with to install, you will get the dreaded “Sorry, Not Enough Memory” error from the Palm App Catalog application (and any other installer, like fileCoaster or PreLoad, which uses the palm appInstaller API).  It doesn’t matter whether you have 192MB free in your /var partition, it will max out at just under 64MB of application usage.

Update #2: I have now created a Linux Application called “Fair Dinkum App Limit” (org.webosinternals.fairdinkum), which removes both the “hang” and the arbitrary application limit.  You can find it in Preware.  Just install it (no need to even run anything – if it is installed, it’s working), and you’re ready to install more applications than you can poke a stick at …

Fair Dinkum App Limit works by simply putting a couple of wrapper scripts in /usr/local/bin, which returns a size of zero when du is called, and returns the output of “ipkg -o /var list_installed” when “ipkg -o /var list” is called.  In the future, the wrappers will be made much more sophisticated than they are right now to prevent operation outside of the specific cases where they need to fool LunaSysMgr, and to also include a safety buffer so that users do not fill the /var partition.  This is a tactical response to a problem that people using homebrew applications are experiencing.  Hopefully, Palm will provide the long term solution for limits on application installation in a release in the near future.

Notes for Palm, if you happen to read this:

1) We fully appreciate that the usage of the ipkg database in /var for homebrew applications is a choice that the homebrew community has made, and is not supported by Palm.

2) We fully agree that the use of “ipkg -o /var list” instead of “ipkg -o /var list_installed” would work perfectly fine for the way that Palm is officially using the ipkg database in /var/usr/lib/ipkg, but we contend that the “list” and “list_installed” commands have clear intended usage, and the one to find the list of installed applications for checking available disk space should be “list_installed”.

3) We fully realise that the initial version of the FairDinkum scripts are unsafe.  Returning a zero value for du is a temporary solution while we work out how to achieve the same result safely.  The intention is to only return false values when du is being called from LunaSysMgr, and to make sure that a safety buffer is kept so that users do not fill the /var partition.

4) I would be very happy to discuss these issues with anyone at Palm (Chuq and Google both have my email address), and would hope that we can together architect a solution for supporting homebrew application installation which does not require these work-arounds.

5) We have designed these work-arounds in a way which does not collide with Palm OTA Updates, and in a way that we can update them automatically using the Preware installer application, and in a way that we can cause them to self-remove when Palm releases a long term solution.

Update #3:

It seems that there is yet another limit on application installation imposed by LunaSysMgr.  Once the used space on /var crosses around 60%, LunaSysMgr will once again refuse to install applications.

I’m going to need to webOS Doctor my Pre yet again (to reallocate 2GB to /var) to determine whether this limit is a fixed percentage, or a fixed number of blocks.

Update #4:

The limit is 102400 free blocks.  Mystery solved.  That also means the Fair Dinkum App Limit cannot cause your /var to be filled.

Update #5:

Thanks to Carl Thompson, an improved version of Fair Dinkum App Limit which does not alter the operation of ‘du’ has been released.

2009 10 12 WebOS Internals Comments Off on The Definitive Analysis of Palm Pre App Install Limits and the Palm App Catalog Hanging Permalink

The WebOS Internals Meta-Doctor

Palm supplies Palm Pre owners with this wonderful recovery tool called the webOS Doctor.  Part of the charter of the WebOS Internals project is to ensure that anything we (or anyone following instructions we publish or installing packages we develop) do can always be reverted using the webOS Doctor.

Usually, a Palm Pre is required to be activated on the Sprint network before it can be used.  This is not possible for a Palm Pre in Australia.

So we need to allow the Pre to be used without activation, and there are a number of information sources showing how this can be done.  There are also some dubious sites that redistribute modified versions of the webOS Doctor (which is a clear violation of copyright law, since it contains proprietary Palm software).  Note that WebOS Internals is always careful to comply with all copyright laws (copyright law is the foundation upon which open source licenses are based).

So we need a way for a Pre owner (who has the right to use the webOS Doctor on their own Pre) to modify the webOS Doctor that is specific to their particular version of the Palm Pre before using it to flash that modified firmware onto their Pre.

That’s where the WebOS Internals “Meta-Doctor” comes into play.

I have created a tool which will download, unpack, patch, and repack a webOS Doctor image, applying a number of transformations along the way to:

  1. Bypass the need for activation
  2. Enable Palm Profile access
  3. Set developer mode on by default
  4. Increase the size of the /var partition to 2GB

You can find this tool in the WebOS Internals source code repository at https://github.com/webos-internals/meta-doctor

Do not redistribute modified versions of the webOS Doctor created using this tool – it is for end-user use only.

I’ve created a forum thread on PreCentral for discussion about this tool.

2009 10 06 WebOS Internals Comments Off on The WebOS Internals Meta-Doctor Permalink

Palm Pre GPS doesn’t like my hemisphere

It seems the Palm Pre GPS was never tested in the southern hemisphere – my new Pre’s GPS reports Lat: 394.6, Long: 138.6

24926.609       PUB     call    460             :1.26   com.palm.location       //getCurrentPosition  «string=“{}”, string=“com.sfmpllc.sendmylocation 1058”»
24926.641       PRV     call    238             :1.68   com.palm.phone  /com/palm/phone/tel_getradiotype      «»
24926.661       PRV     return  238     0.020   :1.26   :1.68           «string=“success”, string=“CDMA”»
24926.751       PRV     call    239             :1.68   com.palm.phone  /com/palm/phone/tel_getbsinfo «»
24926.786       PUB     call    461             com.palm.luna   org.freedesktop.DBus    /org/freedesktop/DBus/AddMatch        «string=“interface=org.freedesktop.DBus,member=NameOwnerChanged,arg0=com.palm.location”»
24926.920       PUB     return  460             com.palm.location       com.palm.luna        «string=“{“errorCode”:0,”timestamp”:1.254820510841E12,”latitude”:394.593215,”longitude”:138.681593,”horizAccuracy”:150,”heading”:0,”velocity”:0,”altitude”:0,”vertAccuracy”:0}”»

24926.609       PUB     call    460             :1.26   com.palm.location       //getCurrentPosition  «string=“{}”, string=“com.sfmpllc.sendmylocation 1058”»

24926.920       PUB     return  460             com.palm.location       com.palm.luna        «string=“{“errorCode”:0,”timestamp”:1.254820510841E12,”latitude”:394.xxxxxx,”longitude”:138.xxxxxx,”horizAccuracy”:150,”heading”:0,”velocity”:0,”altitude”:0,”vertAccuracy”:0}”»

The latitude value should be 34.6 degrees South (or -34.6 degrees).

That would explain why Google Maps isn’t working.

Now I need to work out how to replace the Coordinates java class in /usr/lib/luna/java/location.jar, so that the getLatitude method returns a number between -90 and +90 …

I wonder how many WebOS applications will then barf on a negative latitude value …

The PreCentral thread has more information on other GPS tweaks.

2009 10 06 WebOS Internals Comments Off on Palm Pre GPS doesn’t like my hemisphere Permalink

The Palm Pre lands in Australia

Thanks to the generosity of the PreCentral and WebOS Internals communities, I am now the proud owner of a Palm Pre.

There is just one catch  – since I live in Australia (which uses a proper cellular communications standard), the CDMA Palm Pre that I was able to import from the USA will never work as a phone here (yes, I knew this before I purchased it).  I plan to also purchase an unlocked GSM/UMTS Pre when one becomes available (maybe I’ll buy a German one and then swap the two keyboards).

After founding the WebOS Internals project, and using the Pre Emulator in the development of Preware, it is great to have a real device to get the full Pre experience.

If you want to keep up to date with the activities of the WebOS Internals group, just follow @webosinternals on Twitter.  You can also find a list of articles about WebOS Internals in our Press_Room.  We hang out in the #webos-internals IRC channel on Freenode, and have a webos-internals-announce Google group.

I still use my trusty old Treo 650 as my daily phone, which allows me to not have to worry about reflashing the Pre to try out things, as I don’t keep any real personal data on it.

I guess this also closes the OpenMoko chapter of my open source development activities.  I was involved with OpenMoko from the very start, but always said that an OpenMoko device with a hardware keyboard was my preferred form factor, and the Pre seems to satisfy that personal hardware form factor preference whilst still being open enough on the software side to attract my interest.  I wish those who are continuing the OpenMoko path the best of success.

I’ll document my experiences with the Pre in subsequent posts …

2009 10 06 WebOS Internals Comments Off on The Palm Pre lands in Australia Permalink

Syncing Treo650 with Google Contacts using CompanionLink for Google

In preparation for a possible move from my Treo 650 to the new Palm Pre, I’ve decided to try and synchronise my contacts between the Treo and Google Contacts.

So I’m evaluating CompanionLink for Google as a possible tool to achieve this. Another option might be GooSync.

I tried syncing some sample contacts in both directions, with the following results:

  1. Google Contacts to PalmOS Contacts

    2. Syncing a new contact from Google Contacts to PalmOS Contacts results in the following fields being synched correctly:

    • Name
    • Title
    • Company
    • Home Phone
    • Work Phone
    • Mobile Phone
    • E-mail (synched with the Work Email field in Google Contacts)
    • Fax (synched with the Work Fax field in Google Contacts)
    • Home Address
    • Work Address
    • Other Address
    • Notes

    The following Google Contacts fields are not synched:

    • Home Email
    • Other Email
    • Home Fax
    • Pager
    • Other Phone
    • IM Fields (Google Talk, Skype, Jabber, etc)
    • Custom Field

  2. PalmOS Contacts to Google Contacts

    3. Syncing a new contact from PalmOS Contacts to Google Contacts results in the following fields being synched correctly:

    • Name
    • Title
    • Company
    • Work Email (synched with the first E-mail field in PalmOS Contacts)
    • Home Phone
    • Work Phone
    • Mobile Phone
    • Home Address
    • Work Address
    • Other Address
    • Google Talk (synched with the IM field in PalmOS Contacts)
    • Notes (synched with a combination of the Custom and Note fields in PalmOS Contacts)

    The following PalmOS Contacts fields are not synched:

    • Secondary E-mail entries
    • Other Phone
    • AIM
    • MSN
    • Web site

I then tried duplicating contacts to see if I could determine the primary synchronisation key. Duplicating a contact in PalmOS Contacts and then synchronising did not result in a duplicated contact in Google Contacts. However, changing the E-Mail field in the duplicated contact in PalmOS Contacts was enough to cause it to be created as a second separate record in Google Contacts. So it seems that the PalmOS E-Mail field (which syncs with the Google Work Email field) is the primary key.
Interestingly, even while the PalmOS Contacts HotSync conduit is set up to sync with Google Contacts, the syncing with the Palm Desktop still happens. Indeed, the deletion of a record in the Palm Desktop is reflected in PalmOS Contacts on each sync, but it seems does not trigger a corresponding deletion in Google Contacts (perhaps there is some QuickSync vs SlowSync thing happening here). Modifying the record in Google Contacts which had already been deleted in PalmOS Contacts (through the Palm Desktop) did cause it to be reinstated in PalmOS Contacts on the next sync.
Adding a new E-Mail field to the PalmOS Contacts record (before the existing field) causes that new field to be the one that is synched with the Google Contacts Work Email field. So it seems that synchronisation happens between the first E-Mail field in PalmOS Contacts and the Work Email field in Google Contacts, and that only one Email address is ever synchronised between the two. If there is no Work Email field in Google Contacts, then all E-Mail fields in PalmOS Contacts are deleted. Additional Email fields in Google Contacts are not replicated in PalmOS Contacts. If an additional E-Mail field is added to PalmOS Contacts, then synchronisation exits with an error on the first attempt (“Can have at most one primary email address, found 2”), and prevents other fields being synchronisedbut then succeeds on the second attempt.  As long as a Work Email field is synching properly, other non-synching Email fields on both sides are retained (but not synched, even though other non-Email fields are synched).

2009 05 20 Treo 650 Comments Off on Syncing Treo650 with Google Contacts using CompanionLink for Google Permalink

Connecting a Treo650 to a Freecom DataTank 2

  1. Install bluez2-utils from Optware
  2. Install the following kernel modules: bluetooth, hci_usb, l2cap, bnep, rfcomm, hidp
  3. Create /dev/rfcomm0 as follows:
    mknod /dev/rfcomm0 c 216 0
  4. Enable routing from ppp0 to eth1 (don’t do this if you use ppp for your internet connection):

    /etc/init.d # diff -u routing.orig routing
--- routing.orig        Sat Mar 22 18:57:23 2008
+++ routing     Sat Mar 22 15:14:29 2008
@@ -37,6 +37,7 @@

        # lo & eth0 always accepted (also if WAN port IP not set)
        /sbin/iptables -A INPUT -i $INIF -j ACCEPT
+       /sbin/iptables -A INPUT -i ppp0 -j ACCEPT
        /sbin/iptables -A INPUT -i lo -j ACCEPT

        # get IP address from WAN port
@@ -150,6 +151,7 @@
          /sbin/iptables -A FORWARD -j TCPMSS -o $EXIF --clamp-mss-to-pmtu -p tcp --tcp-flags SYN,RST SYN
        fi
        /sbin/iptables -A FORWARD -i $EXIF -o $INIF -m state --state ESTABLISHED,RELATED -j ACCEPT
+       /sbin/iptables -A FORWARD -i $EXIF -o ppp0 -m state --state ESTABLISHED,RELATED -j ACCEPT


        grep -q ^proxy_server=checked /etc/master.conf
@@ -163,6 +165,7 @@
          fi
        else
          /sbin/iptables -A FORWARD -s $SUBNET -i $INIF -j ACCEPT
+         /sbin/iptables -A FORWARD -s $SUBNET -i ppp0 -j ACCEPT
        fi

        # port forwarding

  5. Edit /etc/ppp/options as follows:

    noauth
    crtscts
    lock
    local
    proxyarp
    ktune
    192.168.1.XX:192.168.1.YY
    ms-dns 192.168.1.ZZ

    (edit the last two lines to suit your network topology, the first IP address
    is your gateway device, the second IP address will be assigned to the client,
    and the third IP address is your DNS server)
  6. Add the following line to /etc/dnsmasq.conf:
    interface=eth0,ppp0
2008 03 22 NSLU2-Linux
Treo 650 Comments Off on Connecting a Treo650 to a Freecom DataTank 2 Permalink

Five new NSLU2 firmware releases in five days

In the last five days, we have made five new NSLU2 firmware releases:

2007-12-31 – Unslung 6.10-beta Release
2007-12-30 – SlugOS 4.8-beta Release
2007-12-29 – OpenWrt/NSLU2 Kamikaze 7.09 Release
2007-12-28 – Angstrom/NSLU2 2007.12 Release
2007-12-27 – Debian/NSLU2 Stable 4.0r2 Release

All of these new releases are available at

http://www.slug-firmware.net/

See http://article.gmane.org/gmane.comp.misc.nslu2.linux/20610 for
an explanation of the pros and cons of each different firmware
distribution, and the installable packages available for each.

Thanks to everyone in the NSLU2-Linux, OpenWrt, Angstrom, OpenEmbedded
and Debian projects who contributed to these releases.

Remember, if you find any of the firmware or packages that the
NLSU2-Linux project provides useful, feel free to make a donation to
the project at

http://www.nslu2-linux.org/wiki/Main/Donate

We are currently in need of about $500 to buy a RAID controller card
and some disks for our autobuild machine to support all this new
firmware with up-to-date package feeds …

2007 12 31 NSLU2-Linux
OpenMoko
OpenWRT Comments Off on Five new NSLU2 firmware releases in five days Permalink

Setting the OpenMoko timezone

If you want to set the timezone on your phone correctly, do the following:

  1. ipkg install tzdata
  2. ipkg install your desired tzdata-* packages.  For instance, I use “tzdata-australia“.
  3. Enable your desired timezone by symlinking it to “/etc/localtime“.  Adjust the following example command line for your locality.
    • ln -s /usr/share/zoneinfo/Australia/Adelaide /etc/localtime
  4. The “date” command should now show the correct time for your timezone.  If it is not correct, then install the “ntpclient” package, and use it to set your clock.

Note that this technique should work on any OpenEmbedded-based Linux distribution.

    2007 09 01 NSLU2-Linux
    OpenMoko     Comments Off on Setting the OpenMoko timezone Permalink

    Intercepting hotplug on the Freecom FSG-3

    The Freecom FSG-3 wireless storage router has four USB ports, and has support for hotplug built into the kernel.  This makes it ideal for use as a docking station for OpenMoko phones.

    Unfortunately, it does not have the normal hotplug agent scripts that you expect to find on a desktop Linux distribution.

    So you have to roll your own:

    1. Run “mv /sbin/hotplug /sbin/hotplug.freecom“
    2. Create a new “/sbin/hotplug” shell script (the following is an example of how to automatically enable USB networking for an OpenMoko phone): 
      #!/bin/sh

case $1 in
  ( usb )
    case $PRODUCT/$INTERFACE in
      ( 1457/5122/212/2/6/0 ) # OpenMoko GTA01 cdc-ether
        case $ACTION in
          ( add )
            ifconfig usb0 192.168.0.200 up
            ;;
          ( remove )
            ifconfig usb0 192.168.0.200 down
            ;;
        esac
        ;;
    esac
    ;;
esac

/sbin/hotplug.freecom "$@"

    3. Run “chmod ugo+x /sbin/hotplug” to ensure that your new hotplug script is executable.
    4. See http://linux-hotplug.sourceforge.net/?selected=usb for the list of environment variables you can use to distinguish different devices.
    2007 09 01 OpenMoko Comments Off on Intercepting hotplug on the Freecom FSG-3 Permalink

    Replacing dropbear with openssh

    I prefer to use OpenSSH rather than Dropbear on my devices.  The main reason is to get sftp support (which is required by sshfs).  Another reason is to get better support for agent forwarding (which is essential for bouncing from one machine to another without leaving your private keys all over the internet).

    To do this on OpenMoko (or any other OpenEmbedded-based distribution for that matter, for instance SlugOS or Angstrom):

    1. Edit /etc/init.d/dropbear by replacing “DROPBEAR_PORT=22” with “DROPBEAR_PORT=2222” (or any other unused port).
    2. Run “ipkg install -force-depends openssh” to install openssh.
    3. Make sure you have set a root password before rebooting (use “passwd” to set it).
    4. Reboot (dropbear will restart on the new port, and openssh will start on the normal ssh port).
    5. Check that openssh is now serving on port 22 by logging into the device over ssh.
    6. Run “ipkg remove -force-depends dropbear” to remove dropbear.
    7. Then run “ipkg install openssh-sftp” to install support for the sftp protocol which sshfs uses.

    2007 09 01 NSLU2-Linux
    OpenMoko     Comments Off on Replacing dropbear with openssh Permalink