After some analysis, I found that disabling all the feeds in Preware reduced the “hang” from more than 30 seconds to less than a second.

Looking through the output of ‘dbus-util –capture’ showed that the “hang” was during the call to the queryInstallCapacity method of the com.palm.appinstaller service, the routine that the Palm App Catalog application uses to determine whether there is enough space to install a new application from the Palm App Catalog.  Yes, this is the method which is at the heart of the dreaded “Too many applications” errors that people are seeing when they have a number of homebrew applications installed and try to install a new application from the Palm App Catalog.

Watching the output of ‘ps -ef’ during calls to this method (you can call it manually using luna-send) showed that palm was calling “ipkg -o /var list”.  Curious.  Why would you want to know the *complete* list of all *available* applications when determining whether there is room to install one known new application.  I suspect that Palm should be calling “ipkg -o /var list_installed” instead (which just lists the installed applications).  Note that Palm doesn’t use feeds the way that Preware does, so for Palm’s official use of ipkg, list and list_installed would return the same thing in their testing, but list_installed is definitely what they should be using to determine the size of installed applications.

The plot thickens when you use strace (which Palm conveniently includes in the official firmware image) on the running LunaSysMgr process.

It seems that LunaSysMgr calls “ipkg -o /var list” to get a list of installed packages (the probably intend to just get the list of installed packages, but when you have Preware installed and have feed configuration files in /var/etc/ipkg/*.conf, it actually returns a list of all *available packages).

LunaSysMgr then does an execve of “/usr/bin/du -s /var/usr/palm/applications/package” for each package in that list.  (BTW Palm, you seem to have a bug in the logic of that code, cause it’s running du on random garbage strings after the end of the real package list)

Yes, that’s right.  A call to queryInstallCapacity spawns a new program (”du”) once for each package returned by “ipkg -o /var list”.  No wonder the UI hangs for 30 seconds or more!

A single “du -s /var/usr/palm/applications/*” would be a far more efficient way to get exactly the same information, but again, Palm would not see this difference in testing because they do not support the third-party Preware application usage of ipkg feeds.

You can imagine that this behaviour is probably related to the app install limit that many people are experiencing too.  Unfortunately, I’ll have to reduce my /var partition size down from it’s current 2GB size (courtesy of the WebOS Internals Meta-Doctor) to be able to investigate this one.

Now the webos-internals.org developers need to develop a new method of installing homebrew applications so that this bug in Palm’s appInstaller service is not triggered.

In the meantime, the work-around is to go into the Preware Preferences screen, hit the “Feeds” button in the top-right corner, and disable all the feeds while you use the Palm App Catalog application in another card.  No need to exit the Feeds screen, just toggle all the button to “Off”, and then toggle them back to “On” when you’re finished with the App Catalog.

For the solution to this problem, see Update #2, below.

I’ve created a thread on PreCentral where this issue can be discussed.  As I uncover more information, I’ll publish my finding here.

Update #1: I’ve now webOS Doctored my Pre in the name of science, and have tested the limits of installing applications.

If you run “du -s /var/usr/palm/applications/*”, and add up all the numbers in the first column, then as soon as you hit the 62367 1K blocks limit of the addition of the sizes reported by that “du” command and the size of the app you with to install, you will get the dreaded “Sorry, Not Enough Memory” error from the Palm App Catalog application (and any other installer, like fileCoaster or PreLoad, which uses the palm appInstaller API).  It doesn’t matter whether you have 192MB free in your /var partition, it will max out at just under 64MB of application usage.

Update #2: I have now created a Linux Application called “Fair Dinkum App Limit” (org.webosinternals.fairdinkum), which removes both the “hang” and the arbitrary application limit.  You can find it in Preware.  Just install it (no need to even run anything – if it is installed, it’s working), and you’re ready to install more applications than you can poke a stick at …

Fair Dinkum App Limit works by simply putting a couple of wrapper scripts in /usr/local/bin, which returns a size of zero when du is called, and returns the output of “ipkg -o /var list_installed” when “ipkg -o /var list” is called.  In the future, the wrappers will be made much more sophisticated than they are right now to prevent operation outside of the specific cases where they need to fool LunaSysMgr, and to also include a safety buffer so that users do not fill the /var partition.  This is a tactical response to a problem that people using homebrew applications are experiencing.  Hopefully, Palm will provide the long term solution for limits on application installation in a release in the near future.

Notes for Palm, if you happen to read this:

1) We fully appreciate that the usage of the ipkg database in /var for homebrew applications is a choice that the homebrew community has made, and is not supported by Palm.

2) We fully agree that the use of “ipkg -o /var list” instead of “ipkg -o /var list_installed” would work perfectly fine for the way that Palm is officially using the ipkg database in /var/usr/lib/ipkg, but we contend that the “list” and “list_installed” commands have clear intended usage, and the one to find the list of installed applications for checking available disk space should be “list_installed”.

3) We fully realise that the initial version of the FairDinkum scripts are unsafe.  Returning a zero value for du is a temporary solution while we work out how to achieve the same result safely.  The intention is to only return false values when du is being called from LunaSysMgr, and to make sure that a safety buffer is kept so that users do not fill the /var partition.

4) I would be very happy to discuss these issues with anyone at Palm (Chuq and Google both have my email address), and would hope that we can together architect a solution for supporting homebrew application installation which does not require these work-arounds.

5) We have designed these work-arounds in a way which does not collide with Palm OTA Updates, and in a way that we can update them automatically using the Preware installer application, and in a way that we can cause them to self-remove when Palm releases a long term solution.

Update #3:

It seems that there is yet another limit on application installation imposed by LunaSysMgr.  Once the used space on /var crosses around 60%, LunaSysMgr will once again refuse to install applications.

I’m going to need to webOS Doctor my Pre yet again (to reallocate 2GB to /var) to determine whether this limit is a fixed percentage, or a fixed number of blocks.

Update #4:

The limit is 102400 free blocks.  Mystery solved.  That also means the Fair Dinkum App Limit cannot cause your /var to be filled.

Update #5:

Thanks to Carl Thompson, an improved version of Fair Dinkum App Limit which does not alter the operation of ‘du’ has been released.

Usually, a Palm Pre is required to be activated on the Sprint network before it can be used.  This is not possible for a Palm Pre in Australia.

So we need to allow the Pre to be used without activation, and there are a number of information sources showing how this can be done.  There are also some dubious sites that redistribute modified versions of the webOS Doctor (which is a clear violation of copyright law, since it contains proprietary Palm software).  Note that WebOS Internals is always careful to comply with all copyright laws (copyright law is the foundation upon which open source licenses are based).

So we need a way for a Pre owner (who has the right to use the webOS Doctor on their own Pre) to modify the webOS Doctor that is specific to their particular version of the Palm Pre before using it to flash that modified firmware onto their Pre.

That’s where the WebOS Internals “Meta-Doctor” comes into play.

I have created a tool which will download, unpack, patch, and repack a webOS Doctor image, applying a number of transformations along the way to:

1. Bypass the need for activation
2. Enable Palm Profile access
3. Set developer mode on by default
4. Increase the size of the /var partition to 2GB

You can find this tool in the WebOS Internals source code repository at http://git.webos-internals.org/?p=tools/meta-doctor.git

Do not redistribute modified versions of the webOS Doctor created using this tool – it is for end-user use only.

I’ve created a forum thread on PreCentral for discussion about this tool.

24926.609       PUB     call    460             :1.26   com.palm.location       //getCurrentPosition  «string=“{}”, string=“com.sfmpllc.sendmylocation 1058”»

24926.920       PUB     return  460             com.palm.location       com.palm.luna        «string=“{”errorCode”:0,”timestamp”:1.254820510841E12,”latitude”:394.xxxxxx,”longitude”:138.xxxxxx,”horizAccuracy”:150,”heading”:0,”velocity”:0,”altitude”:0,”vertAccuracy”:0}”»

The latitude value should be 34.6 degrees South (or -34.6 degrees).

That would explain why Google Maps isn’t working.

Now I need to work out how to replace the Coordinates java class in /usr/lib/luna/java/location.jar, so that the getLatitude method returns a number between -90 and +90 …

I wonder how many WebOS applications will then barf on a negative latitude value …

The PreCentral thread has more information on other GPS tweaks.

There is just one catch  - since I live in Australia (which uses a proper cellular communications standard), the CDMA Palm Pre that I was able to import from the USA will never work as a phone here (yes, I knew this before I purchased it).  I plan to also purchase an unlocked GSM/UMTS Pre when one becomes available (maybe I’ll buy a German one and then swap the two keyboards).

After founding the WebOS Internals project, and using the Pre Emulator in the development of Preware, it is great to have a real device to get the full Pre experience.

If you want to keep up to date with the activities of the WebOS Internals group, just follow @webosinternals on Twitter.  You can also find a list of articles about WebOS Internals in our Press_Room.  We hang out in the #webos-internals IRC channel on Freenode, and have a webos-internals-announce Google group.

I still use my trusty old Treo 650 as my daily phone, which allows me to not have to worry about reflashing the Pre to try out things, as I don’t keep any real personal data on it.

I guess this also closes the OpenMoko chapter of my open source development activities.  I was involved with OpenMoko from the very start, but always said that an OpenMoko device with a hardware keyboard was my preferred form factor, and the Pre seems to satisfy that personal hardware form factor preference whilst still being open enough on the software side to attract my interest.  I wish those who are continuing the OpenMoko path the best of success.

I’ll document my experiences with the Pre in subsequent posts …