FuguTabetai.com

Amazon cloud drive, photos, rclone, and odrive

Maybe a year ago or so (I forget when it was announced) Amazon introduced a new program for their Cloud Drive service. It was $12 a year for unlimited photo storage. I thought that was ridiculously cheap, so I signed up.

I had some complaints about the service, but nothing major. Mainly, there wasn't a native way to save my iPhoto library to the cloud. The service is only unlimited storage for photos, so you can't backup metadata, just your pictures. The web client is the only way to do uploads, and that would sometimes drop a file here or there. That meant I couldn't upload my entire library in one shot - I wouldn't know which photos were dropped. Also, the files are stored based on file name, which doesn't work well for me. In my iPhoto library I have lots of files with the same name from different camera. That doesn't matter in iPhoto, since it uses a file hierarchy and the names don't conflict, but Amazon Cloud Drive puts all photos in the root of the drive, so you can overwrite older files with the same name with newer files.

I eventually exported my iPhoto library to files with album names or dates or something in the filename, and slowly, over a few weeks, uploaded all the files. For $12 I was very happy with that.

Then I got the iPhone app, and photos were uploaded automatically. That is great!

A few weeks ago though, I got an email from Amazon saying that they are discontinuing the service. I wasn't surprised. It was a great deal for the price. They offered a chance to change to unlimited storage for $60 a year. That is also a great deal. So I switched to that.

I didn't do much for a while though, because I didn't have a good way to get files into Amazon Cloud Drive.

But then I found RClone, which bills itself as Rsync for the cloud. And it basically works like an rsync. Since I switched to unlimited storage, that can support directories. rclone worked great for me, I was able to back up a lot of personal data that should be offsite, and also do a real back-up of my iPhoto library. Nice. So I highly recommend the combination of rclone and Amazon Cloud Drive.

One thing that I would really like is the convenience of Dropbox to be able to keep files in sync. Amazon Cloud Drive is really just a big place to put files, but it doesn't have a way to do sync across files. There is a solution that can help with that though: odrive. odrive is a service that lets you add cloud storage providers (Amazon, Google, Dropbox, and others) and then it will manage those files for you. It allows for sync on top of the services that it supports. So by installing the odrive client on my machines, I now have an easy way to sync files between those machines. Another great thing about odrive is that it doesn't download everything. I have 172GB in my Amazon Cloud Drive right now, and I don't have that much space on my machines. At any given time I only want to work with a small set of files. odrive will create placeholder files and only downloads things as you need them. So I can set up one folder that I am currently working with as actively synced, and the rest of the cloud drive doesn't take any space on my machine. It is very nice.

The free client doesn't automatically keep a folder in sync, so while placeholder files will show up, I might have to click on them to download the files, but that isn't a big problem with my workflow. The free client also might only support one or two accounts, but since I only need one account, that is fine by me.

I still have some complaints about Amazon Cloud Drive - mostly that the pictures are dumped into the root folder, so I have maybe a hundred thousand files in there, but otherwise I'm very happy with the combination of Amazon Cloud Drive, rclone, and odrive working together. Check it out!

Packaging up a Java application on OSX with recent versions of Java

So, I've started hacking on a very, very old project of mine that is a Swing-based Java application. I got it running again on OSX but things have changed a lot since I last was doing much coding on it. It used to be that Apple vended Java for OSX, but now Oracle does, and the way that you package up a Java application as an OSX application has changed.

Oracle has some documentation on how to package up an application, but I have to admit that it took me a lot longer than it should have to get this to work. First, Oracle distributes an Ant task for packaging the JAR. I don't use Ant for my project. Actually, I don't remember how I was packaging up the application before. I think I just hand-crafted some directories and dropped an Info.plist file in there that worked.

I was a bit intimidated by that, but it does look like someone has done things by hand. This also looked pretty complicated.

So I ended up installing ant. I had previously installed homebrew on my machine, so that was as simple as "brew install ant". Then I needed to put together a build.xml. I know a bit about that, but not much. I use eclipse for this project, but haven't done anything smart with how it is set up. It just uses the default java builder. I think I even have it set up to put the compiled code in the existing directory structure with the Java files. Not really great. Anyway, I made a simple build.xml that pulls in the libraries I need and some other resources. When I tried to run the resulting application though, I get a failure:

LSOpenURLsWithRole() failed with error -10810

That was not helpful. Running from the command line didn't help, just output that error. Running via java -jar did help though: it couldn't find some classes it needed. Oh, right, I need to set the classpath in the Manifest file. So if you get that error, check to see if your classpath is set correctly. Remember that when you run something via -jar, the -classpath option is completely ignored, and it takes the classpath from the JAR's manifest file. Here is what my final build.xml looked like:

<project name="GMAO" default="bundle-GMAO" basedir=".">        
    <taskdef name="bundleapp"
             classname="com.oracle.appbundler.AppBundlerTask"   
             classpath="../GMAO_libs/appbundler-1.0.jar" />
    <!-- See the lib reference here, this is why you need to use the lib directory! -->

	<path id="build.classpath">
	  <fileset dir="${basedir}">
	     <include name="lib/*.jar"/>
	  </fileset>
	</path>

	<pathconvert property="manifest.classpath" pathsep=" ">
	  <path refid="build.classpath"/>
	  <mapper>
	    <chainedmapper>
	       <flattenmapper/>
	       <globmapper from="*.jar" to="*.jar"/>
	    </chainedmapper>
	  </mapper>
	</pathconvert>
	
	<target name="create-jar" description="Create GMAO Jar">
		<jar destfile="GMAOGUI.jar" basedir="." includes="**/**.class,../common/**/**.class,images/**,docs/**,*html,*xml,*xsl">
			<manifest>
				<attribute name="Main-Class" value="com.FuguTabetai.GMAO.GMAOGUI"/>
				<attribute name="Class-Path" value="${manifest.classpath}"/>
			</manifest>
		</jar>
	</target>
	
    <target name="bundle-GMAO" depends="create-jar">
        <delete dir="appBundle" failonerror="false"/>
        <mkdir dir="appBundle"/>
    	<echo message="JAVA_HOME is set to = ${java.home}" />
        <bundleapp outputdirectory="appBundle"
            name="GMAO"
            displayname="GMAO"
            identifier="com.FuguTabetai.GMAO.GMAOGUI"
            mainclassname="com.FuguTabetai.GMAO.GMAOGUI"
        	icon="images/GMAOGUI.icns">
        	<runtime dir="${java.home}/.."/>
        	<option value="-Dswing.volatileImageBufferEnabled=false"/>
            <!-- The following is important and should point to your build -->
            <classpath file="GMAOGUI.jar" />
            <!-- You can have multiple instance of classpath if you 3rd party or
                 dependent jars in different locations -->
        	<classpath file="lib/TableLayout.jar" />
        	<classpath file="lib/commons-logging-1.2.jar" />
        	<classpath file="lib/helpgui-1.1.jar" />
        	<classpath file="lib/jcommon-0.7.0.jar" />
        	<classpath file="lib/jfreechart-0.9.3.jar" />
			<classpath file="lib/jnlp.jar" />
    		<classpath file="lib/skinlf.jar" />
    		<classpath file="lib/swingfx.jar" />
    		<classpath file="lib/xnap-commons-0.9.5.jar" />
        	<classpath file="lib/xpp3_min-1.1.3.4.0.jar" />
        	<classpath file="lib/xstream-1.2.jar" />
        	<classpath file="lib/gmao_common.jar" />
        </bundleapp>
    </target>
</project>

The Appbundler documentation was useful in adding some additional properties. I know that the build.xml could be better, and I'll probably improve it, but I wanted to note it here because I know in a few years I will want to figure out why I did this.

Excruciatingly slow Swing performance while loading / painting images on a Macbook Pro Retina 13" under Java 1.8

So, in my previous post I mentioned that I was doing some profiling of a Java Swing application. The application was super slow. An operation that used to take less than a second (basically loading an ImageIcon and painting it) would take about three minutes (I timed it) under Java 1.8.0_31 on OSX 10.9.5. It made my app very slow to use. Once the image was loaded, things were fine, except some other operations (Java2D Transforms) were unusable. That made the app almost unusable under certain conditions.

I spent some time looking around on the internet, and found lots of people complaining about issues that might be related. Since this application is one that I wrote, I could change it if there was a fix out there.

Mostly discussion focused around Oracle's implementation being OpenGL only, and not using the Mac Quartz rendering anymore. That is something that I've known about since Java 1.6 when Apple dropped their own implementation. That is probably the cause of the problem, but there is nothing that I can do about it, aside from downgrading to Java 1.6, which I don't want to do because I want to be a cool guy living in cool times and using Java 1.8 features and maybe even some closures.

One thread hinted that BufferedImage of TYPE_INT_ARGB_PRE instead of TYPE_INT_RGB are significantly faster. It didn't make a measurable (wall clock) difference in my application. I was disappointed.

The actual problem seems to be with a lack of implementing VolatileImage in BufferedImage. It looks like you can use VolatileImage directly and gain back some performance, but further digging indicated that I could set the JVM Property -Dswing.volatileImageBufferEnabled=falseand for whatever reason, all my performance problems went away. That isn't super useful for software that you distribute because seriously, how many users are going to know how to set JVM properties? I guess once I get around to packaging up my software as an application again I'll be able to set that anyway, but I hope that this is fixed in an upcoming Java release.

It is disappointing to me that Swing graphics had such a horrible regression and it has apparently been around for two Java versions on OSX. I really like Swing because I can run my application on OSX, linux, and Windows (and I sometimes do!) but it doesn't seem like it is getting as much love as it used to. Which is strange, because Android is dominant in the mobile space and clearly graphics performance is good enough there for all sorts of applications and games.

Profiling in Java

I've recently started doing some manga translation again, but what that means is that I've tried to get my GMAO program running on my newer MacBook. Just trying to run the OSX app that I packaged a few years back brought up a dialog box asking me to install the 1.6 JRE. I don't really want to do that, we're in the 21st century now, and I want to run Java 1.8. So I downloaded that JRE. Still, I get the message. It turns out that the type of launcher I use (info.plist XML or whatever) is the style that Apple supported which only works with their JRE, which is only available to 1.6. And is more or less deprecated now.

So I found out that if I download the JDK (and just the JRE) I get the command line tools, and then I can pretty easily start up GMAO from the command line from the .app. Which still is far from ideal, so I guess I need to re-package it to use the new launcher format. First though, since I can at least run GMAO on the new laptop, I started to do some translation. Great.

Except, it takes about three minutes (wall clock minutes) to load a new image. That is terrible. It used to take on the order of seconds, if that. So I guess there is something in the APIs that have changed to make whatever I'm doing super slow - I know my code doesn't do anything that should take that long. So I'll have to track that down.

I installed Eclipse and copied my code over. Wow, over 800 warnings. Mostly about non-parameterized types. I should probably fix those. I'm able to compile under Java 1.8 though, so that is good.

Where is all that time going when I load an image? Let's try to profiling. So, I try to use jvisualvm. I can look at the heap and stuff, but CPU profiling is not supported: "failed to create jmx connection to target application". I tried many things, but I just couldn't get things working. I enabled some logging and a net connection was timing out.

RMI uses two ports for communication, which is pretty dumb if you are behind a firewall. Anyway, I don't think that was the problem here, I think it was having trouble finding the host to get the RMI information from.

Finally I found some java options that I could use to get the connection to work. On the JVM running the app I want to profile:

java -Xmx2048m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote.rmi.port=3334 -classpath $CP com.FuguTabetai.GMAO.GMAOGUI

I was able to start up jvisualvm and connect to the JVM that way. The profiling was helpful, but didn't get me as far as I wanted. By the way, if you are on OSX and want to start jvisualvm you can just use jvisualvm, but to run Java Mission Control I had to find out where that binary was:

find /Library/Java -name jmc
/Library/Java/JavaVirtualMachines/jdk1.8.0_31.jdk/Contents/Home/bin/jmc
/Library/Java/JavaVirtualMachines/jdk1.8.0_31.jdk/Contents/Home/lib/missioncontrol/Java Mission Control.app/Contents/MacOS/jmc

Here are some other interesting notes. Using the HPROF option just showed me some garbage, it isn't accurate. See this excellent post for more information about that. This post was super helpful in tracking down my connection problems with JMX. Google might have a nice lightweight java profiler but I haven't read that yet. This stackoverflow post pointed me in the right direction for finding the critical java.rmi.server.hostname parameter (and maybe com.sun.management.jmxremote.rmi.port.) I'll probably need to look more into how to do Java app bundles in this brave new Java 1.8 world.

What is my mail server rejecting?

I've recently migrated my hosting from one VPS to a new host. It was a lot more work than it should have been. I'll try to be much better at keeping my host up to date and maintained.

At any rate, I finally have my mailserver up and running on the new host. I implemented greylisting and some other spam prevention measures. Sometimes I wonder what is being rejected and why. Here is a super simple perl one-liner that can look at the mail.log file and spit out what hosts are rejected, and what the rejection reasons are (independently.)

grep reject mail.log | perl -n -e 'END { foreach my $key (sort {$hosts{$a} <=> $hosts{$b}} keys %hosts) { print "$hosts{$key}\t$key\n"; } foreach my $key ( sort { $reasons{$a} <=> $reasons{$b} } keys %reasons) { print "$reasons{$key}\t$key\n";}} if (m/RCPT from (.*?): .*?: (.*);/) { $hosts{"$1"}++; $reasons{"$2"}++}'

More on Baldur's Gate on a Kindle Fire HD 8.9" with GemRB

I've spent a bit more time playing around with my Kindle Fire and GemRB. Planescape: Torment seems like it needs more work before that will be fully playable, but Baldur's Gate 1 and 2 are supposed to be very well supported. I would like to play through those, so I set them up. I played Baldur's Gate 1 many years back, but I remember very little.

This time through I want to play as a Fighter/Mage and import the character into Baldur's Gate 2. There is a nice mod that adds the Baldur's Gate 1 content to the Baldur's Gate 2 engine, and lets you play through them both. That is called Baldur's Gate Trilogy WeiDU. I installed that. I also followed more or less exactly this post from gog.com to enhance the gameplay which includes some other mods and restores some extra content. This time around I took some screenshot with the game in 1920x1200 resolution with the original fonts, then with some large fonts (overran the content areas for the most part) and finally at 960x600 (my preferred resolution) with the droidserif font. It looks pretty good, but the font is still a bit big, and the "p" is cut off (probably it is too wide for the space allocated to it?) but still it is all very readable.

Unfortunately I have some some problems. I keep finding hard crashes when I explore around the map. They are repeatable, and I have a save game for them so I'll try to get in touch with the GemRB community to see if they can help. My guess is that the BG2 engine with the BGT mod and the additional content plus other mods that I added just hasn't been tested much.

So I think I will just install Baldur's Gate 1, and the suggested mods for that, and then try again.

RPGs on the Kindle Fire HD 8.9"

I've had a Kindle Fire HD 8.9" model for a few months now. I really like it. I am able to read email (personal and work) and set up calendar entries (personal and work.) I can do some light web browsing, and check up on FaceBook with it.

Every once in a while there are deals on Apps. In a recent deal, I picked up the game CHAOS RINGS from the Amazon Appstore.

It is a pretty 3-D game with what look to be pre-rendered (or at least fixed camera position) backgrounds. The story is simple, and very Japanese. Combat is a JRPG style system where the two opposing forces line up and take turns attacking each other. I love turned based combat, so that is a bonus in my book, although I prefer games that blend strategic with tactical combat like the old Gold Box games, or Infinity Engine games. Still, this game is very easy to pick up for a few minutes, grind out a few battles, and put back down. They have a few puzzle stages that have all been variations on a theme (sliding block puzzles, teleporting block puzzles, things like that) but they have all been fairly easy. The maps on the overwold have so far been very simple, there is an automap, and zero need to take notes.

The economy seems pretty poorly balanced to me. The only thing I think you need gold for is to buy keys to open locked chests, and at this point I have more gold than I know what to do with. You can buy weapons and armor (as well as "gems" and items) but for everything except the item, there is a linear progression in power and zero choice involved. I either find something better than what is for sale in the dungeons, or I buy the best that is available (it hardly makes a dent in my amassed wealth) and that is it. The items can be somewhat useful, but so far the battles haven't been well tuned either and I think that giving up a turn to apply a buff just isn't worth it.

So I'm still going through this RPG, even though the story seems a bit simplistic, contrived, and the characters are annoying. It is a fun diversion.

GemRB and Infinity Engine Games

The other thing I came across recently is GemRB. GemRB is a "Game Engine Made for pre-Rendered Backgrounds" that is an implementation of the Infinity Engine that powered the BioWare and Black Isle Studios RPGs (Baldur's Gate, Planescape: Torment, Icewind Dale, etc.)

I had known about GemRB for a while, but since I heard that Planescape: Torment wasn't really playable in the current build, didn't go to too much effort to try it out. The other day though, Good Old Games had a D&D bundle for a ridiculous price which included six or seven infinity engine games for like $20. So I bought it, and then started to play around with getting it to run on my Kindle Fire HD 8.9".

First off, I basically followed this forum thread on how to set up GemRB for Android. I ran into a few problems, but not too many.

First, you have to install the game on windows. That went well. Then, you have to decide what resolution you want to play at. There are mods that modify the game engine and data files to allow resolutions that were not supported at the time the game was written. In my case, I thought I would try out 1920x1200, which is the native panel resolution on the Kindle Fire HD 8.9".

That didn't work out so well; I was able to play the game, but the controls were so tiny that I had a real problem reading text and hitting controls.

I then re-sized the game using the tweaks and fix packs and the widescreen mod. There is an option to install for GemRB, which I did. I changed the resolution to half the native panel resolution, so 960x600. The controls are still a bit too small, but they are at least mostly reliable. Text is still smaller than I would like, and a bit hard to read due to the raster fonts being scaled.

It turns out that GemRB supports TTF fonts, and people have been able to get that to work on the android version so I'm going to look at that next.

I wanted to find out what fonts are already on my system (so maybe I can just set the fontpath to that), so I started up an ADB shell session. It looks like the Kindle Fire HD 8.9" (first generation) fonts are stored in /system/fonts/, and the following are available:

AndroidClock.ttf

AndroidClock_Highlight.ttf

AndroidClock_Solid.ttf

Baskerville-Bold.ttf

Baskerville-BoldItalic.ttf

Baskerville-Italic.ttf

Baskerville.ttf

Caecilia-Bold.ttf

Caecilia-BoldItalic.ttf

Caecilia-Italic.ttf

Caecilia.ttf

Clockopia.ttf

Code2000.ttf

DroidNaskh-Regular.ttf

DroidSans-Bold.ttf

DroidSans.ttf

DroidSansArmenian.ttf

DroidSansEthiopic-Regular.ttf

DroidSansFallback.ttf

DroidSansGeorgian.ttf

DroidSansHebrew-Bold.ttf

DroidSansHebrew-Regular.ttf

DroidSansMono.ttf

DroidSansThai.ttf

DroidSerif-Bold.ttf

DroidSerif-BoldItalic.ttf

DroidSerif-Italic.ttf

DroidSerif-Regular.ttf

Georgia-Bold.ttf

Georgia-BoldItalic.ttf

Georgia-Italic.ttf

Georgia.ttf

HYGothicMedium.ttf

HYMyeongJoMedium.ttf

Helvetica-Bold.ttf

Helvetica-BoldItalic.ttf

Helvetica-Italic.ttf

Helvetica-Light.ttf

Helvetica-Medium.ttf

Helvetica.ttf

Lohit-Bengali.ttf

Lohit-Devanagari.ttf

Lohit-Tamil.ttf

LucidaConsole.ttf

LucidaSansWGL-Bold.ttf

LucidaSansWGL-BoldItalic.ttf

LucidaSansWGL-Italic.ttf

LucidaSansWGL.ttf

MHeiM-Big5HKSCS_E.ttf

MTChineseSurrogates.ttf

MYingHeiSMedium.ttf

Palatino-Bold.ttf

Palatino-BoldItalic.ttf

Palatino-Italic.ttf

Palatino.ttf

Roboto-Bold.ttf

Roboto-BoldItalic.ttf

Roboto-Italic.ttf

Roboto-Regular.ttf

STBShusongRegular.ttf

STKaiTi.ttf

TBGothicBold_213.ttf

TBGothicMed_213.ttf

TBMinchoBold_213.ttf

TBMinchoMedium_213.ttf

Trebuc-Bold.ttf

Trebuc-BoldItalic.ttf

Trebuc-Italic.ttf

Trebuc.ttf

Verdana-Bold.ttf

Verdana-BoldItalic.ttf

Verdana-Italic.ttf

Verdana.ttf

I think I'll try Droid Serif Regular.

In the GemRB.cfg file, which is /sdcaard/Android/data/net.sourceforge.gemrb/files/GemRB.cfg, I had to edit it and add in

CustomFontPath=/system/fonts/

The thread I linked to says to edit the override/pst/fonts.2da file, but that file did not exist. Knowing how these things sometimes work, I copied the file /sdcard/Android/data/net.sourceforge.gemrb/files/unhardcoded/pst/fonts.2da over to /sdcaard/Android/data/net.sourceforge.gemrb/files/override/pst/fonts.2da and edited that to include the DroidSans-Regular file.

Unfortunately, that didn't work. My fonts.2da file was seemingly deleted on program launch? I came across a message suggesting that the GemRB binary directory might be getting unpacked on launch, but that you could put the fonts.2da file in the game's override directory, so I added it and modified it in the /sdcard/gemrb/pst/override/ directory. That caused the program not to be able to launch, and the log file revealed that it couldn't find the font DroidSerif. It turns out that I had a space like: "DroidSerif -Regular" so that is not going to work. After fixing it, it worked! But a lot of the text was too big at 24 point... And when I tried to talk to someone the game crashed. Setting the size back to 14 for two of the entries seemed to fix it. Not bad. Not great, but not bad.

Jordan Mechner's The Making of Prince of Persia

A long time ago, when I was still a young kid, my dad bought an Apple //e (or maybe a //+, but eventually we ended up with a //e.) That event likely changed my life, and definitely set me on the path that ended up being my passion and career: a software developer. Before I got anywhere near programming though, I spent a lot of time doing all sorts of things on that computer. What I enjoyed most were computer games, and one of the most amazing games I had seen at an early age was Karateka.

Karateka was amazing because it had large characters that were animated very well. They were as close to lifelike as we got back then. It was written by someone called Jordan Mechner, who of course I had no idea who he was. The name stuck in my head though. A few years after Karateka, he came out with a game called Prince of Persia, which much later became a video game franchise and a movie. Back in the past when I was a kid, I played this game. It was a great game. It was hard. I never finished it. I also never actually bought the game, I illegally copied it from someone.

Well, recently Jordan Mechner released the source code to Prince of Persia, the original game! With that announcement, I spent of a bit of time on his website, and found that he also had written a book, the Making of Prince of Persia. I've got a link on the left to the Amazon version, I suggest the Kindle book because it is pretty cheap, like $8. I bought it because I was feeling a bit guilty about not paying for the games I had played. Karateka, which I did finish, and from which I learned one important bit of information: never run straight up to a princess if you have the opportunity to meet a princess. Walk slowly and respectfully towards her. I never did get very far on Prince of Persia - it was hard! - but I did enjoy playing. I think that even without the time limit Prince of Persia would have been hard for me back then. It would probably remain hard for me today, since there is a lot of mapping that you have to do, and I don't know if I have the patience for it now.

I highly recommend watching the GDC Postmortem (linked on the left) of Prince of Persia, and buying the book. It isn't really a technical programming book, it is really more of a diary about a young man trying to find a career for himself. I didn't know that Jordan Mechner wanted to be a screenwriter, but that plays a large role in the book as well. I had a hard time putting the book down, and really enjoyed reading through the book, partly as nostalgia, and partly as an interesting look at a young man trying to find his way in life.

In fact, I've been really struck recently by how young people really can make a large difference and make a big impact. The most important thing is not experience (that helps, but can also just prevent you from doing something because you know how hard it can be to actually accomplish difficult tasks) but is just the idea of doing something. If you have an idea, don't let people stop you, don't let your idea of what you need to be to accomplish something, just get started and get out there and do it. I'm looking at all these amazing people who accomplish amazing things before the age of 30 - John Lennon, Steve Jobs, Anne Frank, Mozart, Shawn Fanning (well …)

It makes me excited to see what my young son will do when he is young. I'm proud of the things that I accomplished, but I haven't changed the world. What is amazing to me is that you can change the world, even if it is only in the way of creating an amazing game that inspires others.

Eclipse, Subversion, and Moneydance CSV Importers on OSX

I use Moneydance to track my personal finances. Recently Tokyo Mitsubishi UFJ bank decided to drop support for downloading transactions in Microsoft Money format and switched to a CSV format. That is kind of annoying because I like to download my transactions from the bank and jam them into Moneydance. It means I don't have to type them in. But now they are in CSV format, which isn't natively supported

There is a beta CSV import extension for Moneydance though, so I tried that out. It has support for formats from some banks, and then has a "custom reader" format that you can use to set up the fields that your bank sends. That worked out well. The problem is that the imported transactions from the bank were garbled. The text didn't show up as proper Japanese, it was more like goobledegook.

That is a clear indication that the encoding on the file was set wrong. I know the files my bank sends are in Shift-JIS, but I changed the format to UTF-8 in Emacs since UTF-8 is the format that all text files should be saved in. Unfortunately, the extension does not import as UTF-8 and since it does not give an option for selecting the encoding I assume that they just take the platform default or something.

A quick check of the code shows that indeed, they use a Java FileReader (who thought it was a good idea to have the filereader open text files in the platform default encoding? It is never a good idea to assume a default and not allow programmers a way to change it.) So to fix it, you simply need to use a FileInputStream and an InputStreamReader with the encoding set. No problem.

It looks like Google has a SVN repository for the project. I figure I'll open up Eclipse and use that to make the local change and import my text file. Oh, my version of Eclipse is really old. So let's download a new version. That was pretty painless.

Now, how do I hook Eclipse up with Subversion? It looks like there are two solutions, the more popular of which is Subclicpse. I was able to install that without any trouble. I was a bit worried about the version of Subversion that my mac came with (1.6.17 according to svn –version) and a mismatch there, but let's give it a go.

Wait, they don't have any information telling me how to use Subversion. Well, some British guy wrote something up that makes it pretty clear. Following along with him, I got up to where Subclipse would try to pull down the information and … Some sort of error about a JavaHL library! What? Well, that seems to be pretty well explained on the Subclipse website, so I downloaded JavaHL (the Java subversion bindings apparently) and tried again. Now I get a version mismatch! It says "Incompatible JavaHL library loaded. 1.7.x or later required." But there were no versions listed on the download page! And by the way, that download from open.collab.net does not list any different versions, only different operating systems! I downloaded the correct version for my version of OSX (I still haven't downloaded to the latest version) but that did not work! What do I do now? The documentation says that it installs svn itself (it did, in /opt/subversion/bin) but that version is the same as the one that came with OSX by default anyway. So why would there be a version mis-match? Also, they made me register to download their package, which was kind of annoying, but forgivable. As long as they understand that some guy named Fugu Tabetai working as a consultant for Fish Eating Incorporated that makes less that $25 million a year probably isn't a good business opportunity for them.

A little searching shows that other people have had this problem as well and the solution is to install a newer version of Subversion than the one that open.collab.net installs from wandisco, whatever that is. Apparently wandisco is a subversion provider of some kind. In my case, since I am on 10.6 I needed their 10.6.x package from their subversion download page. After downloading that though, it still seems like I have the same old version of subversion and not some sneaky new 1.7.5 version.

Guess what? Rebooting did not solve the problem either. But doing a sudo rm -Rf /opt/subversion and re-installing the wandisco problem did get me the proper version (after appending /opt/subversion/bin to my PATH) in terminal. So maybe now it will work in Eclipse? Let's give that a go!

Hey what do you know? It worked! I was able to suck down the code for the CSV importer. Of course, then I realized another problem: since the importer is an extension for Moneydance, I now need to find the Moneydance library files (JARs) that is links against so I can actually compile the thing. Then I have to figure how to compile and package it in the extension format that it needs after I make my changes. Luckily, all of that is available at the Moneydance developer page.

Adding the two JARs in the developer kit did get the project to almost completely compile, except for six errors that popped up. It looks to me like I'm linking against old JARs, because the documentation says that the errors Eclipse is complaining about are not true (e.g., this class doesn't implement some method, but it shows in the API docs that it does.) So perhaps I downloaded an old version of the developer's kit somehow. I don't see how that is possible though; I downloaded the only one that is up there. Well, I've posted in the group for the csv importer and will see if that is actually something that they can help me with. If not, I'll email the Moneydance folks. This should be a simple problem that I can actually fix, and it would improve my life, so I really want to get this done.

After a day's delay, I heard back from the maintainer of the CSV import plugin and he passed me different versions of the Moneydance development libraries. They were created in 2011 sometime vs 2006 which are the ones that came from the Moneydance website. The project compile successfully with them, so now I have no excuse not to actually add this feature.

The GUI was created in Netbeans' Matisse which is supposed to be a nice automated GUI creation system. It generates regular old Java code though, so I might as well edit that directly in Eclipse myself and then worry about what to do on the commit later.

I need to modify ImportDialog.java's creation of a CSVReader and also the one in TransactionReader.java. I did all that, and now I am caught up on the failing unit tests (nothing that I did - these were pre-existing failures) from what looks like a test that can't compile anyway. After making those modifications and building on my machine (generating my own keys to sign the plugin and all) things looked great, and the import went fine. I've sent the changes on to the maintainer, so maybe we'll see this show up at some point!

Cocoa Programming

I've been doing some Cocoa Programming for Mac OSX lately. I've been using Aaron Hillegass' "Cocoa Programming For Mac OSX" and have really had an easy time following along with the book. Before ordering the book, I tried to dive into OSX programming using online material, and while I was able to get some things done, I was having a real hard time trying to wrap my head around general concepts (how does the general cocoa interaction model work? how do I try to understand when to hook things up with connections in interface builder and when to code things? What things are done for me automatically?) and found that this book has been good at explaining these things.

The book itself is pretty easy to follow and has extensive screenshots, not normally something that I would think is important in a programming book, but for XCode and Interface Builder this is surprisingly helpful. Oh, I have to drag from this to that, or that is the pane in the inspector I need to be looking at. A surprsingly large amount of "programming" so far has involved knowing what key to type in which box in interface builder to get some control to watch some value in some object. And knowing which objects that Apple already provides has keys that can be watched to do what you want.

I'm coding on snow leopard (10.6) and the book was written for 10.5 so there are some disconnects there, but so far I haven't run into any errors with the code that I can't figure out quickly. The biggest so far has been an example in Chapter 11 which looks like it is a bit hairy to fix. The amount of commitment I need to make as a student to something that is easy (binding in Interface Builder) compared to writing a (granted, very small!) custom class and using it to basically just type cast is pretty big. It seems like a simple type cast operation could also be accommodated in the bindings / core data paradigm (and it is, using transformations, and a small custom class) but at any rate I have really been enjoying this book.

It makes it much easier to get a handle on where to start, and since I have programming experience (C / C++, Java user interfaces, and perl mostly) I have been able to get through about a chapter per session while doing the examples and challenges.

One of the controversial things seems to have been the introduction of "dot notation" in Objective-C 2.0. This post on Big Nerd Ranch explains pretty well how I feel about it. It is confusing to me. I like the @property and @synthesize tags added to the language a lot; being able to use dot notation sounds convenient, but only if the class is accessing properties. Otherwise it can be confusing. Is that a method call? Is it doing anything tricky with memory? Do I have to worry about it and check the setter / getter selector? I'm going to stick with bracketed notation until I get a better handle on these things.

I also like that Objective-C 2.0 introduced garbage collection, but I also want to do explicit memory management with release / retain for a while.

I've been really impressed with how easy it is to create user interfaces in XCode / Interface Builder. All I have to compare with is Java's Swing platform, and I never used any GUI interface builders with that. I know things are getting better in that area with NetBean's Matisse and some of the Eclipse plugins, but every time I have looked at those they don't seem to save you from writing much of the boilerplate code that you need to navigate user interfaces. With XCode and Interface Builder you really don't have to write much code and things are saved out as data.

I am sure that similar systems exist for Java out there by now, but I don't know where I would start with them and they are certainly external to stock Java vended from Sun (Oracle?)

The other thing that has really impressed me is Core Data. Apple makes it really easy to do the common things, and not hard to difficult things. In my experience with Java, it is as hard to do easy things as hard things (they are both surprisingly difficult.)

I've got a project in mind for Cocoa on OSX, which will probably take a few months (I do not have much free time!) but once I finish up with that I am really interested in looking at iPod / iPad development. It's too bad that there are major differences in the APIs available to the two platforms, but that might be the next thing I look into. I guess I would have to get an iPod Touch or iPad at that point though. :-)

Perl on Mac OSX does not like ~

This caused me to lose about thirty minutes of my precious "not working, not sleeping" time (currently at about two hours a day.)

I was trying to run some file test operations on some files in OSX using Perl (one of the newer ones, 5.10 maybe) that have spaces in their names. In general, I absolute hate how processing files with spaces is so difficult. One thing I like to do is change the IFS (internal file separator) in bash while doing things with files. This blog post shows a really nice example of doing that.

Anyway, I was doing something basically like:

if (-e $file && -f $file) { ... }

for $file where $file was something like "~/tmpLibrary/covers/Sanderson, Brandon - Mistborn.jpg". It was failing all the time. I was also getting mystifying "failed trying to stat file with newline in it" error messages when I knew the file name had no newlines. I still don't know what that was about. Unfortunately, for files that I knew existed and were regular, the above test was failing.

Why?

After much experimentation, lots of googling (there was nothing on this) I figured it out. And remembered that I knew this was a problem at some point, and must have forgotten. Perl barf on "~/tmpLibrary/bar.jpg". It will work fine with absolute file paths: "/Users/devans/tmpLibrary/bar.jpg". Oh. That is strange, but I guess I can understand that. There isn't any real reason that Perl should expand the path with the user home location. I wouldn't be surprised if it, and would be (was!) surprised when it didn't, but whatever.

So, remember that kids. Don't count on home directory expansion. Not that I could find anything about that (or spaces in filenames, which work just fine) for perl file operator tests, but there you go.

And I spend my free time doing stuff like this for fun. I am a sad, sad man.

Stripping DRM from Ebooks

I found a good post on how to remove DRM from ebooks at http://www.makeuseof.com/tag/how-to-strip-mobi-and-prc-ebooks-of-encryption/. They have a link to some python scripts that can remove DRM from some forms of ebooks as long as you know the PID for the book that you bought.

This morning I purchased Brandon Sanderson's "The Well of Ascension", the second book in the Mistborn series. I found the first book during Tor.com's launch ebook giveaway, (you can get it too!) and really enjoyed the book. I wanted to read the rest. (Apparently, I might not be the only one.) So I checked online, and I could get the second book for about $14 from Fictionwise. Sounds great. The only problem is that the books they sell there have DRM, Digital Restrictions Management. I am not able to read books that are encrypted with DRM on my preferred ebook reading platform: FBReader on my OLPC with Ubuntu installed on it. So I decided to try to remove the DRM. That would restore my rights as the owner of the book to archive it, so that I can read it in a month, six months, five years, or twenty years. As long as I ensure that I have the regular unencrypted file and software to read it, I should be fine.

If I did nothing about the DRM I would only be able to read the book on the computer that I used to download it. A 15" notebook. It isn't really all that portable.

I was able to strip the DRM as outlined in the link above, but the resulting mobipocket file came up empty when I tried to load it on FBReader. Bummer. So I tried another approach. I took the unencrypted mobipocket file, and loaded it up into the OSX Stanza ebook reading software. Then I saved it again as an ePub file, a more open format. That did open ok in FBReader, and now I can read the book that I purchased on any hardware that I like.

I am a bit disappointedthat I needed to pay $14 for the book. I would have preferred $7 or so since I do not get a physical copy, but ebooks are actually more convenient for me. On Amazon.com the book is actually $7.99 for a new, physical copy (or the Kindle copy, which I am not able to buy, but could use if I could after stripping the DRM) that includes lots of costs for printing, shipping to warehouses, distribution, whatever. Ebooks are a lot simpler when it comes to distribution: you ship them over the internet, with perhaps some up-front computation to encrypt the book using some sort of DRM scheme. Costs would be lower without the DRM. Customers would be happier because things are easier to use. People who want to buy books probably are not the people that are going to go and upload the files to the internet. People who just want to get the book for free can already do that. I can't see how DRM is really helping the industry, but that is the standard for books right now.

Thankfully, it is now possible to get non-DRM'd music files, from Amazon or Apple's iTunes store (but you need to make sure the stuff is iTunes plus still I think?) Hopefully video will go the same way.

I would really like to get a Kindle but I won't do that until I can get one that works in Japan. Until then I will make do with what I have. Even once I get a Kindle though, I would like to make sure that my books do not have DRM on them so that I have control of my files, and what I can do with them is not dictated by a third party (regardless of whether or not I think that the system is reasonable enough, and non-intrusive enough to use.)

BTW, you can use the MobiDeDRM if you get the Kindle PID (type '411' from the Setting menu, according to this blog post.)

I hate Fujitsu (AKA upgrading the hard drive on an artificially limited Fujitsu Machine)

My mother-in-law's laptop is "broken". The laptop is a Fujitsu FMV-Biblo MG50S, if you check the page there you will see there are a few others with similar specs. I took a look at it last week. A quick check of the hard drive (first on list of things to check because "I can't put any more pictures on it") and that is the problem: 30GB on a 30GB drive. Nice.

Summary: I hate you Fujitsu. Marketing droids added a hidden something somewhere (looks like it was hiding on the MBR) that made Windows see only 30GB of an 80GB drive, and when I did a clone to a new hard drive (160GB) it showed up as 30GB. So to be clear: when cloning a drive using CloneZilla, PNG, or Acronis True Image and the like, if you copied the hidden MBR your new 160GB drive would show up in Windows as only 30GB, even though the Disk Utilities management program would show the full disk size.

Absolutely crazy. Click the "Read More" link to read more about this insanity. For the impatient: do not copy the hidden MBR and you will be fine. Also, I now prefer Clonezilla to PNG. And the GParted boot CD rocks. Also, I hate Fujitsu. If you need to buy a new laptop you really should only consider Apple (coupled with a TimeCapsule) or ThinkPads. For netbooks, do whatever you want but back them up somehow.

read more (1450 words)

Emacs longlines-mode

I have been a fan of auto-fill-mode (and flyspell-mode for that matter) for a long time. Unfortunately, when I think about it I much prefer free-flowing text. The problem with auto-fill-mode is that it will throw \n characters into your text file when you need to wrap lines. Most email clients will automatically wrap text, and will do it at the size that is best for the user. Or at least, that is the theory.

Anyway, I just discovered longlines-mode. This mode in Emacs will wrap long lines (hence the name) but does not insert \n characters into your text file. Also, when you copy text, it copies as long unbroken lines. So this is really nice. I wonder why it took me so long to find this mode?

It makes me wonder what other marvelous mysteries emacs is hiding from me.

Using NTT DoCoMo's P906i as a tethered bluetooth modem for internet access with Mac OSX 10.5.7

Shamus Young's PixelCity Screensaver

I'm a big fan of Shamus Young's Twenty Sided Tale blog. It is very well written, generally very interesting, has geeky computer stuff, fun game-related stuff, DRM related ranting, and generally is just a very interesting blog. I don't really know how Shamus does it; it looks like he is married and has N kids where there is an N that exists such that N >= 1, writes funny stuff for his blog, and has very high quality posts overall. I've been trying to make my blog a bit better by emulating some of Shamus' posts: write clearly, use pictures whenever possible, and try to pick interesting topics.

I generally am only able to use pictures, and only then infrequently. But I'm trying. For some really interesting and funny stuff, see the DM of the Rings, an exploration of what Lord of the Rings would look like if it was a Dungeons & Dragons campaign. Very tongue-in-cheek humor.

What I'm interested in publicizing this time though is a very interesting programming project that Shamus took on in the completely unflattering light of the public. Programming is a lot of fun, but you certainly have to be of a certain mindset to enjoy it. In a way, programming is like building a city from the ground up: you have to start out planning the zoning, the roads, the infrastructure, utilities, then the kinds of buildings you need, how to build them, how to manage them, etc. It is even more like building a city from the ground up when you are writing a program about building a city from the ground up, which is exactly what Shamus did. His series on PixelCity starts out with a simple goal: automatically build a city completely algorithmically. Wait, actually that isn't really such a simple goal.

Anyway, start with the link above and read through the whole series. By the time you get to the end you should have a good understanding (from a high level) of what programming projects are like. As a reward, you can download a Windows Screensaver that builds a city for you and then flies you around it. It is seriously cool. He has released the source code to the community, so I expect that a Mac and Linux version will show up soon. If not, that is something that I would be interested in looking into, but until I get my wedding planned and done, I don't have much time for those kinds of projects. I'm lucky to find the time to put out a half-coherent blog post every once in a while.

So, if you are interested in what it looks like, check out this very nice video Shamus posted on YouTube:

Setting up an AFP (Apple Filesharing Protocol) on Ubuntu and a Firefly iTunes Media Server

One of the things I've been meaning to do for a while is set up my Ubuntu machine to share out the music I have on it. I run Amarok on the machine and love it, but that doesn't help when I'm super lazy and don't want to reach over for the linux machine keyboard when I have a perfectly good laptop in my lap*. (* Of course, I do have a VNC server set up on the machine so I could VNC in and start up Amarok that way, but it somehow feels like cheating.)

First step in getting the machine to share out music: set up an AFP server do the other machines in the house (mostly Macs) can see it. That was a lot easier than I expected: just follow the instructions on this post. Great! That seemed to work well. I think. I already had samba up and running on the machine and I am guessing that is what is currently showing up in the Finder. I'll check it out on R.'s machine when I can pry her away from it. The one thing that I did do was to change ATALK_MAC_CHARSET to 'MAC_JAPANESE' and ATALK_UNIX_CHARSET to 'UTF8'. It was pointed out over on this Japanese blog entry that that would be a good idea. I also set up a share for my data folder. I was impressed that this went so smoothly because you need to compile the service from source in order to enable encrypted passwords on the server. It went really smoothly though.

Once you have AFP set up, you need to set up Avahi to broadcast the server. This guide is a really nice explanation of how to set up Avahi. So once that is done, you can move on to the next step in the process: and set up Firefly on the system. That setup was also really smooth, with the exception that for some reason, if you change the default password the service does not seem to work. I have no idea why that would be the case, but do have a vague memory of the same thing happening a few months ago when I set it up. Annoying, but not such a big deal. Once I hit up the webpage for the service, set up the proper directory for the music, and did the scan, the share showed up in iTunes just fine. Nice.

iPhoto 09's Faces Feature

I recently bought iLife 09 and have been using the Faces feature in iPhoto a lot. The Faces feature will look through all your pictures and identify people's faces. Then you can put a name to the face, and gradually iPhoto learns to spot pictures of that person. It is an amazing feature. Facial recognition has been a promised feature from Artificial Intelligence since the 80s, and this (along with Picasa's Name Tags) is the first really commercial product that I have seen facial recognition in. Since I'm a computer scientist by trade, I'm well aware of how these kinds of things work beneath the covers, and while there has been some press coverage saying that this feature isn't ready for mass release, I disagree. I think it does a good job, has a great interface, and more than that, is really fun to use. I find that once I put a name to a face, I want to go through and see what other pictures I can find that the person is in. I also have been adding metadata to my pictures in some way (going back to file name for really old pictures) that show who is in a picture, so it is interesting to see how well iPhoto compares to my tags.

Once you have added a bunch of names to faces, iPhoto has a nice cork-board of faces that you can click on to find all pictures they are in, or have iPhoto show you more pictures that it thinks they might be in. Adding names to faces is very easy. When you see a picture, you hit the "Name" button and then get a picture like the one to the left. People who have a known name have their name below their face, and for people whose name isn't known you see either "unnamed" or iPhoto asking "Is this X?" where X is someone you have already named. It is really impressive. iPhoto does a very good job of noticing faces - it doesn't always notice all faces, but it gets most of them - and it does a good job of suggesting names when it thinks it might know who someone is.

Once you have named a few people and they show up on the list of faces, you can easily scan through lots of pictures and confirm or deny iPhoto's guesses. You can go through and quickly click once to confirm a face, or twice to deny it. You can also name a guess as someone else with a control-click if it threw someone else into the list of guesses. It makes it really easy to go through and check all the pictures iPhoto thinks someone might be in. Then once you have done a round of confirmation and rejection it will re-search the database for more pictures. Getting it to get down to zero suggestions for each person is addictive. Of course, even if there are no more suggestions, that does not mean that all pictures of that person have been found. If you are going through yourself and find some untagged pictures, you can tag them and then iPhoto will do more guessing based on the new data point. For obsessive compulsive people you might stuck in a loop where you want to go through every picture and make sure that everyone is named. That might not be a good thing...

iPhoto is also wrong sometimes, and guess that strange things might be faces. In the middle picture on the right, names have been blurred to protect the innocent and stomach-face-bearing people. Also, iPhoto doesn't know when a face is a human, or whether it is just a face-like object to sometimes hilarious results.

Things that disappoint me about this feature in iPhoto: there should be better intergration with the Address Book. After a software update, iPhoto will suggest people from your address book, which is great. It does this based mostly on email addresses to keep track of people, so you can add someone's email address from address book and you won't get duplicate name suggestions. I wish that in address book though that it would know about the faces in iPhoto and you could get the faces gallery on the Address Book entry. As it is now, you have to go into iPhoto yourself, pic a picture, then drag it over to Address Book. You also lose the nice face cropping that iPhoto does for you. That's too bad. I hope that in future updates that put that kind of functionality into Address Book.

Another issue is that iPhoto 09 adds Flickr and Facebook support, but at least in Flickr it doesn't look like they send the extra faces meta-data. They should make a Flickr "note" around the face with the name, or at least put a list of the recognized people into the photo. As it is now I feel like I have to go in and manually type the names into the description field, which is exactly what I was trying to automate out of the picutre. (uh, no pun intended.) I haven't checked if they do that for Facebook yet. Also the upload interface isn't as nice as what I have been using, Connected Flow's Flickr Export. That has a lot more options and gives you a much better update on upload progress. I'll probably keep using it since I prefer it - you can create new sets with a description and it just generally is more powerful.

Otherwise I am really happy with Faces. It is a fun way to spend time going through your old pictures. I'm afraid that now I'm addicted to trying to name everybody that has ever been in one of my digital photos...

OSX Password generator script

http://www.codepoetry.net/products/passwordassistant

Found a nice script to open up the OSX password generator window. Might come in handy if you need to generate a bunch of passwords. I'm still looking for a good replacement password safe for OSX / linux / windows (preferably one that works on all three.) I have been using Keyring (an open source Palm application) for ages, and still have my Treo 600 as basically a portable password store (but I don't have it with me at right this moment, so this script will come in handy.)

Emacs, Japanese, Putty, Windows, and text entry

I don't know why but I have had to set this up a few times now. If you try writing Japanese via Putty into Emacs, and things do not work for you (instead it looks like Emacs is interpreting things as control characters in some way) then the following magic incantation might help you:


;; Set up Japanese input and coding systems
(set-language-environment "Japanese")
(set-terminal-coding-system 'utf-8-unix)
(set-default-coding-systems 'utf-8-unix)
(set-buffer-file-coding-system 'utf-8-unix)
(prefer-coding-system 'utf-8-unix)
(set-keyboard-coding-system 'utf-8) ; This is the magic for windows putty Japanese input


The important bit is the set-keyboard-coding-system. The other things are important to some degree, but the keyboard setting is what determines whether emacs will beep at you or put up some pretty Japanese text.

I actually have Putty, Emacs, and Gnu Screen all playing nicely together now, which is great. But this is only true for version 23.x of Emacs, since 22.x doesn't seem to play well with Japanese in a terminal under screen...