Recently AWS started charging for Redshift snapshots. I noticed an increase in my AWS bills and decided to dig into the reason. The cost explorer was quite nice, giving me a summary of spending over the past few months.

Deleting my snapshot was an easy choice, since this data was used for analytics in my Redditor project and it only contained data up to 2016.

Next, I decided to look into whether I could reduce the DynamoDB monthly costs. This was a mystery to me, since AWS reported that the table only used up 14.6 GB. The free tier allowed up to 25 GB.

Yet, every month, I was getting billed for an extra 22 GB used.
After reading the detailed pricing documentation, I found the answer. Amazon explained in their pricing page that, “DynamoDB measures the size of your billable data by adding the raw byte size of the data you upload plus a per-item storage overhead of 100 bytes to account for indexing.” With some simple calculations, I arrived at the same range as my monthly costs:
Item count of 366,867,285 * 100 bytes = 36.6 GB
36.6 – (25 – 14.6) = 25 GB over the free tier limit

While the first free 25 GB was not enough for my use cases, it turned out that AWS allows up to 25 Write Capacity Units (WCUs) and 25 Read Capacity Units (RCUs) of provisioned capacity on the free tier, which is also barely enough for Redditor’s word frequency explorer. I decided to increase the read capacity to 25 RCUs, with each read unit allowing 4 KB of data transfer per second. A typical request to get the counts for word phrases over a period of several years returned about 100 KB of uncompressed data.

A quick calculation shows that a single request for ngram counts already uses up all of the RCUs alloted for a second!
100 KB / (4KB / s) = 1s (the request takes at least a second on DynamoDB)
As shown in the Chrome network performance tab, the requests took 2-3 seconds.

Solution

The solution for small projects is to use a MySQL key-value table where the time series data is stored in a single column.

    --------------------------------------------------------------------------------------------------------------------------------------------+

    | key                                   | series                                                                                            |

    +---------------------------------------+---------------------------------------------------------------------------------------------------+

    | example                               | 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,34,49,52,62,94,116,77,138,126,175,123...............................|

    +---------------------------------------+---------------------------------------------------------------------------------------------------+

This works perfectly for read-only data where the series column does not need to be modified. I used this approach for storing web link frequency counts: https://github.com/yuguang/reddit-comments/tree/master/project. Using some simple Spark code, I filled in the data for missing months as 0 and imported the converted timeseries CSV into MySQL: https://github.com/yuguang/reddit-comments/blob/master/serving_optimization/optimize_timeseries.py. The result is that the response times are now under 150ms!

The manuscript for the journal paper that I co-authored has been accepted for Software: Practice and Experience. The paper is titled DataMill: A Distributed Heterogeneous Infrastructure for Robust Experimentation and describes the second version of DataMill, a performance evaluation infrastructure, that I helped to build.

Today I cleaned up the database for Fiddle Salad and Python Fiddle. Both use the same Django back-end for code storage. While browsing tags, I noticed that often both CamelCase and lowercase spellings were used for tags. Since I was working on a tag suggest feature earlier this week, I decided to convert all tags to lowercase so that tag suggestions would not be redundant. An additional benefit is further normalization of the data. Fortunately, I found a fork of django-taggit, the Django app I used for tagging, that supported enforcing lowercase tags everywhere. Two management commands were already present for normalizing data, mergetags and lowercasetags. django-taggit had two fields for each tag, a name and slug. lowercasetags converted all tag names to their lowercase form. mergetags takes at least two tag slugs and merges all tags into a single destination tag. The result is that all associations are moved to a single tag. While mergetags is suitable for manually resolving redundant data, the number of tags on Fiddle Salad is too large. I wrote an command to automate this process:

Because performance is not a concern for a single-time data processing script, I did not bother to optimize the queries nor run-time. This script would be useful for anyone who wants to normalize tags in the same manner, so it is in a git repository. Finally, I tested the new command on a clone of the production database.

When all went fine, I ran lowercasetags and mergealltags on both Fiddle Salad and Python Fiddle. Now I was really impressed with the results as I clicked through the tags on both sites. The tags on Fiddle Salad were much better organized as they were ordered by popularity. While looking through the tags, I noticed that “test” was among the top. I decided to add ‘test’ to the list of stopwords for django-taggit. These stopwords are removed during save so that they are not associated with new snippets.
Now that the tags are normalized, I am ready to move on and deploy tag suggestions.

I have been using e17 for about a year. I haven’t encountered any bugs, and it’s definitely stable enough for daily usage. On upgrading e17 earlier this week to 0.17.6, I found the monitors to be handled separately. I considered this to be a regression in the software. I installed the upstream development version 0.18.7, since it was in the Portage tree to see if it fixed the problems. Other people had the same multiple monitor or dual-monitor problems with e18. I played around with the settings to no avail. Installing 0.17.5 again by editing the 0.17.6 ebuild didn’t help.
I decided to install the old 0.17.5 ebuild which is no longer in the Pportage tree. Upon close examination, EFL (Enlightenment Foundation Libraries) in the recent ebuilds replaced the separate packages used in the 0.17.5.ebuild.
In the 0.17.5.ebuild:

In the 0.17.6.ebuild:

I decided I had enough of the split screen methodology. One reason people don’t switch software is because they have to change the ways they interact with them. In my case, interacting with each monitor separately would lead to productivity loss while learning the new methodology and in the future. For example, the taskbar in the newer versions only show windows from one monitor. If I use two taskbars, I would have to keep track of where I put each window. User interfaces should get out of the way as much as possible so that users would not have to perform such unproductive chores.
Time to download the old ebuilds and rollback e17.

The general procedure is to download old ebuilds as necessary and remove packages that block other packages from being emerged. I went on to reinstall terminology, e17’s native terminal.

Finally, to prevent future upgrades from undoing the work, mask newer versions of e17 and terminology in /etc/portage/package.mask/e17

When I finished my undergrad degree, I regretted taking CS 467. I spent about 80% of my coursework time on that class. Now that I see the same notation and set theory used in a statistics textbook I am reading, I would say the course was totally worth it. It would have helped if I had taken the assignments lightly, since the assignment questions were always a puzzle to students. The time would have been better spent working through examples in the textbook. Looking back at my elective courses, I am glad I didn’t take a CS elective that was mostly programming, such as CS 349. Classes like CS 360 enhance math skills which are helpful in graduate studies.

We got a new server to be set up to build binary packages for DataMill. There was already RAID set up on it, and Linux automatically takes control of the disks. The first thing to do if you get errors such as unable to open /dev/sdb1: Device or resource busy or error checking /dev/sdc1 status: No such file or directory is to run fdisk to erase all partitions. Then reboot with parameters nodmraid nomdadm from your live CD, for example the System Rescue CD. After the reboot, I stopped the RAID controller and continued on with formatting.

to avoid the error. There are other options for creating a btrfs volume, such as specifying raid levels for data and metadata.

I mounted the btrfs volumes with -o compress=zlib during the install. To keep the compression when files are overwritten, these options must be included in /etc/fstab.

When compiling the kernel, RAID and LVM are not necessary since they are included in btrfs. LVM capabilities, such as growing a volume, are also available.
I used dracut to generate an initramfs that supports btrfs RAID boot with modifications to the configuration file /etc/dracut.conf.d.

I then ran the command dracut --hostonly --force 'initramfs-genkernel-x86_64-3.12.13-gentoo' 3.12.13-gentoo which overwrote the file /boot/initramfs-genkernel-x86_64-3.12.13-gentoo.

Installing a JDK on ARM has several challenges. First, there is no binary icedtea for ARM. Second, building icedtea generates circular build time dependencies.
On my first attempt, I just ran the following commands:

However, it soon met an error:

Later, when I checked on gentoo-packages, it didn’t have an arm ebuild. Oracle has hardfloat and softfloat binary JDKs for ARM, so I went on to install them. They can be used to bootstrap an icedtea build.

You may not have a browser installed on your ARM board, so I suggest just uploading the Oracle tarballs to a server and using wget to download them.
To fix the control panel error when you don’t have a desktop environment installed, you need to remove lines from the ebuild and digest it.

These lines need to be removed:

Once removed, generate the manifest for it.

If you run emerge dev-java/oracle-jdk-bin, it will succeed.

I recently received several SABRE Lite BD-SL-i.MX6 boards for running ARM benchmarks on DataMill. To install Gentoo on it, you need a USB-to-serial cable converter. It came with a 4GB SD card, which is large enough to get you started.
Overview of the installation:

• Reset board and format SD card (optional)
• Compile kernel with btrfs support (optional)
• Copy boot script and kernel
• Extract stage3 and portage snapshot
• Configure the install
• Backup and clone (optional)

Reset board and format SD card

When I first tried to boot images extracted to the SD card on the board, it always got stuck at the U-Boot prompt. It turned out that there were environment variables left over from previous uses for these boards. Here is the way to reset board variables:

After I managed to boot Linux images on it, I went onto formatting the card for a Gentoo install. I used sfdisk so that formatting can be scripted with a file which I saved as mmc_partitions:

I formatted the second partition as btrfs so that compression could be used on the small SD card. You may want to format it as ext3 to save time. If you have problems getting it to boot, check to make sure your btrfs-progs are not newer than the btrfs version the kernel is built to support. sys-fs/btrfs-progs-0.20_rc1 worked for the January 2014 kernel source.

Compile kernel with btrfs support

The git tree for the kernel is available at https://github.com/boundarydevices/linux-imx6/. There are two main branches, one for Android and one for non-Android. Use the most recent one for non-Android.

Make the following changes to the kernel config:

Gentoo requires a devtmpfs filesystem to mount at /dev while selecting btrfs also selects LIBCRC32C. The CRC32c CRC algorithm is used by btrfs for checksums.
Now cross compile the kernel

Copy boot script and kernel

Download 6x_bootscript-20121110 available from a blog post and rename it 6x_bootscript. Get the kernel from one of the i.MX6 builds if you decided to skip building it.

uImage is located in /usr/src/linux/arch/arm/boot if you compiled the kernel.

Extract stage3 and portage snapshot

Download the latest stage3 tarball and extract it

If you get errors trying to run emerge --sync, it is because downloading the portage tree is not optional unlike other guides indicate. Simply download it from your nearest mirror and extract it

Configure the install

If you compiled the kernel with btrfs, edit /mnt/p1/etc/fstab file so that zlib compression is enabled

Set kernel boot argument in /mnt/p1/etc/inittab. Change the console= argument to ttymxc1. eg "$bootargs console=ttymxc1,115200 vmalloc=400M consoleblank=0 rootwait"
The rest of the process is the same as for other ARM boards, such as the Trimslice. The guide for it is available at http://dev.gentoo.org/~armin76/arm/trimslice/install.xml.

Backup and clone

First make the backup in a folder

After unmounting the SD card and testing it on the device, put in a new one and clone it with the following commands

The Funtoo ARM Guide has up to date sections on setting the root password and using swclock. If you decide to use swclock, update the last shutdown time to set the clock with touch /tmp/mnt/p1/lib/rc/cache/shutdowntime after extracting the tarball.

The recent video I made for Python Fiddle didn’t prove to be a hit with on 60 views so far, but the one for the JavaScript IDE has been on the top or second place in referrals for Fiddle Salad. One explanation would be YouTube’s video ranking algorithm takes into account community factors such as subscribed channels and the videos in your channel.
The number of visitors to my site for Wijmo Books is also disappointing but almost to be expected. At least the site was a lot of fun to build and still looks spiff.

One of the bugs still in the Waterloo Course Planner is the handling of prerequisite sentences that end with “* students only.”. The fix I made was rather simple. Though converting existing test cases to unit tests did not help because none of the older grammar rules were changed and therefore the tests were not broken, it did help in the development of new grammar rules. Python unit tests does a nice job of pinpointing the exact place where the expected results differ from running the code.

“* students only.” appearing in a string results in the tokenizer printing out a list of invalid tokens. There are two ways of ignoring strings in PLY:

1.  t_ignore_ in the tokenizer
2. change t_ignore_ to a token and add a new rule that returns an empty string in the parser

I used the second method this time, since the semicolon preceding “* students only.” would also need to be ignored. Semicolons are treated as a signal for an “and” clause otherwise. The new rule looks like the following: