Archive for the 'Fiddle Salad' Category

Merge Tags with django-taggit

Jan 01 2015 Published by under Fiddle Salad,Programming,Python Fiddle

Today I cleaned up the database for Fiddle Salad and Python Fiddle. Both use the same Django back-end for code storage. While browsing tags, I noticed that often both CamelCase and lowercase spellings were used for tags. Since I was working on a tag suggest feature earlier this week, I decided to convert all tags to lowercase so that tag suggestions would not be redundant. An additional benefit is further normalization of the data. Fortunately, I found a fork of django-taggit, the Django app I used for tagging, that supported enforcing lowercase tags everywhere. Two management commands were already present for normalizing data, mergetags and lowercasetags. django-taggit had two fields for each tag, a name and slug. lowercasetags converted all tag names to their lowercase form. mergetags takes at least two tag slugs and merges all tags into a single destination tag. The result is that all associations are moved to a single tag. While mergetags is suitable for manually resolving redundant data, the number of tags on Fiddle Salad is too large. I wrote an command to automate this process:

from django.core.management.base import BaseCommand, CommandError
from taggit.models import Tag, TaggedItem
from django.core.exceptions import ObjectDoesNotExist

class Command(BaseCommand):
    help = 'merges all tags automatically'

    def merge(self, extra_slugs, dest_slug):
        try:
            dest_tag = Tag.objects.get(slug=dest_slug)
        except ObjectDoesNotExist:
            raise CommandError('Destination Tag "%s" does not exist' % dest_slug)

        for slug in extra_slugs:
            try:
                tag = Tag.objects.get(slug=slug)
            except ObjectDoesNotExist:
                raise CommandError('Tag "%s" does not exist' % slug)

            items = TaggedItem.objects.filter(tag=tag)
            count = items.count()
            for i, item in enumerate(items):
                if i % 20 == 0:
                    self.stdout.write('Merging %s %d/%d\n' % (slug, i+1, count))
                obj = item.content_object
                if not obj:
                    return
                obj.tags.remove(tag)
                obj.tags.add(dest_tag)
            tag.delete()

            self.stdout.write('Successfully merged tags into "%s"\n' % dest_slug)


    def handle(self, *args, **options):
        for tag in Tag.objects.all():
            if Tag.objects.filter(name=tag.name).count() > 1:
                tags = Tag.objects.filter(name=tag.name).order_by('id')
                dest = tags[0].slug
                extras = []
                for tag in tags[1::]:
                    extras.append(tag.slug)
                self.merge(extras, dest)

Because performance is not a concern for a single-time data processing script, I did not bother to optimize the queries nor run-time. This script would be useful for anyone who wants to normalize tags in the same manner, so it is in a git repository. Finally, I tested the new command on a clone of the production database.

bash-4.1$ python manage.py lowercasetags
Lowercasing 1/1621
Lowercasing 21/1621
.
.
.
Lowercasing 1621/1621
bash-4.1$ python manage.py mergealltags
Merging jquery_1 1/46
Merging jquery_1 21/46
Merging jquery_1 41/46
Successfully merged tags into "jquery"
Successfully merged tags into "jquery"
Merging stylus_1 1/7
Successfully merged tags into "stylus"
Merging hello_1 1/10
Successfully merged tags into "hello"
Merging test_1 1/147
Merging test_1 21/147
Merging test_1 41/147
Merging test_1 61/147
Merging test_1 81/147
Merging test_1 101/147
Merging test_1 121/147
Merging test_1 141/147
Successfully merged tags into "test"
Merging me_1 1/2
Successfully merged tags into "me"
Merging no_1 1/5
Successfully merged tags into "no"
Merging one_1 1/16
Successfully merged tags into "one"
Merging things_1 1/3
Successfully merged tags into "things"
Merging learning_1 1/4
Successfully merged tags into "learning"
Successfully merged tags into "body"
Merging week-one_1 1/1
Successfully merged tags into "week-one"
Merging studio_1 1/33
Merging studio_1 21/33
Successfully merged tags into "studio"
Merging internet_1 1/36
Merging internet_1 21/36
Successfully merged tags into "internet"
Merging assignment_1 1/6
Successfully merged tags into "assignment"
Merging homework_1 1/6
Successfully merged tags into "homework"
Merging lessons_1 1/1
Successfully merged tags into "lessons"
Merging code_1 1/12
Merging tags_1 1/7
Successfully merged tags into "tags"
Merging two_1 1/6
Successfully merged tags into "two"
Merging salcedo_1 1/3
Successfully merged tags into "salcedo"
Merging page_1 1/15
Successfully merged tags into "page"
Merging music_1 1/4
Successfully merged tags into "music"
Merging table_1 1/7
Successfully merged tags into "table"
Merging band_1 1/9
Merging texas_1 1/1
Successfully merged tags into "texas"
Merging biography_1 1/2
Merging assignment-two_1 1/2
Successfully merged tags into "assignment-two"
Merging website_1 1/9
Merging a_1 1/2
Successfully merged tags into "a"
Merging words_1 1/1
Successfully merged tags into "words"
Merging section_1 1/2
Successfully merged tags into "section"
Merging header_1 1/1
Successfully merged tags into "header"
Merging ui_1 1/4
Successfully merged tags into "ui"
Merging first_1 1/8
Successfully merged tags into "first"
Merging random_1 1/1
Successfully merged tags into "random"
Merging internet-studio_1 1/5
Successfully merged tags into "internet-studio"
Merging angularjs_1 1/11
Successfully merged tags into "angularjs"
Merging i_1 1/2
Successfully merged tags into "i"
Merging lines_1 1/1
Successfully merged tags into "lines"
Merging row_1 1/1
Successfully merged tags into "row"
Merging alex-alpha_1 1/1
Successfully merged tags into "alex-alpha"
Merging assignment-one_1 1/2
Successfully merged tags into "assignment-one"
Merging google_1 1/1
Successfully merged tags into "google"
Merging man_1 1/4
Successfully merged tags into "man"
Merging nick_1 1/1
Successfully merged tags into "nick"
Merging cartoon_1 1/1
Successfully merged tags into "cartoon"
Merging batman_1 1/2
Successfully merged tags into "batman"
Merging code_1 1/7
Merging the_1 1/1
Successfully merged tags into "the"
Merging animation_1 1/2
Successfully merged tags into "animation"
Merging band_1 1/4
Merging assignment-one-of-three_1 1/2
Successfully merged tags into "assignment-one-of-three"
Merging status_1 1/1
Successfully merged tags into "status"
Merging python_1 1/2
Successfully merged tags into "python"
Merging cat_1 1/1
Successfully merged tags into "cat"
Merging none_1 1/7
Successfully merged tags into "none"
Merging adam_1 1/2
Successfully merged tags into "adam"
Merging school_1 1/3
Successfully merged tags into "school"
Merging website_1 1/9
Merging biography_1 1/2
Merging bootstrap_1 1/8
Successfully merged tags into "bootstrap"
Merging datamill_1 1/5
Successfully merged tags into "datamill"
Merging gentoo_1 1/2
Successfully merged tags into "gentoo"
Merging dobschal_1 1/1
Successfully merged tags into "dobschal"
Merging weimar_1 1/1
Successfully merged tags into "weimar"

When all went fine, I ran lowercasetags and mergealltags on both Fiddle Salad and Python Fiddle. Now I was really impressed with the results as I clicked through the tags on both sites. The tags on Fiddle Salad were much better organized as they were ordered by popularity. While looking through the tags, I noticed that “test” was among the top. I decided to add ‘test’ to the list of stopwords for django-taggit. These stopwords are removed during save so that they are not associated with new snippets.
Now that the tags are normalized, I am ready to move on and deploy tag suggestions.

One response so far

Too Lazy to Click: A Decline in Traffic with a Tips Panel

Oct 17 2013 Published by under Fiddle Salad,Programming

After launching the a new tips panel on Fiddle Salad that shows on top of other dialog windows whenever a fiddle is opened, I found a decline in traffic. Specifically, page views decreased. Does that mean the tips were poorly done? To investigate, I compared the views in the past 30 days to the previous period and looked at the traffic for different pages. The top pages ranked by page view saw an increase in ratio compared to the rest. My hypothesis is that when opening a saved fiddle, users were deterred by the tip panel that opened. I found the data to back up the hypothesis.
google analytics traffic
The analysis is that there were 0 visitors to the pages in the past 30 days because they just didn’t bother to check off “Show tips on startup”. This made sense, as the tips panel often blocked pieces of code. My plan is to just show the tips panel once a day so that opening saved fiddles wouldn’t show it.

Update:
I found out that the new tab page in Chrome stable was changed, removing the apps. This would explain the decline in traffic on the CoffeeScript IDE page. This change has been there for 4 months on the Chrome beta channel that I use on Windows.

No responses yet

Using Encapsulation to Make an Awesome Angular JS Editor in Fiddle Salad

Sep 26 2013 Published by under Fiddle Salad,Programming

Angular JS is a popular MVC framework by Google, and I received a bug report that Fiddle Salad wouldn’t reload the preview as it does in all other cases. This problem was noticed before, but it wasn’t a priority since pressing the refresh button as required in all other web development methods solves the problem. However, this was because I was attempting to solve it at the wrong level by modifying a method. As I’ve learned, there are dependencies between the execute, reset, and initiation of the class involved.

Fiddle Salad’s CodeRunner class has just 5 public methods. They are

  • execute()
  • add_file()
  • remove_css()
  • reset()
  • debug()

as in the class diagram on the wiki at https://github.com/yuguang/fiddlesalad/wiki/2.-Classes. Therefore, any implementation of a CodeRunner just needs to support these 5 methods. The debug method could be refactored into a super class because it would generate the template for the new execute() method.

debug: ->
  ###
  Debug opens a new window with the code loaded in the page. External CSS and JS files are loaded through head tags.
  It assumes all external resources are stored in the view model.
  ###

  template =
    css: _.template '<link rel="stylesheet" type="text/css" href="<%= source %>" />'
    js: _.template '<script type="text/javascript" src="<%= source %>"></script>'
    html: _.template """
          <!DOCTYPE html>
          <html>
            <head>
              <title>Fiddle Salad Debug View</title>
              <script src="http://leaverou.github.com/prefixfree/prefixfree.min.js"></script>
              <style>
                <%= css %>
              </style>
            </head>
            <body>
              <%= body %>
              <%= headtags %>
              <script type="text/javascript">
                <%= javascript %>
              </script>
            </body>
          </html>
          """

  # initialize array of head tags
  headTags = new Array
  # for each external resource in view model
  _.each viewModel.resources(), (resource) =>
    # get the file type of the resource, call mapped template with resource, and append generated HTML to head tags
    headTags.push template[@filetype resource.source()](source: resource.source())
  headtags = headTags.join('')
  # get JavaScript and CSS code from the engine
  javascript = engine.get_code LANGUAGE_TYPE.COMPILED_PROGRAM
  body = engine.get_code LANGUAGE_TYPE.COMPILED_DOCUMENT
  css = engine.get_code LANGUAGE_TYPE.COMPILED_STYLE
  # call the template for the window with the head tags and code
  html = template.html {javascript, css, body, headtags}
  # open window with generated HTML
  window.open = 'data:text/html;charset=utf-8,' + encodeURIComponent(html)

If I just change the last line, the debug method turns into a reset method.

@window.location = 'data:text/html;charset=utf-8,' + encodeURIComponent(html)

The rest of the class still needs to be completed, but they are as obvious as appending to an array when adding JS/CSS files.

No responses yet

Show tips on startup in 4 lines of code

Aug 17 2013 Published by under Fiddle Salad,Programming

Fiddle Salad has accumulated many features, but where are they? In keeping with one of its principles, features are hidden until the situation calls for them. One example is the JavaScript errors that are shown on hover over the results window. Recently, I’ve decided showing an unobtrusive tips window is as good as hiding features to reduce UI clutter. So, let’s start with 4 lines of Knockout code.

TipsPanel = ->
  @startup = ko.observable(not store.get('hideTipsOnStartup'))
  @startup.subscribe((checked) ->
    store.set('hideTipsOnStartup', not checked)
  )

I start by setting startup to a storage value using store.js. It uses HTML5 local storage with cookies for fallback. Since I want to show the tips panel by default and hideTipsOnStartup would not be set, startup is set to the opposite hideTipsOnStartup. JavaScript’s ! and CoffeeScript’s not casts an expression to a boolean, as required by Knockout’s checked binding.
Next, I add a manual subscription to the observable so that its value is stored each time the checkbox is checked or unchecked.
Finally, the checkbox is bound to tips.startup where tips is a TipsPanel instance.

<label><input data-bind="checked: $root.tips.startup" type="checkbox"/> Show tips on startup</label>

No responses yet

Grabbing a Share of the Pie – Marketing Fiddle Salad Correctly

Aug 01 2013 Published by under Fiddle Salad,Programming

The idea of running Fiddle Salad came in a weird way. At first, I was just making a Python to JavaScript compiler and putting it on a website. It was simple enough, as the compiler was already built for it. The back-end for saving and authentication through Google, Twitter, and Facebook used the same one as Python Fiddle. After the site was launched, I noticed that not many people were interested in writing Python. Also, I got a taste of CoffeeScript, which is now my favorite language.

I think the demise of the Python-only fiddle came as a combination of low traffic and no saves. If nobody ever sees it or uses it, then it is better gone. Fortunately, I had the chance to design and develop the next generation of Fiddle Salad to support other languages.

6067OT_3_13

The real Fiddle Salad was launched in April of 2012. I remembered calling the previous version Python Fiddle. Last January, I decided to do an experiment with the CoffeeScript page by creating a separate app in the Chrome App Store. It turned out to double my number of daily visitors. Last May, it was featured on Lifehacker as one of the best apps for development. So why not do it again?

I waited until the right time. Last month and the previous one, the project received major updates which I will be discussing on the Fiddle Salad blog. Meanwhile, I published more apps in the Chrome App Store.

The difference now is that instead of an app supporting more features that show up in searches, I’ve limited the features that are marketed. This way it targets specific audiences. JavaScript programmers don’t want to find out about the features built for CoffeeScript, and a programmer looking for a CoffeeScript IDE won’t bother with an app that doesn’t have CoffeeScript in the title.

 

No responses yet

Planning For the Next Version of Fiddle Salad: How I Nearly Jumped to My Next Project

The work done on Fiddle Salad this month would not have been possible without last month’s planning. Furthermore, Fiddle Salad would not have been my idea if I did not invest time in building Python Fiddle. Python Fiddle was really the end product of 9 years of dreams of running a high performance computer and the result of my experience using Gentoo Linux. So I bought a computer to build Python Fiddle, which also turned out to be necessary to run the latest IDE and development tools to build Fiddle Salad.  When I started working with the Python interpreter in JavaScript,  it was horrendously slow. It took about 20 seconds to load and took up almost 1GB of memory. Any text editor except Vim without syntax highlighting was quick enough to edit the 12MB source code file.

Fiddle Salad is an evolution of both the original idea and code base that belonged to Python Fiddle. Now it is really Fiddle Salad that’s driving the development of Python Fiddle, because they share much of the code base. 

So this is the third major milestone, which I almost gave up on before I embarked on it. Before I started work on this milestone, actually a day or two before I planned, I suddenly noticed huge, discouraging signs. They came as shocking surprises. For example, I discovered a hidden option in an application I have used often before that had some of the functionality I was going to build. If that wasn’t enough, it was actually quite popular and many people probably knew that feature. As another example, I discovered another application that was more innovative in certain aspects than the application I planned to build. I got still more examples, but they aren’t worth repeating here.

As a habit, I reached for my next plan and the best tools I have available. I then realized that I would be throwing away about 8 months of work and the plans for this month, which worked out so well. Although I had no reason and no incentive at all to work on Fiddle Salad, I did so only because I enjoyed every moment of it. I believe that’s what we are all here for, the very drumbeat of the universe.

In the end, those serious signs got swallowed up by my project, as I managed to either include their ideas or integrate them right into it. Fiddle Salad is really the culmination and peak of all live web development environments, having the best features in all of them and in my imagination.

No responses yet

Designing Fiddle Code Storage

Feb 17 2012 Published by under Fiddle Salad,Programming,Python Fiddle

I just had an aha! insight 5 minutes ago when working on the design to be implemented next month. The previous storage format used string splitting and joining. Everyone who’s worked with it recognizes the mistake immediately, though the solution didn’t come until I spent a bit of time designing the system.

Previously, get_code and set_code belonged to the abstract factory. Looking at the way things are used, it may better facilitate the implementation of the local history feature if a code storage class was used. I took a snapshot when I got the idea for the JSON format, so things are still a bit disorganized:

The realization is that this design doesn’t require the back-end to store all the different languages used in the fiddle. It allows both PythonFiddle, which stores only Python, and FiddleSalad, which mixes languages, to use the same storage backend. They already do, but I’m getting ready to go to the next stage. The /python/ and /coffeescript/ URL structure is still good for SEO and linking purposes.

No responses yet