{"id":1403,"date":"2015-01-01T20:14:31","date_gmt":"2015-01-02T01:14:31","guid":{"rendered":"http:\/\/yuguangzhang.com\/blog\/?p=1403"},"modified":"2015-01-01T20:19:33","modified_gmt":"2015-01-02T01:19:33","slug":"merge-tags-with-django-taggit","status":"publish","type":"post","link":"http:\/\/yuguangzhang.com\/blog\/merge-tags-with-django-taggit\/","title":{"rendered":"Merge Tags with django-taggit"},"content":{"rendered":"

Today I cleaned up the database for Fiddle Salad<\/a> and Python Fiddle<\/a>. Both use the same Django back-end for code storage<\/a>. While browsing tags, I noticed that often both CamelCase and lowercase spellings were used for tags. Since I was working on a tag suggest feature earlier this week, I decided to convert all tags to lowercase so that tag suggestions would not be redundant. An additional benefit is further normalization of the data. Fortunately, I found a fork of django-taggit, the Django app I used for tagging, that supported enforcing lowercase tags everywhere. Two management commands were already present for normalizing data, mergetags<\/tt> and lowercasetags<\/tt>. django-taggit had two fields for each tag, a name and slug. lowercasetags<\/tt> converted all tag names to their lowercase form. mergetags<\/tt> takes at least two tag slugs and merges all tags into a single destination tag. The result is that all associations are moved to a single tag. While mergetags<\/tt> is suitable for manually resolving redundant data, the number of tags on Fiddle Salad is too large. I wrote an command to automate this process:<\/p>\n

[cc lang=”python”]
\nfrom django.core.management.base import BaseCommand, CommandError
\nfrom taggit.models import Tag, TaggedItem
\nfrom django.core.exceptions import ObjectDoesNotExist<\/p>\n

class Command(BaseCommand):
\n help = ‘merges all tags automatically’<\/p>\n

def merge(self, extra_slugs, dest_slug):
\n try:
\n dest_tag = Tag.objects.get(slug=dest_slug)
\n except ObjectDoesNotExist:
\n raise CommandError(‘Destination Tag “%s” does not exist’ % dest_slug)<\/p>\n

for slug in extra_slugs:
\n try:
\n tag = Tag.objects.get(slug=slug)
\n except ObjectDoesNotExist:
\n raise CommandError(‘Tag “%s” does not exist’ % slug)<\/p>\n

items = TaggedItem.objects.filter(tag=tag)
\n count = items.count()
\n for i, item in enumerate(items):
\n if i % 20 == 0:
\n self.stdout.write(‘Merging %s %d\/%d\\n’ % (slug, i+1, count))
\n obj = item.content_object
\n if not obj:
\n return
\n obj.tags.remove(tag)
\n obj.tags.add(dest_tag)
\n tag.delete()<\/p>\n

self.stdout.write(‘Successfully merged tags into “%s”\\n’ % dest_slug)<\/p>\n

def handle(self, *args, **options):
\n for tag in Tag.objects.all():
\n if Tag.objects.filter(name=tag.name).count() > 1:
\n tags = Tag.objects.filter(name=tag.name).order_by(‘id’)
\n dest = tags[0].slug
\n extras = []
\n for tag in tags[1::]:
\n extras.append(tag.slug)
\n self.merge(extras, dest)
\n[\/cc]
\nBecause performance is not a concern for a single-time data processing script, I did not bother to optimize the queries nor run-time. This script would be useful for anyone who wants to normalize tags in the same manner, so it is in a git repository<\/a>. Finally, I tested the new command on a clone of the production database.
\n
\nbash-4.1$ python manage.py lowercasetags
\nLowercasing 1\/1621
\nLowercasing 21\/1621
\n.
\n.
\n.
\nLowercasing 1621\/1621
\nbash-4.1$ python manage.py mergealltags
\nMerging jquery_1 1\/46
\nMerging jquery_1 21\/46
\nMerging jquery_1 41\/46
\nSuccessfully merged tags into \"jquery\"
\nSuccessfully merged tags into \"jquery\"
\nMerging stylus_1 1\/7
\nSuccessfully merged tags into \"stylus\"
\nMerging hello_1 1\/10
\nSuccessfully merged tags into \"hello\"
\nMerging test_1 1\/147
\nMerging test_1 21\/147
\nMerging test_1 41\/147
\nMerging test_1 61\/147
\nMerging test_1 81\/147
\nMerging test_1 101\/147
\nMerging test_1 121\/147
\nMerging test_1 141\/147
\nSuccessfully merged tags into \"test\"
\nMerging me_1 1\/2
\nSuccessfully merged tags into \"me\"
\nMerging no_1 1\/5
\nSuccessfully merged tags into \"no\"
\nMerging one_1 1\/16
\nSuccessfully merged tags into \"one\"
\nMerging things_1 1\/3
\nSuccessfully merged tags into \"things\"
\nMerging learning_1 1\/4
\nSuccessfully merged tags into \"learning\"
\nSuccessfully merged tags into \"body\"
\nMerging week-one_1 1\/1
\nSuccessfully merged tags into \"week-one\"
\nMerging studio_1 1\/33
\nMerging studio_1 21\/33
\nSuccessfully merged tags into \"studio\"
\nMerging internet_1 1\/36
\nMerging internet_1 21\/36
\nSuccessfully merged tags into \"internet\"
\nMerging assignment_1 1\/6
\nSuccessfully merged tags into \"assignment\"
\nMerging homework_1 1\/6
\nSuccessfully merged tags into \"homework\"
\nMerging lessons_1 1\/1
\nSuccessfully merged tags into \"lessons\"
\nMerging code_1 1\/12
\nMerging tags_1 1\/7
\nSuccessfully merged tags into \"tags\"
\nMerging two_1 1\/6
\nSuccessfully merged tags into \"two\"
\nMerging salcedo_1 1\/3
\nSuccessfully merged tags into \"salcedo\"
\nMerging page_1 1\/15
\nSuccessfully merged tags into \"page\"
\nMerging music_1 1\/4
\nSuccessfully merged tags into \"music\"
\nMerging table_1 1\/7
\nSuccessfully merged tags into \"table\"
\nMerging band_1 1\/9
\nMerging texas_1 1\/1
\nSuccessfully merged tags into \"texas\"
\nMerging biography_1 1\/2
\nMerging assignment-two_1 1\/2
\nSuccessfully merged tags into \"assignment-two\"
\nMerging website_1 1\/9
\nMerging a_1 1\/2
\nSuccessfully merged tags into \"a\"
\nMerging words_1 1\/1
\nSuccessfully merged tags into \"words\"
\nMerging section_1 1\/2
\nSuccessfully merged tags into \"section\"
\nMerging header_1 1\/1
\nSuccessfully merged tags into \"header\"
\nMerging ui_1 1\/4
\nSuccessfully merged tags into \"ui\"
\nMerging first_1 1\/8
\nSuccessfully merged tags into \"first\"
\nMerging random_1 1\/1
\nSuccessfully merged tags into \"random\"
\nMerging internet-studio_1 1\/5
\nSuccessfully merged tags into \"internet-studio\"
\nMerging angularjs_1 1\/11
\nSuccessfully merged tags into \"angularjs\"
\nMerging i_1 1\/2
\nSuccessfully merged tags into \"i\"
\nMerging lines_1 1\/1
\nSuccessfully merged tags into \"lines\"
\nMerging row_1 1\/1
\nSuccessfully merged tags into \"row\"
\nMerging alex-alpha_1 1\/1
\nSuccessfully merged tags into \"alex-alpha\"
\nMerging assignment-one_1 1\/2
\nSuccessfully merged tags into \"assignment-one\"
\nMerging google_1 1\/1
\nSuccessfully merged tags into \"google\"
\nMerging man_1 1\/4
\nSuccessfully merged tags into \"man\"
\nMerging nick_1 1\/1
\nSuccessfully merged tags into \"nick\"
\nMerging cartoon_1 1\/1
\nSuccessfully merged tags into \"cartoon\"
\nMerging batman_1 1\/2
\nSuccessfully merged tags into \"batman\"
\nMerging code_1 1\/7
\nMerging the_1 1\/1
\nSuccessfully merged tags into \"the\"
\nMerging animation_1 1\/2
\nSuccessfully merged tags into \"animation\"
\nMerging band_1 1\/4
\nMerging assignment-one-of-three_1 1\/2
\nSuccessfully merged tags into \"assignment-one-of-three\"
\nMerging status_1 1\/1
\nSuccessfully merged tags into \"status\"
\nMerging python_1 1\/2
\nSuccessfully merged tags into \"python\"
\nMerging cat_1 1\/1
\nSuccessfully merged tags into \"cat\"
\nMerging none_1 1\/7
\nSuccessfully merged tags into \"none\"
\nMerging adam_1 1\/2
\nSuccessfully merged tags into \"adam\"
\nMerging school_1 1\/3
\nSuccessfully merged tags into \"school\"
\nMerging website_1 1\/9
\nMerging biography_1 1\/2
\nMerging bootstrap_1 1\/8
\nSuccessfully merged tags into \"bootstrap\"
\nMerging datamill_1 1\/5
\nSuccessfully merged tags into \"datamill\"
\nMerging gentoo_1 1\/2
\nSuccessfully merged tags into \"gentoo\"
\nMerging dobschal_1 1\/1
\nSuccessfully merged tags into \"dobschal\"
\nMerging weimar_1 1\/1
\nSuccessfully merged tags into \"weimar\"
\n<\/code>
\nWhen all went fine, I ran lowercasetags<\/tt> and mergealltags<\/tt> on both Fiddle Salad<\/a> and Python Fiddle<\/a>. Now I was really impressed with the results as I clicked through the tags on both sites. The tags on Fiddle Salad<\/a> were much better organized as they were ordered by popularity. While looking through the tags, I noticed that “test” was among the top. I decided to add ‘test’ to the list of stopwords for django-taggit. These stopwords are removed during save so that they are not associated with new snippets.
\nNow that the tags are normalized, I am ready to move on and deploy tag suggestions. <\/p>\n","protected":false},"excerpt":{"rendered":"

Today I cleaned up the database for Fiddle Salad and Python Fiddle. Both use the same Django back-end for code storage. While browsing tags, I noticed that often both CamelCase and lowercase spellings were used for tags. Since I was working on a tag suggest feature earlier this week, I decided to convert all tags […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[203,23,146],"tags":[214,212,213],"class_list":["post-1403","post","type-post","status-publish","format-standard","hentry","category-fiddle-salad","category-programming","category-pythonfiddle","tag-database","tag-django","tag-normalization"],"aioseo_notices":[],"_links":{"self":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts\/1403","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/comments?post=1403"}],"version-history":[{"count":4,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts\/1403\/revisions"}],"predecessor-version":[{"id":1410,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts\/1403\/revisions\/1410"}],"wp:attachment":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/media?parent=1403"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/categories?post=1403"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/tags?post=1403"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}