TL;DR (or Show Me The Code): working example, full sources.

Quick start

IronWorkerNG (their upload/management tool)

We will need their management tool to upload code.

gem install iron_worker_ng

The credentials

IronWorkerNG reads the credentials from a file named iron.json.
To get it, we need to login to https://hud.iron.io/dashboard, click on the key-icon and download iron.json to the root directory of the project.

Resolving dependencies on CPAN modules

Iron.IO doesn’t come with any perl module other than what is in core, so we need to bring in all our dependencies.

For that we will use Carton

$ cpanm -nq Carton   # install carton

add our dependencies to cpanfile

$ cat cpanfile
requires 'Data::Printer';
requires 'File::Slurp';
requires 'Mojolicious';

and install them to a ./local/ directory

$ carton install

Seting up the worker

The configuration for the worker is defined in a <service>.worker file, so we create a simple one that tells it to run our perl script and to upload our dependencies (generated by Carton, above) together with the code.

$ cat iron-pl.worker
runtime 'perl'
exec 'iron-pl.pl'
dir 'local/lib/perl5'

This will upload our code together with the ./local/lib/perl5 directory (as ./perl5 on their system)

For more information, check the .worker documentation.

Writing the code!

This is what will be run when you schedule some tasks.
We can go crazy on that one, but here I’d like to highlight just a few things:

#!/usr/bin/env perl
use 5.010; use utf8; use strict; use warnings;
(...)

use lib 'perl5';

We need to load the dependencies from the local directory, thus the use lib above.

(...)
my %args = @ARGV;

They pass arguments as ('-d', '/task/', '-e', 'production', '-id', '5277ca9ed16f93360109d0aa', '-payload', '/task/task_payload.json') so we can load them as a Hash and access it as $args{-d} :–)

The data passed from the scheduled task to the worker comes in a .json file, so we need to parse that

my $payload = Mojo::JSON->new->decode( read_file($args{-payload}) );

For more info (and executable code), check the full sources on github.

Deploy

With all pieces ready, we can upload the code to their platform using their management tool.

$ iron_worker upload iron-pl.worker
    ------> Creating client
            Project 'ironing-board' with id='5277a2b987a3b90005000044'
    ------> Creating code package
            Found workerfile with path='iron-pl.worker'
            Detected exec with path='iron-pl.pl' and args='{}'
            Merging dir with path='local/lib/perl5' and dest=''
            Code package name is 'iron-pl'
    ------> Uploading code package 'iron-pl'
            Code package uploaded with id='5277af07c7abc62bd5098755' and revision='4'
            Check 'https://hud.iron.io/tq/projects/5277a2b987a3b90005000044/code/5277af07c7abc62bd5098755' for more info

Add tasks to the queue

$ iron_worker queue iron-pl -p '{"tags":["iron.io","perl"]}'
    ------> Creating client
            Project 'ironing-board' with id='5277a2b987a3b90005000044'
    ------> Queueing task
            Code package 'iron-pl' queued with id='5277ca9ed16f93360109d0aa'
            Check 'https://hud.iron.io/tq/projects/5277a2b987a3b90005000044/jobs/5277ca9ed16f93360109d0aa' for more info

Check results

We can either go straight to the web interface using the url listed above or use their management tool:

    $ iron_worker log 5277ca9ed16f93360109d0aa
    ------> Creating client
            Project 'ironing-board' with id='5277a2b987a3b90005000044'
    ------> Getting log for task with id='5277ca9ed16f93360109d0aa'
    Iron-Play v0.0.1
    2013-11-04T16:26:10
    Environment: {
        HOME              "/task",
        LANG              "en_US.UTF-8",
        LD_LIBRARY_PATH   ".:./lib:./__debs__/usr/lib:./__debs__/usr/lib/x86_64-linux-gnu:./__debs__/lib:./__debs__/lib/x86_64-linux-gnu",
        LOGNAME           "nobody",
        MAIL              "/var/mail/nobody",
        OLDPWD            "/task",
        PATH              ".:./bin:./__debs__/usr/bin:./__debs__/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games",
        PWD               "/task",
        SHELL             "/bin/sh",
        SUDO_COMMAND      "/usr/bin/ruby run.rb --sleep 60 -e production -n 1 -j /mnt/iron-jail",
        SUDO_GID          1000,
        SUDO_UID          1000,
        SUDO_USER         "ubuntu",
        TERM              "unknown",
        USER              "nobody",
        USERNAME          "root"
    }
    Arguments: {
        -d         "/task/",
        -e         "production",
        -id        "5277ca9ed16f93360109d0aa",
        -payload   "/task/task_payload.json"
    }
    Payload (/task/task_payload.json): \ {
        tags   [
            [0] "iron.io",
            [1] "perl"
        ]
    }

    Top users for tag 'iron.io':
    AnaelFavre: 417
    thousandsofthem: 446
    Travis R: 6935

    Top users for tag 'perl':
    ikegami: 106098
    mpapec: 10324
    TLP: 40585

    Done.

But, but..

Of course, this example is not very useful, manually scheduling tasks and getting results from calling iron_worker log <id> is not really something interesting :–)

To be useful it would need to be automatically triggered by some event and produce usable results (updating a databse, notifying another service, etc..) but I assume you already know this if you’re searching for how to run perl on IronWorker.

So, go crazy and drop me a hello on the comments bellow if this post was somehow helpful.

Cheers! \o/

So, for my future self, here it is:

As example, lets pick github’s IP.

[carlos@multi ~]$ host github.com
github.com has address 192.30.252.129

The key thing to remember is that it is the same as forward dns, just on a “funny” zone name.

In our example, the reverse dns record is: 129.252.30.192.in-addr.arpa
(and the zone 252.30.192.in-addr.arpa)

Find the authority over that zone:

[carlos@multi ~]$ dig 129.252.30.192.in-addr.arpa

; <<>> DiG 9.8.1-P1 <<>> 129.252.30.192.in-addr.arpa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21414
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;129.252.30.192.in-addr.arpa.   IN      A

;; AUTHORITY SECTION:
252.30.192.in-addr.arpa. 60     IN      SOA     ns1.p16.dynect.net. ops.github.com. 7 3600 600 604800 60

;; Query time: 72 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Fri Oct 18 03:25:41 2013
;; MSG SIZE  rcvd: 113

Get the PTR record for that ip:

[carlos@multi ~]$ dig PTR 129.252.30.192.in-addr.arpa

; <<>> DiG 9.8.1-P1 <<>> PTR 129.252.30.192.in-addr.arpa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29787
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;129.252.30.192.in-addr.arpa.   IN  PTR

;; ANSWER SECTION:
129.252.30.192.in-addr.arpa. 3189 IN    PTR ip1b-lb3-prd.iad.github.com.

;; Query time: 29 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Fri Oct 18 03:28:44 2013
;; MSG SIZE  rcvd: 86

Directly from authority

And if you’re trying to troubleshoot whether your ISP correctly updated the PTR records you requested, you can ask the authority directly and avoid waiting for propagation:

[carlos@multi ~]$ dig PTR 129.252.30.192.in-addr.arpa @ns1.p16.dynect.net

; <<>> DiG 9.8.1-P1 <<>> PTR 129.252.30.192.in-addr.arpa @ns1.p16.dynect.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51495
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;129.252.30.192.in-addr.arpa.   IN      PTR

;; ANSWER SECTION:
129.252.30.192.in-addr.arpa. 3600 IN    PTR     ip1b-lb3-prd.iad.github.com.

;; AUTHORITY SECTION:
252.30.192.in-addr.arpa. 86400  IN      NS      ns1.p16.dynect.net.
252.30.192.in-addr.arpa. 86400  IN      NS      ns3.p16.dynect.net.
252.30.192.in-addr.arpa. 86400  IN      NS      ns2.p16.dynect.net.
252.30.192.in-addr.arpa. 86400  IN      NS      ns4.p16.dynect.net.

;; Query time: 476 msec
;; SERVER: 2001:500:90:1::16#53(2001:500:90:1::16)
;; WHEN: Fri Oct 18 03:31:39 2013
;; MSG SIZE  rcvd: 172

Just simple resolution

Now, for completeness sake, if all you want is the reverse dns, remember that host works on both ip and name.

[carlos@multi ~]$ host github.com
github.com has address 192.30.252.130
github.com mail is handled by 30 ASPMX3.GOOGLEMAIL.com.
github.com mail is handled by 30 ASPMX2.GOOGLEMAIL.com.
github.com mail is handled by 20 ALT1.ASPMX.L.GOOGLE.com.
github.com mail is handled by 10 ASPMX.L.GOOGLE.com.
github.com mail is handled by 20 ALT2.ASPMX.L.GOOGLE.com.
[carlos@multi ~]$ host 192.30.252.130
130.252.30.192.in-addr.arpa domain name pointer ip1c-lb3-prd.iad.github.com.
[carlos@multi ~]$ 

On our system, /dev/SuperFoo/sub-bar is a submodule of SuperFoo.git repository.

carlos@carlosdev /d/SuperFoo (master)> git log -1
commit c2abbad057a02a2bd0d7c1e1c74048da6ef88234
Author: Chuck Norris <chuck@example.com>
Date:   Thu Oct 10 02:35:16 2013 +0000

    Updated to latest translation

carlos@carlosdev /d/SuperFoo (master)> cd sub-bar
carlos@carlosdev /d/S/sub-bar ((2279a91))> git log -1
commit 2279a9187b023f79cf274f52a76fe5059f119914
Merge: fc57e1a a3110e8
Author: Chuck Norris <chuck@example.com>
Date:   Thu Oct 10 02:24:15 2013 +0000

    Merge branch 'master' into currenttranslations

so what?

They are disconnected, despite being shown/accessible as a regular subdirectory of your main project.

again?

carlos@carlosdev /d/SuperFoo (master)> vi sub-bar/templates/home/login.html.haml cgi/fooish.cgi
..(change change change)..
carlos@carlosdev /d/SuperFoo (master)> git status

# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#   (commit or discard the untracked or modified content in submodules)
#
#       modified:   sub-bar (modified content)
#       modified:   cgi/fooish.cgi
#
no changes added to commit (use "git add" and/or "git commit -a")

sub-bar/ files are not part of your repository!

Look again:

carlos@carlosdev /d/SuperFoo (master)> git diff
diff --git a/sub-bar b/sub-bar
--- a/sub-bar
+++ b/sub-bar
@@ -1 +1 @@
-Subproject commit 2279a9187b023f79cf274f52a76fe5059f119914
+Subproject commit 2279a9187b023f79cf274f52a76fe5059f119914-dirty
diff --git a/cgi/fooish.cgi b/cgi/fooish.cgi
index c607f79..2b1ce17 100755
--- a/cgi/fooish.cgi
+++ b/cgi/fooish.cgi
@@ -1,4 +1,3 @@
-#!/usr/bin/perl

 use strict;
 use warnings;
carlos@carlosdev /d/SuperFoo (master)> 

Ok, got it. But what’s the deal? (or, What could possibly go wrong?)

This happened before, more than once. Not hypothetical.

carlos@carlosdev /d/SuperFoo (bugfixes)> git status
# On branch bugfixes
# Your branch is behind 'origin/bugfixes' by 1813 commits, and can be fast-forwarded.
#
nothing to commit (working directory clean)

carlos@carlosdev /d/SuperFoo (bugfixes)> git rebase -p
Successfully rebased and updated refs/heads/bugfixes.

All good. Let’s go fix our ticket!

carlos@carlosdev /d/SuperFoo (bugfixes)> vi lib/README.txt 
carlos@carlosdev /d/SuperFoo (bugfixes)> git diff
diff --git a/sub-bar b/sub-bar
index 8b6fa45..eac1423 160000
--- a/sub-bar
+++ b/sub-bar
@@ -1 +1 @@
-Subproject commit 8b6fa45a258e37293d472c81844a3c37e921b6f9
+Subproject commit eac1423fcb1812e0ff958712231dddc06687733c
diff --git a/lib/README.txt b/lib/README.txt
index 293a71f..e9a6e01 100644
--- a/lib/README.txt
+++ b/lib/README.txt
@@ -1,4 +1,4 @@
 This directory holds shared code between all sub-projects. Please keep it clean.

-See https://internal.example.com/wiki/code-rules
+See https://example.thirdparty-inthecloud.net/wiki/code-rules

Looks good, simple enough.

carlos@carlosdev /d/SuperFoo (bugfixes)> git commit -am 'Fix wrong url'
[bugfixes 9ecfc7d] Fix wrong url
 2 files changed, 2 insertions(+), 2 deletions(-)

Cheers! We just rolled back sub-bar by 1 month.

carlos@carlosdev /d/S/sub-bar ((eac1423))> git log -1 8b6fa45a258e37293d472c81844a3c37e921b6f9
commit 8b6fa45a258e37293d472c81844a3c37e921b6f9
Author: Michelle Yeoh <mi@example.com>
Date:   Thu Oct 3 11:52:38 2013 +0000

    Translated using Weblate (Portuguese)

    Currently translated at 100.0% (448 of 448 strings)

carlos@carlosdev /d/S/sub-bar ((eac1423))> git log -1 eac1423fcb1812e0ff958712231dddc06687733c
commit eac1423fcb1812e0ff958712231dddc06687733c
Author: Angelina Jolie <angie@example.com>
Date:   Mon Sep 2 02:10:57 2013 +0000

    Tidy

Next time bugfixes gets released, the version of sub-bar in production will be from 1 month ago.

Not just minor changes..

carlos@carlosdev /d/S/sub-bar ((eac1423))> git diff --stat eac1423..8b6fa45
.. snip ..
78 files changed, 5871 insertions(+), 1435 deletions(-)

Why did that happen?

Because it is a separate repository, git only tracks three things about the submodule:

• url of the external repository
• path where the files will be put/cloned
• the commit id from that repository that you want to use.

They are:

carlos@carlosdev /d/SuperFoo (bugfixes)> cat .gitmodules
[submodule "sub-bar"]
    path = sub-bar
    url = git@bitbucket.org:exampledotcom/sub-bar.git

carlos@carlosdev /d/SuperFoo (bugfixes)> git ls-tree bugfixes
..snip..
100644 blob 2d918f0b794c059d6900e7211cd34e22aa395a77    Makefile   ← file
040000 tree 486d5dafd1d4fd7ff4f8cf3b273d5912e84666ae    api        ← directory
160000 commit eac1423fcb1812e0ff958712231dddc06687733c  sub-bar    ← submodule (commit id to use)
..snip..

When we did the rebase, the changes included updates to the version of sub-bar that the SuperFoo repository should be using.
But git DOES NOT checkout that new version inside the sub-bar directory.

Git knows we should be using commit 8b6fa45
But on sub-bar directory we have eac1423 (from before the rebase)

It doesn’t know (or care) if the commits are newer, older, etc.
So, when we did git commit -am'...' without paying attention to that sub-bar mention that appeared on git diff or git status, we told git that we want it to use commit eac1423, which is what we have checked out at the moment.

This is really confusing. I don’t care about sub-bar and I’m not touching it, how do I avoid breaking things?

First and foremost, when git status or git diff shows sub-bar, do not ignore it.

The root cause is that when you move your repository to a different state, git won’t automatically move the submodule to the new expected commit.
So we fix it with git submodule update.

The name might be counter-intuitive, but git submodule update doesn’t change/write/commit/push the submodule, it just runs a checkout of the expected commit.

Second, don’t git add sub-bar;
(Unless updating the submodule to a new version is what you’re trying to do.)

Caveats

Bitbucket

Bitbucket (as of 2013-10-10) won’t show changes to submodules on their diff.
That means reviewers won’t be able to see that you made a mistake!

git commit -a / git commit .

Please, don’t do it. Ever.
This will happily commit every change you have on disk.
It is an horrible practice, much more dangerous when submodules are involved.
Always review your changes with git status and git diff.
Don’t commit anything you didn’t mean to.

What happened is that I did only half of the work.
I defined method_missing but I forgot to define respond_to? accordingly.
The result is that it worked when I called it directly on the instance, but failed if an association was involved.

To give an example, say you have a class like this:

1
2
3
4
5
6
7
8
9
10
11
12
13

class A < ActiveRecord::Base
  def example
    true
  end

  def method_missing(method, *args)
    if method.to_s =~ /example/
      example
    else
      super
    end
  end
end

Calling example directly on your instance works just fine.

1
2
3
4
5

>> A.new.my_example
=> true

>> A.create!.example_me?
=> true

All fine, but as soon as you get an association in the middle of things:

1
2
3

class B < ActiveRecord::Base
  belongs_to :a
end

It just doesn’t go well anymore:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

>> b = B.new(:a=>A.new)
=> #<B id: nil, a_id: nil, created_at: nil, updated_at: nil>


>> b.a.example?
NoMethodError: undefined method example?' for #<A id: nil, created_at: nil, updated_at: nil>
 from /var/lib/gems/1.8/gems/activerecord-2.3.5/lib/active_record/associations/association_proxy.rb:220:inmethod_missing’
from (irb):55

>> b = B.create!(:a=>A.create!)
=> #<B id: 4, a_id: 7, created_at: “2010-03-08 21:15:01”, updated_at: “2010-03-08 21:15:01”>
>> b.a.failing_example
NoMethodError: undefined method failing_example' for #<ActiveRecord::Associations::BelongsToAssociation:0xb70491d0>
 from /var/lib/gems/1.8/gems/activerecord-2.3.5/lib/active_record/associations/association_proxy.rb:220:inmethod_missing’
from (irb):57

Now, this last error is a bit clearer but I don’t remember running into it at the time.
If I had just followed the association_proxy:220 hint right away… ;)

What happens is that b.a doesn’t return the instance but rather an AssociationProxy instance that provides ActiveRecord’s extended functionality and this proxy relies on A#respond_to? to correctly forward method calls to the actual instance.

What I should have done is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

class A < ActiveRecord::Base
  def example
    true
  end

  def method_missing(method, *args)
    if method.to_s =~ /example/
      example
    else
      super
    end
  end

  def respond_to?(method, include_private = false)
    if method.to_s =~ /example/
      true
    else
      super
    end
  end
end

1
2
3
4

>> B.create!(:a=>A.create!).a.example?
=> true
>> B.new(:a=>A.new).a.failing_example
=> true

There are some much better write-ups on this topic, if you want to read more:

• Using method_missing and respond_to? to create dynamic methods
• Solving the method_missing/respond_to? problem

Now, I must be honest here: what I was doing was a big code smell :)
It taught me the lesson to use method_missing properly and was even quite fun to debug and all that but what I really needed and end up doing in that case was a group of delegates here and there and voilà, it was all cool and clean.

Remote Pair Programming

The focus

The challenge

The safety feeling

The learning

The loneliness

The results