carlos@blog ~/notes>

\o/

External Commands - Vim Tricks

While editing code, I frequently cycle between changing the code and running some commands: running unit tests, linting tools, style checker, restarting services, etc.

I generaly do that directly from inside vim with :!command and then repeating it with :!(up arrow)

While this is good, I learned a trick a while ago that still ranks as one of my favorites.

The trick

Add this to your .vimrc

1
map \] :wa<bar><UP><CR>

Then run your command as :wa|!command and repeat it with just \] directly on normal mode.

If \ and ] are not next to each other on your keyboard layout, just change the mapping to something that is very easy and convenient to type :)

The advantage

This is quicker to type, it saves all the files before running the command and it still allows you to run that one-off :!command in between your cycle without it getting in the way of your repetition.

That’s it. Happy Vimming!

iDrac Virtual Console Crashing on Linux

iDrac

iDrac (or any other IPMI) is great and can really save your time and your life, but it is really frustrating to find out that your out-of-band virtual console crashes on you at the moment you need it the most.

Given that this particular problem has struck me a few times already I decided to write a post to my future self and save me some trouble.

Scenario

After receiving an alert and failing to connect to the host, you log into the iDrac’s web interface and click “Launch” on the Virtual Console option.
It downloads a file called viewer.jnlp.
You start it with javaws viewer.jnlp and, after a few confirmation dialogs, it explodes!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
carlos@multi ~/Downloads> javaws viewer.jnlp
This application does not specify a Codebase in its manifest. Please verify with the applet's vendor. Continuing. See: http://docs.oracle.com/javase/7/docs/technotes/guides/jweb/security/no_redeploy.html for details.
This application does not specify a Codebase in its manifest. Please verify with the applet's vendor. Continuing. See: http://docs.oracle.com/javase/7/docs/technotes/guides/jweb/security/no_redeploy.html for details.
Application title was not found in manifest. Check with application vendor
KVM/VM Client Version: 5.01.01 (Build 22)
ProtocolAPCP.receieveSessionSetup : reconType = 101
capabilities..4
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f1268a72550, pid=14296, tid=139717301577472
#
# JRE version: OpenJDK Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: OpenJDK 64-Bit Server VM (25.45-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  0x00007f1268a72550
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/carlos/Downloads/hs_err_pid14296.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
fish: Job 1, “javaws viewer.jnlp ” terminated by signal SIGABRT (Abort)

Looking into the log file gives no better clue.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
---------------  T H R E A D  ---------------

Current thread (0x00007f1289ae0000):  JavaThread "iDRAC7 Virtual Console Client" [_thread_in_native, id=14329, stack(0x00007f1278112000,0x00007f1278213000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f1268a72550

Registers:
RAX=0x00007f1268a72550, RBX=0x0000000000000378, RCX=0x0000000000000000, RDX=0x00007f12782102cf
RSP=0x00007f12782102e8, RBP=0x00007f1278210550, RSI=0x00007f12698cf477, RDI=0x00007f12698cf590
R8 =0x00007f1298984908, R9 =0x0000000000000000, R10=0x00007f1268d5e380, R11=0x0000000000000202
R12=0x0000000000000000, R13=0x00007f1269e15eb0, R14=0x00007f12782107b8, R15=0x00007f1289ae0000
RIP=0x00007f1268a72550, EFLAGS=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000014
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f12782102e8)
0x00007f12782102e8:   00007f1269800440 0000000000000000
(...)

There is a lot of information there but going thru java stacktraces of your virtual console tool while your server is down is undesired at best.

Solution

Fortunately the solution is simple.

Open the viewer.jnlp file.
There will be a bunch of OS and architecture-specific lines like this:

1
2
3
4
5
6
7
8
9
 <resources os="Windows" arch="x85">
   <nativelib href="https://server.example.com:443/software/avctKVMIOWin32.jar" download="eager"/>
   <nativelib href="https://server.example.com:443/software/avctVMAPI_DLLWin32.jar" download="eager"/>
 </resources>
 <resources os="Windows" arch="amd64">
   <nativelib href="https://server.example.com:443/software/avctKVMIOWin64.jar" download="eager"/>
   <nativelib href="https://server.example.com:443/software/avctVMAPI_DLLWin64.jar" download="eager"/>
 </resources>
 (...)

Delete all of them, except for the relevant one. In my case:

1
2
3
4
  <resources os="Linux" arch="x86_64">
    <nativelib href="https://server.example.com:443/software/avctKVMIOLinux64.jar" download="eager"/>
   <nativelib href="https://server.example.com:443/software/avctVMAPI_DLLLinux64.jar" download="eager"/>
  </resources>

That’s it. Running javaws viewer.jnlp should work fine now.

I believe this is not the “correct” solution but it gets you going.

Talk: Feedback

This is a talk I gave at work. It’s the second time I give a similar talk so I’ve decided to post it up. Unfortunately I have no slides nor video so I’m posting my notes :–)

Feedback

Feedback: collecting information about present and past of a process to drive changes to the process itself

Crucial to improve, if you don’t know at what speed you’re running you can’t adjust to the speed limit.
If you’re not aware of problems, you can’t fix them.
If you don’t see logs and exception stacktraces you won’t know that your application crashes every time someone from Russia visits your website.

Phrased like this sounds like a good thing but when you think about the actual process of giving/receiving feedback it changes a bit.
Who likes performance appraisals? Code reviews? Submitting the system you worked on for 6 months to 3rd party review and scrutiny?

Why are we afraid of giving feedback?

When we ask for opinion/review, generally get “it’s ok/cool/nice”.
People point out obvious problems but it usually doesn’t go further.

There are usually something that could be different.
A better way, an alternative way, an experimental way.
Not necessarily better but if you look at something and think “I would have done it differently”, maybe you should say that.
The other person might like your idea or it might trigger another idea in her head.

Another example.
You know that person who always do something when presenting that distracts people from the presentation?
They repeat the same phrase again and again or they mispronounce a word or whatever and at the end of the talk everybody is whispering about how many times he did this or that?
That person could be a much better presenter if he could fix that behavior but it won’t happen if he is not aware of it!

So, why are we so bad at giving feedback?

To answer that we need to look at another question.

Why are we so bad at receiving feedback?

People don’t like being criticized.
Humans are wired like that :(
Hearing criticism doesn’t change who you are but we don’t like it anyway.
We go into defensive mode, we need to show we did the best thing. We need to prove the person wrong!

To avoid confrontation, nobody wants to give nor receive feedback.

Why is it bad?

If you both don’t give feedback not to hurt or make others angry and you don’t actively seek feedback, it only happens when a problem occurs (or someone explodes). That’s the time when emotions are high and openness is low.

That makes people less willing to seek improvement generating a vicious cycle.

How to fix it, then?

Ermm..

  • Improve on how we provide feedback
  • Improve on how we receive feedback

How to receive feedback

You can’t force people to be nicer or to do things in a way that pleases you.
But you can improve on how you handle it, so focus on that.

First, don’t take it personally.
Face everything as an opportunity to revisit a topic and think if anything could be improved.
Be open. You don’t have to agree but be willing to consider ideas.
If it is a complain or an aggressive criticism, try to filter out the offending part and think about the root cause of that complain; Could you have done anything different so that there would be no complain?

How to give feedback

Avoid doing it if you’re angry or in bad mood.
Consider what is the goal of your feedback and focus on having a constructive conversation.
Don’t expect the person to agree. You will be lucky if they at least listen :–)
Consider that your idea/suggestion might not be better or might be missing parts of the picture.
Even better, assume you’re wrong; that there is something you don’t know.
Guide the conversation to try to understand where you’re wrong.

Wrap up

Feedback is good, is crucial to improve (as person, as professional, as team).
Most people are bad at both giving and receiving and avoid it as much as possible.
There is no silver bullet, no easy solution.
Be aware of the problem. Consider how you could do it better.
Search about the topic, find more information :)
And most important, practice. Stop avoiding it.

Extra

Focus on the right stuff!

Criticism is driven by the frustration and fears of the giver, not from the needs of the recipient. The underlying assumption is that the recipient somehow “should know better” and needs to be set straight. The implied message is that the recipient’s intentions are questionable, that there is something wrong with the recipient that the giver of criticism knows how to fix. In criticism, the problem is all in the recipient.
In contrast, feedback has an air of caring concern, respect, and support. Far from being a sugar cookie, feedback is an honest, clear, adult to adult exchange about specific behaviors and the effects of those behaviors. The assumption is that both parties have positive intentions, that both parties want to be effective and to do what is right for the company and other people. Another assumption is that well-meaning people can have legitimate differences in perception. The person offering the feedback owns the feedback as being his reaction to the behavior of the other person. That is, the giver recognizes the fact that what is being offered is a perception, not absolute fact.
— Gary R. Casselman & Timothy C. Daughtry

Punishing honest mistakes stifles creativity. I want people moving and shaking the earth and they’re going to make mistakes.
— Ross Perot

Links

Those are generally written for managers but the reasoning is the same.

Running Perl Workers on Iron.IO

Despite not being listed as an option on their documentation, it is possible to run perl on their IronWorker service.

TL;DR (or Show Me The Code): working example, full sources.

Quick start

IronWorkerNG (their upload/management tool)

We will need their management tool to upload code.

gem install iron_worker_ng

The credentials

IronWorkerNG reads the credentials from a file named iron.json.
To get it, we need to login to https://hud.iron.io/dashboard, click on the key-icon and download iron.json to the root directory of the project.

Resolving dependencies on CPAN modules

Iron.IO doesn’t come with any perl module other than what is in core, so we need to bring in all our dependencies.

For that we will use Carton

$ cpanm -nq Carton   # install carton

add our dependencies to cpanfile

$ cat cpanfile
requires 'Data::Printer';
requires 'File::Slurp';
requires 'Mojolicious';

and install them to a ./local/ directory

$ carton install

Seting up the worker

The configuration for the worker is defined in a <service>.worker file, so we create a simple one that tells it to run our perl script and to upload our dependencies (generated by Carton, above) together with the code.

$ cat iron-pl.worker
runtime 'perl'
exec 'iron-pl.pl'
dir 'local/lib/perl5'

This will upload our code together with the ./local/lib/perl5 directory (as ./perl5 on their system)

For more information, check the .worker documentation.

Writing the code!

This is what will be run when you schedule some tasks.
We can go crazy on that one, but here I’d like to highlight just a few things:

#!/usr/bin/env perl
use 5.010; use utf8; use strict; use warnings;
(...)

use lib 'perl5';

We need to load the dependencies from the local directory, thus the use lib above.

(...)
my %args = @ARGV;

They pass arguments as ('-d', '/task/', '-e', 'production', '-id', '5277ca9ed16f93360109d0aa', '-payload', '/task/task_payload.json') so we can load them as a Hash and access it as $args{-d} :–)

The data passed from the scheduled task to the worker comes in a .json file, so we need to parse that

my $payload = Mojo::JSON->new->decode( read_file($args{-payload}) );

For more info (and executable code), check the full sources on github.

Deploy

With all pieces ready, we can upload the code to their platform using their management tool.

$ iron_worker upload iron-pl.worker
    ------> Creating client
            Project 'ironing-board' with id='5277a2b987a3b90005000044'
    ------> Creating code package
            Found workerfile with path='iron-pl.worker'
            Detected exec with path='iron-pl.pl' and args='{}'
            Merging dir with path='local/lib/perl5' and dest=''
            Code package name is 'iron-pl'
    ------> Uploading code package 'iron-pl'
            Code package uploaded with id='5277af07c7abc62bd5098755' and revision='4'
            Check 'https://hud.iron.io/tq/projects/5277a2b987a3b90005000044/code/5277af07c7abc62bd5098755' for more info

Add tasks to the queue

$ iron_worker queue iron-pl -p '{"tags":["iron.io","perl"]}'
    ------> Creating client
            Project 'ironing-board' with id='5277a2b987a3b90005000044'
    ------> Queueing task
            Code package 'iron-pl' queued with id='5277ca9ed16f93360109d0aa'
            Check 'https://hud.iron.io/tq/projects/5277a2b987a3b90005000044/jobs/5277ca9ed16f93360109d0aa' for more info

Check results

We can either go straight to the web interface using the url listed above or use their management tool:

    $ iron_worker log 5277ca9ed16f93360109d0aa
    ------> Creating client
            Project 'ironing-board' with id='5277a2b987a3b90005000044'
    ------> Getting log for task with id='5277ca9ed16f93360109d0aa'
    Iron-Play v0.0.1
    2013-11-04T16:26:10
    Environment: {
        HOME              "/task",
        LANG              "en_US.UTF-8",
        LD_LIBRARY_PATH   ".:./lib:./__debs__/usr/lib:./__debs__/usr/lib/x86_64-linux-gnu:./__debs__/lib:./__debs__/lib/x86_64-linux-gnu",
        LOGNAME           "nobody",
        MAIL              "/var/mail/nobody",
        OLDPWD            "/task",
        PATH              ".:./bin:./__debs__/usr/bin:./__debs__/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games",
        PWD               "/task",
        SHELL             "/bin/sh",
        SUDO_COMMAND      "/usr/bin/ruby run.rb --sleep 60 -e production -n 1 -j /mnt/iron-jail",
        SUDO_GID          1000,
        SUDO_UID          1000,
        SUDO_USER         "ubuntu",
        TERM              "unknown",
        USER              "nobody",
        USERNAME          "root"
    }
    Arguments: {
        -d         "/task/",
        -e         "production",
        -id        "5277ca9ed16f93360109d0aa",
        -payload   "/task/task_payload.json"
    }
    Payload (/task/task_payload.json): \ {
        tags   [
            [0] "iron.io",
            [1] "perl"
        ]
    }

    Top users for tag 'iron.io':
    AnaelFavre: 417
    thousandsofthem: 446
    Travis R: 6935

    Top users for tag 'perl':
    ikegami: 106098
    mpapec: 10324
    TLP: 40585

    Done.

But, but..

Of course, this example is not very useful, manually scheduling tasks and getting results from calling iron_worker log <id> is not really something interesting :–)

To be useful it would need to be automatically triggered by some event and produce usable results (updating a databse, notifying another service, etc..) but I assume you already know this if you’re searching for how to run perl on IronWorker.

So, go crazy and drop me a hello on the comments bellow if this post was somehow helpful.

Cheers! \o/

Dns and Ptr Records Quick Notes

Every once in a blue moon I get to troubleshoot, help someone with or explain how reverse dns works but somehow it keeps getting misplaced inside my brain.

So, for my future self, here it is:

As example, lets pick github’s IP.

[carlos@multi ~]$ host github.com
github.com has address 192.30.252.129

The key thing to remember is that it is the same as forward dns, just on a “funny” zone name.

In our example, the reverse dns record is: 129.252.30.192.in-addr.arpa
(and the zone 252.30.192.in-addr.arpa)

Find the authority over that zone:

[carlos@multi ~]$ dig 129.252.30.192.in-addr.arpa

; <<>> DiG 9.8.1-P1 <<>> 129.252.30.192.in-addr.arpa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21414
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;129.252.30.192.in-addr.arpa.   IN      A

;; AUTHORITY SECTION:
252.30.192.in-addr.arpa. 60     IN      SOA     ns1.p16.dynect.net. ops.github.com. 7 3600 600 604800 60

;; Query time: 72 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Fri Oct 18 03:25:41 2013
;; MSG SIZE  rcvd: 113

Get the PTR record for that ip:

[carlos@multi ~]$ dig PTR 129.252.30.192.in-addr.arpa

; <<>> DiG 9.8.1-P1 <<>> PTR 129.252.30.192.in-addr.arpa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29787
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;129.252.30.192.in-addr.arpa.   IN  PTR

;; ANSWER SECTION:
129.252.30.192.in-addr.arpa. 3189 IN    PTR ip1b-lb3-prd.iad.github.com.

;; Query time: 29 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Fri Oct 18 03:28:44 2013
;; MSG SIZE  rcvd: 86

Directly from authority

And if you’re trying to troubleshoot whether your ISP correctly updated the PTR records you requested, you can ask the authority directly and avoid waiting for propagation:

[carlos@multi ~]$ dig PTR 129.252.30.192.in-addr.arpa @ns1.p16.dynect.net

; <<>> DiG 9.8.1-P1 <<>> PTR 129.252.30.192.in-addr.arpa @ns1.p16.dynect.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51495
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;129.252.30.192.in-addr.arpa.   IN      PTR

;; ANSWER SECTION:
129.252.30.192.in-addr.arpa. 3600 IN    PTR     ip1b-lb3-prd.iad.github.com.

;; AUTHORITY SECTION:
252.30.192.in-addr.arpa. 86400  IN      NS      ns1.p16.dynect.net.
252.30.192.in-addr.arpa. 86400  IN      NS      ns3.p16.dynect.net.
252.30.192.in-addr.arpa. 86400  IN      NS      ns2.p16.dynect.net.
252.30.192.in-addr.arpa. 86400  IN      NS      ns4.p16.dynect.net.

;; Query time: 476 msec
;; SERVER: 2001:500:90:1::16#53(2001:500:90:1::16)
;; WHEN: Fri Oct 18 03:31:39 2013
;; MSG SIZE  rcvd: 172

Just simple resolution

Now, for completeness sake, if all you want is the reverse dns, remember that host works on both ip and name.

[carlos@multi ~]$ host github.com
github.com has address 192.30.252.130
github.com mail is handled by 30 ASPMX3.GOOGLEMAIL.com.
github.com mail is handled by 30 ASPMX2.GOOGLEMAIL.com.
github.com mail is handled by 20 ALT1.ASPMX.L.GOOGLE.com.
github.com mail is handled by 10 ASPMX.L.GOOGLE.com.
github.com mail is handled by 20 ALT2.ASPMX.L.GOOGLE.com.
[carlos@multi ~]$ host 192.30.252.130
130.252.30.192.in-addr.arpa domain name pointer ip1c-lb3-prd.iad.github.com.
[carlos@multi ~]$ 

Git Submodules: How (Not) to Use It.

Intro (or What’s a git submodule?)

On our system, /dev/SuperFoo/sub-bar is a submodule of SuperFoo.git repository.

carlos@carlosdev /d/SuperFoo (master)> git log -1
commit c2abbad057a02a2bd0d7c1e1c74048da6ef88234
Author: Chuck Norris <[email protected]>
Date:   Thu Oct 10 02:35:16 2013 +0000

    Updated to latest translation

 

carlos@carlosdev /d/SuperFoo (master)> cd sub-bar
carlos@carlosdev /d/S/sub-bar ((2279a91))> git log -1
commit 2279a9187b023f79cf274f52a76fe5059f119914
Merge: fc57e1a a3110e8
Author: Chuck Norris <[email protected]>
Date:   Thu Oct 10 02:24:15 2013 +0000

    Merge branch 'master' into currenttranslations

so what?

They are disconnected, despite being shown/accessible as a regular subdirectory of your main project.

again?

carlos@carlosdev /d/SuperFoo (master)> vi sub-bar/templates/home/login.html.haml cgi/fooish.cgi
..(change change change)..
carlos@carlosdev /d/SuperFoo (master)> git status

# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#   (commit or discard the untracked or modified content in submodules)
#
#       modified:   sub-bar (modified content)
#       modified:   cgi/fooish.cgi
#
no changes added to commit (use "git add" and/or "git commit -a")

sub-bar/ files are not part of your repository!

Look again:

carlos@carlosdev /d/SuperFoo (master)> git diff
diff --git a/sub-bar b/sub-bar
--- a/sub-bar
+++ b/sub-bar
@@ -1 +1 @@
-Subproject commit 2279a9187b023f79cf274f52a76fe5059f119914
+Subproject commit 2279a9187b023f79cf274f52a76fe5059f119914-dirty
diff --git a/cgi/fooish.cgi b/cgi/fooish.cgi
index c607f79..2b1ce17 100755
--- a/cgi/fooish.cgi
+++ b/cgi/fooish.cgi
@@ -1,4 +1,3 @@
-#!/usr/bin/perl

 use strict;
 use warnings;
carlos@carlosdev /d/SuperFoo (master)> 

Ok, got it. But what’s the deal? (or, What could possibly go wrong?)

This happened before, more than once. Not hypothetical.

carlos@carlosdev /d/SuperFoo (bugfixes)> git status
# On branch bugfixes
# Your branch is behind 'origin/bugfixes' by 1813 commits, and can be fast-forwarded.
#
nothing to commit (working directory clean)

carlos@carlosdev /d/SuperFoo (bugfixes)> git rebase -p
Successfully rebased and updated refs/heads/bugfixes.

All good. Let’s go fix our ticket!

carlos@carlosdev /d/SuperFoo (bugfixes)> vi lib/README.txt 
carlos@carlosdev /d/SuperFoo (bugfixes)> git diff
diff --git a/sub-bar b/sub-bar
index 8b6fa45..eac1423 160000
--- a/sub-bar
+++ b/sub-bar
@@ -1 +1 @@
-Subproject commit 8b6fa45a258e37293d472c81844a3c37e921b6f9
+Subproject commit eac1423fcb1812e0ff958712231dddc06687733c
diff --git a/lib/README.txt b/lib/README.txt
index 293a71f..e9a6e01 100644
--- a/lib/README.txt
+++ b/lib/README.txt
@@ -1,4 +1,4 @@
 This directory holds shared code between all sub-projects. Please keep it clean.

-See https://internal.example.com/wiki/code-rules
+See https://example.thirdparty-inthecloud.net/wiki/code-rules

Looks good, simple enough.

carlos@carlosdev /d/SuperFoo (bugfixes)> git commit -am 'Fix wrong url'
[bugfixes 9ecfc7d] Fix wrong url
 2 files changed, 2 insertions(+), 2 deletions(-)

Cheers! We just rolled back sub-bar by 1 month.

carlos@carlosdev /d/S/sub-bar ((eac1423))> git log -1 8b6fa45a258e37293d472c81844a3c37e921b6f9
commit 8b6fa45a258e37293d472c81844a3c37e921b6f9
Author: Michelle Yeoh <[email protected]>
Date:   Thu Oct 3 11:52:38 2013 +0000

    Translated using Weblate (Portuguese)

    Currently translated at 100.0% (448 of 448 strings)

 

carlos@carlosdev /d/S/sub-bar ((eac1423))> git log -1 eac1423fcb1812e0ff958712231dddc06687733c
commit eac1423fcb1812e0ff958712231dddc06687733c
Author: Angelina Jolie <[email protected]>
Date:   Mon Sep 2 02:10:57 2013 +0000

    Tidy

Next time bugfixes gets released, the version of sub-bar in production will be from 1 month ago.

Not just minor changes..

carlos@carlosdev /d/S/sub-bar ((eac1423))> git diff --stat eac1423..8b6fa45
.. snip ..
78 files changed, 5871 insertions(+), 1435 deletions(-)

Why did that happen?

Because it is a separate repository, git only tracks three things about the submodule:

  • url of the external repository
  • path where the files will be put/cloned
  • the commit id from that repository that you want to use.

They are:

carlos@carlosdev /d/SuperFoo (bugfixes)> cat .gitmodules
[submodule "sub-bar"]
    path = sub-bar
    url = [email protected]:exampledotcom/sub-bar.git

carlos@carlosdev /d/SuperFoo (bugfixes)> git ls-tree bugfixes
..snip..
100644 blob 2d918f0b794c059d6900e7211cd34e22aa395a77    Makefile   ← file
040000 tree 486d5dafd1d4fd7ff4f8cf3b273d5912e84666ae    api        ← directory
160000 commit eac1423fcb1812e0ff958712231dddc06687733c  sub-bar    ← submodule (commit id to use)
..snip..

When we did the rebase, the changes included updates to the version of sub-bar that the SuperFoo repository should be using.
But git DOES NOT checkout that new version inside the sub-bar directory.

Git knows we should be using commit 8b6fa45
But on sub-bar directory we have eac1423 (from before the rebase)

It doesn’t know (or care) if the commits are newer, older, etc.
So, when we did git commit -am'...' without paying attention to that sub-bar mention that appeared on git diff or git status, we told git that we want it to use commit eac1423, which is what we have checked out at the moment.

This is really confusing. I don’t care about sub-bar and I’m not touching it, how do I avoid breaking things?

First and foremost, when git status or git diff shows sub-bar, do not ignore it.

The root cause is that when you move your repository to a different state, git won’t automatically move the submodule to the new expected commit.
So we fix it with git submodule update.

The name might be counter-intuitive, but git submodule update doesn’t change/write/commit/push the submodule, it just runs a checkout of the expected commit.

Second, don’t git add sub-bar;
(Unless updating the submodule to a new version is what you’re trying to do.)

Caveats

Bitbucket

Bitbucket (as of 2013-10-10) won’t show changes to submodules on their diff.
That means reviewers won’t be able to see that you made a mistake!

git commit -a / git commit .

Please, don’t do it. Ever.
This will happily commit every change you have on disk.
It is an horrible practice, much more dangerous when submodules are involved.
Always review your changes with git status and git diff.
Don’t commit anything you didn’t mean to.

Method_missing and Respond_to?: Doing It Properly

Sometime ago while using method_missing to implement some functionality I got the weird behavior that it would work only most of the time but not always.
In retrospect it’s now pretty obvious but in the heat of the moment it took me about half a day of investigation and talking before I figured it out.

What happened is that I did only half of the work.
I defined method_missing but I forgot to define respond_to? accordingly.
The result is that it worked when I called it directly on the instance, but failed if an association was involved.

To give an example, say you have a class like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
class A < ActiveRecord::Base
  def example
    true
  end

  def method_missing(method, *args)
    if method.to_s =~ /example/
      example
    else
      super
    end
  end
end

Calling example directly on your instance works just fine.

1
2
3
4
5
>> A.new.my_example
=> true

>> A.create!.example_me?
=> true

All fine, but as soon as you get an association in the middle of things:

1
2
3
class B < ActiveRecord::Base
  belongs_to :a
end

It just doesn’t go well anymore:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
>> b = B.new(:a=>A.new)
=> #<B id: nil, a_id: nil, created_at: nil, updated_at: nil>


>> b.a.example?
NoMethodError: undefined method example?' for #<A id: nil, created_at: nil, updated_at: nil>
 from /var/lib/gems/1.8/gems/activerecord-2.3.5/lib/active_record/associations/association_proxy.rb:220:inmethod_missing’
from (irb):55

>> b = B.create!(:a=>A.create!)
=> #<B id: 4, a_id: 7, created_at: “2010-03-08 21:15:01”, updated_at: “2010-03-08 21:15:01”>
>> b.a.failing_example
NoMethodError: undefined method failing_example' for #<ActiveRecord::Associations::BelongsToAssociation:0xb70491d0>
 from /var/lib/gems/1.8/gems/activerecord-2.3.5/lib/active_record/associations/association_proxy.rb:220:inmethod_missing’
from (irb):57

Now, this last error is a bit clearer but I don’t remember running into it at the time.
If I had just followed the association_proxy:220 hint right away… ;)

What happens is that b.a doesn’t return the instance but rather an AssociationProxy instance that provides ActiveRecord’s extended functionality and this proxy relies on A#respond_to? to correctly forward method calls to the actual instance.

What I should have done is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
class A < ActiveRecord::Base
  def example
    true
  end

  def method_missing(method, *args)
    if method.to_s =~ /example/
      example
    else
      super
    end
  end

  def respond_to?(method, include_private = false)
    if method.to_s =~ /example/
      true
    else
      super
    end
  end
end
1
2
3
4
>> B.create!(:a=>A.create!).a.example?
=> true
>> B.new(:a=>A.new).a.failing_example
=> true

There are some much better write-ups on this topic, if you want to read more:

Now, I must be honest here: what I was doing was a big code smell :)
It taught me the lesson to use method_missing properly and was even quite fun to debug and all that but what I really needed and end up doing in that case was a group of delegates here and there and voilà, it was all cool and clean.

Solutionatic Pair Programming

Remote Pair Programming

A few weeks ago, Larry O’Brien and I set out to experiment with Remote Pair Programming.
He wrote about it on his SDTimes column: Windows & .NET Watch: Problematic pair programming

We both liked the experience and would be rather happy to be able to work on a long term project that way (anyone hiring a remote pair?), but since his column gives more emphasis to the down sides, I thought I could write my point of view which is more on the up side ;)

In general, I agree on the down sides but I got so excited with the advantages I saw that I would barely remember mentioning the problems if asked.

The focus


Working in pair forces you to focus on the work.
There is just no way you can be there and not be paying attention to whatever it is that you’re supposed to do.
You get so focused in the work and the communication that the environmental noise that would otherwise annoy you just seem to disappear.
The speed in which you make progress can vary with a lot of factors but you are *guaranteed* to make some progress. All. The. Time.

The challenge


The fact that someone is watching your every step means that you keep challenging yourself to do things the best way you can.
You just can’t look sloppy to your pair.
That means you avoid cutting corners or knowingly doing the wrong thing out of laziness or “to fix it later”.
If you do something that feels wrong, you’ll be called on it and have to defend your point, so, unless it’s worth the trouble, you just work a bit harder and go for the right thing.

The safety feeling


You certainly feel more confident about every line of code you write.
Anything absurdly wrong will surely get catch by one or the other.
Even the small errors or typos get spotted really fast.
Sure, things can still be wrong, bugged or bad designed but the fact that the code was co-created and reviewed by at least one more person gives you a comforting feeling.

The learning


Working with someone else full time means you’re continuously learning or at least exercising a different approach than your own.
You learn new tricks.
You get to defend your points.
And you question everything.
At the end of the day, you feel actively improving in many things.
Even your English, if you’re not a native speaker ;)

The loneliness


Working remotely means you’re almost always alone.
When your clients are in different countries, thousands of miles away, that’s guaranteed.
It’s just too expensive, time consuming and a tiresome task to travel for a meeting.
If there is something I miss every now and then it’s having someone at close reach that I can share work stuff and work together.
Working in a pair was a comforting step in that direction.

The results


We shared the resulting project on Github.
If you’d like to poke around, it’s available at http://github.com/carloslima/pasteme

Overall, I consider it was a big win over the solo experience.
There is only one thing that could be much better and that’s the IDE/Tools support.
While sharing a VM using VNC mostly worked, it didn’t come close to the experience you get when you use Saros, Bespin or even Google Wave.
Being able to use all your monitors, drag windows around and organize it the way you like better would definitely improve the experience.

Have a different opinion or want to suggest a different tool?
Just drop me a comment!

RSnapshot-style Backups on Dreamhost

DHSnapshot

A while ago I needed to find a good setup for backing up some machines.

I wanted to use RSync to do snapshots-like backups.

This setup is:
* Off-site
* Storage efficient (the bulk of my backup was unlikely to ever change)
* Network efficient (only transfer changed files)
* Keeps versions going back a few months

Now, DreamHost offers 50GB of free space for personal backups and that seemed a good fit for this particular situation.

Then I found RSnapshot and it was exactly what I needed.
Unfortunately, RSnapshot doesn’t backup to an external server.
It’s meant to work in a setup where the server hosting the backups connects to your data sources and pull the files.
That totally makes sense and is probably the best setup but it wouldn’t work for me.

DreamHost doesn’t have RSnapshot installed on their backup servers and the only access users have to that machine is SFTP and RSync.
I needed something that would work with just that.

I came across a post about backing up DreamHost websites to DreamHost Backups that showed a workaround to the lack of SSH access to the backups service.

So I took the idea, mixed it up with the stuff I wanted from RSnapshot and wrote a small perl script to do it.

It’s called dhsnapshot and is published at GitHub: http://github.com/carloslima/dhsnapshot

At this moment, it’s not very flexible: it’s hardwired to keep 7 daily, 4 weekly and 6 monthly backups and it’s also limited to a single backup source (you can only point it to one source directory)
But it’s not hard to change, or even make it configurable.
I might do it if I ever get the need or motivation :)

One way or the other, it should be reasonably simple to setup (instructions on GitHub).

I’d be glad to know if it helped anyone.
So, please, drop me a comment if you fnd it useful.

Hosting Your GIT Repository on DreamHost

A few weeks ago I needed a private remote Git repository.
Since I already have a DreamHost account, I didn’t want to pay for GitHub just to be able to set my repository as private.

DreamHost does not officially support Git on their Web Panel, so you must set things up manually.

There is a Git entry on their Wiki but it looked a bit messy, so I decided to write a cleaner tutorial.

Keep in mind that I’m using WebDAV.
It allows more than one user to push to the same repository, but it’s also slower.

You can use SSH but it’s a lot more complex to setup.
If that’s what you want, read the DreamHost Wiki.

Setup a WebDAV folder

Login to DreamHost Web Panel
Go to Goodies –> Htaccess/WebDAV
Select the domain name you want to use
Click “Set Up A New Directory”

DH Panel/WebDAV

Create the remote repository

Git cannot create the remote repository, it only operates on existing ones, so we need to create an empty repository locally and manually upload it to DreamHost.

So, there we go, open a console and..

carlos@ubuntu:~/dev$ mkdir blank.git
carlos@ubuntu:~/dev$ cd blank.git/
carlos@ubuntu:~/dev/blank.git$ git --bare init
Initialized empty Git repository in /home/carlos/dev/blank.git/
carlos@ubuntu:~/dev/blank.git$ touch git-daemon-export-ok
carlos@ubuntu:~/dev/blank.git$ git --bare update-server-info
carlos@ubuntu:~/dev/blank.git$ mv hooks/post-update.sample hooks/post-update
carlos@ubuntu:~/dev/blank.git$

Upload the blank repository and rename it

Use Nautilus or any other file manager that supports WebDAV to upload blank.git to Dreamhost and rename it to something meaningful. For the purpose of this example, let’s call it project.git

On Ubuntu:

Places->Connect
Places->Connect
Places->Connect
Places->Connect

After it finishes uploading, rename the folder from blank.git to project.git

When I renamed the file, I got an error message saying it failed, but refreshing showed it actually worked.

Ready!

If you’re starting fresh and all you need is a blank repository, then you’re set.
Just clone your repository and start working!

carlos@ubuntu:~/dev$ git clone http://[email protected]/git/project.git
Initialized empty Git repository in /home/carlos/dev/project/.git/
Password:
warning: You appear to have cloned an empty repository.

carlos@ubuntu:~/dev$ cd project
carlos@ubuntu:~/dev/project$ (work, work, work)
carlos@ubuntu:~/dev/project$ git add README
carlos@ubuntu:~/dev/project$ git commit -m "your commit message"
[master (root-commit) 5bbe5f6] your commit message
0 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 README
carlos@ubuntu:~/dev/project$ git push origin master
Password:
Fetching remote heads...
refs/
refs/tags/
refs/heads/
updating 'refs/heads/master'
from 0000000000000000000000000000000000000000
to 5bbe5f6507fa39293bdc9674ca4ae2e0a1d2f15e
sending 3 objects
done
Updating remote server info

carlos@ubuntu:~/dev/project$ git pull
Password:
From http://[email protected]/git/project
* [new branch] master -&gt; origin/master
Already up-to-date.
carlos@ubuntu:~/dev/project (master)$

Optional: Push your project repository to DreamHost

Otherwise, if you already have a project repository that you’ve been working locally, now is the time to push it to DreamHost.

Instead of cloning the new repository, push your project there first:

carlos@ubuntu:~/dev$ cd real_project/
carlos@ubuntu:~/dev/real_project$ git config remote.upload.url http://[email protected]/git/project.git/

  # It is important to put the last '/'; Without it, the server will send a redirect which git-http-push
  # does not (yet) understand, and git-http-push will repeat the request infinitely.

carlos@ubuntu:~/dev/real_project$ git push upload master
Password:
Fetching remote heads...
refs/
refs/tags/
refs/heads/
updating 'refs/heads/master'
from 0000000000000000000000000000000000000000
to a10703d8e400ca9df1b19345975718935c083905
sending 107 objects
done
Updating remote server info
carlos@ubuntu:~/dev/real_project$

Then confirm it worked and start fresh by cloning it.

carlos@ubuntu:~/dev/real_project$ cd ..
carlos@ubuntu:~/dev$ git clone http://[email protected]/git/project.git/
Initialized empty Git repository in /home/carlos/dev/project/.git/
Password:
got a10703d8e400ca9df1b19345975718935c083905
walk a10703d8e400ca9df1b19345975718935c083905
got 574596c4cc435461515aa1a4c3cdd0e93af947f3
got 067f993be7432ac27e8a6e9636dea53dcc3d8632
got 475be0881778acd2de7404175fa323823e4d1ac0
walk 067f993be7432ac27e8a6e9636dea53dcc3d8632
(...)
got b7b5d32db9dd30c9ea28434b125781eb4a3e95b2

carlos@ubuntu:~/dev$ cd project/
carlos@ubuntu:~/dev/project$ git log --oneline
a10703d Adds a beautiful whitespace! :)
067f993 Adds project description.. or sort of
80c2e22 removes rerun.txt
07e3cd2 Initial commit

carlos@ubuntu:~/dev/project$ git pull
Password:
Already up-to-date.

carlos@ubuntu:~/dev/project$ git push origin master
Password:
Fetching remote heads...
refs/
refs/tags/
refs/heads/
'refs/heads/master': up-to-date
carlos@ubuntu:~/dev/project$

And you’re good to resume working on your project :)

Bonus: saving your WebDAV password

Now, if you’re thinking that typing the WebDAV password over and over again kind of suck, you can save it so that git won’t ask you anymore.

carlos@ubuntu:~/dev$ echo "machine www.example.com login bob password secret" >> ~/.netrc

There is only one thing to keep in mind. When you save your password like this, you need to drop the bob@ from the urls.
So, instead of referring to your repository as http://[email protected]/git/project.git/ you need to use just http://www.example.com/git/project.git/
If you do this after you finished everything and cloned your repository, then git will have already saved the “wrong” url into its config file and will keep asking you for the password.

To fix this you can either clone again using the correct url or fix git’s config manually by doing:

carlos@ubuntu:~/dev/project$ git config remote.origin.url http://www.example.com/git/project.git/
carlos@ubuntu:~/dev/project$

Now, just for reference :)

carlos@ubuntu:~/dev$ more /etc/issue.net
Ubuntu 9.10
carlos@ubuntu:~/dev$ git --version
git version 1.6.3.3
carlos@ubuntu:~/dev$ date
Thu Feb 25 08:55:18 BRT 2010
carlos@ubuntu:~/dev$