JQ is a nifty tool to parse, slice and dice json data .

I know I can do this with python/ruby/whatever but it’s fun to try something new ..

Instance offering for Linux + EU Ireland

curl https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/20160126001708/index.json | jq '[.products| to_entries | .[].value | select (.attributes.operatingSystem == "Linux" and .attributes.location=="EU (Ireland)")  | {key: .sku, value: .}]|from_entries'

Pricing info

curl https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/20160126001708/index.json | jq '.terms.OnDemand | .. | .priceDimensions? | select (. != null) |map_values (.)| map (. + {sku: .rateCode|split(".")[0]}) | .[] ' 

It would be cool to somehow combine the two (per instance pricing) but for now this is ‘good enough’.

1. Variables
   QCOW_FILE=file.img
LV_NAME=RootDisk
2. Find out the size of the file

   qemu-img info ${QCOW_FILE}

> image: file.img
> file format: qcow2
> virtual size: 60G (64424509440 bytes)
> disk size: 56G
> cluster_size: 65536
> Format specific information:
compat: 0.10

3. Set the file size

   SIZE="60G"

4. create a logical volume with matching size.

   lvcreate -L${SIZE} -n${LV_NAME} vg0

5. “mount” the image file as a nbd device

   qemu-nbd -c /dev/nbd0 ${QCOW_FILE}

6. copy the contents from nbd0 to the new logical volume

   dd if=/dev/nbd0 of=/dev/vg0/${LV_NAME} bs=1M

7. disconnect the volume

   qemu-nbd -d /dev/nbd0

TIL bash is a bit tricky to get right:

1. When you chain multiple processes via pipes under bash , the exit code available in $? is actually the return code of the last process in the chain. This turns out to be very relevant if you pipe the output to something like the tee utility which almost always will return successfully. In order to catch the return code use set -o pipefail  before you execute the command.
2. Let’s say you want to wrap a command in a bash script and that command takes a bunch of parameters. A quick and dirty way would be to call the command like this from your bash script $COMMAND $*. This would pass to the command all the arguments supplied to the script. In most cases this ok but if one of the arguments contains a space, bash will split it and serve it as two separate arguments to your command. So instead of $COMMAND  -argument "First Second"  bash will send this as $COMMAND -argument "First" "Second" . The workaround is to use $@ and wrap that in quotes. So : $COMMAND "$@"

Inspired by the project of this year’s SNE students I’ve decided to explore this thing on my own.

Some quick notes on my findings so far:

• Google increases the string length as it runs out of keyspace space. Since late July 2013 they use 6 upper/lower/digits for the unique part of the short URL
• As of December 2014 the fill ratio for the 6 digit keyspace is about 4%.
• I think that the keys are not random and must be generated via some algorithm. I came to this conclusion because the 4 and 5 character keyspaces are completely full, i.e. any lookup returns ‘success’ or ‘removed’ . I assume it would be inefficient to randomly generate the string and then check if the string is not already used. As the keyspace fills up, most of the random strings would have been already used. Also this means that there’s no correlation between the time or the length of the URL.

Sourcecode here: https://github.com/cosu/urlshort

You need mongo running on the localhost to save the results.

If your transfers from outside the EC2 cloud to the new m3 instances fail, then it might be due to jumbo frames configured by default on this new type of instance.

I have an external machine which also uses jumbo frames and during the tcp handshake the machines advertise maximum segment sizes (MSS) of 8960 and 8961bytes respectively.

I’m not really sure if this is Amazon’s fault or a misconfiguration on my side but the fix is to lower the MTU back to 1500 on the instance. Sure, intra-area transfers might be slower but at least I can copy stuff from outside.

ip link set dev eth0 mtu 1500

Follow up from the previous post.
Quick and dirty howto adapted from instructions found at dnscrypt.org

The first step is to install unbound:

brew install unbound

Next, download the dnssec anchor to /usr/local/etc/unbound/

sudo unbound-anchor -a /usr/local/etc/unbound/root.key

Add the following things in /usr/local/etc/unbound/unbound.conf at the end of the server section. Note that unbound will be running under the nobody account and I’m assuming that dnscrypt-proxy is already running on port 40! For this, you must change the plist of dnscrypt-proxy. See the previous previous post for that.

auto-trust-anchor-file: "/usr/local/etc/unbound/root.key/root.key"
interface: 127.0.0.1
access-control: 127.0.0.1/8 allow
chroot: ""
username: "nobody"
do-not-query-localhost: no
forward-zone:
name: "."
     forward-addr: 127.0.0.1@40

Copy the plist config for launchd to /Library/LaunchDeamon:

sudo cp -fv /usr/local/opt/unbound/homebrew.mxcl.unbound.plist /Library/LaunchDaemons

Start unbound:

sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.unbound.plist

Test it with:

dig +dnssec . @localhost

You should see something like:

 
; <<>> DiG 9.8.3-P1 <<>> +dnssec .
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44948
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1 

If the ad flag is there then the response is dnssec validated.

dnscrypt provides encryption between clients and dns resolvers. Here’s how to install it on OSX via homebeew and how to change the default resolver to something else besides OpenDNS’ server.

brew install dnscrypt-proxy

Go to System Preferences -> network -> advanced -> dns and set 127.0.0.1 as DNS server

Launch dnscrypt-proxy and see if it works

sudo /usr/local/opt/dnscrypt-proxy/sbin/dnscrypt-proxy --local-address=127.0.0.1:53
nslookup google.com

if that works then time to test with an alternate server

sudo /usr/local/opt/dnscrypt-proxy/sbin/dnscrypt-proxy --local-address=127.0.0.1:53 --resolver-address=176.56.237.171:443 --provider-name=2.dnscrypt-cert.resolver1.dnscrypt.eu --provider-key=67C0:0F2C:21C5:5481:45DD:7CB4:6A27:1AF2:EB96:9931:40A3:09B6:2B8D:1653:1185:9C66

This is one of the servers made available on the dnscrypt page. There are a bunch of servers there to chose from (the closer, the better) or you can build your own dnscrypt-aware resolver. It all goes well it’s time to next copy the homebrew launchd plist file to its proper place so that dnscrypt starts at boot time.

Copy the supplied plist file in the right place:

sudo cp -fv /usr/local/opt/dnscrypt-proxy/homebrew.mxcl.dnscrypt-proxy.plist /Library/LaunchDaemons

Open the file with an editor and replace/append the parameters for dnscrypt-proxy. This step is required only if you are unhappy with the OpenDNS resolver. One of the major drawbacks of the opendns resolver (and the reason why I’ve stopped using opendns altogether ) is that it hijacks searches from the Firefox address bar and redirects them to their search engine.

My version of the file looks like this

< ?xml version="1.0" encoding="UTF-8"?>
< !DOCTYPE plist PUBLIC "-/Apple/DTD PLIST 1.0/EN" "http:/www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>homebrew.mxcl.dnscrypt-proxy</string>
    <key>KeepAlive</key>
    <true></true>
    <key>RunAtLoad</key>
    <true></true>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/opt/dnscrypt-proxy/sbin/dnscrypt-proxy</string>
      <string>--local-address=127.0.0.1:53</string>
      <string>--resolver-address=176.56.237.171:443</string>
      <string>--provider-name=2.dnscrypt-cert.resolver1.dnscrypt.eu</string>
      <string>--provider-key=67C0:0F2C:21C5:5481:45DD:7CB4:6A27:1AF2:EB96:9931:40A3:09B6:2B8D:1653:1185:9C66</string>
      <string>--user=nobody</string>
    </array>
    <key>UserName</key>
    <string>root</string>
    <key>StandardErrorPath</key>
    <string>/dev/null</string>
    <key>StandardOutPath</key>
    <string>/dev/null</string>
  </dict>
</plist>

If you’re using a custom resolver, like above, you must supply all three parameters – resolver-address, provider-name and provider-key!

And now for the last step, enable dnscrypt-proxy in launchd:

sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.dnscrypt-proxy.plist

No more DNS leakage!

From here we could have further refinements, like installing unbound for local DNS caching and DNSSEC support, etc.

If syslog starts complaining about neighbor table overflow  then some limits need to be increased.

Current values:

grep . /proc/sys/net/ipv6/neigh/default/gc_thresh*

/proc/sys/net/ipv6/neigh/default/gc_thresh1:128
/proc/sys/net/ipv6/neigh/default/gc_thresh2:512
/proc/sys/net/ipv6/neigh/default/gc_thresh3:1024

Add to sysctl.conf

net.ipv6.neigh.default.gc_thresh1 = 512
net.ipv6.neigh.default.gc_thresh2 = 2048
net.ipv6.neigh.default.gc_thresh3 = 4096

Reload sysctl parameters

sysctl -p