Sunday, September 23, 2018

figuring out s3 bucket size

You can calculate the size of your buckets with commandline tools, or using the analytics feature.  However, all of these incur charges.

Alternatively you can run a 'usage report', save it as a csv , and then do some math

cat report.csv | awk -F, '{printf "%.2f GB %s %s \n", $7/(1024**3 )/24, $4, $2}' | sort -n

Tuesday, August 28, 2018

Time to talk about google app engine, flex , standard , php, and google cloud storage.

I've been doing some experimenting using GAE to make a service like cloudinary.  It's pretty simple to do, using imagick, which is installed in GAE.  But here are some simple discoveries;

1)  GAE flex, sounds like GAE standard, but it's not.  if using php55 it's quite different.  with php72 now launched in standard, it's closer to flex, you need a front_controller, and your app.yaml is simpler
2) GAE standard scales really quickly, like to 10k instances in 1 to 2 mins.  Flex takes a LOT longer, think 30 mins
3) GAE can use imagick, imagick is not fast, and cpu intensive , fetching an image from a GCS bucket and rendering it cropped or resized is a 1-2 second operation.

you could use appengines image api, but for php it's limited to getservingurl functions. unless you want to write protobuf interfaces .

In python, it's way more useful, you can do all the usual transforms, and it's not cpu intensive on your appengine instance, meaning you don't scale to 10k instances.  However it's not as fast as GCE , as below

So instead, I used GCE because

1) you can use libvips instead of imagick, 400ms for image resizing and serving, compared to 1-2 seconds with GAE, or 800ms with GAE and image api
2) you can use preemptible instances 

The savings are huge, for my estimates, $400k + per year between GAE and GCE

two other findings, about php and the google wrappers;  they are cool, you can do things like

if (file_exists("gs://bucket/foo/bar"))

which is cool, right ? , not really, when you dig in.  under the hood, that does two bucket calls.  because it tests if it's writable, even if your operation doesn't care about writes.  It's more efficient to use curl, and test the contents returned etc, and then use the contents later in you code.  This becomes more apparent (it did for me) when using a GLB, with nodes around the world. users hitting nodes in Australia hitting a multi-region bucket in the US, were a low slower, than the same users hitting a node in the US.  This is due to the multiple transcontinental bucket calls.  using curl and a single call eliminated the performance differences

Monday, May 21, 2018

load testing

I have used apache bench, and more recently tools, but it still wasn't ideal.  I came across this tool which will do a better job for basic testing

Thursday, March 29, 2018

Sky Q tuning with a Meter

Using a sky Q compatible LNB, you can still tune with a regular digital tuner.  The analogue inline tuners won't get enough voltage to work right.

I use a satlink ws-6933, and set it to a vertical transponder and hooked it to the vertical output of the LNB, without doing that you will get a signal measurement with no quality measurement

Below are the screen shots of sky Q signal on the meter and in the sky Q UI too.

Wednesday, January 31, 2018

FreeBSD in GCE

Yes there are supported images of amd64 freebsd available in GCE to boot, but I needed an i386 variant so here are the steps I took.

I downloaded the mini memstick image, converted it to a disk format virtualbox recognised

VBoxManage.exe convertfromraw FreeBSD-11.1-RELEASE-amd64-mini-memstick.img FreeBSD-11.1-RELEASE-amd64-mini-memstick.vdi  --format vdi

Then made a virtual machine with that as it's disk, booted and chose the shell option, then remounted as read/write, and put this into /boot/loader.conf

mount -o rw /dev/ufs/FreeBSD_install /


then shutdown the machine (gracefully!)

now I have an image that supports install via a serial port, just need to make it the right format for GCE to consider as a boot disk

VBoxManage.exe clonehd FreeBSD-11.1-RELEASE-amd64-mini-memstick.vdi disk.raw --format raw

tar -Sczf /tmp/compressed-image.tar.gz disk.raw

now put compressed-image.tar.gz in a bucket and create an image from it.

create a virtual machine with this as the primary disk and a 20GB second disk. Boot , and connect serial port , and install to the second disk.  before finishing the install make sure you update fstab on the second disk, so it can be used as a primary disk.

lastly fixup your nic MTU

ifconfig_vtnet0="SYNCDHCP mtu 1460"

figuring out s3 bucket size