Sunday, September 23, 2018

figuring out s3 bucket size

You can calculate the size of your buckets with commandline tools, or using the analytics feature.  However, all of these incur charges.

Alternatively you can run a 'usage report', save it as a csv , and then do some math

cat report.csv | awk -F, '{printf "%.2f GB %s %s \n", $7/(1024**3 )/24, $4, $2}' | sort -n




Tuesday, August 28, 2018

Time to talk about google app engine, flex , standard , php, and google cloud storage.

I've been doing some experimenting using GAE to make a service like cloudinary.  It's pretty simple to do, using imagick, which is installed in GAE.  But here are some simple discoveries;

1)  GAE flex, sounds like GAE standard, but it's not.  if using php55 it's quite different.  with php72 now launched in standard, it's closer to flex, you need a front_controller, and your app.yaml is simpler
2) GAE standard scales really quickly, like to 10k instances in 1 to 2 mins.  Flex takes a LOT longer, think 30 mins
3) GAE can use imagick, imagick is not fast, and cpu intensive , fetching an image from a GCS bucket and rendering it cropped or resized is a 1-2 second operation.

you could use appengines image api, but for php it's limited to getservingurl functions. unless you want to write protobuf interfaces .

In python, it's way more useful, you can do all the usual transforms, and it's not cpu intensive on your appengine instance, meaning you don't scale to 10k instances.  However it's not as fast as GCE , as below

So instead, I used GCE because

1) you can use libvips instead of imagick, 400ms for image resizing and serving, compared to 1-2 seconds with GAE, or 800ms with GAE and image api
2) you can use preemptible instances 

The savings are huge, for my estimates, $400k + per year between GAE and GCE

two other findings, about php and the google wrappers;  they are cool, you can do things like

if (file_exists("gs://bucket/foo/bar"))

which is cool, right ? , not really, when you dig in.  under the hood, that does two bucket calls.  because it tests if it's writable, even if your operation doesn't care about writes.  It's more efficient to use curl, and test the contents returned etc, and then use the contents later in you code.  This becomes more apparent (it did for me) when using a GLB, with nodes around the world. users hitting nodes in Australia hitting a multi-region bucket in the US, were a low slower, than the same users hitting a node in the US.  This is due to the multiple transcontinental bucket calls.  using curl and a single call eliminated the performance differences

Monday, May 21, 2018

load testing

I have used apache bench, and more recently artillery.io tools, but it still wasn't ideal.  I came across this tool which will do a better job for basic testing https://github.com/wg/wrk

Thursday, March 29, 2018

Sky Q tuning with a Meter

Using a sky Q compatible LNB, you can still tune with a regular digital tuner.  The analogue inline tuners won't get enough voltage to work right.

I use a satlink ws-6933, and set it to a vertical transponder and hooked it to the vertical output of the LNB, without doing that you will get a signal measurement with no quality measurement

Below are the screen shots of sky Q signal on the meter and in the sky Q UI too.



Wednesday, January 31, 2018

FreeBSD in GCE

Yes there are supported images of amd64 freebsd available in GCE to boot, but I needed an i386 variant so here are the steps I took.


I downloaded the mini memstick image, converted it to a disk format virtualbox recognised

VBoxManage.exe convertfromraw FreeBSD-11.1-RELEASE-amd64-mini-memstick.img FreeBSD-11.1-RELEASE-amd64-mini-memstick.vdi  --format vdi

Then made a virtual machine with that as it's disk, booted and chose the shell option, then remounted as read/write, and put this into /boot/loader.conf

mount -o rw /dev/ufs/FreeBSD_install /

boot_multicons="YES"
boot_serial="YES"
comconsole_speed="115200"
console="comconsole,vidconsole"

then shutdown the machine (gracefully!)

now I have an image that supports install via a serial port, just need to make it the right format for GCE to consider as a boot disk

VBoxManage.exe clonehd FreeBSD-11.1-RELEASE-amd64-mini-memstick.vdi disk.raw --format raw

tar -Sczf /tmp/compressed-image.tar.gz disk.raw

now put compressed-image.tar.gz in a bucket and create an image from it.

create a virtual machine with this as the primary disk and a 20GB second disk. Boot , and connect serial port , and install to the second disk.  before finishing the install make sure you update fstab on the second disk, so it can be used as a primary disk.

lastly fixup your nic MTU

ifconfig_vtnet0="SYNCDHCP mtu 1460"


Thursday, November 30, 2017

Controlling an LG soundbar

As part of my home automation configuration, I wanted to figure out how to control my soundbar.

Seems it's not documented, there isn't an API, but there is an Android App.  So I installed Genymotion, set the nic to be bridged and used wireshark to sniff what the app was doing.  I saw udp packets for each key, so copied the data, and used it like this....

Mute: 080312002a0808011204081a1002 followed by 080312002a0808011204081a1001 
Vol down: 080312002a080801120408191002 then 080312002a080801120408191001
Vol up: 080312002a080801120408181002 then 080312002a080801120408181001
Power off: 080312002a080801120408011002 then 080312002a080801120408011001 
switching inputs (optical/bluetooth/home):  080312002a0808011204084d1002 then 080312002a0808011204084d1001

So, to send this udp data to a sound bar at 1.2.3.4 address 

echo "080312002a080801120408011002" | xxd -r -p > /dev/udp/1.2.3.4/9741 

netcat seemed quite slow for some reason

Tuesday, November 21, 2017

caching a webcam

I Recently tried to use a webcam on the internet and it was unusable because too many people were using it.

Taking a peek, it was clear it was simple jpgs being refreshed by client side JS , using the built in webui in the camera itself, the hardware just couldn't handle too many users.

So I setup some urls and cloudfront to cache, and then an nginx proxy which would do the right thing with headers to make the images cache friendly.  Then reached out to the webmaster and talked them through what to do

Next step is to consider changing the client JS to not just increment a number in the url, which is ignored by the CDN, but maybe find some other way to cache to you get better than one frame a second.


server {
listen  8888;

access_log      /var/log/nginx/cam1.access.log;
error_log       /var/log/nginx/cam1.error.log;

server_name cam1.test.com;
location / {

    add_header Pragma public;
    add_header Cache-Control "public,max-age=1,s-max-age=1";

    proxy_pass http://1.2.3.4:120/;
    proxy_set_header Host 1.2.3.4;
}

}


server {
listen  8888;

access_log      /var/log/nginx/cam2.access.log;
error_log       /var/log/nginx/cam2.error.log;

server_name cam2.test.com;
location / {

    add_header max-age 1;
    add_header s-max-age 1;
    add_header Pragma public;
    add_header Cache-Control "public";
    proxy_pass http://1.2.3.4:121/;
    proxy_set_header Host 1.2.3.4;
}

}

server {
listen  8888;

access_log      /var/log/nginx/cam3.access.log;
error_log       /var/log/nginx/cam3.error.log;

server_name cam3.test.com;
location / {

    add_header max-age 1;
    add_header s-max-age 1;
    add_header Pragma public;
    add_header Cache-Control "public";
    proxy_pass http://1.2.3.4:122/;
    proxy_set_header Host 1.2.3.4;
}
}

figuring out s3 bucket size