Desra Blog

I recently faced dealing with some badbots and scrapers,  it was in a LAMP stack with varnish at the edge.  I decided to deal with it in varnish, as I always try handle as many tasks at the edge as I can, and leave apache to serve php. So, I thought about the problem a bit, and decided to use a token bucket, nothing unusual about that. (I had to modify the source to allow passing values instead of defaulting to 1 token).  However I went a bit further and decided that different pages are 'worth' more than others, i.e. they are more sensitive.  For example, accessing the homepage vs accessing account pages.  This required a patch of the throttle mod to allow you to pass the 'cost' of a page, so more than 1 token is removed from the bucket.  For now it just logs, but I intend to send a user that is exceeding the request rate to a different backend server that will give them fake data to devalue their scraping. you could detect user agent strings or other ...

The $GENERATE directive in a zone file includes nibble mode, which means you can do things like this ; reverse IPV6 zone file for example.com $TTL 2d ; default TTL for zone 172800 secs $ORIGIN 0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa. @ IN SOA ns1.example.com. hostmaster.example.com. ( 2003080800 ; sn = serial number 12h ; refresh = refresh 15m ; retry = update retry 3w ; expiry = expiry 2h ; min = minimum ) ; name servers Resource Recordsfor the domain IN NS ns1.example.com. ; the second name servers is ; external to this zone (domain). IN NS ns2.example.net. ; PTR RR maps a IPv6 address to a host name ; hosts in subnet ID 1 $GENERATE 0-4294967295 ${0,15,n}.0.0.0.0.1.0.0.0 PTR 2001-0db8-0000-0001-0000-0000-${0,15,n}.example.com. After adding the zone o...

Google puts them into SPF records ?! dig +short txt _cloud-netblocks.googleusercontent.com | sed 's/"//g; s/ip4://g;' | tr ' ' '\n' | grep include | cut -d ':' -f2 | xargs dig +short txt  | sed 's/"//g; s/ip4://g;' | tr ' ' '\n' | grep '/' responds with 8.34.208.0/20 8.35.192.0/21 8.35.200.0/23 108.59.80.0/20 108.170.192.0/20 108.170.208.0/21 108.170.216.0/22 108.170.220.0/23 108.170.222.0/24 162.216.148.0/22 162.222.176.0/21 173.255.112.0/20 192.158.28.0/22 199.192.112.0/22 199.223.232.0/22 199.223.236.0/23 23.236.48.0/20 23.251.128.0/19 107.167.160.0/19 107.178.192.0/18

I've been back to Freebsd lately, and using freebsd10 rc2. I noticed an option to compile access lists into the kernel module for ipfilter.  I got it to work, but thought I'd note down how ... firstly the kernel doesn't know the option to compile access lists so add this instead makeoptions     CFLAGS+=-DIPFILTER_COMPILED or you could add to /etc/make.conf now, take your ruleset in a file and create ip_rules.c and ip_rules.h  by doing this /sbin/ipf -n -cc -f <filename> copy them into  /usr/src/sys/contrib/ipfilter/netinet edit ip_rules.c and take out the if statement for NetBSD by deleting #if (__NetBSD_Version__ >= 399000000) #else then, if you have in and out rules, there is a bug in generating the ip_rules.c , work around it by adding frentry_t *ipf_rules_out_[1] = {         (frentry_t *)&out_rule__0 }; and adjust [1] to the amount of rules you have. then compile th...

I use centos in work, behind a proxy that uses NTLM auth. some tips; install CNTLM, which access a proxy to NTLM proxies :-) configure it so it listens for localhost connections /etc/cntlm.conf Username        username Domain          domain Password        password Proxy           upstream.proxy.addr:8080 NoProxy         localhost, 172.18.32.*, 127.0.0.*, 10.*, 192.168.* Listen          3128 Gateway yes Allow           127.0.0.1 Deny            0/0 edit yum.conf echo "proxy=http://127.0.0.1:3128" >> /etc/yum.conf edit maven settings.xml /usr/local/apache-maven-3.0.5/conf/settings.xml   <proxies>     <proxy>       <id>optional</id>       <active>true</active>     ...

Based on a clean install of CentOS-6.4-x86_64-minimal.iso , this step builds the RPM's and shares them via apache as a repo to install. --- /etc/init.d/iptables stop yum groupinstall "Development Tools" yum install unzip createrepo ws-commons-util wget java-1.6.0-openjdk-devel.x86_64 ant ant-jdepend genisoimage mysql mysql-server ws-common-utils MySQL-python tomcat6 httpd.x86_64 wget http://ftp.heanet.ie/mirrors/www.apache.org/dist/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz tar -zxvf apache-maven-3.0.5-bin.tar.gz mv apache-maven-3.0.5 /usr/local/ export PATH=/usr/local/apache-maven-3.0.5/bin:$PATH wget http://www.us.apache.org/dist/cloudstack/4.0.2/apache-cloudstack-4.0.2-src.tar.bz2 bunzip2 apache-cloudstack-4.0.2-src.tar.bz2 tar -xvf apache-cloudstack-4.0.2-src.tar cd apache-cloudstack-4.0.2-src/deps wget http://zooi.widodh.nl/cloudstack/build-dep/cloud-iControl.jar wget http://zooi.widodh.nl/cloudstack/build-dep/cloud-manageontap.jar w...

after a fresh install of  ubuntu 12.04 using  http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.0.0-incubating/html-single/Installation_Guide/#management-server-installation-overview works out like this echo " deb http://cloudstack.apt-get.eu/ubuntu precise 4.0" > /etc/apt/sources.list.d/cloudstack.list wget -O - http://cloudstack.apt-get.eu/release.asc|apt-key add - apt-get update apt-get install cloud-client-ui apt-get install mysql-server nfs-kernel-server cloud-setup-databases cloud:secret --deploy-as=root:password cloud-setup-management now go to  http://server:8080/client/ and login as admin/password you should read the manual and do nfs etc like this mkdir -p /export/primary mkdir -p /export/secondary echo "/export *(rw,async,no_root_squash)" >> /etc/exports exportfs -a put this into /etc/default/nfs-kernel-server LOCKD_TCPPORT=32803 LOCKD_UDPPORT=32769 MOUNTD_PORT=892 RQUOTAD_PORT=875 STATD_...