Two more tools for loading data into Graphite:

Firstly, following up with the Pinoccio Gardenbot project, I was using a quick and shoddy Python script to push a single Pinoccio Scout’s data into Graphite.

Since I wasn’t able to find an existing tool to push scout data to Graphite I put together a small nodejs script that slurps all (numeric) scout data from the Pinoccio streaming API and forwards it to Graphite. I also used this as a chance to get comfortable with Node.js, and this represents my first published Node.js work.

Pinoccio2Graphite on Github

Secondly, another bountiful stream of data that I’ve been collecting recently is that of Docker container metrics. Again I took the oppurtunity to write the tool in the language of the resource I was supporting, which for the case of Docker means Go. Go is an incredibly fun language to work with once some of the core concepts crystallize. I look forward to using Go and some of the exceptional Go-built tools (etcd, nsq, more Docker!) for systems work of the future.

Docker2Graphite (I have SND: sysadmin naming disease) is a little less polished than the Pinocchio tool. Memory and CPU accounting metrics are collected, however blkio data is not yet available. Hopefully when all metrics are collected I’ll have a decent library for polling all types of sysfs metrics. This tool also represents my first public Go tool, and all comments and advice are appreciated.

Both tools run as foreground processes¹, be sure to use a terminal muxer (screen, tmux) or a flexible init (Upstart, Supervisor) to keep the processes running long term.

¹: Go’s flexible and (memory)cheap goroutines actually make daemonization extremely difficult

When it comes to digitizing, storing, and playing back multimedia: nothing compares to having an actual computer in your entertainment center. No console or proprietary media device can compete with the flexibility, power, and expressive input that a regular PC allows.

Thankfully building a home theater PC (HTPC) with ~10TB of storage will only set you back about $1200 these days, with storage prices always falling. Some expertise with hardware and operating systems is required, however many communities and guides can be found online. Building a media PC can be an excellent platform for learning about storage, media formats, and backup concerns.

Parts

• Case: selected not only for the number of drives it can hold, but for how quiet and form fitting the design is.
• CPU: For basic media boxes that won’t be doing video capture or modern 3d gaming, a CPU with an integrated GPU will suffice, and save from needing to buy (or fit into a case!) a graphics card.
• Motherboard: Motherboards can now be found with 6-12 SATA ports and either real or virtual RAID controllers built in. This greatly simplifies storage concern and saves on another PCI card.
• RAM: 8GB of “high performance” (but none of that “EXTREME” nonsense, or RAM with coolers twice the size of the chips) memory.
• SATA large capacity storage drives: As of this writing 4TB drives can be had for $150. Consider splurging for 2 spares and doing RAID6
• SSD medium capacity drive for operating system. Not a hard requirement, but it can boost performance of the entire system and simplify settin up and maintaining the storage array.

Our build

Here’s a (relatively out of date) list of parts for our build:

• Case: Fractal Design’s Define Mini This case wound up being larger than I expected, but not more than I needed. It has a little room to grow, and is exceptionally quiet, even with 5 spinning drives at work. Another unexpected delight was the cable tray built into the side of the case “under” the motherboard. The panel on that side also comes off, and a number of holes around the motherboard tray allow cables to be run in a separate (and hidden) space.
• CPU: Since I don’t do any video capture or plan to do any modern gaming on a living room television I went with a relatively new “integrated GPU” CPU. Specifically Intel’s i3-3225 Ivy Bridge CPU (with integrated Intel HD4000 GPU) . This CPU/GPU combo has been able to play every high def video I’ve thrown at it, even under the load of other encoding
• Motherboard: This was a bit of a tossup, since most of my favorite manufacturers have either changed ownership, names, or both over the past ~8 years since I’ve built a home PC (assisting IT has its material perks). I wound up selecting the AsRock H77M for it’s featureset, 6 SATA ports, RAID support, and other amenities.
• RAM: I went with 2x4GB of G.Skill Ripjaws series RAM. It won on price and not being wildly tacky.
• Storage drives: Amazon happened to run a deal on 3TB Seagate Barracuda’s which really started this whole build. I have monitoring enabled on the drives and the array, but no problems so far.
• Boot drive: I went with a Kingston V300 since it was on sale on NewEgg at the time. This drive doesn’t have to be reliable, and can be backed up onto the storage array.

Build

The actual build process was relatively straight forward, other than a number of rewirings in order to fully take advantage of the cable tray. If you’re new to PC building Tom’s Hardware has a pretty good guide for actually putting components together. As usual the only advice is: be wary of electrostatic shock, and DON’T FORGET THE THERMAL GREASE!

The BEAST

Check out that cable job!

Software

One aspect of building an HTPC that can be overwhelming is the sheer number of choices for software. Rather than view it as overwhelming, consider it for it’s freedom of choice. Once you find an established solution that works for you, make that your default, and never stop experimenting!

• OS: Entirely up to the comfort of the users. I would of course recommend Linux, however some especially draconian DRM schemes have yet to be implemented on Linux, meaning Netflix, Amazon Prime Video, and some other popular services are not available¹. Again the biggest influencing factor here should be user comfort.
• RAID: Yes, for consumer hardware in 2014, software RAID is a more than acceptable choice. For Linux users this means MD.
• Media Player: A program to play any and all file formats without its user interface getting in your way. Mplayer (an incredibly flexible media tool) and VLC (great at complex subtitles and handling multiple tracks) are always highly recommended.
• Media Center: The daily driver of a media box. There are many wonderful media center projects out there: XBMC, Plex, MythTV(a solid PVR too) and more. We’re currently using Plex’s hosted product for our media at home.
• Ripping Software: For backing up DVD media, nothing beats HandBrake
• Emulators: Don’t forget, most previous generation game consoles can be used via emulation software these days. With the hardware in this box we’re able to play all of our older console favorites from Atari to N64 without breaking a sweat. Getting ROMs is still a legal grey area, however even the Internet Archive will hook you up with torrents.

¹: Update 9/19: Apparently updates coming for the Linux NSS package will enable enough of the DRM that Netflix will now support Linux!

Welcome to the new Octopress powered era of timsogard.com!

While the previous site was beautifully designed, it was also powered by a combination of Blosxom (a painfully-dead-no-wait-sort-of-forked-and-community-maintained perl static blog system) and custom Perl CGI code. While I happen to love Perl, I would counsel anyone to steer clear of doing front end work with it. Thaggravations, combined with a minor incapacity to self document, led to a whole lot of nothing on the updates front.

I had been hoping to find a Python powered static site generator since it’s my strongest language, however Octopress/Jekyll is one of the most mature and strongly recommended static site generators out there (not surprising since it powers GitHub’s pages product). Also, being a relatively new project, and one that powers the homepages of a lot of open source projects, there are tons of clean and creative themes and styles available.

I’ll be porting the old design back onto this site at time goes on. Until then enjoy the clean mnml theme from Isaac Overacker.

Hobby electronics and hardware hacks have always impressed and inspired me, however it wasn’t until 2012 that I picked up a soldering iron to call my own. Since then I’ve been getting into drones and embedded hardware and dreaming about robots. I’ve even been sniffing around the internet of thangs movement that everyone is so exited about. Looking forward to collecting and using that data.

Earlier this summer, a friend from Sauce Labs told me about the Pinocc.io microcontroller. Pinoccio boards are small, Arduino compatible, 802.15 mesh networked (with wifi handoff) microcontrollers that can be expanded with stackable “backpacks”. Pinoccio boards are designed to be driven via ScoutScript through the Pinocc.io HQ. The Pinoccio HQ web interface provides a great set of tools for building simple Pinoccio projects and provides a websocket API which can be monitored for data from the scouts.

Here at home, Sean and I had some plants which were proving to be incredibly sensitive to moisture in both the air and their potting soil (such is life in the Outer Richmond) and I decided that building a Gardenbot on the Pinoccio platform would be a lot of fun.

Some of the tasks required for this build were:

• Building a sensor circuit with a DHT22 Temp/Humidity sensor, a TSL2561 light sensor, and a handful of DFRobot Soil Moisture sensors.
• Creating a custom Pinoccio sketch to incorporate the sensor libraries, poll the sensors and deliver custom reports to Pinoccio HQ.
• Writing a poller for the Pinoccio API to ingest data into Graphite.
• Setting up alerts based on moisture and humidity levels.

Sadly the project got put on hold awaiting a new tool (toy). However I wanted to put up what was complete and highlight some of the Pinoccio Arduino code. I received a lot of help from the folks in #pinoccio on freenode for the custom reports, and want to document it for others.

Custom Reports

While ScoutScript allows for sending custom reports, they are currently limited to a simple key-value pair, where both key and value must be strings. Submitting a multi-value report can still be done by hand building the necessary JSON string, however given the amount of work already going into the custom firmware sketch, it was easier to mimic the built-in reporting.

Building this functionality seemed straight forward, but wound up being enlightening to some of the inner workings of the Pinoccio firmware.

The biggest mystery was getting the JSON key strings correct. The Pinoccio firmware stores most strings inside a keymap, to allow efficient reuse. Adding a key to the key map seemed straight forward enough, and worked as expected through serial console testing, but the streaming API was still receiving gibberish. The trick turned out to be that the space saving string map functionality was used even for scout-to-scout messaging purposes. This meant that any lead scout which needed to accept custom strings had to have those strings in the same position in their key map. In order to satisfy this map synchronization all that is required is ensuring the keyMap() calls happen in the same order at startup. See the setup{} blocks in the Gardenbot-pinoccio.ino and lead_garden_sketch.ino files for more.

Another gotcha was that the Pinoccio StringBuffer class uses vsnprintf, which within the Arduino libraries only supports string and int types by default. In order to work around this I had to convert float values into strings before sending them into the API report. The Ardunio avr-gcc libraries have a dtostr method which handles this. This was used for all the values coming from the DHT sensor.

Many thanks to @soldair and Blathjis from the #pinoccio IRC channel for their timely help.

Gardenbot-Pinoccio on GitHub

Graphite is salvation from the metrics systems of old, which demanded m4 macros, XML data definitons, and the blood of your first born. Graphite only requires an initial retention config and accepts plaintext. Getting data back out again is also a breeze: after building up graphs in the built-in composer, you can get the take the graph image URL and add &format=json to get the plot points in json.

I needed a Graphite instance for personal projects and I’ve been on a kick of containerizing everything, I built a Dockerfil for Graphite 0.9.2 via nginx/uwsgi in a container. The source is available at GitHub: drags/docker-graphite. Docker index: drags/graphite¹

Check out the README for setup instructions. Open an issue for discussion.

¹: This is just a personal build of Graphite in Docker and does not support clustering. Check out other images on the Docker registry for some excellently maintained containers.

I really like arena shooters, especially from the iD Software heritage. One of the great things about iD is that they tend to opensource their engines after 5 years, which is great not only for the long term player communities; but provides the base of numerous 3rd party and fan projects as well.

Along with the engine code, a lot of multiplayer assets for the older games are also free to use, and servers can be run without requiring a license. Add community made mods, levels, and models and you have a deep pool of interesting games to pick from, all free.

I’ve started packaging these shooters into Docker containers so they can be set up by anyone on the quick. The first game to be containerized has been my longtime favorite: Quake II. Here’s the docker index build for drags/quake2. As usual the source is on GitHub: drags/docker-quake2

Next on the list is Warsow, followed by some iD Tech 4 derivatives: ioquake3/openarena, CPMA, Urban Terror

At awe.sm we had a small team, but a proportionally large number of git repositories to keep track of (we strove to be an SOA and had a love of composability). We wanted to see repo activity interspered with deploy and monitoring announcements from our chatops bots. Initially we tried general git repository watching plugins, but the overhead of tracking ~100 repositories was unreasonable.

Luckily GitHub provides the events API which emits messages for all repo activity: pushes, pull requests, issues, team membership, forking, starring, and more. Building a simple scraper with SupyBot and Requests was short work.

SupyBot is a decent general purpose IRC bot, and Hubot has become somewhat of a standard in the chatops space; nevertheless, I wanted to cleanup this plugin and throw it out on the net in case others have the same need.

Get it on GitHub: drags/supybot-github-event-announce

Absolutely solid SolidSteel show from DJ Irk, recommended by @SeanJCourtney

I love a good gif

Shoreleave is one of the best characters on Venture Brothers, which puts him in the “god tier” as far as cartoon characters go.