Adding a simple progress bar to dd

I recently ran into an issue where I wanted to move several KVM based virtual machines from one server to another server. There’s several ways you can accomplish this depending on what you want to do. In my case I was using LVM for the disk backend, so simply copying the disk image files wasn’t an option. It boiled down to two basic options.

  • Put system in single-user mode, rsync the contents over, and reinstall grub
  • Use dd and copy the whole LVM volume over piped through ssh

The advantage using the rsync method is that you’re only copying the files you need over, thus less data transfer happens. But then you run into needing to re-run grub (which generally isn’t a problem). In addition, if you’re using LVM within the LVM volume for the VM and the volume group is named the same, you run into some interesting issues. The advantage for using dd is that you can get a literal copy of the disk image and just start the VM back up without any other steps. Of course, this will only work if the volumes are the same on both ends.

So I decided to go with dd but ran into a problem of seeing the progress of a 15G volume copy. I did some digging around and found a blog post that mentioned using a command line application called ‘bar‘ so I decided to give it a shot! Its a fairly simple application that just creates a basic progress bar based on the data being piped into it. If you’re running Gentoo, the package is called app-admin/bar.

Here’s the command I ended up running:

dd if=/dev/lvm/cholula-disk | bar -s 15g | \
    ssh -c arcfour $host "dd of=/dev/lvm/cholula-disk"

When ran, it gives you output similar to:

6.0GB at   17.9MB/s  eta:   0:08:32   40 [=========================                 ]

The downside is that you need to specify the block device size before hand, but for something simple like this its quite nice. Of course I could just use one of the many dd forks out there which include progress bars but this is quick, dirty, and simple!

I used the arcfour cipher mainly to reduce the CPU overhead and increase the throughput, but you should probably never use this cipher on an untrusted network as it does have weaknesses. I didn’t try doing throughput tests on other ciphers, but it would be interesting. It took me approximately 10-12 minutes to copy a 15G volume over a gigabit network which isn’t too bad.

Another trick you can do is utilitize the LVM snapshot feature and create a snapshot of the running volume. If any data changes on the volume, it won’t be copied over obviously, but it will at least let you do a cold “live” migration of sorts.

Why you should go to Beaver BarCamp 3

Beaver BarCamp is tomorrow in Corvallis and you should go to it! Why?

  • Its completely free and open to EVERYONE
  • Free food and drinks!
  • You get a free T-Shirt (if you register)
  • Cool and interesting topics & ideas will be discussed!
  • Meet other cool people from the area

But what is a BarCamp exactly?

BarCamp is an ad-hoc gathering born from the desire for people to share and learn in an open environment. It is an intense event with discussions, demos and interaction from participants who are the main actors of the event. — barcamp.org

beaver barcampThis is the third incarnation of Beaver BarCamp in Corvallis and is bound to be the largest thus far. Many people I have talked to have a hard time understanding what happens at a barcamp, and if its only for technical people. In the past, our barcamp has been tech focused, but that’s been primarily because the outreach for the event has been mostly directed at OSU EECS students. Barcamp’s are designed so that the people who attend also are the presenters. The more people who attend, the more variety you’ll have at the event. You don’t need to have a full-fledge presentation prepared, just an idea, a room (which we’ll provide), and people to talk to!

How do you give a session? Make sure you get to Kelly Engineering at 10AM, put your presentation and name on a sticky note, pick a room & time and stick the wall. We’ll adjust rooms based on the popularity of the session to ensure there is enough room. We have all of the meeting and classrooms available in Kelly essentially, so we shouldn’t have any problems finding a room! Food and drinks will be provided for free thanks to the sponsorship of the OSEL for breakfast, lunch, and dinner. We’ll also have a snack break in the afternoon!

Some fun non-technical examples you could see at a BarCamp:

  • How to kayake
  • How to brew beer
  • How to grow chickens in your backyard

Don’t feel like giving a session yourself? That’s OK! We won’t pressure you at all. You can come and attend just one session to see what its like and don’t need to spend all day there.

I plan on giving sessions about the following tomorrow:

  • What is Calagator and why Corvallis needs it
  • What’s the next step for the Corvallis Social Tech scene?
  • Ignite Corvallis brainstorm / roundtable discussion

I hope to see you there tomorrow!

Get Involved: TrustTheVote.org

Government is something we often consider something we really can’t get involved in as average citizens outside of public office. But with the recent grass roots campaign that President Obama orchestrated, it goes to show that we as citizens do have the ability to affect influence in government. One of the major steps in creating a more open government is establishing a standardized set of guidelines and rules for digital voting in our country.

Recently at the OSU Open Source Lab, we’ve been in talking with the Open Source Digital Voting Foundation (OSDV) with hopes that we can help host some of their sites and infrastructure. What and who exactly are they? They’re a non-partisan and non-profit organization based in Silicon Valley that strives to try and fix many of the issues that we face today in digital voting. They have three primary goals.

  1. Resolving the root causes for the problems that exist in voting technology today,
  2. Developing guidelines for assessing, testing, and verifying whether an existing or new digital voting device, system, or service is truly high assurance and high veracity, and
  3. Defining, designing, and building a reference implementation of an open source high assurance digital voting system and service.

Excited? Interested in learning more? Live in the Portland, Oregon area? You’re in luck! The OSL and Portland Open Source Entrepreneurs (POSSE) is hosting an event at Cubespace this Wednesday from 6-8PM with a presentation by the OSDV founders. Their presentation will:

  • Introduce the project, its motivation, founding, and development efforts to date
  • Walk through the TrustTheVote technology road map and review major projects underway
  • Discuss development philosophies and approaches including experience-driven design and test-driven agile development
  • Review opportunities for systems architects, software developers, SDQA/test specialists, and user experience designers
  • Cover plans to expand the volunteer developer teams, future opportunities for senior members of technical staff, and opportunities for you to get involved

Get Involved! See you there!

The EeePC netbooks are awesome!

eeepc1000As some of you may have noticed, I usually carry around a tiny little black laptop when I’m at Beer and Blog or at a coffee shop. Since people ask about it so much, I decided to write a few blog posts about it with this being the first.

What is it? Its an EeePC 1000 from ASUS (the same guys that make motherboards). Its a new type of laptop that is generally referred to as a netbook mainly because it lacks some of the features that a normal laptop has such as a cdrom, being heavy, a multi-core power sucking CPU, and a huge screen. Outside of that, its like any other laptop with a few other enhancements. In my case, this EeePC has these features which I love about it.

  • Long battery life, claims one day computing (6+hrs)
  • Solid State Drives (SSD) - no moving parts & better battery life!
  • Multi-touch track pad
  • Very usable keyboard (92% from a normal laptop)
  • Lightweight (2lbs 15oz)
  • Fast 1.6 Ghz Intel Atom processor
  • 802.11n Wi-Fi and Blue-tooth
  • 1.3MP web cam

I’ve used my trusty 15″ PowerBook for many years and I still use it, but its become more of a hassle to carry around with me. I’m a UNIX Admin, so I don’t need much on my laptop (xterm, Firefox, Thunderbird, and pidgin) to get work done and the EeePC was a perfect fit. The first models that ASUS releases for the EeePC didn’t appeal to me since they had a screen and keyboard that was too small for me to deal with. A good friend of mine (who also has the habit of being an enabler for me :P) purchased one for himself and kept raging about it. I was actually considering getting an upgrade for my PowerBook but didn’t want to pay $2K for a new MacBook, so instead I decided to get this EeePC for  around $500 (its down to $430 now).

Asus gives you two options for Operating Systems on these laptops: Windows XP or Linux (Xandros). Since I’m a Gentoo developer, I decided to give it a try on this laptop. I encountered a few issues of course, but overall I have all the hardware working. The only annoying thing so far is the wireless driver not being included in the mainline kernel, but that’s going to change soon I hope. There is an open source driver but its a little finicky to deal with on networks using any form of security passwords. But it does work!

In the upcoming posts, I plan to write about the following:

  • How I installed Gentoo on it
  • Gentoo tweaks I use on it
  • Window management
  • Firefox tweaks (yes, you need them!)
  • How netbooks are helping promote Linux to the masses

I love my EeePC and you should get one too! :)

Switched to WordPress and here’s why

In my first post on my new blog I mentioned that I decided to use Drupal for my main blog site. As you can probably notice by the footers on this page I’m no longer using Drupal and decided to go back to WordPress. I noticed several people at the last Corvallis Beer and Blog asked me about it. That made me think I should write a blog post about my journey, or at least the short version of it.

There were several reasons for this but it basically boiled down to using the right tool for the right job. Drupal is an amazing platform for managing and generating dynamic html content. Its plugin and theming system is one of the best out there. Unfortunately, I’m no Drupal guru (even though we host them at the OSL) and had several issues getting some things working like I wanted.

So why WordPress? Well, it seems like everyone in my region raves about it plus I tried the 2.7 version and saw how simple it was to administrate and setup. The ability to install plugins without ssh’ing into the machine was  a key feature that I just loved. Here’s a few of the things that I really like about WP.

  • Installing/Upgrading plugins easily
  • Categories/Tags are easy to setup
  • TinyMCE is installed and works flawlessly
  • Comment spam is easier to handle
  • The editing flow is less busy
  • Found a theme that is simple, elegant, and easy to customize the header image

While I agree Drupal can essentially do all the things above, it requires more researching, googling, tweaking, and trail and error to get it right. I spent a week or two combing the interwebs trying to find a defining site or blog post that explained everything. I found several sites that had decent information, but it wasn’t explained in a manner that made sense to me. The whole concept of taxonomy is just strange to me and in WP its stupidly simple. To me Drupal is like the Gentoo of blog software. It’s great and awesome if you know how to tweak it and use it properly, but for a blogging n00b like me, its just too much hassle for me to do deal with. I think Drupal has some great documentation, its in no comparison to the simple ones like Gentoo has (which is a large part of why people can install it).

So what does Drupal need to do? A few simple things would have made my life easier:

  • Create a page or site describing in detail how to setup similar features that modern blogs like WordPress have (categories, tags, etc)
  • Implement module installation via Drupal
  • Implement module upgrades via Drupal
  • Create a Blog centric theme similar to K2
  • Make it easier for people to use Drupal but keeping it simple for normal bloggers

I may go back to Drupal someday but I need to understand it more. I really think Drupal is an awesome project and its ability is far superior than any other CMS out there. But for a blog its just a bit too hardcore for the average blogger. Hopefully that will change in a few years.

Come see me play my trumpet!

trumpetThat’s right, come see my musical talents next Tuesday night from 6:30 - 8PM at Charlies Rhythm and Brews in Albany, Oregon. This will be our first gig ever as a group! We’ve been working hard since last September or so working out a set and we’re finally ready for the big scene. The group I’m in is jazz combo called The Infallible Collective. The group details are as such:

  • Trumpet - Me!
  • Electric Upright Bass - W.G. “Malkiel” Davies
  • Guitar - Doug Meyers
  • Drums - Kevin Van Walk

We’ll be playing a mixture of jazz standards (with a twist!) along with some originals compositions by the group. The style of Jazz we’ll be playing will range from ’50s & ’60s bop to Latin Swing & Bossa Nova to modern trip-hop and jams. It should be a good time! I’ll be posting a Facebook event about it soon.

See you there!

Another great Beer ‘n Blog in Corvallis

Beer and Blog at Fireworks was awesome tonight! Thanks so much to Ocean for getting the pizza’s lined up for us. It never ceases to amaze me about the tech community here in Corvallis. It may be smaller than Portland, but its certainly just as full of energy and ideas!

fireworksTonight I overheard some discussions about getting other user groups going on a more regular basis centered around ideas created at Beer and Blog. I’m excited to hear new ideas like this coming out of these meet ups. It was certainly one of the goals and hopes I had when I decided to organize. Related to that, I’m planning on heading up to Portland on Saturday and help at the Calagator code sprint. While I don’t plan on helping with much coding, I do hope to get some feedback on some usability questions I had regarding a Corvallis Calagator based site I was working on. The Corvallis tech community is facing the same problems Portland had, but on a much smaller scale. There’s getting to be more and more events happening and people need an easy way to track it.

I’m actually hoping to utilize Calagator as more of a community driven calendar that includes events such as live music, arts events, and other community related events. While  I’m personally interested in a unified live music calendar, I’d love to see it used for a more wide range of events.

But back to Beer ‘n Blog tonight! It almost turned into a disaster for me tonight. I had just gotten to Fireworks when my pager started going off in a frenzy. By the end of it, my iPhone had about 100 text messages on it. I soon noticed that a group of servers in a set of racks next to each other appeared to have a power failure of some kind. Thankfully one of my students was still at work and took care of it. Almost everything came back online without any help from us which is amazing. I’m still waiting on the full story on what happened but it appears one of the electricians working on the power upgrade had tripped something on the power rail those servers were on. Talk about a headache! :(

Even though we didn’t have a real “topic” driven session tonight, I think a lot of people got help with their blogs and learned a few new tricks! Next week we’ll have Dawn Foster from Portland at Cloud 9 for a talk about Yahoo Pipes. I can’t wait!

A new era has begun

Yesterday was the inauguration of President Barack Obama, our next President of the United States.  Its so refreshing say that! I’m thrilled to see change in our government with progressive ideas. While I’m interested in seeing the important issues that plague our nation get resolved, I’m also interested to see how our new President will utilize information technology, specifically the use of open source. Throughout his campaign, it was known that he uses several open source technologies such as Apache and PHP for his main website and as well as change.gov. Its refreshing to know that he’s keen on realizing how important technology is on improving the nation. His first step was updating and improving the whitehouse.gov website by including a blog and several RSS feeds. While a minor detail, I think it’ll become very important in the years to come for his administration.

The grass roots campaign he ran was incredible and proved that the power of a community of people is powerful. The nature of his campaign acted almost like an open source project in the sense that it was a group of volunteers that had a very specific goal. CNN even noticed that correlation on election night by mentioning The Cathedral and the Bazaar, an essay describing the open source theology.

I can’t wait to see if Obama will do anything that would help promote and support open source in government and beyond. I’m very interested obviously because that’s what I do with my job, but I also feel that its important to help save tax payers money. I’d be ecstatic if his administration noticed what we’re doing here at the OSL and would help us out, but that’s a pretty far fetched dream. I have a feeling that he has far more important tasks to worry about, but one can dream.

Congrats President Barack Obama, you’ve proved to all of us that with determination and planning, anything is possible!

Happy Holidays, ruby on rails, and … thunderstorms?

I’ve been spending the last week back home in northeast Kansas with my family for the holidays. I thankfully missed the snowpocalypse in Portland by ONE DAY and had an uneventful flight to Kansas. If I would have decided to leave one day later, I probably would have been stuck in Oregon for another 2-3 days. Talk about being lucky!

Somehow I didn’t escape wacky weather totally. In the last four days, the temperature has gone from 15F to 45F to 67F to 27F. Only in Kansas will you have balmy weather one day and snow/ice the next (literally). Not to mention, I even got to experience a thunderstorm late last night! This type of weather never happens in Oregon so its nice to have some variety like this once in a while!

I’ve done a surprisingly good job of not doing any work related things while on vacation. The only things I’ve done have been taking care of a few minor issues and replied to a few tickets so that clients don’t think we’re totally ignoring them. I really needed this break so its been pretty refreshing.

The only tech related project I’ve been doing lately is getting familiar with calagator (an open source calendar aggregator used by the PDX tech community), and also even teaching myself Ruby and Ruby on Rails (RoR). I must be a nerd to use my downtime to teach myself something new like this.

I’ve always had this negative view on RoR mainly because of the past experience I’ve had and heard from other people. Setting up a server to run a rails app its quite different than something like php. Not to mention that there’s a lot of wrong ways to run the application. I’ve always heard and even experienced performance issues with rails applications, but it seems to have changed in the last year or so.

I’ve had several great online chat conversations with Igal Koshevoy in the last few days talking about the various issues that Ruby and Rails has had over the last few years. It seems as though the RoR community has worked pretty hard to try and combat a lot of the bad PR its had and fixed many of the technical faults it had. They’ve even gone so far as to fork the Ruby source into a project called Ruby Enterprise Edition. It includes some important performance fixes and other various improvements that benefit RoR applications. Overall, I’m fairly impressed with the design and simplicity of Rails for web applications. I can see the appeal for it from a development point of view. Now I just need to play with the backend some more to find an ideal way to host it.

I’m heading to Kansas City and Topeka in the next few days to see some more friends. I haven’t seen some of these friends in a few years so it’ll be a nice finale to a great holiday vacation! I head back to Oregon next Tuesday and can’t wait to get back to town. I actually miss the rain :)

Drupal for blogging

I’m attempting to get back into the blogging scene with Beer and Blog starting up and all. I decided to go with Drupal since we use it quite a bit at the OSL and I need some more experience with it. Its a bit overkill for a general blog, but being a UNIX administrator, I enjoy the customization and flexibility it provides!

So far I’ve enabled the following modules as they were recommended online quite a bit:

  • Pathauto
  • Mollum
  • Global redirect
  • Markdown

I’ll play with the theme as I go along but I wanted to at least get a blog up and going! If you guys have any suggestions for using drupal as a blog site, please let me know.

Woohoo!