PneumaticTube Update

Last year around this time I created a little command-line application to upload files to Dropbox from Windows. Since then I've added a few features, so I figured it was time to post an update.

The biggest change is support for chunked uploading. Dropbox requires chunked uploading for files larger than 150MB, and even for smaller files it's a nice feature to have because you can get progress updates on your upload. This feature took a little work, because the original library PneumaticTube was based on, DropNet, didn't have any async support and its successor, DropNetRT didn't have any chunked uploading support. But after a little porting work, PneumaticTube can handle uploading large files just fine and now has a couple of different progress indicator options.

The other big feature change is the ability to upload a whole folder to Dropbox. This makes PT a much more convenient tool for backup operations. Right now it only supports the top level of a folder (no recursion), but that may change in the future.

There've also been several bug fixes. One of these bugs was somewhat surprising: the progress indicators during chunked uploading caused crashes under PowerShell ISE. I'm posting a little more about this bug because info about it was scarce on Google, and maybe this will help someone else who has a similar problem.

In order to show upload progress indicators in-place (rather than writing "X % Complete" on a new line over and over again), I use Console.SetCursorPosition to move the cursor back to the beginning of the line and overwrite the old value. It turns out that the PowerShell console in PS ISE isn't a real console; among the differences between the standard terminal and ISE is the fact that ISE doesn't support SetCursorPosition or other cursor operations. Calls to those methods will throw IOExceptions.

PneumaticTube now handles this case by trying to access Console.CursorTop in the constructor of each progress display implementation. If access to CursorTop throws an exception, the progress display classes stop trying to report progress using those methods. This keeps scripts which call PneumaticTube under ISE (or another console which doesn't support those cursor operations) from failing. I also added command line switches for disabling progress reporting entirely as a workaround for other possible console support issues.

If you want to give PneumaticTube a try, you can download it directly from the releases page or install it using chocolatey. Happy uploading!

Writing ID3 Tags for Windows From the Command Line

I will preface this by saying the reason I needed to figure this out is a little dumb. See, I'm one of the few remaining people in the world who still has a Windows Phone 7 device. I bought an HTC HD7 back in 2011 and it's still going strong. I've been eyeing the 8.1 phones for a while, but I haven't been able to justify a new phone while this one still works. So here I am, still putting podcasts on my phone with the Zune software, like an animal.

The Zune software is really my only complaint about the experience, and for a while I avoided it completely because the phone is capable of downloading podcasts via wifi; since I wasn't using it for music, I never had to see the Zune logo and all was well. There were just two problems:

1. There's no way to add an arbitrary RSS feed to the phone's podcast list. If the podcast isn't in the Zune Marketplace, the phone can't get it on its own.
2. Once in a while, it would be in the process of downloading podcasts right when it lost wifi signal (if I picked it up and walked out the door at the wrong time), and the downloads would get stuck in limbo and require manual cancellation.

Someday I might write about the incident that finally tipped the scales and made us decide to eliminate iTunes from our household; it's a long tale of the woes of those of us who refuse to declare our allegiance entirely to just one of the feudal lords. Suffice it to say that I was already moving to sync my iPod Shuffle using MediaMonkey to a DRM-free music collection stored on our Synology. And the Synology is perfectly capable of handling the podcast downloads (also synced to the iPod). And the Synology is totally comfortable with arbitrary RSS feeds. And the Synology doesn't get stuck in the middle of a download if it loses connectivity. So the problem was now 95% solved; the last 5% is the part which might be useful to other Windows users out there.

The podcasts were on the Synology, and they could be synced over wifi to the phone via the Zune software on my laptop. As long as the phone was connected to wifi and plugged in to charge, it would magically get all of the podcasts synced to it.

But weirdly, the Zune software only seemed to recognize a few of the podcasts as being podcasts; everything else it thought was music. This meant it showed up with incorrect art on the wrong menu of the phone.

After a few blind alleys and a lot of messing around with settings, I figured out the Zune software's problem - it was making the incredible assumption that a file was only a podcast if the ID3 tag had "Podcast" as the genre. Depending on how you look at it, this is either perfectly reasonable or totally crazy. But from a practical perspective, it definitely wasn't functional. Lots of podcasts don't set the genre on their files at all; others set it to something like "talk" or "comedy". And woe to the Windows user, some of them have the audacity to use ID3v2.4 tags, the blindingly bleeding-edge standard from the year 2000!

ID3 (the "standard" metadata container for .mp3 files) is a mess. It's right up there with EXIF for being both staggeringly useful and horrifically inconsistent. It provides tons of useful data (artist names, notes, track info, album art, duration, ratings, and much, much more) in a theoretically cross-platform format which lives right inside the file. I say theoretically because there are disagreements between applications on what to store in which frame, some applications add their own arbitrary data, and not all applications agree on or properly handle the text encoding.

Anyway, the latest version (again, from 2000) is 2.4. But Windows only natively supports 2.3. This is why you can use a tag editor to add, say, a genre to an MP3 file and then look at it in Explorer (under Properties->Details) and see a blank genre. If you force the tag editor to use v2.3 tags, the metadata will show up just fine in Explorer. Why doesn't Windows natively support v2.4? No idea. I did some Googling, but no answers were forthcoming.

The Zune software also requires v2.3 tags (it probably just uses the native info from Windows). Which brings me to the problem: in order for these podcasts to show up correctly on my phone, I needed to set the genre and force the tags to be rewritten in v.2.3 (for the album artwork and other info to work). Preferably in a script.

As with just about anything else you might want to do in this world, there's a Python library for that. In this case, Mutagen, which can do pretty much anything you want with audio metadata. Putting together the actual script to do what I wanted ended up only being slightly harder than "import antigravity":

from mutagen.mp3 import MP3
from mutagen.id3 import ID3, APIC, TCON, TCOP, WOAR, error
import argparse

parser = argparse.ArgumentParser(description='Write the Podcast genre tag')
parser.add_argument('-m', '--mp3', help='Target mp3', required=True)
args = vars(parser.parse_args())

tags = ID3()
try:
	tags.load(args['mp3'], None, True, 3) 
except error:
	tags.save(args['mp3'])

try:
    tags.add(
        TCON(
            encoding=3, 
            text= u'Podcast'
        )
    )

    tags.update_to_v23()    
    tags.save(args['mp3'], 2, 3, '/')
except error:
    pass

I just run each podcast file through that script and it force-sets the genre to "Podcast" and rewrites the metadata as v2.3 tags. After that, everything syncs as expected.

I realize almost no one else in the world will have my exact problem, but I'm hoping this can help someone else who's fighting with ID3 tags on Windows.

By the way, the script works in Python 2.7.6 - I haven't tested on any other version. YMMV.

An Even Better PowerShell Forecast

After posting the last update to my PowerShell weather script, I was looking at the sheer awkwardness of the pre-made SOAP request. Basically, in order to send a SOAP request to the web service, I just kept a ready-made SOAP envelope in a file alongside the script, prepped it with some simple search-and-replace for the parameters, and used Invoke-WebRequest to POST it.

Here's the old code:

$uri = "http://graphical.weather.gov/xml/SOAP_server/ndfdXMLserver.php"
$lat = "40.019444"
$lon = "-105.292778"
$start =  Get-Date -format "yyyy-MM-dd"

$body = Get-Content .\ndfd.soap
$body = $body.Replace("[lat]", $lat).Replace("[lon]", $lon).Replace("[start]", $start)

[xml]$envelope = Invoke-WebRequest $uri -Method post -ContentType "text/xml" -Body $body
[xml]$weather = $envelope.Envelope.Body.NDFDgenByDayResponse.dwmlByDayOut.'#text'

It's awkward and very lazy, but it works. Still, it was bothering me. After a little more research, it turns out that if you've got a WSDL, you can just generate a proxy straight out of PowerShell (much the same way you'd generate proxy classes to work with a SOAP service from Visual Studio). The cmdlet you need is New-WebServiceProxy (naturally). Here's the updated code using New-WebServiceProxy and dumping the pre-made SOAP request:

$uri = "http://graphical.weather.gov/xml/DWMLgen/wsdl/ndfdXML.wsdl"
$lat = "40.0269"
$lon = "-105.251"Po
$start =  Get-Date -format "yyyy-MM-dd"

$Proxy = New-WebServiceProxy -uri $URI -namespace WebServiceProxy

[xml]$weather = $Proxy.NDFDgenByDay($lat, $lon, $start, 7, "e", "Item24hourly")

Just generate the proxy and call the method. Notice that this also means I get to drop the awkward line of code which extracted the XML I cared about from the return SOAP envelope. Much cleaner.

A Better PowerShell Forecast

In my last post, I wrote about getting the forecast from the National Weather Service from the command line in PowerShell. But the final version of the script I left off with wasn't that great; it basically just wrote a bunch of strings out to the console, which isn't a very PowerShell-esque way to do things. PowerShell likes objects, and it especially likes when we can commands like Where-Object and Select-Object to do interesting things with them.

So, let's update that script a bit so we get objects with properties back, instead of just strings. While we're at it, we can fix a bug with the old version - it's only checking the high temperature value for nil, and the web service we're calling can return nil for any of these values.

Here's the old version of the output loop:

for($i = 0; $i -lt 7; $i++){
    if(-not $days[0].value[$i].nil) {
        ("High of " + $days[0].value[$i] + ", low of " + $days[1].value[$i] + ", " + $days[2].value[$i].'weather-summary')
    }
}

Here's the new version:

$startDate = (Get-Date)

for($i = 0; $i -lt 7; $i++){
    $dayOfWeek = (($startDate).AddDays($i).DayOfWeek).ToString()

    $high = $days[0].value[$i]
    if($high.nil) { $high = "[unknown]"}

    $low = $days[1].value[$i]
    if($low.nil) { $low = "[unknown]" }

    $condition = $days[2].value[$i].'weather-summary'
    if($condition.nil) { $condition = "[unknown]"}

    New-Object -TypeName PSObject -Property ([ordered]@{"Day" = $dayOfWeek; "High" = $high; "Low" = $low; "Conditions" = $condition})
}

Let's break that down a bit.

First, we're adding $dayOfWeek to so that we can label the days; it makes the output nicer and helps me when I forget which day of the week it is.

For each of the other values we want to return (the high, low, and conditions) we're checking to see if the XML value returned by the service was nil; if so, we're just going to set the value to 'unknown' and move on.

The last line is the new and interesting bit. PowerShell lets you create custom objects on the fly with the New-Object cmdlet. And one really convenient option for that cmdlet is -Property, which lets you pass in a hash which will defined the properties of the new object. So, by calling New-Object in a loop like this, the output of the script is a list of objects which all have the properties Day, High, Low, and Conditions.

Even better, PowerShell 3 supports ordered hashes (notice the ordered keyword at the front of the hash). This forces the hash (and the object generated from it) to preserve the order in which we declared the properties.

The upshot of this is that when we run the script, the output looks like this:

Day                          High                         Low                          Conditions
---                          ----                         ---                          ----------
Sunday                       57                           34                           Chance Rain Showers
Monday                       58                           36                           Mostly Sunny
Tuesday                      73                           39                           Mostly Sunny
Wednesday                    79                           44                           Mostly Sunny
Thursday                     70                           40                           Partly Sunny
Friday                       67                           41                           Partly Sunny
Saturday                     70                           42                           Partly Sunny

And since each of those is an object, we can now pipe the output through other cmdlets, which allows things like:

.\forecast.ps1 | Where-Object {$_.Low -gt 40}

This command is piping the output through Where-Object and only returning the days with a Low greater than 40 degrees:

Day                          High                         Low                          Conditions
---                          ----                         ---                          ----------
Wednesday                    79                           44                           Mostly Sunny
Friday                       67                           41                           Partly Sunny
Saturday                     70                           42                           Partly Sunny

Much more useful. For extra fun, try .\forecast.ps1 | Out-GridView. Once you turn your script outputs into objects, you can do nearly anything with them.