Part II - The Ultimate Linux Home Router - Tumbleweed and Customizing the Environment

Friday, December 20, 2013

Part I - Tumbleweed (optional)

This is optional and you'll have to decide for yourself if switching your build to Tumbleweed is the way to go. Basically, this step involves "rebranding" your installation away from the openSUSE 13.1 and over to Tumbleweed. Tumbleweed does not have a version. When a new package of any kind is available for your installation, it will show up as an update. If you have automatic updates enabled, this will ensure you're running the latest version of everything, including the kernel, all the time.
There are a few reasons you may not want to do this:

You have some kind of kernel module or modification that needs to be compiled for each kernel. In my case, the external network adapter falls under this category. Every new kernel version will require me to recompile the kernel module and install it. If the kernel is updated automatically and the system is rebooted, it will revert to the module that's built into the kernel. If that module is the same version that exists today (and I have no reason to believe it would not be), my router is going to fail. It's a risk I'm willing to accept, since I will be routinely checking up on the system and installing the kernel module is pretty simple. I also, hope, one day to script this part out somehow.
Rolling updates can be unreliable. And, indeed, Tumbleweed isn't as widely used and tested as the default distribution.

Ultimately, I'm using tumbleweed because I'm choosing to expose some services to the wide open internet and I want everything to be as up-to-date as possible for security purposes.
That said, I don't believe any of the remaining steps explicitly requires you to use Tumbleweed.
Here's some more information so that you can make the decision yourself.

Switching up the repositories

Connect to your box via PuTTY and login.
We're going to be firing off a lot of commands that require root access, so step one is to:

$ su

Enter your password

# zypper ar --refresh http://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/ Tumbleweed
# zypper ar --refresh http://download.opensuse.org/distribution/openSUSE-current/repo/oss/ 'openSUSE Current OSS'
# zypper ar --refresh http://download.opensuse.org/distribution/openSUSE-current/repo/non-oss/ 'openSUSE Current non-OSS'
# zypper ar --refresh http://download.opensuse.org/update/openSUSE-current/ 'openSUSE Current OSS updates'
# zypper ar --refresh http://download.opensuse.org/update/openSUSE-non-oss-current/ 'openSUSE Current non-OSS updates'
# zypper rr openSUSE-13.1-1.10 repo-debug repo-debug-update repo-debug-update-non-oss repo-non-oss repo-oss repo-source repo-update repo-update-non-oss
# echo '[main]' > /etc/zypp/vendors.d/Tumbleweed.conf
# echo 'vendors = suse,opensuse,obs://build.opensuse.org/openSUSE:Tumbleweed' >> /etc/zypp/vendors.d/Tumbleweed.conf
# zypper lr

That last command will list out the repositories. You should see this.

# | Alias                            | Name                             | Enabled | Refresh
--+----------------------------------+----------------------------------+---------+--------
1 | Tumbleweed                       | Tumbleweed                       | Yes     | Yes
2 | openSUSE Current OSS             | openSUSE Current OSS             | Yes     | Yes
3 | openSUSE Current OSS updates     | openSUSE Current OSS updates     | Yes     | Yes
4 | openSUSE Current non-OSS         | openSUSE Current non-OSS         | Yes     | Yes
5 | openSUSE Current non-OSS updates | openSUSE Current non-OSS updates | Yes     | Yes

If you're missing any of those, run the appropriate "zypper ar --refresh" command above to add the missing repository. If you've got more than what is above, run zypper rr where is the name of the alias in the your list that isn't in the one above. This is very important. You won't be able to reliably upgrade the distribution if you have more than these repositories enabled.
Now run:

# cat /etc/zypp/vendors.d/Tumbleweed.conf

You should see:

[main]
vendors = suse,opensuse,obs://build.opensuse.org/openSUSE:Tumbleweed

If you see something different, run "rm /etc/zypp/vendors.d/Tumbleweed.conf" and issue the "echo" commands above again (along with the cat command to ensure it's correct).
We're ready to refresh the repositories and convert this install to a Tumbleweed install.

# zypper refresh

You're going to get prompted several times with a message similar to this:

Retrieving repository 'Tumbleweed' metadata ---------------

New repository or package signing key received:
Key ID: xxxx
Key Name: xxxx 
Key Fingerprint: xxxx
Key Created: Mon 18 Feb 2013 12:09:00 PM EST
Key Expires: Wed 29 Apr 2015 01:09:00 PM EDT
Repository: (repository name)

Do you want to reject the key, trust temporarily, or trust always? [r/t/a/? shows all options] (r):

Type "a" and hit enter each time.
This could take a few moments depending on your internet connection or server performance. You've already had a cup of coffee, either switch to wine or surf Hacker News or something while you wait. When it's finished:

# zypper dup
# reboot

You've probably gotten a new kernel at this point, so it's a good idea to reboot the bugger. Hopefully, you get the same IP address from DHCP. If not, log in locally and:

$ sudo /sbin/ifconfig

To find your IP address.

Part II - Fish Shell (optional)

I have been using Fish Shell for a while. It's a great productivity enhancer for the infrequent sysadmin. It's by no means required, but I strongly recommend it. More about that here.
If you visited the site you probably discovered that there's an easy way to install via zypper. I haven't been able to get this to work in quite a while on openSUSE, so we're going to build it.

# zypper in git ncurses-devel autoconf gcc-c++ cmake lynx
# cd ~
# git clone git://github.com/fish-shell/fish-shell.git
# cd fish-shell
# autconf
# ./configure && make && make install
# echo '/usr/local/bin/fish' >> /etc/shells

Note: We've installed a C++ compiler, autoconf and cmake, which are development tools that allow for compiling of applications. Generally speaking, it's a bad idea to leave these installed on a production machine. Later you might want to remove these packages if you don't need them (and you can take that advice for any package that you don't need).

Set Fish Shell as the Default Shell (Optional)

Fish is not precisely compatible with bash. Normally this isn't much of a problem provided each script you need to run identifies itself as a bash script. I have never had an issue setting fish as the default shell, but if you're concerned you can skip this and invoke fish by simply typing "fish" at a $ or # prompt. To set it as default:

# chsh -s /usr/local/bin/fish
# exit
$ chsh -s /usr/local/bin/fish
(Enter your password)
$ su
(Enter it once again)

Syntax Highlight in Nano

I prefer the GNU nano text editor over the more common vi and others. It's a simple 'notepad'-in-text-mode-like editor. The openSUSE includes a number of syntax highlighting options, however, they're not turned on by default and there are some missing. We'll download new syntax highlighting rules and configure the environment to use them as well as add a command to your profile to prevent nano from wrapping lines (fatal to configuration file editing).

# cd /usr/local/src
# git clone https://github.com/nanorc/nanorc.git
# make
# echo 'set nowrap' >> ~/.nanorc
(enter your password)

Other Useful Packages

# zypper in man

Part I - The Ultimate Linux Home (and possibly Small Business) Router based on openSUSE

Introduction

This is a multi-part post that walks through the creation of what I consider to be "The Ultimate Linux Based Home Router". Of course, you're feelings on what is truly The Ultimate will likely differ from mine, however, since this is my blog, I'm defining.
My apologies for typos or bad grammar along the way. My secret real reason for putting this together is to ensure that I have a log of what I did. I have very little time to devote to blogging, so proof-reading has only been partially done.

Distribution

I picked OpenSUSE: Tumbleweed. There are many Linux Router distributions like ClearOS and such that offer turn-key solutions. I tried them and didn't like that a lot of the features I was interested in were add-ons that cost money. These guys offer a clear advantage--they're really dead simple to setup and maintain. If you want easy, go that route. If you want free, and want to learn a bit along the way, follow along. My choice of openSUSE is purely because I'm comfortable with it. I've been working with OpenSUSE for about a decade.

Features - Work in Progress

High performance routing and all of the features you'd expect from a home router, DHCP, DNS, etc.
Active Directory without Windows Server
NTP Server
Certificate Server
Authenticated transparent proxy with filtering for the children
Advertisement filtering and privacy enhancing capabilities for every device on network
Secure Shell with port forwarding
Traffic shaping
Intrusion Prevention
Guest wireless access for children's wireless devices and other untrusted devices (these will be filtered the most aggressively, including an SSL interception/replay with a local certificate from our new CA)
Home web server

At the time this was published, advertisement filtering, secure shelling, NTP and Active Directory are completed. I ran into some issues with being double NATed that have to wait until Comcast is open so I can have a static IP provisioned. I'll publish the remaining steps as I get them completed.

But Why?

Partly because I can. Mostly because this year I remarried and added two (amazing) children to my life. One of them is in second grade and is just starting to regularly use the internet. He also has an older friend across the street who has no internet access, so his phone is hooked up to my wireless. We're nuts in this house about adult supervision on the Internet (yes, we're those kinds of parents). At the same time, I know we won't be around all the time, so I want my children's (and their friend's) devices to be highly restricted. Filters suck. And I know there will be times when I will need to open up access to some things, so by adding authentication to the proxy, I or my lovely wife will be able to login with an account that is less restricted.
As far as wireless is concerned, I hate the idea of giving out my wireless password to devices I don't own. Google's Android, by default, uploads the wireless keys and stores them in a retrievable format, meaning Google essentially has the keys to everyone WiFi network any android device has connected to (there's an opt-out to this feature, but most people don't). We opt out on our main network with our devices. Any other devices that want access will have to go through our guest network, which includes an SSL intercepting firewall (don't worry, I'll make sure to document how to skip that feature).

Hardware

I repurposed an old MiniITX media center PC. It's got a lot of hardware that is unnecessary, but it's powerful enough to fit the bill for a solid home router. The specs are:

2.4 GHz Intel Core(TM) i3 CPU M370
4GB onboard memory
Built-in Ethernet and Wifi
USB 3.0 External Ethernet

BIOS / Hardware Preparation

This is going to be different for everyone, so you'll have to hunt around, but here is what you want to configure. Depending on your hardware, some of these options may not be available. Don't worry, they are all optional.

SATA should be in AHCI (not IDE) mode.
Power on when power is lost.
Decide what to do about HyperThreading
Disable any hardware that you won't be using (Parallel ports, sound cards)

If you can't switch SATA to AHCI, later in this document when I'm referring to the main hard drive, I'll be calling it 'sda'. This may show up as 'hda' for you. Just switch it around. With regards to HyperThreading, most of the recommendations for Linux Routers advise to disable it. On my previous home-grown router (on much less powerful hardware), I ended up enabling it because it routinely fell over with too much traffic when it was disabled. I am disabling it on this hardware.
At this point, I do not have the USB Ethernet adapter even plugged into this machine. I also have an existing router in place that I do not want to disturb, yet (that way my other computer can continue to surf the internet while I'm building this new router). If you have more than one Ethernet adapter on board, make sure you only plug one cable in and make sure the cable is plugged into the ethernet adapter that you want to be your Internal/Non-internet facing adapter.

The GUI Installation

Download the correct version of openSUSE 13.1 for the processor that your new router has installed. Burn it to a DVD or follow the instructions to make a thumb drive out of it. Boot!

Welcome Screen

Set your language and accept the license agreement (don't worry, you're not signing your life away!)

Installation Mode Screen

Select New Installation. Deselect Use Automatic Configuration

Clock and Time Zone

Select Hardware Clock Set to UTC. I'm not sure that this matters, but being a guy who does a lot of work with data, I take a hard line on the idea that UTC is the only way a date/time should ever be stored.
Select Change.... Hidden behind this menu is the NTP settings. Since we're going to be configuring this server with Samba 4.1 and Active Directory, time synchronization is very important.

Change Date and Time

Select Synchronize with NTP Server. Select an NTP server and select Save NTP Configuration. I've always used us.pool.ntp.org.
Select Run NTP as daemon and finally Synchronize now for good measure.
Select Accept

Desktop Selection

I wanted the server to have as many resources available for actually serving things, so I went minimal here. You may decide you want the GUI, and if so, KDE is the default and I'd recommend sticking with that based on my past experiences with openSuSE.
Select Other.
Select Minimal Server Selection (Text Mode).

Suggested Partitioning - Reloading an existing Windows or Linux machine from scratch

First things first, if you're reloading a machine that has a different operating system on it, you might miss the fact that the openSuSE installer is likely trying to preserve your existing partition. You also, likely, have different hardware than me so the partition strategy I used might not fit. Use common sense, or do what I did and prefer shiny new things. Since I'm writing this as much for me -- to track what I did -- as for you, here's what I did.
There are some important things to consider here. I'm running on one high-performance SSD and have no option to have a second disk installed internally due to the form factor of the case I am stuck with. If you have an SSD and an HDD, you may want to partition out areas of the drive that experience frequent changes (/var comes to mind). This ensures logging and such isn't constantly writing to your SSD and reducing its lifespan. Select Create Partition Setup....
Select 1: (your disk details here)
Select Use Entire Hard Disk
I select Propose Separate Home Partition primarily because that's, minimally, how I've always done it. I also selected Use Btrfs as Default File System because... shiny. You probably should stick with ext4 for everything.

Create New User

Fill this out with values that are meaningful to you. If you're going to enable SSH and open it up to the world -- and allow password authentication via SSH, you'll want to select a miserably difficult password.
Select Use this password for system administrator.
Select Receive System Mail.
Deselect Automatic Login.

Installation Settings

We're going to customize the software later, but a few things are worth setting up here.
Under Firewall and SSH, select enable and open next to SSH service will be disabled, SSH port will be blocked. The rest of our customization we're going to do remotely via SSH, so we want this turned on.

Installation

Get some coffee...

Text Mode

If you've followed this exactly, you're rebooted into a text-mode YaST2 installation @ linux screen. If you installed KDE, you have something different. The options will probably be similar, but since I've elected to skip the GUI, I can't confirm that. Set your hostname and domain name. I'm using the same domain name that my Active Directory domain name will use. It's also a domain I registered to myself on the internet. If you're using a domain name that can be valid on the Internet, make sure you register it because we're going to configure the DNS to be split-brain and if someone else registers that name, you'll be unable to access that domain. Believe it or not, this was a problem I had with my last installation. Alternatively, you can use something that ends in ".local" and hope that ICANN doesn't open that up for new registrations.

Network Configuraiton

The external network card I'm using is problematic. The default support for it in the Kernel uses a kernel module that is really flaky and will simply "forget" the card, requiring a reboot to find it again. The manufacturer provides a driver that I'll be compiling and installing later, so for now it's unplugged and I'll only be configuring the adapter that's on board.
I also have a WiFi card in this device and at a future point will be configuring this router to act as a very filtered guest network. The neighbor kids don't have internet service at home and since they visit frequently, they have my WiFi password on their phone. Google's Android default setting includes uploading that password to Google where it's stored in plain text (unless you explicitly disable that), I'd like to have a separate, isolated, WiFi network for my friends/family/neighbors and children's devices. This subnet will also have the most restricted settings as it pertains to proxy filtering. I'm happy to let my family/friends borrow my internet when they're here, but I'd prefer if you weren't using it to download porn.
That said, the default settings here are fine. We'll fuss with most of this later, right now we want a bootable box that we can SSH into.

Test Internet Connection

Hopefully this works for you! In 12.3, it didn't for me. 13.1 appears to be perfect.
The installer will download updated packages. Considering 13.1 was released only a few short weeks ago, I'm impressed at how much as already been superseded with new packages.

Online Update

Let the update run now and select Accept when the list of packages requiring updates is displayed.

Final Steps

Allow the system to reboot itself and you might also want to eject that CD at this point.

Release Notes

Does anybody really read these? Use your other computer and look the release notes up online if you really want to see them.
Click next and finish and say hello to your login prompt (it may take a second or two to come up).
Login with the user name and password you set during installation.

Setting up / Checking SSH

I couldn't connect to the box via SSH despite selecting to enable SSH on the firewall at installation. Here are the steps I took to enable it via YaST2:

$ su
(Enter your password)
# yast

Head over to Security and Users.
Select Firewall.
Select Allowed Services.
Select Service to Allow.
Choose "Secure Shell Server"
Select Add
Select Next
Hit F9 to quit. Run the following command:

# /etc/init.d/sshd start
# chkconfig sshd on
# ifconfig

That last command will show you the IP address that was assigned via DHCP. If you intend, like I do, on doing the rest of this installation on a remote windows box, it's time to download PuTTY. Open putty, put the IP address from above in the Host Name field and change the following settings (optional - this is how I like it). On the logging tab, set Logging to "Printable Output".
On the Window tab, set Columns to 160, Rows to 96 and Lines of Scrollback to 99999 (Because 100000 is one line too many).
On the Appearance tab, Change the font to Consolas 9-point and Font Quality to Default.
Stay tuned for Part II

FIX: Visual Studio 2012 takes dramatically longer to build than Visual Studio 2010

Sunday, August 26, 2012

Symptoms

Aside from a very agitated developer, the solution or projects within the solution in question used Code Contracts in Visual Studio 2010 and the Code Contract library is not installed or working with Visual Studio 2012. You can verify that this is the case by simply pulling up a project's properties. If you have no code contract tab, you have no functioning add-on. Go and install it. (As of this writing, you will have to run devenv /setup from a VS2012 Command Prompt to make it all work)

The Fix

Install the latest Code Contracts. Reopen your solution. Check the Code Contracts page and make sure they are configured optimally (cache enabled and background processing enabled). It appears that even using identical settings to Visual Studio 2010, it still resulted in the compile taking far longer on Visual Studio 2012 with any code contracts enabled. Perhaps the Code Contract's library just isn't ready for VS 2012 RTM.
Though Code Contracts are a great feature, I simply didn't need them for this project so I disabled them entirely and my build times went back to normal. I'll follow up when I discover what the real cause was and am able to turn everything back on.

Getting Burned by System Center Configuration Manager (and some help to avoid it!)

Wednesday, August 8, 2012

A coworker sent me this great story about HP deploying a task sequence in Configuration Manager and destroying all (or at least a substantial number) of their PCs and workstations (other helpfuls for more details).

It's interesting in that it highlights a battle I had to fight a while back and goes back to a phrase I probably utter once a month. "SCCM is the most dangerous tool we own." Along with a couple of succinct examples as to what an administrator of SCCM could do in ten minutes to utterly destroy anything connected to it.

As the author indicated, business owners, users, and managers, often simply see it as a "patching tool" like Windows Update with a handful of other features. Microsoft has been doing a fantastic job with patch reliability, most people own Windows computers and understand what patching is and simply expect it works 99% of the time and therefore any tool associated with patching is assumed to be simple and elegant (two words most IT folks wouldn't immediately jump to when describing Microsoft Update, but I can't think of the last time my mom or dad called about a computer problem and the culprit was MU).

Also, as the author indicated, it's painfully easy to overlook something and accidentally deploy something unintentionally with a far greater scope than intended. In the story above it was a task sequence that included formatting the drive. Someone not familiar with SCCM or in a shop that doesn't use all of its features may scratch their head wondering why something like this would exist. It was unlikely that, as the author states, it was just a simple reformatting. It was more likely a whole operating system deployment.

Don't use it, it's too dangerous!

This is often the knee jerk reaction that organizations go to after a minor or major catastrophe, and I'm willing to bet that's what Australia's CommBank is wrestling with right now. I've even argued against the use of the previous version of SMS 2003 and "it's too dangerous" was one of several of the bullet points.

Out of the box, it is too dangerous. The way people typically architect the entire solution (which is to say, they don't) is too dangerous.

I'm going to go into more detailed steps with screens in a future blog posts assuming that they are still an issue in SCCM 2012 (and I have no doubt that some of them are) but I wanted to touch a few things that I did to mitigate the risk so that the benefits could be enjoyed.

Dramatically restrict access to the All System's Collection and make sure all administrators repeat "Don't use All Systems for anything, ever" at least once a day for a year.

At a minimum, access to do anything to or with this collection should be restricted to one or two people, preferably two people that don't actually work with the system day-to-day. SCCM has very granular access controls (so granular that few people bother to use it and I'm told it's been dialed back in 2012 to strike a good balance). The issue above was an administrator accidentally including All Systems as criteria for advertisement of a task sequence. This wouldn't happen where I work. Aside from the collection being restricted at the ACL, everyone who administers it understands the mantra of "Don't use All Systems."

Rinse and repeat for All Workstations and All Servers.

Every new roll-out of anything should be phased to collections with well defined membership criteria.

At a minimum, you need three categories for deployment. Since SCCM is targetted at larger organizations, you likely need more than six.

The first category is "hopeless victims". These are the workstations of your experts and volunteers. Include people that have regular backups and that understand, fully, the dangers. These folks should also understand that it is their responsibility to report problems --- any problem --- immediately, if they suspect it was from SCCM. Servers in this category would have to be pure development, with impacts to them being minimal if they went down.

The second category is "dev/test". These are servers your organization would survive a couple of days without at moderate/low impact. For larger organizations, this would be at least two groups.

The third category is "production". For organizations with redundant systems, I'd insert at least one additional category "Node A" of redundant systems, followed by subsequent nodes before going to applications that have servers that are a single point of failure.

That's the simplest implementation. On the workstation side, it's a good idea to create collection criteria that spreads user impact evenly across functional areas. Make this judgement based on the number of people that can be out sick for a day before a department fails. Don't deploy anything to more than that many of those user's PCs in a 24 hour period.

Make sure management knows the risks, understands, and is on-board. Make it formal.

Thankfully, the management staff from my level up is fantastic. They understood the dangers and were willing to sign off on a policy.

Here are our rules:

Nothing gets deployed company wide on the same day it's advertised regardless of its scale. Regardless if the roll-out is a screen saver for Marketing that goes to all customer facing users, a security patch that isn't being actively exploited or cannot be mitigated through other means or a general operating system deployment that goes to all workstations, the advertisement is at least a business day in the future. The reason for this is to give a window to account for administrator error. I've personally been saved by this rule. I advertised a full Microsoft Office 2007 Professional installation to nearly the entire company (at the time about a 1GB install, multiplied by ~5,000 workstations many of which didn't meet the minimum requirements for that version). That 24-hour buffer allowed me to review where the deployment was going, and reverse it.

Each roll out group is given one day's buffer. To the above point: The first group (the "hopeless victims") are the only ones to receive the rollout after one day, and they're given at least one day to provide feedback. If you've picked the right people for your hopeless victims, you won't have to send an e-mail to let them know to "watch out", they'll scream properly at the first hint of a problem. The reason should be obvious: containment. As you roll out, the risk for problems is highest initially. As each group is added successfully, the risk is reduced while the surface area is increased.

Policies are meant to be broken

No exceptions, except. Identify every exception you can. Some of these are personnel issues--Marketing wants a new screen saver deployed company wide, they just finished testing it and want it there tomorrow. For my own job protection, I wouldn't do something like this without C- level executive sign off. Decide what's enough accountability if things go horribly wrong. Most deployments of this nature are not emergencies, they're eagerness by people who don't know and shouldn't have to care about the risks (that's your job!).

The "real" emergencies almost always have more than one option. These are the "PATCH RIGHT NOW!" situations due to malware infection. Patching the problem is the most obvious solution, but during an emergency it's important to remember the bold friendly letters of The Hitchhiker's Guide to the Galaxy (Don't Panic!). The few minutes it takes to step away and analyse a problem are far more valuable than the hours or days it takes to undo your poorly planned solution. What are you trying to prevent? In the midst of an emergency, it's difficult to see beyond the "gut reaction" solution. (System 1 says "I'm trying to patch the vulnerability to prevent an infection", System 2 says "I want to minimize the impact to my customer's personal information/my business transactions/my (specific) intellectual property). It might be better to pull the plug to the internet for a few hours than to deploy a poorly tested solution. Understand the solution, rank your options from lest impacting/most effective to most impacting/least effective. Pick a few and start there. Much of this falls into having a good plan for emergency management that includes the "Who", "What", "Why" and "When" so you can figure out the "How" as the bovine excrement hits the rotating blades of the air circulation system. It's worthy of another post and I'll do my best.

And another thing ...

I have specifically avoided mentioning my employer. This is my experience and is not limited to my current employer. This is also my personal blog. It is not sanctioned by my employer. It is not written by me as an agent of the company I work for. It is my opinion. If you choose to take my advice, imagine that I'm a crazy person who has never seen a computer and has no business writing on anything computing related.

Out of respect for my best friend and coworker, everywhere you see "I", I should have wrote "we". My experience was a result of (at a minimum) one brilliant mind sharpening my own. I don't have permission to use his name (I haven't asked but will correct this post when I do).

And finally, at least some of the information presented has been gathered by the great number of other sources (through forums, blog posts and other heaping piles of awesomeness). But they weren't gathered today. They were gathered during crisis and combined with my experience, knowledge and sometimes just (Oh S*** Trial and Error). If you pioneered the above lessons, let me know. Send me a link and I'll update the post.

AppliancesConnection.com (and GE Capital) ... Adventures (and failures in) User Experience (Updated 1x)

Thursday, December 22, 2011

Background

After two service calls to fix an old dishwasher, I decided I'd had enough of my beautiful bride having to hand-wash 3/4 of what came out of our failing GE Profile dishwasher. I did some research and landed on a Bosch model that was both highly rated by its owners and recommended by Consumer Reports. The problem is that no local retailer carries this specific model. Being sensitive to the fact that I purchased the last dishwasher without enough research, I wanted this model. And heck, I buy everything else online, why not a semi-major appliance?

Solving Cart Abandonment at the Expense of an Angry Blogger

A lot has been written on preventing cart abandonment, and I won't say that they got it all wrong. I clicked "Add to Cart", did a quick retailmenot.com look. They actually have coupon codes named RETMENOT??, I saw this as funny and won't take issue with the whole "why don't they just offer that as a deal" element. Clearly they know a lot of customers are going to use that service to find coupon codes. They also didn't require me to set-up an account, and instead just e-mailed me a password (we'll skip the security implications -- that they're likely storing this password in plain-text in a database -- for another post).

When presented with payment options, I was offered 12-month financing if I filled out a quick credit app. I had intended on doing the equivalent of paying cash (I pay my credit cards off every cycle), but when offered an option to simply pay it off in chunks over a few months with no interest, my weakness to loss aversion kicked in and I told myself that funny little lie that somehow I'll pocket a small discount due to the interest earned with that money remaining in my investment account for a few months.

After completing my order and printing my authorization form as instructed (I felt dirty doing this, but I was on my bride's laptop which didn't have PDF Creator installed but did have a USB Laser printer attached). Then, I headed out for a small trip with the family. Upon returning, I discovered the order was on hold and I was required to submit proof of identification and fax or e-mail my authorization letter from GE Capital to AppliancesConnection.com. This seemed bizarre. I've got three other online accounts that I signed up and used same day and I've never been asked for such a sensitive piece of documentation. Coupled with the fact that they e-mailed me my account password, I was not confident about how this sensitive information was going to be stored. The inconvenience of having to scan this all in (and redact most of my drivers license) on what was in my mind "a done deal" was enough to make me cancel the order. Or, that's what I should have done. This dishwasher is hard to find, and it's the one I wanted. They were the only retailer of three that carried it and the only one with a delivery timeframe that was acceptable (my bride's poor fingers!). I'll likely never do business with them again, but they got this one.

Moan and Complain, that's what the Internet is for. STFU, how would you solve this?

This is a solved problem. Amazon.com, buy.com and newegg.com have figured it out. Amazon even uses GE! Granted, I don't know AppliancesConnection.com's balance sheet and negotiating position with their payment provider, but if this is GE saying "pay us more to eliminate hassling your customers" and they're doing so claiming that the fees are to offset additional fraud, they're lying. It's a revenue booster. I could have easily forged the parts of my license they required me to send. And in the end, they were delivering to my billing/home address, which GE verified during the credit check. At some point, a dude is going to be walking this product into my foyer and I'll be signing for it.
Shop for credit providers and find one that isn't stuck with policies pre-2002.
Negotiate a better or equal solution that isn't quite such an awful user experience. While still messy, AppliancesConnection.com could have requested a secondary credit account with matching shipping/billing information, and only require the added scrutiny if the item is not being shipped to a matching billing addresses. This seems like it would be more effective than asking for my ID with everything but my name/address redacted. Even that seems unnecessary, though.
At a minimum, ... prepare your customer for this. What followed after submitting my order was this strange progression of e-mails, one of which claiming that I had opened up a support ticket with the order (I was puzzled reading this on my phone). The credit authorization did have a section at the bottom informing the merchant to treat the transaction as they would if it were done face-to-face (laughable). I half wonder what would have happened if I had just ignored the e-mail. Would someone have called eventually?

So you jumped through the hoop anyway, STFU

You're right. At this point, I've attached the required information with the bits redacted. With how clumsy this was, I'm having second thoughts even as I write this. Will delivery scheduling be this messy? If one other thing ends up odd about this order, I'm cancelling it and probably going brick-and-mortar with my second choice dishwasher carried by a local retailer. I have a truck.

The difference: A delightful user experience

User Experience is the new customer service. If I complete a transaction and it's easy, or even delightful, it's the equivalent of being rushed to the front of the line and having a sales associate offer to help you load the product into your car. If, then, something goes wrong between the payment processing point and delivery that requires me to call customer service, I'm going to be far more forgiving and assume it's a one-off. Based on how that turns out, I'll probably do business with that merchant again. In fact, if the inconvenience is handled very well with discounts or other perks to offset the inconvenience, I may seek that retailer out first because they've now proven they know how to make things right when things go wrong. They'll be predictable if something like that inevitably happens again.

User Experience will probably be the only Customer Service I encounter when interacting with you. Do it like everyone else and I'll have my only incentive will be seeking out the best price. Do it right, and I'll start at your site and pay more for a product knowing the results will be predictably good.

Send weird, cryptic e-mail messages from do-not-reply addresses and make unusual requests for documentation, and you might get an ugly blog post on a blog nobody reads. Still, I've probably told at least 8-16 people about my only marginally bad customer experience.

UPDATE . . . 6:15 PM same day as post

The mystery solved

I kept thinking about this and it seemed so off that I had to review everything again.
After reviewing my approval documentation more closely, I discovered wording that implied I had actually applied for a more generic credit card (think Visa, Master, American Express or Discover card if nobody had ever heard of them). It's a GE Capital card (Ta Da!). So my card is accepted wherever GE Capital is accepted. Wait, what?! Where exactly? This is why I was asked for additional documentation during checkout. AppliancesConnection.com did what they'd be required to do if they were presented with a Visa/Master card that was in the just approved but not mailed yet non-card card state, so they were instructed to use the rather traditional protocol of requiring additional documentation ... except that method doesn't work online and it works even worse when the customer thinks they've just performed part of the check-out routine. Being a familiar, though infrequent experience, I would have understood what was going on if the GE Capital card was a Visa/Master/American Express/Discover Card. Perhaps there's a really good incentive (zero fees?) for landing in on the negative side of both a generic and a retail store-branded credit card, but I can't find one. Feel free to convince me.

This post was proof-read by my dog. Unfortunately, she died several years ago.

Subscribe to: Posts ( Atom )