Impostor Syndrome

I mentioned previously that I failed to publish one of my blog articles for over a year due to impostor syndrome, but then I realized that maybe some people don't know what that is or if I was even being serious.  So, first of all, yes, I was serious, sort of.  I do think I suffer from what's colloquially known as Impostor Syndrome, but not to the extremes that some do.  The basic gist of the problem is that I always worry that people will find out that I have absolutely no idea what I'm doing, despite having a fairly successful career as a software developer.  There's more information at Wikipedia if you're curious.  It's the opposite of the Dunning-Kruger Effect, in which incompetent people tend to overestimate their own level of competence, which sadly is what most really successful business people have from what I've seen over the years.  We, as a society, tend to reward overconfident incompetence instead of self-doubting excellence.  I've read that it's significantly more common in women than men, but I personally believe it affects far more developers than are willing to admit it, based on behavior I've witnessed over the years.  It could just be that more women are willing to admit to it.  I've decided to share my personal experience with this problem, what I've been able to do to overcome it to a degree, and how it still affects a lot of things in my career, usually for the worse, in the hopes that bringing more attention to the problem will help others who are similarly afflicted.  I'm going out on a major limb here and I hope that it doesn't collapse under my weight.

We've got wee, not so wee, and friggin huge

The sheer enormity of the volume of knowledge in Computer Science and software development is frankly overwhelming.  As the old adage goes, the more I learn, the less I know.  There are new developments constantly coming out, and it's impossible to keep up with them all, especially given my relative lack of formal education which has left me to pick up some of the more fundamental parts of the discipline in a more ad-hoc manner over the years.  I feel like I'm constantly catching up with where I should have been years ago.  It doesn't help that the landscape has changed drastically in that time. What was once considered PHD level material now seems to be considered by some to be basic knowledge.  Object-oriented programming, functional programming, reactive programming, aspect-oriented programming, set theory, graph theory, bayesian statistics, machine learning, AI, computer graphics, big data, distributed systems, operating systems, compilers, interpreters, security, encryption, dynamic typing, static typing, weak typing, strong typing, data structures, search algorithms, consensus algorithms, and on and on and on.  And that doesn't include all the peripheral knowledge like project management, agile methodologies, version control systems, bug tracking software, etc.  It's a lot to know, and it's seemingly more and more all required knowledge based on what you read from the peanut gallery.  And those are just a sampling of the things I either know or know I need to know, and I keep finding new examples all the time.  The backlog on my reading list is huge, and I don't have the time or energy to actually catch up.  It would take years of dedicated full-time effort, and that's without a job or a family to maintain.

This one goes to 11.  It's one louder.

So, that's the first part of the problem.  It's impossible to understand all these topics to a significant depth.  Some people are fine just skating by on the surface, understanding enough to talk about it, but I personally don't feel qualified to talk on a subject until I have really absorbed it.  And that takes time, and with things like programming, a lot of practice.  I'm stuck with an unfortunately limited subset of these topics that I really understand, and a larger subset that I sort of get but couldn't really speak to to any depth.  Some would say I'm lacking in some of the basics, but I also really know some of the not-basics, which makes it really difficult for people to gauge my competence level until they've actually worked with me.  My biggest concern is that I'll have a conversation like the one from Spinal Tap where Nigel keeps insisting that his amp is louder because it has an 11 on the knob, where others only go up to 10.  He knows enough about amps to know that higher numbers are louder, but completely misunderstands what the numbers actually represent and assumes that 11 is just louder.  Some people are content to pretend they understand the technology while spouting off incorrect information to people who don't understand it enough to call BS.  As Mark Twain said, "It is better to keep your mouth closed and let people think you are a fool than to open it and remove all doubt."  I opt to keep quiet, but when I hear people talking with confidence on so many topics, I falsely assume they actually understand what they're talking about and I feel lacking by comparison.  I project my ethics and thought processes on to others who don't necessarily share them.

How does he not get fired?

It doesn't help people with this problem that many developers exhibit a tendency which has been labeled as "feigning surprise" (see Hacker School Rules), in which they pretend to be shocked when someone doesn't know something that they have learned.  This has a severe negative emotional impact on people who already think they are impostors.  It makes them want to speak up less, and just retreat into their safety zone.  I personally think this can be pretty tightly coupled with impostor syndrome, but I can't necessarily prove it, it's just a hunch based on my own experiences.   I think, at least in some cases, that the person feigning surprise is not consciously faking it but is actually surprised that someone else doesn't know something they do.  They feel like they're a fraud, and certainly the other person shouldn't also be a fraud.  It's like a weird combination of nihilism and narcissism that doesn't make any logical sense to others, and it comes across as demeaning and only serves to worsen the problem.

This is an NP hard problem

I think a lot of my opinions on interviewing in my previous post revolve around this problem, honestly.  I tend to clam up in interviews when they hit an area I don't understand extremely well, out of fear that I'll say something extremely stupid on the subject and just cast away all doubt as to my imposterosity.  In reality, often times, they're purposely testing how the subject reacts to a situation in which they don't know the answer. Despite my logically understanding this, my psychological reaction is to flee inward rather than reveal my ignorance, so I look even worse by not working it out with the interviewer.  My whole career is just a house of cards waiting to crash down as soon as someone realizes I really don't understand all of the intricacies of directed acyclic graphs and paxos consensus algorithms. This can come across as incompetence in interview settings, and I've done poorly on a number of interviews and lost jobs because of it.

Nevermind the man behind the curtain

Given how interviews are conducted, I'm amazed every time I get to a new job how much it's basically like the old job.  I've had a couple times where I finally felt like I got in; I was now part of the elite who worked at these places with these ridiculous interviews.  I sure fooled those guys; they're gonna feel pretty silly that they hired me once they realize I'm not a demigod.  Come to find out that my new coworkers were basically of the same competence level as my old ones; some good, some not so good. I imagine it's that way at places like Google, Microsoft, and Amazon, despite all their pretense of hiring only the best of the best.  They still make stupid decisions and release the same crappy, bug-filled software as the rest of us.  They just don't do it in the open, so people romanticize that things must be perfect behind their shroud of secrecy.

I'm good enough, I'm smart enough, and dog gone it, people like me

So, what can someone with impostor syndrome do to combat it?  There are a few things I've found that have helped me immensely, but each is a constant battle.  Just being aware that I have this problem, and that it's a known, not-uncommon issue has helped tremendously.  I can objectively look back over my career and realize that I have actually done well at most jobs I've had. When I left one job, my boss told his boss that they were losing their best developer,  and at another job I was kept on after nearly everyone was let go in order to help transition things to the parent company. Even at the places where I felt completely unworthy after the difficult interview, yet somehow managed to land the job, I ended up being well-respected and valued because of my pragmatism, my attention to detail, and my ability to ramp up quickly.  You would think that would be validation enough, but I tend to dismiss those things because those people obviously didn't see how much I was struggling.  When I feel especially fraudulent, I remind myself of those experiences so I can feel more calm and confident, and it helps.

Another thing that has helped immensely is setting aside time to actually research areas where I feel I'm lacking. If I can at least have a cursory knowledge of an area, enough to actually have a decent conversation on the subject, I feel a lot less incompetent.  This has proven the most difficult, because it's often led to going down a rabbit hole of related subjects that I can never possibly learn to the degree I would prefer.  Nevertheless, it's still been a boon, and I feel like I'm in a much better position than I was a few years ago.  I'm working on being less dismissive of my own ideas, as it's turned out that I've had some good ideas over the years that have worked out pretty well, but I still have a tendency to discount my opinions or second guess myself.

Oh, he comes from a broken home.  So... no coffee then?

I guess that's all the advice I have.  There's no miracle cure here, but it is manageable.  Be objective, be logical, and remind yourself that you actually have done a good job from time to time.  Most of all, remember that nearly everyone else is faking it just as much as you are, if not more.  Some people are just better at hiding it than others.

Being a part of something special

In my career, I can think of only one time I've felt like I was a part of something truly special.  At Liquid Web, I was part of small team of developers and engineers that built out Liquid Web's cloud product called Storm on Demand.  It was a pretty amazing accomplishment for a handful of people over a thirteen month period from inception to public release.  Despite being a very niche player in the market, we were only a few months too late to be the 2nd public cloud provider after Amazon.  Unfortunately, during those few months, several other players also got their clouds out, so when we released, it didn't make quite the splash we had hoped.  Don't get me wrong, it was wildly successful, I just think some people had unrealistic expectations.  It was really an amazing accomplishment, and despite the fact that I've since left the company, I still count it as the best experience of my career thus far.

So, I think I'm about to have my second such experience.  At Rackspace, we've formed a Data Services "practice area" (practice as in law practice, not basketball practice, it took me a while).  A year or two ago, we acquired a startup called ObjectRocket that specializes in providing MongoDB as a service.  Their focus is on providing the best MongoDB experience out there and taking all of the headaches of administrating a MongoDB installation away from the developer so they can just focus on the code.  So now, ObjectRocket, the CloudDatabases teams, and CloudBigData team (I'm on this one) are joining forces to form this group.  There's a ton of work to do to get where we need to be, but it's exciting to be a part of, and even if we only accomplish half of what it has set out to do, it should be huge.

It's also possible that this ends up being a giant disaster and implodes.  Given my conversations with the leadership so far, and based on their personalities and management style, I don't feel like this is a big concern.  We've got the resources of Rackspace behind us, and we have the right type of pragmatism and excitement and technical know-how to make this succeed.

If this sounds like something you'd like to be a part of, please contact me.  We need your help.  I think this is a fantastic opportunity, and I'd love to see some of my friends and former coworkers (those aren't mutually exclusive sets) be a part of it.

Modern code review practices

I might just be curmudgeonly, but I'm about to the point where I won't even bother contributing to a project that doesn't use the Github pull-request model for code contributions.  There's so much pomp and circumstance with some projects, when they should be doing everything they can to welcome contributions.  Other similar systems like Bitbucket are fine, although Bitbucket's code review UI needs some help.  I've tried a lot of other systems over the years, and none of them match the simplicity and joy of Github.

The 90s called and want their code contribution practices back

I've recently begun trying to contribute to Ambari.  Ambari's contribution instructions read like a Microwave Oven's operator manual.  The process is so arcane that I'm surprised that anyone contributes.  You have to sign up for two services, JIRA and Review Board, and then you submit a patch to the JIRA ticket, as well as create a Review Board review with the same patch, and the same information as in the ticket, then link it back to the JIRA ticket and link the JIRA ticket to the review.  Here, read it yourself: Ambari's How To Contribute Page  I don't know if the Apache Foundation dictates this broken process or if the projects just take it on themselves to make the process horrible, but either way, someone needs to be informed that the 90s called and want their code contribution practices back.  Some of you might be saying, "what's the big deal?  Submitting patch files to two places and manually inviting people to review your code when you don't know who should review your code, then manually linking the review back to the ticket, and then enlisting yet another person to actually commit the code on your behalf isn't that bad" (if you're saying this, please punch yourself in the face and save me the trouble).  But there's also another catch that might be a little less obvious: attribution.  If you submit a patch, and then someone else actually commits the patch for you, the git log attributes the patch to the person who committed it, not you.  You may not realize this, but people like receiving credit for their work, even in Open Source.  Some people will simply refuse to contribute if they don't get credit, and frankly that's as it should be.  Can you imagine an actor or a writer doing work and then letting someone else put their name on it?  I can't.  Ambari puts your name on a contributors page, so it's not like there's no attribution, but that's not the same as having it show up on github, which is commonly used as a portfolio these days.

Barriers, barriers everywhere, and not a drop to drink

Openstack's contribution model isn't as terrible as Ambari.  It at least gets attribution right.  Openstack uses a system called Gerrit for code reviews.  You simply have to sign a CLA, sign up for Launchpad, download a magical tool called git-review (it handles all the arcane Gerrit shenanigans for you), and then run git-review on your committed local branch.  Then Gerrit takes it from there.  More info here: OpenStack's How To Contribute page You'll also need to read all of the coding standards, and all the information about how to format your commit messages, and oh yeah, make sure you know everything that's talked about on the mailing list because if you don't conform, your code will be rejected.  You'll likely not pass your code review and then have to re-submit the same branch for review (your new friend it git commit --amend).  Assuming you actually make it through the gauntlet of feedback and pass all the automated tests, which cover things like Openstack's extremely limiting and pedantic coding standards, then Gerrit will automatically merge it for you and push it up to Github.  So, it's usable, even if a bit overblown.  The biggest problem is that the process takes forever.  Even working with the same developer on an Openstack project and another project that's just on Github, it's amazing how much faster progress is made on the Github-based project.  I understand the need for verification of new code before it's released, but this is just a bit much.  Github lets you integrate with other systems to verify pull requests, and it works extremely well and doesn't require a ton of hoop-jumping from the developers.

I'll quit while I'm ahead

I really think it's antithetical to the spirit of Open Source to make it difficult to contribute to your project.  Github gets it right.  You fork my code, write your own code, then submit a pull request.  All the feedback and code is in the same interface, you can update your pull request based on feedback, and when it's good to go, the committer can merge it in.  Done.  If you don't want to use their issue tracker, you can integrate it with most popular issue trackers and have it automatically update tickets for you.  Whatever you do, don't make the contributor have to do this manually.

Full disclosure: I don't work for Github and I'm not a paid shill.  I just really think they nailed this process and there's a good reason they're becoming a de-facto standard.  I wish other projects would catch on.

Even the best software sucks

I rant a lot about user experience.  If you know me, you've probably heard me ramble on about it.  And when I talk about user experience, I don't just mean a slick UI.  There are a lot of aspects to user experience, like performance, responsiveness, obviousness, etc.  I'm not going to go into all of that here, as this is a rant about traveling.

I recently embarked upon a cross-country drive with my family to attend my wife's family reunion.  It was a good trip overall, except for one thing.  Google Maps!  Oh, how I have learned to hate thee.  Now, don't get me wrong, I love Google Maps.  It's saved my bacon on more than one occasion.  There have been times where it has wrong information and has led me astray, but that's understandable.  It's a difficult task to keep that much map data up-to-date and accurate with how frequently things change in our world.  What happened to me was far more annoying.  I use the built-in turn-by-turn navigation feature of Google Maps when traveling by car, and it's almost always been great.  However, a recent update has made it extremely unstable.  It crashes every hour or two.  No notification, it just silently exits and my phone eventually goes to sleep if I don't notice in time.  This wouldn't be a huge deal, except for two factors that the developers seem to be unaware of:

1. Google Maps actually does crash.
2. There are areas of the country that have unreliable or nonexistent cellular data signals. 

You would think this would be common knowledge, but I guess over in California, they have miles and miles of reliable data signals, and they never, ever leave their little bubble.  There's no other explanation for a few of the behaviors of Google Maps.  Let's go into them:

Google, Google, where art thou, Google?

After restarting from a crash, Google Maps forgets that you were mid-navigation.  I can understand that it doesn't want to just assume you didn't exit on purpose.  That's fine. But it could pretty easily prompt you.  "I see you were in the middle of a trip.  Resume?"  This would have saved me from so much pain that I might have forgiven the fact that the app crashed approximately 40 times on my trip.  This would depend upon Google Maps actually storing ANY trip data on the phone itself, which apparently it does not or else you wouldn't have to do the following.

Houston, I'm broadcasting in the blind.  Is anybody out there?

To restore your trip, you have two options.

1. Type in the destination again.  Wait until it finds it.  Click on the navigate link.  Pick the route.  Click navigate again.  Wait for it to get the GPS sorted out.  Go.
2. Click in the search box.  Wait for the recent history to appear.  Click on the navigate link, etc, as above.

So option 2 would be acceptable, if a bit tedious, if Google Maps actually kept your recent history on your device.  Alas, no, it does not.  It has to go out to the internet and get your history so you can start your trip again. Did it never occur to the developers to follow basic mobile app development guidelines and assume that the data connection is unreliable?  I can understand that Google has to spy on everything you ever do and wants to store that data online, but there's absolutely no reason it doesn't also keep a local copy that it just asynchronously updates when the internet reappears.  No data connection?  We got you covered.  But, no, that isn't how they do it.  There's nothing quite so infuriating as having the map die a few miles before a turn in the middle of nowhere and you furiously are trying to get it going again before you miss your turn, only to see a "no connection" error when trying to start the trip again.  Why isn't the trip already downloaded?  Don't you know what local storage is for?  It got to the point where my wife just loaded the trip in Waze on her phone as a backup, and while having the same directions parroted to me from two devices within seconds of each other was kind of amusing at first, it was also a sad statement about how badly Google Maps was performing for this trip.

To err is program

To me, this is just a basic feature of a navigation program that should have been solved years ago.  I'm not even getting into nice-to-have features like displaying speed limits on the map.  Seriously, why isn't that a thing?  You already have the data or you wouldn't be able to calculate how long the trip takes.  So, instead of asking for the bare minimum of caching the trip after you search for it, and not requiring an internet connection to see your recent searches, I'm going to one-up that and say what Google Maps really should provide: the ability to pre-cache an entire trip from your cushy, safe, wifi connection at home that never needs to touch the internet again the entire trip. It should download the entire route and all maps within a reasonable distance of the trip (say, 1 mile on both sides of the road, to cover pit stops and small detours). I've got gigabytes of free space just begging to be used for this. Bring it on, Google.  Based on the difficulty of your interviews, even your HR interns should be capable of providing these features.  They aren't that difficult to program.  Now, where's that bug tracker...

Making sense of the Hadoop ecosystem

I've been learning Hadoop for my job at Rackspace, so I wrote up a rather lengthy treatise as a primer on the various pieces of software in the Hadoop ecosystem.  After asking for some fact-checking from my team, it was suggested that I post it to the Rackspace blog instead, so I did.  They said it was too much awesome for a single post, so they had me split it into two posts, kind of like Kill Bill or the last Twilight movie.  Writing these posts is what got me to finally start this blog, so now you know where to point the blame.

Part 1

Part 2

Things have changed a little in the last month since I wrote the posts.  Maybe I'll do a Part 3 at some point with what I've learned since then.

Edit: Fixed the link to Part 1 - it had changed since they originally published it for some reason.

On recruiting at conferences

I always find it interesting, if a bit sad, that at every conference I attend, every talk has an obligatory "we're hiring" slide.  It's always delivered with such a lackluster, almost apologetic tone, that it probably only serves to turn people off of the option, honestly.  It feels like whoever is presenting was forced to include it by an overeager recruiting department.  The same thing goes for trying to recruit by having a booth or by networking at the conferences.  A girl came up to the table I was having lunch at recently, and very bluntly said she was looking for people to come work for her company.  My response was "so is everyone else".  The fact of the matter is that people who come to conferences are sent there by their employer, which means that a) they're employed and b) their employer thinks highly enough of them to pay a lot of money to send them to a conference.  It's just not a receptive audience.  The best thing you can expect out of conferences is to get your name out there so when people are looking, they're aware of you, and all the awesome stuff you're doing, and how happy all your employees seem, and the fact that you didn't force your presenter to post a "we're hiring" slide in their talk. You're better off investing that effort and money in getting involved in the community by hosting local user groups, contributing back to open source projects, and building a good, developer-friendly reputation.

Anger-Driven Development

If you've been a programmer long enough, then you've come to realize one simple, universal, unassailable truth: all other programmers are morons.  Then if you've been a programmer a little bit longer, you realize that to every other programmer, you *are* the other programmer, but I digress.  There's been a slew of X-Driven Development methodologies talked about over the years: Test-Driven Development, Behavior-Driven Development, etc.  While those are all good in theory, what I find that actually gets things done is what I like to call Anger-Driven Development, or Frustration-Driven Development.

The basic gist of it is "these floors are dirty as hell and I'm not gonna take it any more!".

Here's how it goes:

1. You repeatedly have to do some mundane or menial task that the idiot who designed the system should have automated (note: the idiot might be you).
2. You get sick of doing it, and yell "Khaaaaaaaannnnnnn!!!!!" a lot.
3. You channel that anger, ignore your deadlines, and automate it away so you never have to do it again.

Or alternatively:

1. You have to maintain some software written by a moron.  You know you've been there.
2. You keep having to use some awkward-to-use, obtuse interface that only makes sense to the moron who wrote it.  Good thing he no longer works there or you'd give him  piece of your mind!
3. You cry when nobody's looking (hopefully nobody's looking).
4. You channel the rage and rewrite or wrap the problematic code so that you don't have to waste one more second of your precious life dealing with it.

That's basically it.

1. Situation sucks
2. You get angry enough to do something about it
3. You fix the glitch.
4. ???
5. Profit

The problem with all that anger is, if you bottle it up too much, it tends to come out in powerful bursts.  You become the Hulk and rage on the source of your frustration, which is counter-productive.  Joss Whedon, the wise sage, has taught us the secret in the recent Avengers movie:

I think that's about where we need to be as developers: always angry.  We maintain a base level of righteous indignation about the code we have to work with, but we keep the Hulk at bay by fixing just enough of the brokenness to keep the blood from boiling over.  Always just on the cusp of blowing up, but channeling that anger into productive change.  You let the anger drive you to produce better code, better systems, and better processes just so you can sleep at night, despite someone being wrong on the internet.

I'd like to say I'm at that point of anger zen, but I still let the Hulk out at times.  Fortunately for me, my current situation is very empowering, so I'm able to fix things as I run into them for the most part, and Hulk stays within.  For now.