48 hours of Advanced Technical Support

from Microsoft people who know what the hell they're doing.

I got to work with Microsoft Azure, HDInsight, Hive, Power Map, Microsoft Query against Hive Cluster.  Very cool weekend.  Unique learning experience and I made new friends. About 2/3 made it through.  

The data came from data.seattle.gov  and some of the presentations are very cool.  The suite of business tools was astounding.

I was also interviewed, along with my new friend and the only team mate who survived.  I was so glad she showed up this morning.  That wasn't a sure thing.

But even though I didn't present any of my "findings," and our presentation was, well, very much less than "strong," I was able to actually make it through the process from end to end, and replicate it.  That was what was most important for me - to learn.  So, my new friend and I went up there and gave a 3 minute presentation on our little PowerMap visualization (tres kewl product!), and then sat down.  No way was there any real insight into what we created, but jaizus!  some of the presentations really made it through a process.  They were awesome.

My First

HACKATHON!

I loaded my first Hadoop Cluster on MS Azure and wrote my first Hive query.  

This has been painful, but very, Very worthwhile.  The reason is because Microsoft has a team of experts in various products and this is really all about an intensive training session.  I started out with a group of four guys and one other woman.  About two hours into the "hackathon" proper (as opposed to the leading presentations) three of the guys were gone.  I think at least two of them were here for free food and had no experience with query language, Excel, Dashboard development, databases or their administration.  I'm not sure they could even write code.  The third guy was migrating from policy into coding and wasn't really familiar with query language, Excel, visualization concepts, or analytics, much less the other - basic - items.  

I *intend* to go tomorrow morning.  I left at 8:30pm, an hour before the Microsoft people *could* leave, although a lot of people were wanting to stay the night.  And when I say "a lot," I'd say that out of the original 300 which registered and showed (full room in the morning), there was less than 1/3 left when I left and some of the "kids" were planning on staying up all night.  Microsoft had planned to allow the all-nighter.  I had a conversation with one of the youngsters about, "Why do all the older people leave instead of staying all night?"  

Been there.

Done that.

The feckin' t-shirt has been worn to threads.  I want a bourbon.

But this was a good working experience, but shit, it's like starting up with SSIS & SSRS again.  My eyes are crossed and I feel like an idiot, but if / when I sit down to focus on this again, maybe - just maybe - I'll 'member some of this activity.   I just don't get why MS products are so "complicated" to figure out.  I had to load like twelve new softwares just to get the interface going, but gotta say...  I'm proud of the .csv file I loaded up.

Yep.  That's "Hadoop" - raw files, whatever format you want.  oh joy.  It's a big ole file folder and you specify the parsing routine in the select statement.  

But the thirty day trial versions / access...  I'm not gonna maintain that shit.  So, I guess I'll be dropping back by Hortonworks or somewhere else to spin up some clusters b/c Hive is Apache Hive and so the query language is unchanged.  I go open source b/c I am not gonna pay the corporate $$ but I do need to have access to the technology so I can add to the skillset.

Hate me

It's 77 outside and I'm sitting under my pergola in shade greened from the wisteria and grape climbing between the rafters.  I've been shopping at Pike Street Market all morning with a new friend.  She moved here from Chicago last year.  Yet another transplant who saw and fell in love with Seattle.  The first harvest of Rainier cherries and local strawberries is in.  I spent the morning walking her through some of my favorite stalls in the market, introducing her to the friends I have there as well as the foods I love.  When you're shopping in Pike Street, the vendors usually have samples for you to try before you buy.  The predate Costco by some 60 years in that marketing idea.

 We, Dave and I, are lucky to have this time, this summer vacation.  Even with as much running around as I'm doing, I can choose and now, at this moment, after I've returned from a day running around, and a night with intense technical lectures, and days where I've been working on building my portfolio, refining my resume, and continuing to work on my "Why I left Company XXX story, I've decided it's time to pop a chilled Portuguese rose, slice some cheddar, add some freshly harvested strawberries to the side of the plate and bring my PC outside to finish my correspondence.

And about that, there's always a first "demand" "to know" why "you left."  Suspicion is the first emotion I note from people in a hiring position.  Rarely is it ever curiosity, and then I've only seen curiosity when people are speaking just peer-to-peer.  Being unemployed is treated like a disease at first, but then the next emotion to pass over their demeanor is envy - especially as this is such a lovely spring in Seattle.  People sigh, and god forbid I mention I was out in the sun, at The Market, on Alki Beach, or wandering the woods of Lincoln Park.  One woman's lips zipped so tightly, I thought her seams would burst.  She couldn't have been more than 28, 29, or 30. 

Quite irresponsible of me, I know.  But I am about to return to working on my response email to an Amazon recruiter, giving them five times which would be good to call, I'll be looking up another contact from whom I received an email in LinkedIn. I'm sitting out here on my back patio and I'll work on emails from here.  Yes.  I deserve your envy.

Exploding Heads and Very Good Questions

I shoulda apply the "'What the Hell Are You Thinking' Mind-to-Mind" transfer ring to myself.  Some days I just don't get this "why" I need to push b/c the Meetup I went to today - 2nd lecture blew my brain up into

                 itty

      bitty
                           pieces

Ignoring the fact that I don't know shit about making my own query language structure (so what the hell was I thinking about attending a lecture on one.  I mean, fer chrissakes, I specialize in wearing multi-colored nail polish.

But the talk on google's new product launch, BigTable, very, very interesting.  Especially in light of yesterday's question which asked about that I/O blender effect.  In today's lecture Paul Newson, a Developer Advocate with Google, discussed this in terms of performance impacts because of "bully tenant" / "noisy neighbor" problems.  Anyways, google's rolling out their cloud solution and they've got a developer's kit for you to test performance.  But google's done some algorithmic changes at they've achieved at least three 9s of <well, shit, I forgot exactly what, but I remember thinking, *three 9s, they need five*.>    What the hell.  The free beer was after, so I was only drinking diet coke and it was hot in the room - or I was having hot flashes.  Anyways, they addressed the latency issue to the last 99% of their users... oh yeah... They're up to three 9s on the latency issue when compared to r/w actions of Hbase & Cassandra because of their algorithms and the way they've distributed the loads.  White papers to come at the bottom of this entry.

I was less interested in it from a database problem than I was for the idea of a cloud system which might be able to keep up with the Need For Speed for mobility processing.  But really, what do I know.  Not a fuck of a lot.  That's why I'm out here on Walk About.  But whatever... this was the first time I'd seen the questions of latency addressed.  They also addressed CPU, but as another attendee mentioned, they seemed to have left off addressing questions of R/W actions, bandwidth, etc.  Be that as it may, the distribution, the scaleability of the architecture seems reasonable with the idea of distributed processing, backups, replication, and redundancy.  And now this understanding of the Need for Speed.  Finally.  Something where Cloud Computing looks like it will keep up with mobility demands.  Not that I know anyfink.

So, yes, new metrics need to be added for "tenant" problems - which appears to be the street jargon for the I/O blender effect   However, the fixes appear to be the same budgeting idea that Docker  presented and which Zulily works with, apparently successfully in deploying their iterative application upgrades.

And this link is to a picture worth an extra $500 google credit to add to the $300 when you sign up for their cloud.  And I'd suggest it just so you can get a physical feel.

But there are a lot of smart people out there whose world is different than ours is/was.  I don't know shit, but then I never thought I did.  I just absorbed information.

Original 2006 BigTable WhitePaper
More Big Table Stuff

Relevancy and the Wireless Engineer

I was at a great Meetup last night where one of the speakers was Jeffery Heer of the UW.  His topic was data visualization.  Apparently the Stanford Data Visualization Group moved to Seattle and is now the UW Interactive Lab.  This talk reminded me of just how frustrating some of the problems I used to run into at work were because I couldn't find the right way to "see" the relationships.  Now that I'm out here in the wild I find that there are entire swathes of tools being created which help engineers explore the relationships in the oceans of data I swam in.  If I'd known about these guys then...

The mission of the Interactive Data Lab is to enhance people's ability to understand and communicate datathrough the design of new interactive systems for data visualization and analysis. We study the perceptual, cognitive and social factors affecting data analysis in order to improve the efficiency and scale at which expert analysts work, and to lower barriers for non-experts.

We work in near petabyte territory, if not petabyte, and we couldn't gain access to the tools which would keep us relevant in the industry.  Access to even Excel's PowerPivot would have been tremendously helpful.  As I've mentioned before, R or Python, would have made it easier to work through some of the datasets as well which exceeded the limitations of the Excel we did have.  

It's quite frustrating going to these meetups where they're looking for speakers and there is not one single engineer / "scientist" from any of the wireless carriers is ever there to give a speech.  Where are the wireless engineers to talk about the creation of performance KPIs in complex networks?  Data modeling?  Data wrangling?  The quality of the data?  How to manage missing data, alarm that data is not flowing?  How the acquisition of data affects node performance?  Managing monster sets of data?  What visualization tools are the most useful with heterogeneous network statistics so relationships can be found?   I've met recruiters from Century Link's Cloud Computing group, but that's not exactly a technical speech and they weren't engineers.

From what I've seen we've done a shitton more work with "big data" and predictive modeling and model analysis than most companies I see presenting here.  People are calling data "big" when It's just the first time they've had access to the data stream, much less to the historical data (i.e., retention periods have been extended) so that they can work on predictions, yet the engineers and statisticians from a myriad of small companies are giving the speeches.  Granted, these guys are real statisticians, PhDs., etc. but I'd like to see the theories applied to even more complex data sets.  Then there's the Amazon, Google, Microsoft, Facebook speakers.  They have some interesting talks, but again, so far I haven't heard speeches about performance from them in high traffic situations, or when they're questioned about metrics they have available to guarantee service, the list is not long, or summarized.  They've got a minor fraction of what we were faced with in core engineering.  

We can, and have done better.  And while we don't willingly (any longer, but yes, I have) participate in making configuration & modeling changes on the fly to see the impacts, (i.e., live Ads would be an example) on our networks, we have to during FOA deployments.  Gah!  The hours and years I spent trying to get that data acquired prior to First Office and getting questioned as to its necessity.  IT'S TO BE ABLE TO DETECT FAILURES AT LAUNCH!!!  (ooo...  I just got all shouty-cap.  Deep breath.  Deep breath.)

Relevancy.  That's what is being lost by being siloed away from the dialogues going on in the outside world here.  Wireless engineers and scientists should be participating in these forums, especially if the companies are interested in software defined networks on cloud networks using virtual machines. Mobility is a demanding technology and I am not hearing a thing about it and I think that is quite odd.  When mobility hits "the cloud" space, especially if we don't own the cloud, it's gonna rain dollars and not in a good way - we'll be sending money out - again - without having the data to prove whether or not we're getting what we paid for.  Tracing network failures will be problematic and determining cause will be even more complicated.  

My non-scientific search of world-wide Meetups with the word "telecom" returned about 90 related meetups, 2 in Seattle (which I've promptly just joined). However, there are 90 meetups in Seattle for just analytics alone, not including things like "Big Data" or "Python", "Visualization," etc.  The corporations have cut travel budgets, they silo their engineers, they want to outsource as much as they can and there are no public forums for discussing technical details except things like vendor specific user groups.  They're actively creating a class of engineers who don't know much more than how to do data entry, or other repetitive work.  They're to make decisions based upon what they're told, not upon what the inquiry into the data returns.  

In the interest of capital efficiency, it all comes down to this:  if wireless engineering / "data scientist" / "network planning" (whatever the hell you want to call yourself) types do not have access, and do not participate in this larger world of "big data" discussions the corporations will not  get what they paid for, the networks will not be optimized, and worse - our (oh, I can't use that word any longer) customers will suffer.  Wireless engineers have fiscal responsibility for hundreds (yeah, I know it's more) of millions of dollars of equipment.  If we're not participating in this open-dialogue at this time, if we're not doing more with more data, then we're falling behind, becoming the irrelevant ones, and sure - anyone could do our job.

By the way, here's some more interesting information I came across from last night's talk:

imMens: Real-time Visual Querying of Big Data

And someone sent me this article with a question of "new metrics?" in regards to cloud computing and virtual machines.  Uhm.  Yep.  Just think about the complexity of trying to build a metric around The I/O Blender Effect.  I'd be all over "5 minute data is too high a granularity in measurement interval... I want 1!" and I'd lose.

crafting the perfect cover letter

Cover letters, it turns out, might be that foot in the door, the magic bullet, the all-in-one reveal everyone is looking for.  Cover letters, though, take time to craft.  If they don't, your submission is spam, you're spam, you're looking at the jobs like spam, and the return rate is gonna suck.

All that work for the polished up resume, meh.  Had to cut the summary section out.  I'm not gonna stress over trying to force that step-sister's foot into a combination resume format.  That effort will have to wait for the next phase of rejection and mostly because I've worked so many projects that trying to narrow it down... well, I've taken the bits which garnered me awards or commentary and added those to the cover letter.

I've also started adding some of my stripped out / schema deleted oracle sql code to a github repo.  This was wildly successful in yesterday's interview.  The 30 minute interview turned from testing me in my query skills to "are you sure you'd like to do this?" because I basically would be moving from a "Principal" engineer's position to one lower.  Apparently, "Principal" holds the same level in both Amazon and Microsoft as it does at AT&T.  Good to know.

Anyways, since I'd already checked out the payscale range for position at both payscale.com & Indeed.com.  I just noticed that information is no longer available on my LinkedIn account since I downgraded, but then they source their info from Payscale, so why bother.  My assumed rate of pay would be in line with what would be covered by the job description that I could find.  But really, it's all a "wait and see."  The interview also gave me some better phrases about how to explain my need for fun.  FUN = BUSINESS PROBLEM.  See, I can do business speak bingo.

Business problems are fun, if they're data related.  Either data acquisition, data cleaning, analysis, and reporting.  That phrase really, Really, REALLY worked for me.  I'm all plagarizing it all over the place now.  So, I scrubbed my newly reformatted cover letter and crew-cut resume off again so he could forward to recruiters.

Next up:  I'm putting together a list of questions which would be deal-breakers for me.  Email me your suggestions.

!! summer vacation !!

Woke up late after watching Vikings until 12:30.  Yes, I finally figured out how to use my television... for the first time in over a year.

Went to have coffee with a friend and work on my new cover letter focusing on gaming analytics. Drank cappuccino and had a croissant while signing up for my first aws account to begin learning what Amazon calls "Machine Learning," "Big Data Analytics Options on AWS," "Cloud Watch Metrics and Statistics (Elastic Load Balancing Developer Guide)."  If "we're" prepping for non-proprietary bare metal, "we" should be looking into the big boys and what they're trying to do.  I strongly suggest getting your reading in at both AWS & Microsoft's Azure.  Microsoft only has a two week trial of their cloud services through Azure, whereas Amazon has a feckin year.  Jeff Bazos is brilliant.  Brilliant.

I'm also trying to pick up the language of gaming analytics.  So far I've found some amazing blogs and white papers.  The language is similar to what we get from marketing, and the business side, but you know my focus will be on performance and load balancing.  But here are some links from "what I learned in school today" (while drinking cappuccino and eating croissant)

http://www.ninjametrics.com/blog/video-game-analytics-101-basic-definitions
https://developer.ninjametrics.com/Help/Glossary
http://blog.gameanalytics.com/blog/squeezing-more-juice-out-of-your-game-design.html


Walked the family to Camp Long and back.  The rhodies are in bloom along with the iris and so many other spring flowers.  I won't bore you with yet another set of garden pictures.

I've also been preparing for my first set of "real" interviews.  One of the most difficult things I've come to learn is that data sets, and data complexity large enough to keep me interested are not widespread yet.  Only a few companies have the capital, the interest, and the foresight to capture enough to keep me from boring myself to glassy-eye-edness. 

So, I am beginning to focus down and I'm out here in the sun drinking a glass of wine on a Thursday afternoon.

Life is good.  I think I'll go get laid while I'm at it.  Daaaaviiiiiddddd!!!