Aug 31, 2019

Modeling Knowledge, Understanding, Expertise, and Innovation

Why model Knowledge, Understanding, Expertise, and Innovation?

Modeling is a logical simplification of something that in reality could be quite complex, intangible, or abstract. A model then allows us to logically work with these concepts and influence them in a desirable way.

When it comes to the logical progression from Knowledge to Understanding to Expertise, understanding them in a way we can influence that progression is probably the most important thing we could learn. Without a simplified model, these concepts are hard to grasp and can take decades to understand.

This can also be very useful when building and managing large organizations especially in this hi-tech age of information and knowledge workers. A good model can guide how systems and processes are built so that organizations can drive towards innovation and change.

Knowledge-Understanding (KU) Graph

I have come to think of this learning progression as moving from a 1-dimensional space to higher dimensions. To start with, let us consider Knowledge as something that can be described on a 1-D space as points on a line of infinite knowledge. Thus, I could plot the Knowledge of someone on a line as follow -

Here each point on the line corresponds to some knowledge of the world and highlighted points are the Knowledge the person possesses. Knowledge is a complex enough construct to express in a binary term, but for our modeling, we simplify it as such.

There are practical applications of thinking about Knowledge in this way. We see its use all the time when hiring people when we specify a list of required skills. What we are really asking here is - do you have a Knowledge intersection with these set of Knowledge (Skills)?

Thinking along those lines, we can plot Understanding on the y-axis of the above graph. We frequently refer to the phrase "depth of understanding", and literally plotting it on the y-axis gives us a graphical representation of that phrase. The following diagram illustrates the Understanding of the Knowledge that the above person possesses on a y-axis (2-dimensional space).

I will refer to the above graph as KU graph here. Thinking of Understanding in this way makes understanding Understanding much simpler. The answer to the question "How well do you understand?" can now actually be plotted on a KU graph.

It also nicely separates out the fact that acquiring knowledge is not the same as understanding. Knowing is not understanding. It is far easier to Know. For example, you can read a book and probably know all about camping. It does not mean you understand camping or that experience.

Another example is that of compounding. You might think that if one knows how to add and multiply, they understand compounding, which is so not true. Someone, on the other hand, might not know how to add/multiply or could be really bad at it and they may still understand compounding well and would reap the benefits of that.

In life, we learn Knowledge at school. But using that Knowledge is what builds understanding. And that Understanding of the Knowledge is what comes in handy.

This model though might not be accurate. We have seen examples of prodigies who have been able to do exceptionally well in some field without ever having formally acquired the Knowledge. May be Understanding is not directly related to Knowledge and one can have a lot more Understanding with very little Knowledge. Or maybe Understanding can be acquired faster, almost instantaneously. We will not delve in those edge cases here.


What makes one an expert in something? Can we extend this model to illustrate expertise?

Expertise is a relative term. It is relative to other people. So if someone says they are an expert, they simply mean they can do better than others for that skill.

Thus if we extend the KU graph model into the 3rd dimension, where the z-axis is the KU graphs of other people, we could visualize Expertise graphically via our model and therefore possibly influence it.

Modeling Expertise as a relative construct has other implications that we can see are true in real life. For example, for the same skills and understanding levels, one's Expertise will reduce over time as more people gain a similar understanding in that skillset.

Plotting this 3rd dimension on z-axis allows us to define levels of Expertise and calibrate it over time as KU graphs for a given Knowledge of the masses change. We have typically seen applications of this as skill rating, eg. a 5-star rating or a novice-intermediate-expert classification. The following slice of KU graphs of a set of people for a given skillset depicts this classification.
Thinking of Expertise in this way helps us to plan our way to expertise. It requires us to not only reach those depths of understanding but also reach those depths relative to others in that field.

We can see the analogies of this model in our education system. We do MS in say, Computer Science, after having studied computers for 4 years during undergrad, only so we can improve our understanding in that field. Nevertheless, that does not make us experts. We need to spend years in the industry working on a subset of that Knowledge set to gain understanding to an expert level. Alternatively, we can pick that subset and spend years doing a Ph.D. to reach the levels of being an expert.


Now that we have modeled Knowledge, Understanding, and Expertise, where does Innovation happen? How does that spark of genius ignite? Or simply, how can someone be an innovative person?

If we think of the Expertise scale as ever-changing and KU graphs of the masses evolving for some K, my belief is that Innovation happens at the farthest depths of Understanding. 

One could graphically plot that region on our Expertise KU graph as follows -

Thinking of Innovation in this way can be very powerful since we can direct our way towards Innovation. Not only as individuals but as large organizations in general. We can also determine in what fields we want to be Innovative and how we can reach there.

This concept is not new. We see large amounts of funding directed towards a certain cause with a goal of improving innovation in those areas. Universities have been the most benefactor of this primarily because they organize Knowledge fields wells and thus make it easier to fund the cause. Enterprises dedicate research funds so they can invest in improving understanding in areas where they want to Innovate.


From my two decades of working and leading in various organizations, I have come to realize that having a model where we can drive towards Expertise and Innovation can be very useful. There are systems in place that help with them and having a simple underlying model can prevent them to degrade over time. Also, this path calls for tons of improvisation and calibration and such a model helps everyone understand the direction.

May 11, 2018

Butt time productivity

I have used this term i call "Butt time productivity" as a way to model how some managers view productivity. This becomes really relevant in the software world, where the outcome is not dependent on time spent in front of computers and the best code is probably written when taking a walk or may be in the shower.

Consider two employees, I will call them H and S. H is the hard worker. He is in office at 8am and remains there till 7pm everyday of the week. He never takes vacations and is always available. He is good hearted and fun person to be around. He takes all meeting notes, always on time everywhere and does everything asked for. He will work late at nights and weekend all the time to resolve any issues. The only caveat, he is not the creative kind, yet.

Now consider person S. She is a smart worker. She comes in at 10am and leaves by 4pm. She is often on vacations and not reachable at times. She is whom we would call the good "lazy" engineer. But she is responsible. She can solve problems. She can take a 100 hour task and find a way to do it in 2 hours. She always has brilliant ideas that work and can solve the problem at hand. She solves problems before they become critical. She reduces the task and does it in a way that eliminates future issues.

Now step back a little and watch them both. Wear the hat of that manager. Keep in mind, that in software, the vertical problem sizes and effort are totally unrelated and let's assume you don't understand that. And with that context, you being that manager, who would you promote?

You might argue for S, but the reality is that it will always be H. And many of you might have seen it happen many many times in your careers.

What H excels at and what the organization (or the manager) does not understand, is what I call here as Butt Time Productivity. Think of it as the time ones butt warms the office chair :)

And I think if we are managing creatives and are in that field, we need to be careful about how we measure productivity of a person and how we value hard work. We need to be mindful about size of the problem and size of its effort without allowing criticality to distort that judgement.

That is one of the reason best managers are also leaders who have walked that path and can see beyond linear. They can see beyond the hard working H and the lazy S. They can also see through the skillful rhetoric and presentations on where the creativity sprouted from.

H is not a bad person and S is not super woman. When you consider the spectrum of experience, skills and talent, we all navigate between H and S. This is more about how we see and value productivity. It is also a caution, since H activities can be gamed resulting in crappy politics and BS'ing culture.

In the end, measuring individual productivity is hard. All organizations need both H and S. We need some of both H and S in everyone as well. A great creative organization is built when the relative values of H and S are understood by the leaders. What we value grows. And, what we don't dies. So please don't over value butt time productivity.

Jan 28, 2018

Predicting using Idiot's Bayes

There is a lot of magic that goes into Machine Learning and AI. So it feels great the first time you work out all the math physically and do a prediction given some data.

I set out to implement the Naive Bayes on the pima-indians-diabetes dataset (Google it and you can fine it). There is a lot of greek that goes into understanding the formulae's but for those like me whose dictionary has "greek" meaning "i don't get it", here are all the gory formulae's worked out in a spreadsheet.