Hank Roark

Prototypes, ambiguity, and tradespace exploration

2012-02-13T09:55:00.000-06:00

I ran across an HBR story about how single prototypes that happen early in the system or product development cycle can project outcomes. This is reported to happen because the team becomes focused on the single prototype instead of remaining in the problem space for a longer period of time.

Tradespace exploration, and in particular tradespace exploration rooted in what is valuable to the acquirer or customer, could be one counter measure to this team dynamic. By exploring hundreds or thousands of designs parametrically is one way to avoid settling on optimizing one solution too early in the development process. Further, by mapping to stakeholder utility it keeps the development team focused in the customers problem domain.

May particular research in this area is two fold:
1. Use of multi-attribute tradespace exploration for systems solutions in the competitive commercial space with many sellers competing for many consumers (the current research has been primarily in the context of a monopsony, such as the U.S. Department of Defense or NASA).
2. The use of tradespace exploration to as a value centric approach to target system modularization efforts.

As I get articles out on each of these areas I'll provide additional pointers.

Norms of Validation and the Agile Community

2011-10-13T21:38:00.000-05:00

Had a great dinner with Mike Cottmeyer tonight in Bean Town. At one point we got on the subject of the nature of research and validation expectations. I was reminded that someone told me that different sciences have different validation and correlation expectations because they will reject the null hypothesis.

In high energy physics, the expectation is for the coefficient of determination to be at least 99.9999%.
Depending on which engineering community, 95-99% or greater is expected.
In the social sciences, it is not unusual to accept R2 values of 30-50% as "good enough". Human interactions are considered so complex as that when R2 values are >90% are one has to validate you aren't just measuring the same construct in different ways.

These communities have other norms. A couple norms in these communities are that research must be falsifiable and replicable. Another typical norm is that another minimum level of validation is peer review by people with PhD's and publication in a journal associated with the discipline.

These are the norms these communities. We can argue about them, we can even argue if these communities hold themselves to these norms all the time.

It struck that my observation is that the agile community has their own set of norms for considering work valid and something to be built upon. I would argue that the agile community currently has validation norms well below those of the physical sciences, engineering, and social sciences. Sometimes a poorly done case study (by the standards of the social sciences) or a claim that is published in book form (but not peered reviewed) is sufficient for validation in this community. Sometimes it is just a blog post by a well paid consultant.

As agile software development enters these other communities (such as the engineering systems community) the agile community shouldn't be surprised if they are expected to reach new levels of validation before their findings are accepted.

This is a great opportunity for research. Agile is relatively new and now there should be enough cases out their to start and build serious research upon.

Agilist strike again

2011-05-05T09:45:00.023-05:00

Continuing on my critique of research in agile methods, started with this (relatively) popular post on integrated concurrent engineering compared to agile software methods.,.

I ran across this post by Dean Leffingwell and this post by Chad Holdorff, extolling the "proper" mixture of component vs. feature teams (definitions included in these posts, somewhat) on a project. Problem with this post is that there are not studies showing correlation with the two variables on the graph from Dean, the proposed mixture of component and feature teams, and the productivity and/or quality of output of the teams that do and do not follow this proposal. Additionally, there is no notion of context: why types of systems does this model work for? where does it break down? At best, this mixture is simply a hypothesis that needs to be tested through proper study. They should be marked with "notional" and for illustrative purposes only (i.e., they are fiction).

Now compare the notional mixture of structuring teams with the work of using tools like Design Structure Matrix to organize and improve global product development. A good starting point would be Steve Eppinger's paper, A Model-Based Method for Organizing Tasks in Product Development. Here we have a model, formalized, that has been applied and the outcome studied for productivity changes. There are other, dare I say, mature ways to manage dependencies.

The agile crowd needs to move past heuristics and case studies, and no notion of context, to describe what works and what doesn't work. Further, the definition of work needs include a notion of output quality (and not just absence of defects). There is are better ways than the simple heuristics and one off case studies.

System Architecture Principle 8: Beware of software

2011-05-04T12:24:00.000-05:00

Tagline: Beware of software.

Descriptive version: Software grows very complicated very quickly. It can be of high leverage, but can be potentially dangerous because of how complicated it can become. Further, software does not have a "laws of physics" like physical systems making it difficult to reason about using models and such (the code is the model!).

Prescriptive version: When deciding to allocate functions to software, be aware that you are substantially increasing the internal complexity and number of operating modes of your system.

Discussion: I am not sure if this is an architecture principle, yet. I am certain it will be one with enough time. Certainly of my principles it is the one that has the shortest lifespan.

Software is this very new thing that can quickly grow substantially more complicated than the physical system in which it operates. The number of operating modes of software is estimated as the number of inputs times the number of outputs to some power (Ayaswal2007). Further, software seems to be having a tendency to creep from being something embedded in some component helping that component deliver its functions to an item that becomes a system bus interacting with just about every component in the system. On top of all of this, the work of software is largely hidden in complicated code in bits on software developer's computers making it even harder to reason about.

Citation
B. K. Ayaswal and P. C. Patton. Design for Trustworthy Software. Prentice Hall, 2007.

System Architecture Principle 7: You can't escape the laws of physics

2011-05-03T12:17:00.010-05:00

Tagline: You can't escape the laws of physics (Augustine1996).

Descriptive version: No amount of being clever will allow your system to violate the laws of physics.

Prescriptive version: You can't change the laws of physics; use them, obey them, but don't think for a moment that as an engineer or architect that you can escape them.

Discussion: This is one of the principles which reflects my background, with my undergraduate degree being in physics. I spent the better part of five years (I started in graduate school and decided it was not for me at the time) studying how the universe worked, the models that explained why things happened all around us. As you know, this attempt to understand the universe is very much related to engineering, but in some ways very different than engineering. Engineering seems to be more about, given the set of laws that govern the workings of the universe, how can we (engineers) leverage them to affect the world around us. It is very tempting to confuse this ability to "engineer" the world with the ability to "engineer" the laws of physics. Falling into this confusion would likely lead to very undesirable consequences.

Citation
N. R. Augustine. 1996 Woodru ff Distinguished Lecture Transcript. http://sunnyday.mit.edu/16.355/Augustine.htm, 1996. section title Conceptual Brilliance Doesnt Blind the Laws of Physics.

System Architecture Principle 6: System design drives life cycle costs

2011-05-02T12:13:00.001-05:00

Tagline: System design drives life cycle costs (Blanchard2011) (Crawley2010).

Descriptive version: The systems architecture, and hence the work of the architect, has the largest impact on system life cycle costs.

Prescriptive version: The work of the system architect can drive significant changes in the excepted life cycle costs of the end product. It is important for the architect to understand the life cycle costs constraints (based on perceived value of the acquiring organization and the desired financial margins of the supplying organization) and to make sure the system architecture supports fits within the targets.

Discussion: The principle is support by the notion that highest management leverage is at the very beginning of the project, when the least amount of money has been committed to the project. That is, the very first steps, deciding on the system architecture, is the biggest opportunity to steer a project to a desired life cycle costs.

Citations
B. S. Blanchard and W. J. Fabrycky. Systems Engineering and Analysis. Prentice Hall, 2011.

E. Crawley. Esd.34 lecture 1, September 2010.

System Architecture Principle 5: Systems exhibit emergent behavior

2011-04-29T12:09:00.001-05:00

Tagline: Systems exhibit emergent behavior.

Descriptive version: As the elements of a system are brought together and interact, processes (function) and other intrinsic properties will emerge.

Prescriptive version: An architect must pay as much if not more attention to the functions and the combination of functions as compared to the items of form. It is the proper combination of these internal functions that will result in the external delivered functions that provide value to the beneficiaries. These emergent functions will not simply be the some of the sub-functions.

Discussion: During (Crawley2010), it was stated ``Form by itself delivers no value''. With this I somewhat agree and somewhat disagree. It has been exhibited, by options like the iPhone, that some users will perceive higher benefit to a system when the system has an appealing `industrial design'. This could be an argument for form providing value. On the other hand, one could argue that the function of the device, the iPhone in this example, provides the base value and form provides additional perceived value. With this argument the statement of ``form by itself delivers no value'' might be restated best as ``form provides additional value assuming that functional needs are satisfied''.

Citation
E. Crawley. Esd.34 lecture 1, September 2010.

System Architecture Principle 4: Systems exist to solve a need

2011-04-28T12:05:00.011-05:00

Tagline: Systems exist to solve a need (Shapira) (Crawley2010a).

Descriptive version: Systems exist because humans or organizations have a need and are willing to trade something of value for that need to be satisfied.

Prescriptive version: It is important for the system design / architect to understand the needs of the key stakeholders and design a system that satisfies those needs. Further, the perceived benefit of the system must exceed the perceived cost to obtain and operate the system.

Discussion: In the end, systems are acquired by exchange of money or other things of value in return for the benefits of the system. The system must provide perceived value that is greater than the perceived costs, the difference being the perceived relative benefit of the acquiring organization or person. Note the relation to the perceptions of the acquiring person or organization; this very much is in line with the principle of good architecture, which it is in the eye of the beholder.

This model gives the architect to levers to pull in order to increase the relative perceive value. The first is to focus on the external interfaces, external form, and external deliver function in order to increase perceived value. The second is to manage the internal system design so as to decrease the system life cycle costs

Citations
Y. Shapira. Principles of system architecture. http://en.wikipedia.org/wiki/User:YoavShapira/Principles_of_system_architecture.

E. Crawley. Esd.34 lectures on value exchange model, 2010.

System Architecture Principle 3: What can go wrong will go wrong

2011-04-27T12:05:00.000-05:00

Tagline: What can go wrong will go wrong.

Descriptive version: It is very rare for a system of greater than medium complexity to operate without failure. This applies to both the satisfying the intended needs and anticipating future needs.

Prescriptive version: Robust design, flexibility in design, and design for contingency and emergency operations are critical to the success of a system.

Discussion: The potential history of this ``law'' dates back to an 1877 meeting of the Institution of Civil Engineers (Holt1878):

It is found that anything that can go wrong at sea generally does go wrong sooner or later...

Various other written references to the law of turned up over time, including in the context of stage magic, mountaineering, and as a name for the second law of thermodynamics. The law has been attributed to Capt. Ed Murphy, and engineer from Wright Field Aircraft Lab (Bloch1977), and to an unnamed theoretical physicist (possibly from California Institute of Technology, aka "The Other Technical School"). While it is difficult to pinpoint its origins, the law quickly spread throughout various aerospace engineering cultures (Unknown), into the engineering and science communities, and eventually into popular culture.

In the context of systems architecture, it is important that one consider all possibilities in the design and implementation of the system. Tools and techniques to do this include the consideration for contingency and emergency operations, designs that are robust to variability in operating conditions, to consider flexibility in design so a system can accommodate future unanticipated needs and operating scenarios. This should also be expanded to include consideration of human factors; humans are unpredictable, messy system elements and all care must be taken in the design and operation of systems that require humans to execute any of the system functions.

Citations

A. Holt. Review of the progress of steam shipping during the last quarter of a century. Minutes of Proceedings of the Institution of Civil Engineers, LI:2{10, 1878.

A. Bloch. Murphy's Law, and Other Reasons Why Things Go WRONG. Methuen Paperbacks Ltd, 1977.

Unknown. http://www.catb.org/jargon/html/M/Murphys-Law.html.

System Architecture Principle 2: You can't do everything for everyone

2011-04-07T06:20:00.000-05:00

Tagline: You can't do everything for everyone.

Descriptive version: A system of any size will have many stakeholders with many needs to be satisfied it will be very difficult or very expensive to completely satisfy all of them.

Prescriptive version: An architect must make trades between which needs will be fully satisfied and those that will be partially satisfied or not satisfied at all.

Discussion: Projects of any significance will have multiple stakeholders each with multiple needs that must be satisfied . It is up to the architect to determine the most important stakeholders and needs that must be satisfied . Sometimes this means that needs that are very important to a stakeholder will not be satisfied . It is important for the architect to use high quality (low variability) information when making the decision as to the most important needs to satisfy and which should be partially satisfied or wholly considered out of project scope.

As a supplement to this principle, according to Ed Crawley: "Very many factors will in influence and act on the conception, design, implementation, and operation of a system."

(As a corollary to this principle, the stakeholder with the need that has the least importance will be the most vocal stakeholder, causing no end of grief for the architect.)

(My) Definition of Good Architecture

2011-04-06T09:00:00.000-05:00

Tagline: Good architecture is in the eye of the beholder.

Descriptive version: Ultimately, architecture is a blend of art and science with a tendency to be more art than science. Like all art, good architecture is in the eye of the beholder; that is the judgment of what makes good art depends on the tastes and preferences of the person observing the art.

Prescriptive version: As an architect, one must understand all the stakeholders that will observe their work over its lifespan, and make an attempt to either create something that is "good" within their eyes or make a conscious decision not to appease that stakeholder. All of this must be done while satisfying the system requirements, working within the system and project constraints, and without violating the laws of physics.

(My) System architecture principles

2011-04-06T08:55:00.004-05:00

I recently had cause to be explicit about the principles of systems architecture that I think are important. These are very personal, meaning these have worked for me in the past when architecting a system. I thought I would share them here, in a series of blog posts, for all the world to see. These influence how I think about systems, how I design them, how I critically judge systems, and the guidelines I use to evolve an existing system. I would welcome any feedback on your experiences and thoughts on architecture principles.

Principle 1: (My) Definition of Good Architecture.
Principle 2: You can't do everything for everyone.
Principle 3: What can go wrong will go wrong.
Principle 4:   Systems exist to solve a need.
Principle 5:   Systems exhibit emergent behavior.
Principle 6:   System design drives life cycle costs.
Principle 7:   You can't escape the laws of physics.
Principle 8:   Beware of software.

Integrated Concurrent Engineering vs. Agile Software Development

2011-03-22T20:53:00.000-05:00

Agile inspired software development is certainly all the rage. One could argue that those processes have even crossed the chasm, with mainstream companies adopting various forms of Scrum, XP, DSDM, OpenUP and the like. I certainly fall in the camp of people that started using Agile techniques as soon as I started to understand them (starting as early as the first publications on the c2.com wiki).

Meanwhile, over about the same time (starting in 1994 at JPL, according to Wall in “Reinventing the Design Process: Teams and Models.”), Integrated Concurrent Engineering (ICE) techniques were being tried and adopted in space systems development. Also, some early work on concurrent engineering (CE) was published in 1996 by Prasad in Concurrent Engineering Fundamentals, Volume I: Integrated Product and Process Organization. (Note, this was 3 years earlier than Extreme Programming Explained by Beck.) Prasad described eight fundamental principles of concurrent engineering: "“Early Problem Discovery, Early Decision Making, Work Structuring, Teamwork Affinity, Knowledge Leveraging, Common Understanding, Ownership, and Constancy of Purpose”. ICE includes provisions and support for having a co-located customer and co-located and cross-functional teams; further, ICE is a tool for lean engineering. (A history of ICE is available starting on page 68 of http://esd.mit.edu/people/dissertations/avnet.pdf).

There has been much study (peer-reviewed and published) on the benefits (faster time to completion, lower risk of missing interfaces) and potential drawbacks of ICE (using CE when plain-old sequential engineering would due). There are almost no studies of the same type done in the Agile development world, despite emerging over about the same time period. Why?

Words have meaning

2011-03-17T08:29:00.000-05:00

Why is it that so many people feel it is acceptable to take words and redefine them for convenience? Over on Herding Cats, Glen laments the borrowing and redefinition of words by the agile crowd (a crowd of which I part of until a few years back). I am experiencing one such redefinition: Enterprise Architecture.

I'll start with my quick and dirty definition of system architecture: the function and form of a system and how they relate through the system concept.

To me and some others, enterprise architecture is the structure of an system where the scope is the enterprise (e.g., for profit corporation). The enterprise architecture also describes the enterprise's relationship to outside entities such as capital markets, labor, suppliers, and customers. Further, it describes how the enterprise system will operate and evolve over time.

To others I know, Enterprise Architecture is defined as the information technology architecture at the scope of the enterprise; this is another example of IT co-opting a well defined term. I personally prefer CISR's notion of how enterprise architecture and IT relate: "A firm’s architecture describes a shared vision of how a firm will operate—thus providing a shared understanding of the role of IT." In this way IT is a sub-system of the enterprise system.

Uncertainty Framework

2011-03-14T10:31:00.001-05:00

Over the weekend I came across a paper by McManus and Hastings, "A Framework for Understanding Uncertainty and its Mitigation and Exploitation in Complex Systems". The authors present a taxonomy covering uncertainty, risks & opportunities, mitigations & exploitations, and outcomes. Then the authors cover the existing methods for dealing with uncertainty and which parts of the taxonomy are used in each method.

This is a pretty short paper and well worth the read if you are interested in all in dealing with uncertainty.

White board drawing of the day

2011-02-22T13:19:00.000-06:00

This was seen in my office. If you want projects and products and people to be successful, this might be some of the most important advice:

Deactivating Twitter Account

2011-02-17T09:13:00.001-06:00

Today I deactivated my Twitter account. I think Twitter is a great service for some, but for me if became more of a time suck, delving through pages and pages of 140 character exchanges with nothing really significant being said. This is not meant as a negative toward the people I follow, I would much rather read your blog or see you at a conference where we can interact in a more meaningful, unconstrained way.

Now is the time for me to unplug.

Quote of the day

2011-02-16T12:51:00.003-06:00

You don't get what you want want by trying to eliminate what you don't want.

-- Russ Ackoff

Work Transformation Method Application

2010-12-30T07:59:00.001-06:00

As mentioned in a previous post, one of the projects I have completed as part of the MIT SDM program has been to look at the process of diesel engine aftertreatment system calibration. The project team for this assignment was Genevieve Flanagan (lead), Candice Engler, and I. Also supporting us was Oliver de Weck and Arun Balasubramaniam.

After-treatment systems are those components on diesel engines that remove particulate matter (soot) and nitrous oxides (NOx) from the exhaust. Since the introduction of the U.S. Clean Air act, there have been several levels of decreasing particulates and NOx levels mandated; these are sometimes referred to by tiers (Tier I, Tier II, Tier III, Interim Tier IV, Final Tier IV). Because these are regulatory mandates, a producer of engines must either achieve the mandate or not ship engines. Producers of engines also do not want to compromise performance of the engines to meet regulatory requirements. From a project perspective, this is a project that has fixed scope and schedule.

As producers of these cleaner engines have moved up the tiers they have come to rely more and more on software to control the after-treatment systems in order to meet the regulatory requirements. Part of the process of preparing these engines is to calibrate them and their software in a test environment. Calibration activities are resource intensive, both in terms of human labor and capital for the test cells.

When talking with the after-treatment team the calibration activity was largely treated as one large task, with lots of iteration within the task. The variability in the effort and duration for the calibration of each engine configuration was considered unacceptable by stakeholders. We also learned that the after-treatment team had built a design structure matrix showing the calibration and other dependencies between the components (physical and software) of the after-treatment system. The DSM was highly coupled, with few small independent components and one large meta-component.

We investigated various mechanism to better plan the calibration activities, including a signal flow graph approach and visibility matrices. We settled on using a work transformation approach to create the engineering iteration model. We used the DSM to determine a series of 24 design steps (things that were later labeled "sensor calibration" and "determine soot model", as examples) required for engine calibration. These calibration steps still happen iteratively (e.g., an activity done during step 2 may require rework in step 1), but now there are identified calibration modes (as opposed to one large task before). The next step will be to build a probabilistic schedule (most likely using the signal flow graph approach) for the whole calibration activity based on the twenty-four calibration modes.

Another interesting outcome of this project was that a DSM that represented the system architecture of the system was able to be reused, after mathematical transformation, as a project planning tool. This means that changes to the system architecture can be used to update the project plan with mathematical linkages to the underlying architectural change, including determining project impact of proposed changes.

MIT Systems Design & Management

2010-12-21T20:29:00.000-06:00

This year I wrapped up my first of two years in MIT's Systems Design and Management (SDM) program on my way to a masters of science degree. SDM is a joint degree from MIT's Engineering Systems Division and Sloan School of Management; I describe the program as an engineering management degree. For me, SDM is a 24 month program. I have been completing the degree remotely while I retain my full-time employment status.

Normally SDMer's start their program with a boot camp in January. Because of unusual circumstances, I have my boot camp experience at the beginning of the second year of the program. For January I will be spending full time on campus completing a few courses and participating in various design challenges, professional development sessions, and cohort building exercises. The required courses include a review of statistics and probability and a course on the human side of technology; if I can fit it in I plan to take another course on technical writing.

The last year of the program has been a very positive experience, even with the required juggling of work, family, and school. The classes have been everything one would expect from MIT. So far my courses have included product design & development, real-options in engineering systems, systems engineering, systems architecture, concepts of supply chains, systems project management, and engineering analysis and design. The projects completed during the classes have been challenging and (I think) beneficial to a better understanding of various systems. I'll plan to write a few follow-up blog posts on the projects:

Use of Real-Options to Value Vehicle Telemetry Designs
Multi-attribute Utility Applied to Integration of Bio-alcohols into Existing Distribution Infrastructure
Use of Design Structure Matrix and Work Transformation to Plan Diesel After-Treatment Calibration Projects

If any is more interesting to you, just post a comment and I'll plan to comment on it sooner rather than latter.

Over the next year I have many more Sloan (aka management) courses to complete, a few ESD courses, and a thesis to write (of which I am still trying to settle on a topic). I'll work to keep everyone updated as the next weeks and months unfold.

Cheers, Hank

Lean and Agile Software Architecture - How Much Up Front Architecture?

2010-11-07T11:51:00.000-06:00

I was recently asked by a colleague a question about how much 'upfront architecture' should be done if the systems development project is going to use and Agile approach. My response is below...hopefully it helps others when they struggle with answering this question.

I would say it depends (big surprise) on several factors – A lot of this comes from Lean thinking, not Agile.

In Lean Architecture (Coplien & Bjornvig) the authors say Lean is typified by a Plan-Do-Act cycle, whereas Agile is typified by a Do-Inspect-Adapt cycle. In other words, Lean = Think then Act, Agile = Act then Think. Both have their place, so knowing when and how much to do of each is important.

Here are some factors that might drive one to more thinking before acting or acting then thinking:
Does spending time up front thinking offset by an increase in ‘fraction correct and complete’ of work (i.e., decrease rework) enough to justify the time thinking. This could be quantified with modeling, but we would need historical information in order to calibrate the model.
How well can you predict the future? Forecasts are wrong, forecasts further into the future are more wrong. If the future is highly variable, it might lead to less up front planning and more adapting. As long as this adapting (rework!) is planned into the project schedule/budget.
How important is it to get it right up front? In space satellites you only really test the satellite completely when you launch it, so you better get it right the first time. In pure software-as-a-service system you can write a few hours or days of code, push it into production, if it’s wrong you can rollback and try it again. I’m guessing our projects are at neither extreme.
Projects at scale require intentional architecture; laying runway for features. A big piece of intentional architecture is defining the interface requirements. (It’s also part of requirements management). Maybe one can be lean by defining the interfaces using self documenting code (javadoc style) with stubs and example clients (so that the clients become part of the documentation, and give one a feel for the elegance or lack thereof of the API). Projects success is highly correlated with the quality and completeness of the interface requirements (including API and human-computer interface) agreed up front.
A study of several systems projects said 5-15% of the total project budget should be spend in CD (concept design phase) and PD (preliminary design phase) to bring about the most successful outcome to the project. This 5-15% cost expenditure, depending on the project, may be up to ½ of the project schedule. (INCOSE Handbook of Systems Engineering).
This probably doesn’t answer the question, but maybe it gives some guidelines that will help you decide which way to go.