Editor | On 18, Oct 2005

By: Noel Leon, Jose Jesus Martinez and Carlos Castillo

Noel Leon, Prof. Dr., noel.leon@itesm.mx (Presenter)
José Jesús Martínez, MSc., josemartinez@itesm.mx
Carlos Castillo, MSc., carlos.castillo@itesm.mx
Center for Innovation of Products and Technology, Tec de Monterrey, (CIDYT, ITESM)
Tel. +52 81 81582012, Fax. +52 81 83581209
Ave. Eugenio Garza Sada #2501 Sur
Col. Tecnológico, 64849 Monterrey, NL, México

ABSTRACT
The TECNOS Award was established in the state of Nuevo Leon, Mexico more than 10 years ago, as a way to stimulate people, enterprises, and institutions that contribute to the competitiveness and to the technical and economical development of the region through highly creative and innovative products and processes. To select the winners, expert criteria had been used up to now, which in many cases was strongly influenced by subjectivity.

This paper describes the methodology developed by request of the TECNOS award steering committee, looking for more consistent, sustainable and repeatable methods. The methodology is based on concepts of TRIZ, Value Engineering and the Kano Model. Particularly the concepts of levels of customer satisfaction, invention level, ideality, and patterns of evolution are applied and combined with metrics sustained on value engineering. Radar diagrams for the pattern of evolution are combined with the levels of inventiveness and paired comparison, on the basis of a 1000 points scale. The methodology is assisted by a VBA program in Excel that facilitates the evaluation process.

This paper relates case studies performed before the methodology will be applied for selecting the winners, identifying repeatability and consistency. The main aspects and questions occurred during the training of the evaluation committees’ members and the testing of the methodology are commented.

1. INTRODUCTION
The government of the Mexican state Nuevo Leon maintains as one of its priorities to impulse scientific and technological innovation in the productive and academic areas. This is a strategy to remain competitive in an international environment. The TECNOS Award is granted to those companies, institutions or people that result winners in a contest where new developed products, processes, publications and patents are presented.

There has been a general perception that, after 10 years of being granted, the TECNOS Award had to be updated in order to ensure a better selection of the winner projects. It is intended to have more transparent evaluation criteria, which better correspond to the current situation of the region and the country; reducing subjective judgment by using a more numeric traceable evaluation procedure. This paper presents an evaluation methodology which is intended to fulfill the established requirements.

2. METHODOLOGY
The proposed methodology is intended to help the evaluation committees grading the competing projects, requiring minimal training. The methodology takes into account that the members of the evaluation committee are not necessarily experts in QFD, TRIZ and Value Engineering, which constitute the foundation of the evaluation criteria.

The evaluation scope is based in three main concepts:
• The differentiation of products, services and processes according to the Kano’s customer satisfaction model.
• The degree of ideality and level of inventiveness of the solution developed.
• The relative position of the project (product, process, service) according to the trends of evolution that apply.

2.1. Kano Model
The model of customer satisfaction developed by Dr. Noriaki Kano, describes the complexities of customer needs and its relationship to customer satisfaction. In the present methodology it is used mainly as first classifying criteria. The Kano model identifies how the performance attributes are perceived by customers. (Fig. 1)

The lower curve of the model reflects the basic customers needs, those basic functions or features that customers normally expects from a product or a service. The absence of any of this as basic perceived functions leads to a big customer dissatisfaction causing complains and lost of confidence. In the middle is the proportional performance attributes curve. The better these functions or features perform, the greater the level of customer satisfaction and also the product cost. The top curve represents those functions that excite the customer; the unspoken or unexpected customer needs that, when satisfied, lead to high levels of satisfaction [1] .

2.2. Ideality (I)
The degree of ideality is a ratio of the useful or desired functions and the harmful effects of a new product, process or service. By this definition the degree of ideality can be a number between 0 and ∞, since it is a quotient. However, for this methodology the ideality scale has been standardized to be in a range from 1 to 5 in the same way as the level of inventiveness scale. [2]

Since the ideality value has to be established at the beginning of the evaluation procedure the evaluations committees are asked to make a comprehensive research of similar products in the patent databases and the web. This research is essential to locate the products or services whose ideality is higher or lower than the one being evaluated.

2.3. Level of Inventiveness (N)
TRIZ establish five levels of inventiveness to classify invents and innovations. These five levels are defined in dependency of the knowledge applied at the moment of its creations.

In figure 2 is shown a percent distribution of patents according to their level of inventiveness as established in classical TRIZ and in recent research done [3] . As the level of invention also has to be established at the beginning of the evaluation procedure in a scale from 1 to 5, evaluators are asked to make a thorough analysis of the patent database related to the product or process being evaluated. This way it is possible to establish more objectively the level of invention.

The combined factor for the inventive level and ideality is computed using following formula:
NI= I*p1+N*p2

Where:
NI= Combined Evaluation of Ideality end Inventiveness.
I= Level of Ideality. (1 < I < 5); p1= Weight factor for I N= Level of Inventiveness (1 < N < 5); p2= Weight factor for N It is possible to assign different weights factors to “N” and “I” in dependence of the perceived maturity level of the previous art product or process. In any case the sum of p1 and p2 should be equal to 1. (p1 + p2 = 1). 2.4. Pattern of evolution.
Although the positioning along the S-curve is a generally accepted way of evaluating product or process development, it was not implemented for this methodology, as it would be too time consuming for the purpose of being used for evaluation committees. However, the relative positioning with respect to the previous art based on the concept of TRIZ patterns of evolution is the kernel of the proposed methodology.

As the positioning using the classical TRIZ patterns of evolution is not well supported, it was decided to use the trends of evolutions as proposed in the software CREAX. The evaluators have to choose from a given list of 32 trends of evolution, which of them better apply or are closely related to the project being evaluated. The evaluators will also determine the level at which each of the evolution trends is present based in the scales used in the evolutionary potential tab of the software CREAX (Fig. 3) However, as in software CREAX different scales are used, we decided to use a unified scale from 1 to 10. Once the selection of trends of evolutions has been done and each selected trend has been given a value, a VBA program based on Microsoft Excel facilitates the evaluation displaying a potential evolution radar diagram as a visual aid as shown in Fig. 4.
The evaluation, based on the evolution potential, is perceived to be an adequate tool for orienting toward an increased competitiveness.

2.5. Value Engineering
The value engineering analysis is a useful technique for comparing alternatives in a reasonably well understood domain, where the customer perceive and state what they expect and what they need [5] . In this methodology, we use value engineering analysis to determine the evaluation team perception of the relative importance of the technological trends that apply for the evaluated product or process.

3. CASE STUDY
This section describes the evaluation of a project using the proposed methodology: An electric device to stimulate muscles of a person who is using a plaster splint after a broken bond. This apparatus works using controlled electric impulses around the affected area and is intended to provide a minimum recovering time after the plaster splint is removed.

As explained, the methodology has four steps:
1. Determine the Kano level of the product.
2. Determine the inventiveness level (N), the level of ideality (I) and calculate the combined factor NI.
3. Determine the technological trends that apply to the product (PE).
4. Get the product global evaluation value.

3.1. Determine the Kano level of the product.
From the three possible levels the highest was choose as the product creates a high customer excitement compared to the previous art (plain plaster splint). It should be noted that this classification has only informative purposes and has no influence in the evaluation.

3.2. Determine the inventiveness level (N), the level of ideality (I) and calculate the combined factor NI.
To determine this level, the evaluation team has to perform a deep search in the internet and in the main patent offices, as the US Patent Office (http://www.uspto.gov), Japan Patent Office (http://www.jpo.go.jp) and the European Union Patent Office (http://www.european-patent-office.org)
In this case a 0.5 value was applied to both p1 and p2.
The values determined for this product are:
I= 2.3, as it uses outer resources to be imbedded in the plaster splint
N= 1.6, as it is seen as an adaptation of existing devices
So, the combined factor NI = 1.6(0.5) + 2.3 (0.5) = 1.95

3.3. Determining the technological trends.
Five trends were found that apply for the evaluated. Based on that selection, the radar diagram of the product is constructed.

Following trends were selected in this case:
• Mono-bi-poli Similar Objects
For this trend a value of 4 was assigned as this product is conceived to have as many identical patches as necessary to stimulate an injured area, which can be of different shapes.

• Reducing System Complexity
To this trend we assigned a value of 7, because the product uses a reduced number of parts per main useful function.

• Controllability
For this trend we assigned a value of 4 since the system implies some control level, even though it does no consider feedback.

• Customer Purchase Focus
This trend was given a value of 3 because the product is yet at a level between reliability and the convenience for the patient.

• Action Coordination.
For this trend we assigned a value of 2 because the system does not have any kind of coordination of the impulses application. It could save energy and increase its performance if it had it.

3.4. Evolution Pattern Summary.

3.5. Paired Comparison
The relative importance of the technological trends is compared on a paired way. This task is shown in figure 10, where each letter indicates the prioritized issue and the index for each letter indicates the degree of importance.

The obtained percentages are the weight factors for the final calculation of the trends of evolution index.

The final integrated inventiveness-ideality-evolution index for this project was 423 points.

4. WORKSHOP TRAINING SESSIONS
4.1. Comments and questions during the training workshop
TECNOS Award has an evaluation committee formed by scientifics, professors and engineers of different fields. These committees have to select and to rank the different projects that compete each year. As mentioned above, these selection and ranking have been based on judgment criteria mainly. In order to start introducing the proposed methodology, as and aid for the evaluation process, three training sessions were conducted.

In session one the basic aspects of TRIZ that are used in the evaluation methodology were explained. Basically following topics were briefly covered: ideality, level of inventiveness, technical contradiction, inventive principles, separation principles and patterns of evolution.

Session two had the main objective of training the committees in the use of the software utility that was developed for the evaluation process. Some interesting questions arose during session two; particularly the use of paired comparisons resulted unclear for 70% of the evaluation committee members. Another question was if the numerical results provided by the proposed methodology were consistent and independent from the evaluators’ background. For answering these questions a statistical validation study was conducted in order to gain a deeper understanding (see section 5).

Session three was oriented toward making the participants to evaluate projects based totally on the proposed methodology and leaving aside their traditional criteria.
50% of the evaluators reported that it was a more time demanding task than the traditional methodology. This perception was caused mainly as this methodology requires that the evaluation committees perform a patent and INTERNET research of the project being evaluated. 30% complained about the software interface demanding it to be friendlier. At the end of the third session 75% of the numerical results provided by the proposed methodology were compared with the results provided by the traditional methodology. One of the projects being evaluated was selected as an outstanding technological process with both methods. 75% of the projects were dismissed with both methodologies..

5. STATISTICAL VALIDATION STUDY
For understanding how much the obtained results depend from the evaluators’ background, a group of engineering students was trained in the methodology and different evaluation teams were build. Each evaluation team was given the same project to be evaluated independently from each other. The project evaluated consisted in a par of glasses designed for close vision. The students were asked to research the patent databases and the web, looking for similar inventions. Depending on the results they had to determine the level of invention and the ideality for the glasses. Each team had also to select among 32 technological trends of evolution, those which apply for this project and then grade each selected trend in a scale from 1 to 10.

In table 3 are shown the results of the evaluations performed by 11 different evaluation teams. The average evaluation result given for the glasses was 506.44. The COV was 0.105 and shows a good coincidence. The combined factor (NI), that relates inventive level and ideality, averaged 2, however with a COV of 0.234 which is relatively high. Another aspect analyzed was which of the 32 technological trends were selected more frequently by this group of evaluation teams. Figure 12 shows a distribution of the technological trends selected by the students in their evaluation process.

Action coordination, dynamization, geometric evolution of linear constructions and degree of freedom were those selected more frequently, among 17 technological trends generally considered. Design methodology, mono-bi-poly similar objects, design point, reducing system complexity, decreasing density and space segmentation were selected just by one student each.

6. CONCLUSIONS
This methodology is still in an experimentation stage. It has been tested for the evaluation of contesting projects in the TECNOS Award. The methodology needs perhaps further testing for being improved so that it can evolve to become a standard evaluation tool which may be used also for the evaluation of innovation projects in enterprises. The integration of methods as TRIZ, VE, and the Kano Model has proved to be useful for leading the evaluation teams to use more objective criteria for evaluating the projects. The evaluation procedure is not very time consuming and aside from the patents database research time, the other steps are very fast and intuitive. An important aspect is that the proposed methodology is changing the way of thinking of the evaluation committees of the TECNOS Award. They are changing from subjective judgments criteria to a more structured and scientific way of evaluation. It is expected that in a near future the improved methodology will be formalized in the TECNOS award as a way of ensuring that contesting innovations projects will be evaluated with more objective and consistent criteria.

7. REFERENCES
[1] Center of Quality Management Journal (1993) special issue on Kano´s methods for understanding customer-defined quality, Vol. 2, No. 4, Fall 1993.
[2] Kaplan Stan, “An introduction to TRIZ The Russian Theory of inventive problem solving”, EEUU, IDEATION INTERNATIONAL, 1996.
[3] Soderlin, Pentti “Thoughts on Substance-Field Models and 76 Standards, Do we need all of the Standards?” Management Consultant. Helsinki, Finland, 2003
[4] CREAX Innovation Suite 3.1. (http://www.creaxinnovationsuite.com)
[5] Otto, Kevin and Wood, Kristin “Product Design”, Prentice Hall. USA. 2001.
[6] Altshuller G.S. “Creativity as an exact science”, EEUU, Gordon & Breach, 1988.
[7] Terninko John, Zusman Alla, Zlotin Boris. “Systematic Innovation: An introduction to TRIZ (Theory of Inventive Problem Solving)”, Saint Lucie Press. USA. 1998
[8] Mann, Darrel “Updating TRIZ”, TRIZCON 2003-proceeding, March 2003.