Difference between revisions of "Measuring High-Growth High-Technology Entrepreneurship Ecosystems"

From edegan.com
Jump to navigation Jump to search
Line 6: Line 6:
 
==Current Version==
 
==Current Version==
  
−
The current version was submitted as an R&R to a Special Issue of Research Policy on June 10th, 2020, with manuscript number RESPOL-D-19-01438R1.  On September 15th, I sent an email to the editors requesting information but received no response. The last status reported by Elvesier (https://ees.elsevier.com/respol/default.asp) was 'Required Reviews Complete' on October 9th, 2020. I wrote to the editors again on October 27th, this time using the Elvesier form, to request another update.
+
The current version. which is currently a 2nd R&R at Research Policy is:
  
 
<pdf>File:MeasuringHGHTEntrepreneurshipEcosystemsV3-3.pdf</pdf>  
 
<pdf>File:MeasuringHGHTEntrepreneurshipEcosystemsV3-3.pdf</pdf>  
 +
 +
===Timeline of Submission===
 +
 +
This paper had a storied submission process:
 +
*The deadline for resubmission was June 15th, 2020. Before this deadline, I emailed the editors and offered them either this version of the paper, which contains no empirics, or an empirical paper without examples or definitions. I received no response.
 +
*This version of the paper was submitted as an R&R to a Special Issue of Research Policy on June 10th, 2020, with manuscript number RESPOL-D-19-01438R1.
 +
*On September 15th, I sent an email to the editors requesting information but received no response.
 +
*I wrote to the editors again on October 27th, this time using the Elvesier form, to request another update. The last status reported by Elvesier (https://ees.elsevier.com/respol/default.asp) was 'Required Reviews Complete' on October 9th, 2020.
 +
*On October 28th, I received an email saying: "Hello Ed. I hope to get back to you shortly. I have two good reviews and I’m waiting on a third. This most definitely will be another R&R. More soon"
 +
*On November 8th, I got an official email about the paper that said: "We have now received the referees' reports on your paper, copies of which I enclose below for your information.  As you will see, the referees make various comments and suggestions for improvement. I have given up on the third reviewer and want to return the paper to you." However, this email only contained one review. I requested clarification and noted that Reviewer 3 had asked for empirics.
 +
On November 11th I got an email that said: "Hello Ed,  This is strange. The comments were in the comments to the editor.  Here they are and they are not worth that much.  This special issue has a specific purpose.  '''You do not need to run regressions!'''" The comments are below as Reviewer 1. They indicate that the reviewer accepted the paper.
 +
 +
===2nd R&R===
 +
 +
Note that there Reviewer 2 never returned any comments.
 +
 +
====Reviewer 1's Comments====
 +
 +
First of all, my sincere apologies to you and the authors for my very late report on the manuscript. Please excuse me for the delay.
 +
 +
The authors study measures of high-growth, high-tech entrepreneurship activity across the U.S. and provide concrete examples of how municipalities can use these to assess various policy initiatives.  The authors claim that policy interventions at the municipal level have a significant impact on pre-venture startups, and that this has been missing from the extant academic literature. 
 +
 +
Regarding my take on the paper: it is well done and given that it has already been revised once, I don't have many more major comments, but I do have one thing I would like to bring to your attention -- although the paper is polished, it doesn't seem like an academic paper with extensive empirical analysis (in fact, it doesn't have even a single regression) or with an analytical model that enhances our understanding of theory (doesn't have a single equation either). It is a collection of case studies and definitions, albeit very nicely done. Given my area of expertise in entrepreneurial finance and my own experience publishing an empirical study in ResPol, I feel ill-equipped to offer a recommendation on whether or not this paper fits the scope of ResPol. I believe that is largely an editorial decision and you would be the best judge for it. So, I will let you decide on this bigger question and will skip my minor suggestions for the author. I hope that is alright.
 +
 +
====Reviewer 3's Comments====
 +
 +
Reviewer #3: I'm recommending a major revision to the paper, which would likely constitute a lot of additional work, but I feel would greatly strengthen its contribution.
 +
 +
The objective of this paper is establishing fifteen measures of HGHT entrepreneurship activity, and giving examples of their application and potential usefulness especially with regard to the behavior of what the author terms policy cartels.
 +
 +
The defining of key terms is a useful contribution of this paper, as are the identification of potentially useful metrics of HGHT entrepreneurship. Furthermore, the examples are often helpful in highlighting their various applications.
 +
 +
In my view, the major flaws in the study are not (1) considering possible downsides of each individual metric, (2) considering possible downsides in using the entire battery of measures, and (3) testing alternatives to this measurement approach. I briefly elaborate on these three points in the following paragraphs.
 +
 +
In the conclusion, the author states that two antecedents to improved policy are "standardized measures, to reduce the information asymmetry between policymakers and constituents…and…a simple but grounded framework that can reduce the expertise required to develop and enact productive startup policy."
 +
 +
First, it is not necessarily true that "standardized measurement" improves outcomes, such as policy, overall. Measurements are rarely if ever neutral (vis-à-vis behavior). Two examples are,
 +
*      First, we often implicitly assume that more information in the hands of decision makers is unambiguously good. But this requires a lot of assumptions that do not hold in real life. An additional measure, say, would influence decision making but perhaps the distortion is welfare decreasing. Nassim Taleb gives the example of "value at risk" (VAR), a metric commonly used in finance, in the foreword of the book Lecturing Birds on Flying (pp. xvi).
 +
*      Second, systematic bias due to gaming metrics incentivized by rewards attached to the measurement. Said differently, an agent would have the incentive to change behavior in the least costly way in order to maximize the payoff associated with the measurement---this behavioral change might have nothing to do with the underlying phenomenon of interest. By way of example, Weisbrod, Ballou, Asch in their book Mission and Money, discuss various measures used in university rankings published by US News & World Report, and how these are often finessed by schools in ways that have little to do with education (see pp. 64-65 for one such discussion).
 +
 +
The point is, any conceived standardized framework is not necessarily better than nothing, and not all frameworks would be of equal value. I would have liked to see a more rigorous discussion of the merits of the framework purposed, which analysis would include at least the following:
 +
*      Discussion of alternative calculations of a measure when applicable. For example, the paper's Measure 1 is a composite measure of three sub-measures: why was this particular normalization of sub-measures (i.e., ranking) used? Why was this particular aggregation method (i.e., summation) used? What are the upsides and downsides of this and other approaches? For example, "the flow of dollars" would have a long right tail, which is obliterated when transformed to a rank. Is this a good or bad thing and why?
 +
*      Measurement is often reductive (in the sense that they constitute a mapping from a high-dimensional, potentially complex space, to a far simpler space). What is left out? Is it important? In part this translates to discussing the difference between the conceptual phenomenon and how it is operationalized?
 +
*      Might a measure be systematically biased or lead to bias if implemented?
 +
*      Tests of the effects of using a framework on various outcomes.
 +
In summary, more skepticism about the usefulness of the metrics and the framework, and more empiricism is necessary.
  
 
==Files==
 
==Files==

Revision as of 12:58, 7 December 2020

Academic Paper
Title Measuring High-Growth High-Technology Entrepreneurship Ecosystems
Author Ed Egan
Status R and R
© edegan.com, 2016

Current Version

The current version. which is currently a 2nd R&R at Research Policy is:

Timeline of Submission

This paper had a storied submission process:

  • The deadline for resubmission was June 15th, 2020. Before this deadline, I emailed the editors and offered them either this version of the paper, which contains no empirics, or an empirical paper without examples or definitions. I received no response.
  • This version of the paper was submitted as an R&R to a Special Issue of Research Policy on June 10th, 2020, with manuscript number RESPOL-D-19-01438R1.
  • On September 15th, I sent an email to the editors requesting information but received no response.
  • I wrote to the editors again on October 27th, this time using the Elvesier form, to request another update. The last status reported by Elvesier (https://ees.elsevier.com/respol/default.asp) was 'Required Reviews Complete' on October 9th, 2020.
  • On October 28th, I received an email saying: "Hello Ed. I hope to get back to you shortly. I have two good reviews and I’m waiting on a third. This most definitely will be another R&R. More soon"
  • On November 8th, I got an official email about the paper that said: "We have now received the referees' reports on your paper, copies of which I enclose below for your information. As you will see, the referees make various comments and suggestions for improvement. I have given up on the third reviewer and want to return the paper to you." However, this email only contained one review. I requested clarification and noted that Reviewer 3 had asked for empirics.

On November 11th I got an email that said: "Hello Ed, This is strange. The comments were in the comments to the editor. Here they are and they are not worth that much. This special issue has a specific purpose. You do not need to run regressions!" The comments are below as Reviewer 1. They indicate that the reviewer accepted the paper.

2nd R&R

Note that there Reviewer 2 never returned any comments.

Reviewer 1's Comments

First of all, my sincere apologies to you and the authors for my very late report on the manuscript. Please excuse me for the delay.

The authors study measures of high-growth, high-tech entrepreneurship activity across the U.S. and provide concrete examples of how municipalities can use these to assess various policy initiatives. The authors claim that policy interventions at the municipal level have a significant impact on pre-venture startups, and that this has been missing from the extant academic literature.

Regarding my take on the paper: it is well done and given that it has already been revised once, I don't have many more major comments, but I do have one thing I would like to bring to your attention -- although the paper is polished, it doesn't seem like an academic paper with extensive empirical analysis (in fact, it doesn't have even a single regression) or with an analytical model that enhances our understanding of theory (doesn't have a single equation either). It is a collection of case studies and definitions, albeit very nicely done. Given my area of expertise in entrepreneurial finance and my own experience publishing an empirical study in ResPol, I feel ill-equipped to offer a recommendation on whether or not this paper fits the scope of ResPol. I believe that is largely an editorial decision and you would be the best judge for it. So, I will let you decide on this bigger question and will skip my minor suggestions for the author. I hope that is alright.

Reviewer 3's Comments

Reviewer #3: I'm recommending a major revision to the paper, which would likely constitute a lot of additional work, but I feel would greatly strengthen its contribution.

The objective of this paper is establishing fifteen measures of HGHT entrepreneurship activity, and giving examples of their application and potential usefulness especially with regard to the behavior of what the author terms policy cartels.

The defining of key terms is a useful contribution of this paper, as are the identification of potentially useful metrics of HGHT entrepreneurship. Furthermore, the examples are often helpful in highlighting their various applications.

In my view, the major flaws in the study are not (1) considering possible downsides of each individual metric, (2) considering possible downsides in using the entire battery of measures, and (3) testing alternatives to this measurement approach. I briefly elaborate on these three points in the following paragraphs.

In the conclusion, the author states that two antecedents to improved policy are "standardized measures, to reduce the information asymmetry between policymakers and constituents…and…a simple but grounded framework that can reduce the expertise required to develop and enact productive startup policy."

First, it is not necessarily true that "standardized measurement" improves outcomes, such as policy, overall. Measurements are rarely if ever neutral (vis-à-vis behavior). Two examples are,

  • First, we often implicitly assume that more information in the hands of decision makers is unambiguously good. But this requires a lot of assumptions that do not hold in real life. An additional measure, say, would influence decision making but perhaps the distortion is welfare decreasing. Nassim Taleb gives the example of "value at risk" (VAR), a metric commonly used in finance, in the foreword of the book Lecturing Birds on Flying (pp. xvi).
  • Second, systematic bias due to gaming metrics incentivized by rewards attached to the measurement. Said differently, an agent would have the incentive to change behavior in the least costly way in order to maximize the payoff associated with the measurement---this behavioral change might have nothing to do with the underlying phenomenon of interest. By way of example, Weisbrod, Ballou, Asch in their book Mission and Money, discuss various measures used in university rankings published by US News & World Report, and how these are often finessed by schools in ways that have little to do with education (see pp. 64-65 for one such discussion).

The point is, any conceived standardized framework is not necessarily better than nothing, and not all frameworks would be of equal value. I would have liked to see a more rigorous discussion of the merits of the framework purposed, which analysis would include at least the following:

  • Discussion of alternative calculations of a measure when applicable. For example, the paper's Measure 1 is a composite measure of three sub-measures: why was this particular normalization of sub-measures (i.e., ranking) used? Why was this particular aggregation method (i.e., summation) used? What are the upsides and downsides of this and other approaches? For example, "the flow of dollars" would have a long right tail, which is obliterated when transformed to a rank. Is this a good or bad thing and why?
  • Measurement is often reductive (in the sense that they constitute a mapping from a high-dimensional, potentially complex space, to a far simpler space). What is left out? Is it important? In part this translates to discussing the difference between the conceptual phenomenon and how it is operationalized?
  • Might a measure be systematically biased or lead to bias if implemented?
  • Tests of the effects of using a framework on various outcomes.

In summary, more skepticism about the usefulness of the metrics and the framework, and more empiricism is necessary.

Files

The files are in: 

E:\projects\MeasuringHGHTEcosystems
/bulk/vcdb4

Notice

This paper was broken into two:

Research Policy Special Issue

This paper is for a Special Issue of Research Policy, organized by/for the UMM grant cohort. The deadline for submission is Nov 30th, 2019. See: https://www.journals.elsevier.com/research-policy/call-for-papers/uncommon-methods-and-metrics

Examples of questions that papers could address are:

  1. What fundamental constructs or elements might constitute a theory or theoretical base for the geographically defined entrepreneurial ecosystem?
  2. What are general definitions of entrepreneurial ecosystems so that entrepreneurial ecosystems can be measured in a consistent way across all sectors?
  3. What key relationships need to be captured at the entrepreneurial ecosystem level?
  4. How should the impact of local entrepreneurial ecosystems on economic growth at the national level be measured?
  5. Whose performance (and what) should be measured? Should researchers look at a mix of short- and long-term measures?
  6. Do existing rankings for entrepreneurial ecosystems measure what they claim to measure?
  7. To what extent are entrepreneurial ecosystems and innovation related?
  8. What are the salient levels of analysis (e.g., cultural, institutional, spatial) to consider when analyzing entrepreneurial behavior?How do the characteristics of entrepreneurial ecosystems vary by country?
  9. By which mechanisms do entrepreneurial ecosystems get established, mature, decline, or get renewed?
  10. What are the trade-offs between attracting entrepreneurs to a city, and solving urban problems such as affordable housing?
  11. Under what circumstances could a university be considered an ecosystem, and how does this interact with entrepreneurial ecosystems?What are more finely grained evaluations of the effectiveness of policy instruments that capture connections and ties across entrepreneurial ecosystems?
  12. To what extent is government policy accelerating or inhibiting the progress of entrepreneurial ecosystems?

Data and Analysis

The paper will use vcdb4 and US Startup City Ranking, as well as a wealth of old McNair material. Sources include (copied to the project folder unless otherwise noted):

Retrieved from "http://www.edegan.com/mediawiki/index.php?title=Measuring_High-Growth_High-Technology_Entrepreneurship_Ecosystems&oldid=48294"