SECTION 1: PROJECT OBJECTIVES AND ACCOMPLISHMENTS
The overall scope of this project concerned the improvement of the prototype realtime numerical weather prediction (NWP) system using the Regional Atmospheric Modeling System (RAMS) at Colorado State University (CSU); its customized application in support of forecast operations and research activities at the National Weather Service (NWS) Weather Forecast Offices (WFOs) in Cheyenne, Wyoming (identifier CYS), and Grand Junction, Colorado (GJT); and an assessment of the value of the RAMS forecasts on operations and research at CYS and GJT. Although a number of changes in personnel, collaborative arrangements, and NWP delivery strategies occurred during the course of this project, that overall scope remained unchanged and significant accomplishments were achieved.
Most importantly, the realtime RAMS forecast system at CSU was improved from the prototype system in use prior to this project to a much more reliable and ongoing fully operational system, with a 36h forecast cycle run once a day from a 00 UTC initialization. A larger nested grid with a finer 12-km grid spacing was implemented that included all of Colorado, most of Wyoming, and parts of adjacent states, including the entire forecast areas for CYS and GJT. A reliable NMC data acquisition and reformatting system was implemented, where the initialization and 6-h forecasts from the NCEP 00 UTC Eta model cycle are used for initialization and time-dependent lateral boundary conditions for RAMS. The dissemination of the RAMS forecasts via the RAMS Homepage (http://rams.atmos.colostate.edu/) was improved by enhancing the quality of the graphical output and by adding a great variety of useful forecast products, including meteorological fields on the surface, constant pressure surfaces, and vertical cross sections, and time series and soundings for numerous locations on the fine grid. In addition, these products are presented in a variety of web-based formats that are tailored uniquely for use by NWS, agriculture, aviation, and fire weather interests. Although the RAMS 00 UTC forecast cycle has run too slowly on the cluster of UNIX workstations to be of optimum use by early-shift forecasters, it has provided very useful "nowcast" information on ongoing mesoscale evolution throughout the day and useful lead-time for forecast timeframes through the afternoon and following night. That daily assessment of the mesoscale NWP performance by RAMS has enabled forecasters and researchers to identify strengths and weaknesses of the model, and to use the model to better understand numerous case studies that were unusual or difficult to forecast.
A major development during the second year (1997) of this project arose due to a high turnover rate of key personnel at CSU and CYS, the resultant difficulty in maintaining a viable mesoscale NWP delivery system, and uncertainty about the status and direction of the project. Most importantly, the initial post-doctoral student at CSU working on the project, Scot Rafkin, left CSU for other employment, and Jim Edwards, a key developer of the RAMS code, left CSU for employment at Forecast Systems Laboratory (FSL). Also, the original Science and Operations Officer (SOO) at CYS and a Co-P.I. on this project, Doug Wesley, left CYS for employment at COMET. Because Edwards continued to work with RAMS at FSL, they had an interest in ongoing model development at CSU. As a result, a formal Memorandum of Agreement (MOA) with FSL was written into this project, wherein FSL and CSU agreed to collaborate on research, development, and technology transfer with RAMS and its physical modules; FSL would provide operational NWP from their version of RAMS to CYS and GJT in a more reliable and timely manner; and CSU would concentrate on model improvements, on development of their operational NWP system as an alternative and backup to the FSL system, and on collaborative research with CYS and GJT forecasters. During this period, the SOO at GJT and an original Co-P.I., Mike Meyers, remained a strong advocate of the NWP provided by RAMS and helped implement the MOA strategies. At CYS, Wesley's successor as SOO, Peter Manousos, forecaster Don Moore, and then the latest SOO, David Copley, were all very important in fostering the project through this period.
The operational NWP effort at FSL using their version of RAMS proceeded on a separate track from that at CSU, but with ongoing collaboration on matters such as data assimilation, initialization and NWP dissemination strategies. System upgrades at FSL led to a considerably faster processing time compared to CSU's version of RAMS, which provided useful lead-time for forecasters and allowed two forecast cycles daily. FSL delivered the RAMS output to CYS and GJT in a GEMPAK format over the Frame Relay Network of the NWS Central Region (CR). This system enabled a wider dissemination potential of the RAMS NWP by allowing other CR WFOs to obtain it, which the WFO in Pueblo (PUB) took advantage of. In addition to this realtime support, FSL helped install local versions of RAMS at CYS, GJT and PUB, to be used for research and case studies, and collaborated with forecasters on some case studies.
During this period, and in accordance with the MOA with FSL, CSU concentrated on improving and testing various physical modules in RAMS. For instance, improvements were made in the bulk microphysics scheme by allowing for more realistic size dependencies on terminal fallspeeds and in interactions between the evolving parameterized hydrometeor types. Furthermore, the implementation of the microphysics scheme was made much more efficient through the use of detailed look-up tables rather than making extensive computations each timestep. An improved surface parameterization was developed that accurately models heat and moisture storage in, and exchanges between, fractional grid-cell coverages of water, snow, soils, vegetation types, and the atmospheric boundary layer. An improved shortwave and longwave radiation scheme was developed that accounts for multiple absorption bands for multiple gaseous constituents and with explicit dependencies on hydrometeors in the microphysics scheme. These have all been implemented or are ready to implement in the realtime RAMS model. Further progress has been made in the development and testing of the Rafkin cumulus parameterization, and we expect to implement it during the coming convective season.
Jason Nachamkin's post-doc entry to this project at CSU in 1998 led to the upgrades of CSU's operational realtime RAMS system noted above. He developed a fruitful research collaboration with the SOOs and other forecasters at CYS and GJT, wherein he assisted in many case studies by conducting either detailed analysis of archived operational RAMS output or performing post-case re-simulations or more customized simulations of the cases. After Nachamkin's departure for other employment in 1999, Ray McAnelly assumed his duties in this project at CSU and continued maintaining the operational realtime model and participating in collaborative research with forecasters at CYS and GJT.
The general assessment of the realtime RAMS NWP from CSU, based on case studies (many are cited below), NWS forecaster feedback, and our own experience, is that the increased resolution provided by the 12-km Grid 2 results in better resolved and more accurate circulations in complex terrain than the coarser Eta and other NCEP models. For mesoscale and synoptic precipitating systems, this results in more accurate spatial detail in the quantitative precipitation forecast (QPF) distribution than in the NCEP models. The detailed microphysics in RAMS is sometimes able to accurately model liquid and frozen precipitation distribution as a function of slight elevation differences. Evaluations of forecast precipitation over extended periods (e.g., Gaudet and Cotton 1998; seasonal accumulation on The RAMS Homepage) show distributions that compare well with SNOTEL and other observations, with some systematic biases due to elevation differences between observation sites and the 12-km topography in the model. Initiation of convection in high terrain is better predicted in RAMS than in the Eta and other NCEP models, along with the propagation of multi-cell storms eastward onto the plains. However, convective precipitation amounts are often over-predicted in RAMS, and isolated convection often is not predicted, due to an inability to resolve convection well on a 12-km grid.
In addition to the slow run-time of CSU's realtime RAMS forecast model, a major assessment of both the CSU and FSL versions is that the NWP dissemination methods (web-based graphics and transmitted files for locally generated images using GEMPAK, respectively) are less than optimal for use by forecasters. This is because their primary means of analyzing observational data, evaluating the NCEP model guidance, and formulating specific forecasts is on the AWIPS system, while the RAMS guidance has to be viewed and evaluated separately in a different and more inconvenient graphical format. Thus while they find the RAMS guidance useful if enough time is available for fully assimilating and evaluating it, operational time constraints make that difficult. Ongoing efforts to rectify this problem by making the RAMS NWP compatible with the local WFO AWIPS system are underway as noted below.
In late 1999, a newer version of realtime RAMS forecast model was implemented on a Linux PC cluster of 16 processors at CSU. The NWS and COMET through the WFO in GJT purchased 25% of this system. This new prototype version has been running quasi-operationally for several months along with the older realtime RAMS model and will soon replace it. The new system runs about 3-5 times faster than the older system, which will provide much greater and useful lead-time to NWS forecasters. This increased performance allows for a 48h forecast cycle to be run twice daily, with a larger Grid 1 covering the lower 48 states, a 12-km Grid 2 of the same size as in the operational version (with potential for 5-km grid spacing), and an additional nested Grid 3 with much finer spacing (currently 3 km). The 00 UTC cycle is dedicated to NWS CR interests under this project along with other interests (e.g., Colorado Agricultural Experiment Station) that funded the new system, with Grid 2 covering all of Colorado, most of Wyoming, and portions of adjacent states as in the older version. Grid 3 (currently 120 km ´ 120 km) is relocatable to support more specific interests: it was applied through the winter to a seedability feasibility study over the northern Colorado Rockies, is currently supporting a wave cloud field project based out of Laramie, and will be used for such operations as fire weather forecasting at the request of NWS forecasters during the upcoming fire season. The 12 UTC cycle is dedicated for DOD-funded interests, with Grids 2 and 3 located at various locations in the lower 48 states.
SECTION 2: SUMMARY OF UNIVERSITY/ NWS/AFWA/NAVY EXCHANGES
There were many meetings between CSU, NWS, FSL, and other NOAA personnel concerning this project. These included brainstorming meetings where modelers and forecasters developed strategies for developing the most useful mesoscale NWP systems, special applications for RAMS NWP such as for fire weather forecast operations, special seminars on the realtime RAMS forecast system given by CSU modelers to forecasters at CYS and GJT, and invited presentations by the CSU P.I., Bill Cotton, at NWS workshops. During his current sabbatical, Bill Cotton has given seminars on the realtime RAMS system in Argentina, Kenya, and India and at the NWS Headquarters in the Alaska Region. Collaborative case studies and research projects involving CSU, NWS forecasters at CYS and GJT, and other NOAA personnel have been documented in many conference papers and presentations and at various workshops.
SECTION 3: PRESENTATIONS AND PUBLICATIONS
Gaudet, B., and W.R. Cotton, 1998: Statistical characteristics of a real-time precipitation forecasting model. Wea. Forecasting, 13, 966-982.
Jones, C.N., J.D. Colton, M.P. Meyers and J.E. Nachamkin, 1999: A heavy snow event during an Arctic outbreak across the central Rockies. Preprints, 17th Conf. on Wea. Anal. and Forecasting (13-17 September 1999, Denver, CO), Amer. Meteor. Soc., Boston, MA), Paper 4.5.
Meyers, M.P., J. Edwards, J. Snook, D. Wesley, J. Nachamkin, W. Cotton, P. Manousos, and P. Wolyn, 1998: Mesoscale forecast model (RAMS) applications in a Weather Service office viewed locally on a N-AWIPS platform. 12th Conference on Numerical Weather Prediction. 11-16 January 1998, Phoenix, Arizona.
Meyers M. P., E. Holloway, C. Peterson, J. E. Nachamkin, J. Edwards and W. Cotton, 1998: Operational application of a Mesoscale Model for quantitative precipitation forecasts over complex terrain. Preprints, 8th Conference on Mountain Meteorology, Flagstaff, AZ.
Meyers M. P., J. Snook, D. Wesley and G. Poulos, 1998: Numerical investigation of the 25 October 1997 blowdown event over the Colorado Park Range using RAMS. Preprints, 8th Conference on Mountain Meteorology, Flagstaff, AZ.
Meyers, M.P., J. Snook, D. Wesley, and G. Poulos, 2000: A Rocky Mountain storm - Part II: The forest blowdown - observations, dynamics and modeling. To be submitted to Wea. Forecasting.
Meyers, M.P., D.A. Wesley, S.C.R. Rafkin, T.L. Jensen, J. Edwards, W.R. Cotton, 1996: Mesoscale model applications in the forecast office. Part I: RAMS model configuration for operations. Proc., 11th Conf. on Numerical Weather Prediction, 19-23 August 1996, Norfolk, VA, AMS.
Nachamkin, Jason E., Multiscale Numerical Prediction of Convection in Complex Terrain. Presentation at the Naval Research Lab, Monterey, CA. 13 October 1998.
Nachamkin, J. E., W. R. Cotton, D. Moore, and M. P. Meyers, 1998: Real--time Forecasting at Colorado State University. Preprints, Second Workshop on Real--Time Mesoscale Numerical Weather Prediction, Boulder, CO. June 17--18, 1998.
Nachamkin, J.E., R.L. McAnelly, M. Weiland, K. Daugherty, D. Copley and W.R. Cotton, 1999: Predictability and structure of an intense orographic snowfall event in eastern Wyoming. Preprints, Eighth Conf. on Mesoscale Processes (28 June - 1 July 1999, Boulder, CO), Amer. Meteor. Soc., Boston, MA, 357-360 (Paper P2.2).
Nachamkin J.E., M.P. Meyers, Colton, Jones, 1999: Real-time prediction of frontal and orographic snowfall using a high-resolution numerical model. Submitted, Weather and Forecasting.
Page, E.M., M.P. Meyers, M. Chamberlain and R. McAnelly, 2000: Operational use of mesoscale models in fire weather forecasting. To appear in Preprints, Third Symposium on Fire and Forest Meteorology (9-14 January 2000, Long Beach, CA), Amer. Meteor. Soc., Boston, MA, Paper 4.1).
Poulos, G., D. Wesley, M.P. Meyers and J. Snook, 2000: A Rocky Mountain storm - Part I: The blizzard - observations, dynamics and modeling. Submitted to Wea. Forecasting.
Snook J.S., M.P. Meyers, G.S. Poulos and D.A. Wesley, 1999: Longer range numerical prediction of the October 1997 Routt National Forest severe wind event. Preprints, 13th Conf. on Numerical Wea. Prediction (13-17 September 1999, Denver, CO), Amer. Meteor. Soc., Boston, MA, Paper J3.3.
Snook J., M. Meyers, D. Wesley, G Poulos, 1999: A Rocky Mountain storm - Part II: The Forest Blowdown - observations, dynamics and modeling. Submitted, Weather and Forecasting, Denver, CO.
Wesley, D.A., M. Meyers, S. Rafkin, T. Jensen, J. Edwards, W.R. Cotton, L. Engebretson, 1996: Mesoscale model applications in the forecast office. Part II: Initial impacts on operations and forecast products. Proc., 11th Conf. on Numerical Weather Prediction, 19-23 August 1996, Norfolk, VA, AMS.
Wesley, D.A., G.S. Poulos, J.S. Snook and M.P. Meyers, 1999: Meso-gamma scale snowfall and wind variability in a Front Range blizzard. Preprints, 17th Conf. on Wea. Anal. and Forecasting (13-17 September 1999, Denver, CO), Amer. Meteor. Soc., Boston, MA, Paper 4.7.
Wolyn, P., 1999: The 27-29 November 1997 southern Colorado snowstorm: Case study and modeling investigation. Preprints, Eighth Conf. on Mesoscale Processes (28 June - 1 July 1999, Boulder, CO), Amer. Meteor. Soc., Boston, MA, 351-356 (Paper P2.1).
SECTION 4: SUMMARY OF BENEFITS AND PROBLEMS ENCOUNTERED
4.1 Important educational benefits resulting from this project at CSU include a greater understanding of mesoscale phenomena and how mesoscale NWP should be interpreted. This learning has resulted from day to day-observational evaluation of the RAMS model performance, from routine comparison of the RAMS NWP with Eta and other NCEP model guidance, and from many case studies with CYS and GJT forecasters. For instance, we've learned that vorticity advection on the scale of the operational Grid 2 is not as important as has been demonstrated on synoptic scales, and that deficiencies in soil moisture specification can result in biases in boundary layer temperature, moisture, and mountain slope solenoidal circulation intensity. Cases studied include tornadic storms, severe hailstorms, flash floods, mountain snowstorms, severe localized snowfall and blizzards on the High Plains associated with intense cyclonic storm systems, and forest blowdown events. These have provided tremendous insight into the mesoscale environment and the multiscale interactions with topography and the larger-scale setting that accompany such a fascinating diversity of weather phenomena.
In addition to those scientific and academic benefits, further educational benefits at CSU include a greater understanding and appreciation of the operational problems and constraints faced by NWS forecasters. Specific forecast products such as time series at numerous locations and products tailored for fire weather forecast operations have been identified, thus enabling us to provide more comprehensive web-based products for specific applications. The problems encountered in developing and maintaining a reliable realtime mesoscale forecast model and disseminating its output have been very educational to students and staff at CSU. Through the course of this project, a succession of undergraduate students in computer science has been instrumental in building, maintaining and upgrading the parallel-processor computer systems involved with the model and in JAVA and other web-page progamming support for disseminating the output.
Although much of the research and development (R&D) work noted earlier on the various physical modules in RAMS was not funded directly by this project, their implementation in the RAMS forecast model provides a valuable test bed for their evaluation and refinement, which leads to further improvements in overall model performance. These symbiotic benefits, between this project and other R&D work at CSU, extend beyond CSU as well. For instance, many of the improvements in NWP at major centers such as NCEP originate from R&D in universities, and are implemented only when fully tested. Similarly, the fruits of this R&D are shared with research laboratories such as FSL and NCAR. Thus this and similar Cooperative Projects are very important in the advancement of NWP in general.
The largest problems encountered in this project have been day-to-day consistency in running the model, the inadequate lead-time of the RAMS NWP for forecasters, and an inability to deliver the NWP in a manner best suited for forecasters. The inconsistency and lead-time problems were partially solved early in the project when FSL began running its version of RAMS and providing its output to CYS and GJT. At CSU, personnel stabilization over the last two years along with hardware upgrades have resulted in a much more reliable system. Over the last six months, the realtime RAMS model at CSU has completed its once daily 36h forecast cycle 79% of the time and has made at least partial runs on 87% of the days. With the experience gained and various improvements being made (e.g., using alternative NCEP or NAVY NWP datasets for initialization and boundary conditions when the Eta model is unavailable), we should be able to achieve and maintain at least a 95% run rate. The slow run-time at CSU is being solved with the transition of the operational realtime system to the newer version of RAMS running on a Linux PC cluster. We hope to improve the utility of the RAMS NWP for forecasters as part of a new Cooperative Project with GJT by disseminating output files electronically through the local WFO LDAD and processing the data for viewing and analysis in AWIPS. That data dissemination system for utilization in AWIPS could then be extended to CYS and other WFOs through CR facilities. Similarly, FSL and CYS have been developing procedures for getting mesoscale NWP from FSL into the CYS AWIPS system.
4.2.1 NWS CHEYENNE (David Copley)
Important operational benefits derived from the collaboration between CSU and NWS CYS include: significant exposure to a true mesoscale model and the ability to integrate mesoscale guidance into the forecast process in real time; a heightened awareness of mesoscale processes over the Cheyenne County Warning Area (CWA); the unique ability to better forecast very localized weather patterns within the CWA; the opportunity to participate in studies and research alongside a University partner; and finally, the ability to use the RAMS model as a post analysis tool.
The cornerstone of the NWS modernization is the new Automated Weather Information Processing System (AWIPS). With AWIPS the forecaster has the capability to analyze and display a vast array of numerical weather information. The goal is to provide better, more accurate forecasts over a smaller aerial footprint. To accomplish this requires a data density several orders of magnitude greater than the traditional synoptic scale used throughout the 1970s and 1980s. We are now working with data spacing on the order of kilometers rather than tens of kilometers in the horizontal and tens of meters versus hundreds of meters in the vertical. The RAMS model is one of a hand full of true prognostic models that fit these requirements. This level of data density is required in all NWS forecast offices if we are to achieve our stated goal.
Forecasters along the Front Range of the Rocky Mountains have always known that the interactions between approaching weather systems and the highly variable terrain are extremely complex. By using the RAMS model in real-time, we were able to more precisely determine some of these interactions leading to better forecast products. We found that the more precise representation of orographic features and the higher density boundary layer provided a distinct advantage in determining such parameters as the location of enhanced precipitation and surface wind fields. We could more precisely locate frontogenetical processes as upper level systems transition over the mountains.
Finally, the collaboration with CSU demonstrated the flexibility of a local mesoscale model and the capacity for research provided by reinitializing such a model for storm post-analysis. Working with a variety of researchers at CSU the Cheyenne staff had the opportunity to explore a number of significant weather events over the local warning area and contributed to approximately six published articles. The ability to tune the RAMS model during post-analysis can highlight specific model parameterizations or better illustrate very subtle atmospheric interactions that may go undetected in real time and at larger scales of motion.
A collaborative effort is not without its problems and this one was no exception. However, during the course of this collaboration we were able to address each problem and in most cases find an acceptable solution. Among the more significant problems associated with the RAMS model collaboration were: a rapid staff turnover at both CSU and the Cheyenne Office; model guidance availability in near real time; and the loss of access to FSL real time RAMS guidance.
As already discussed in other sections, there was a rapid staff turnover at both CSU and the Cheyenne office during the course of this project. Cheyenne lost two Science and Operations Officers during the three years that this project was in effect. At the same time CSU lost an equivalent number of researchers. In some cases the turnover of personnel lead to enhanced project opportunities, for example the access to real time RAMS guidance via FSL. While CSU could recover quickly from personnel turnover it was more difficult for the Cheyenne office to respond.
During the initial phase of the RAMS collaboration, the model was run on a collection of workstations, but the computational overhead made it very difficult to generate model guidance in time to be most useful in forecast operations. Since that time, CSU has found creative solutions to the computational speed problem and at one point FSL was able to enter into a memorandum of agreement with CSU and the Cheyenne office to produce RAMS model guidance in grib format that could be pushed directly into the Cheyenne SAC workstation. This was probably the most productive period in the collaboration. With a mesoscale model run available at approximately the same cycle time as the RUC model Cheyenne forecasters could incorporate RAMS guidance directly into real time forecast products.
Unfortunately, in the late spring of 1999, FSL was forced by circumstance to terminate their real time RAMS guidance feed into the Cheyenne office. While advances in computer power at CSU improved RAMS model availability on the World Wide Web, this was not sufficient compensation to having the ability to display the RAMS guidance in conjunction with the NCEP model suite. Use of real time RAMS guidance went into sharp decline. At approximately this same time, the AWIPS system was installed in Cheyenne. The focused training and large learning curve associated with the AWIPS system further limited the use of RAMS mesoscale guidance. Now, approximately nine months after the AWIPS installation, I can detect renewed interest in real time mesoscale model guidance with the CSU RAMS model being the model of choice.
The collaboration between the Cheyenne weather office and CSU was an enriching experience for all concerned. The staff in Cheyenne was provided with a glimpse of the future of model guidance in the mesoscale. The Atmospheric Sciences Department at CSU gained a much better appreciation for operational forecast concerns. From the perspective of a Forecast Office, the most significant lesson learned would have to be that mesoscale models must be available to every National Weather Service Forecast Office. There are plans to develop a national mesoscale model to add to the suite of NWS model guidance. However, each office needs a local mesoscale model that can be tuned and run by the operational staff. The local model guidance must be integrated into AWIPS alongside the NWS model suite. The local mesoscale model will be used to enhance forecaster confidence in the various model solutions. It will provide a tool for local research and event post analysis. And, it will be a useful training tool for numerical weather prediction. The stated goal of the National Weather Service modernization is to improve forecasts and warnings by providing the public with products that meet their needs. Mesoscale models are just one tool to achieve this goal.
4.2.2 NWS GRAND JUNCTION (Michael P. Meyers)
Local modeling is important in any location, but I believe it is even more important over the intermountain west. During the winter months the RAMS model did a much better job with orographically-forced events (which are most of the events) than other operational models. With more sophisticated physics and better representation of the topography, it did a better job with QPF over the region, especially in the bigger storms that the other models had a tendency to smooth out due to their simplified physics and coarse representation of topography. In the convective season, RAMS has been incredible at times, and has looked awful at times. There have been many situations when RAMS will predict large amounts of precipitation (over inch totals) where other operational models have predicted only modest amounts (less than 0.1) and large amounts are observed. This again is due to the better representation of topography and more sophisticated physics in the model. In many of the situations, RAMS was able to initiate convective-like responses due to mesoscale forcing. In the 10 km grid spacing regime, convective parameterizations have a very difficult time representing convection. However, if the forcing is there, the model will generate the updraft and the resultant precipitation on a mesoscale rather than convective scale, basically simulating the convective complex. Both GJT and CYS have documented cases to this effect. From the forecasters' perspective, they have been impressed. It gives them another tool to peruse, and for the bigger events, the model seems to give a bigger "bang for the bucks." The negative side of RAMS has been the fact that it tends to produce too much precipitation in some of the events, and it sometimes inaccurately predicts the locations of the events. The biggest drawback of this research has been the fact that the model output is not available on the AWIPS architecture at this time. It would facilitate the transfer of information to the forecasters if RAMS were available on AWIPS.
Perhaps the biggest argument for RAMS has been the increase in research projects, many of which have used RAMS in their research. One specific event, a high wind event that occurred over the north-central mountains of Colorado, incorporated RAMS as the primary tool to determine the physical controls on this type of wind phenomena. Subsequently, there have been several events that were forecasted by the staff, in a large part, due to this research.