This protocol was designed to train a machine learning algorithm to use a combination of imaging parameters derived from magnetic resonance imaging (MRI) and positron emission tomography/computed tomography (PET/CT) in a rat model of breast cancer bone metastases to detect early metastatic disease and predict subsequent progression to macrometastases.
Machine learning (ML) algorithms permit the integration of different features into a model to perform classification or regression tasks with an accuracy exceeding its constituents. This protocol describes the development of an ML algorithm to predict the growth of breast cancer bone macrometastases in a rat model before any abnormalities are observable with standard imaging methods. Such an algorithm can facilitate the detection of early metastatic disease (i.e., micrometastasis) that is regularly missed during staging examinations.
The applied metastasis model is site-specific, meaning that the rats develop metastases exclusively in their right hind leg. The model’s tumor-take rate is 60%–80%, with macrometastases becoming visible in magnetic resonance imaging (MRI) and positron emission tomography/computed tomography (PET/CT) in a subset of animals 30 days after induction, whereas a second subset of animals exhibit no tumor growth.
Starting from image examinations acquired at an earlier time point, this protocol describes the extraction of features that indicate tissue vascularization detected by MRI, glucose metabolism by PET/CT, and the subsequent determination of the most relevant features for the prediction of macrometastatic disease. These features are then fed into a model-averaged neural network (avNNet) to classify the animals into one of two groups: one that will develop metastases and the other that will not develop any tumors. The protocol also describes the calculation of standard diagnostic parameters, such as overall accuracy, sensitivity, specificity, negative/positive predictive values, likelihood ratios, and the development of a receiver operating characteristic. An advantage of the proposed protocol is its flexibility, as it can be easily adapted to train a plethora of different ML algorithms with adjustable combinations of an unlimited number of features. Moreover, it can be used to analyze different problems in oncology, infection, and inflammation.
The purpose of this protocol is to integrate several functional imaging parameters from MRI and PET/CT into a model-averaged neural network (avNNet) ML algorithm. This algorithm predicts the growth of macrometastases in a rat model of breast cancer bone metastases at an early timepoint, when macroscopic changes within the bone are not yet visible.
Prior to the growth of macrometastases, a bone marrow invasion of disseminated tumor cells occurs, commonly referred to as micrometastatic disease1,2. This initial invasion can be considered an early step in metastatic disease, but is typically missed during conventional staging examinations3,4. Although the currently available imaging modalities cannot detect bone marrow microinvasion when used alone, a combination of imaging parameters yielding information on vascularization and metabolic activity has been shown to perform better5. This complementary benefit is achieved by combining different imaging parameters into an avNNet, which is an ML algorithm. Such an avNNet allows for the reliable prediction of bone macrometastases formation before any visible metastases are present. Therefore, integrating imaging biomarkers into an avNNet could serve as a surrogate parameter for bone marrow microinvasion and early metastatic disease.
To develop the protocol, a previously described model of breast cancer bone metastases in nude rats was used6,7,8. The advantage of this model is its site-specificity, meaning that the animals develop bony metastases exclusively in their right hind leg. However, the tumor-take rate of this approach is 60%–80%, so a considerable number of the animals do not develop any metastases during the study. Using imaging modalities such as MRI and PET/CT, the presence of metastases is detectable from day 30 postinjection (PI). At earlier time points (e.g., 10 PI) imaging does not distinguish between animals that will develop metastatic disease and those will not (Figure 1).
An avNNet trained on functional imaging parameters acquired on day 10 PI, as described in the following protocol, reliably predicts or excludes the growth of macrometastases within the following ~3 weeks. Neural Networks combine artificial nodes within different layers. In the study protocol, the functional imaging parameters for bone marrow blood supply and metabolic activity represent the bottom layer, while the prediction of malignancy represents the top layer. An additional intermediate layer contains hidden nodes that are connected to both the top and the bottom layer. The strength of the connections between the different nodes is updated during the training of the network to perform the respective classification task with high accuracy9. The accuracy of such a neural network can be further increased by averaging the outputs of several models, resulting in an avNNet10.
All care and experimental procedures were performed in accordance with national and regional legislation on animal protection, and all animal procedures were approved by the State Government of Franconia, Germany (reference number 55.2 DMS-2532-2-228).
1. Induction of breast cancer bone metastases in the right hind leg of nude rats
NOTE: A detailed description of the induction of breast cancer bone metastases in nude rats has been published elsewhere6,8. The most relevant steps are presented below.
- Culture MDA-MB-231 human breast cancer cells in RPMI-1640, supplemented with 10% fetal calf serum (FCS). Keep the cells under standard conditions (37 °C, 5% CO2) and passage the cells 2–3 times a week.
- Wash near-confluent MDA-MB-231 cells with 2 mM EDTA in phosphate-buffered saline (PBS), and then detach the cells with 0.25% trypsin. Determine the cell concentration with a Neubauer’s chamber and resuspend them in 200 µL of RPMI-1640 at a concentration of 1.5 x 105 cells/200 µL.
- Use 6–8 week-old nude rats and keep them under pathogen-free, controlled conditions (21 °C ± 2 °C room temperature, 60% humidity, and 12 h light-dark rhythm). Offer autoclaved feed and water ad libitum.
- Before performing the surgery, inject an analgesic drug (e.g., Carprofen 4 mg/kg) subcutaneously. Anesthetize rats with an isoflurane (1–1.5 vol. %)/oxygen mixture at a flow rate of 2 L/min. Check the anesthetic depth by toe pinching.
- For surgery, use an operating microscope with a 16x magnification.
- Perform a 2–3 cm cut in the rat’s right inguinal region. Dissect all arteries in the right inguinal region, including the femoral artery (FA), the superficial epigastric artery (SEA), the descending genicular artery (DGA), the popliteal artery (PA), and the saphenous artery (SA). Place two removable clips on the FA: one proximal to the beginning of the SEA, and another directly proximal to the beginning of the DGA.
- Ligate the distal portion of the SEA. Perform a cut of the SEA’s wall and insert a 0.3 mm diameter needle into the SEA. Connect a syringe containing the cell suspension from step 1.2 to the needle. Remove the distal clip from the FA and clip the SA instead.
- Slowly inject the MDA-MB-231 cell suspension from step 1.2 (1.5 x 105 cells/200 µL) into the SEA. Remove the needle, ligate the SEA, and remove the artery clips. Close the wound using surgical clips and terminate anesthesia. Monitor the animals daily to assess tumor size and any evidence of pain.
2. Magnetic resonance imaging (MRI)
NOTE: For a detailed description of MRI procedures, please see Bäuerle et al.11.
- Perform MRI 10 days PI using a dedicated experimental scanner (see Table of Materials) or a human MR system with an appropriate animal coil.
- Anesthetize the rat with an isoflurane (1–1.5 vol. %)/oxygen mixture as described above. Place a catheter in the rat’s tail vein and tape it to the tail. Connect a syringe containing the contrast agent (0.1 mmol/kg Gd-DTPA in approximately 0.5 mL).
- Place the anesthetized rat in the MR system. Locate the distal femur and proximal tibia of the right hind leg in an anatomic sequence (e.g., T2-weighted turbo spin echo sequence; TR = 8,654 ms; TE = 37 ms; matrix 320 x 272; FOV = 65 mm x 55 mm; slice thickness = 1 mm; scan time 11:24 min).
- Determine the slices covering the distal femur and proximal tibia of the right hind leg and start the DCE-MRI sequence (e.g., fast low angle shot sequence; TR = 3.9 ms; TE = 0.88 ms; matrix = 256 x 216; FOV = 65 x 54 mm2; slice thickness = 1 mm; 8 slices; 100 time points; scan time = 8:25 min). After 30 s, start injecting the contrast agent over a time period of 10 s.
NOTE: The total time to perform an MRI examination is approximately 20 min per animal.
3. Positron emission tomography/computed tomography (PET/CT)
NOTE: For a detailed description of the PET procedures, please see Cheng at al.12.
- Perform PET/CT imaging 10 days PI using a dedicated experimental scanner (see Table of Materials).
- Keep the animals fasted prior to imaging. Anesthetize the rat as described in step 2.2 and insert a catheter in the tail vein as described above.
- Inject 6 MBq of 18F-Fluorodeoxyglucose (18F-FDG) into the tail vein and wait ~30 min to allow the tracer to distribute properly.
- Perform a CT acquisition (tube voltage = 80 kV, tube current = 500 µA, isotropic resolution = 48.9 µm, duration = 10 min).
- Perform a static PET acquisition (lower/upper discriminatory level = 350/650 keV; timing window = 3.438 ns; duration = 15 min).
4. Alternative imaging strategies
- For an early assessment of MDA-MB-231 cells in the hind leg, inoculate 1.5 x 105 labeled cells /200 µL for bioluminescence (i.e., cells expressing luciferin, MDA-MB-231-LUC13) or fluorescence imaging (i.e., cells expressing green or red fluorescent protein, MDA-MB-231-GFP/RFP13). Use the system for preclinical optical imaging to detect intraosseous MDA-MB-231 cells after tumor cell inoculation14.
- Perform experimental ultrasound using a dedicated scanner after intravenous injection of microbubbles to derive morphological and functional parameters of vascularization comparable to MRI7.
5. MRI analysis
- Use a DICOM viewer15 with a DCE Plugin16 and load the DCE sequence in 4D-mode by clicking the “Import” button in the top menu, selecting the DICOM folder containing the MR images from step 2.4, and clicking “4D Viewer” in the top menu.
- Place a circular 2-dimensional region of interest (ROI), with a target size of 1.5 mm2, in the proximal tibial shaft’s bone marrow of the right hind leg, preferably using image numbers 4 or 5 from the sequence consisting of 8 images, as these center images provide more stable results.
- Start the DCE plugin from the top menu, select “Relative Enhancement” in the “Plot Type” field, and define the baseline range from time points 1 to 5 by typing these numbers into the respective fields. Export the analysis as a .txt file with the respective button and choose “DCEraw.txt” as the file name.
- Open RStudio17 and load the provided DCE-Script.R file via the “File” menu by selecting “Open File”. Run the entire script by selecting “Code”, then “Run Region” and then “Run All” from the menu. Copy the output to the provided template file named “ImagingFeatures.xlsx” (Figure 2).
- In the DICOM viewer, place a second ROI within the back muscle of the animal and repeat steps 5.2–5.4 to obtain the muscle DCE measurements for normalization purposes. Within the spreadsheet “ImagingFeatures.xlsx”, the respective bone measurements are automatically divided by the respective muscle measurements for normalization purposes.
- Repeat steps 5.1–5.5 for all animals and complete the spreadsheet.
6. PET/CT analysis
- Open the PET/CT analysis software and import the data obtained in step 3 by clicking “File”, followed by “Manual import”. Mark the ct.img.hd and the pet.img.hdr files. Click “Open” and select “Import all”.
- Open the datasets by selecting “General analysis”, followed by “OK”.
- Select “ROI Quantification”, followed by “Create”, and then “Create a ROI from a template”. Place a 2-dimensional ROI approximately 4 mm x 6 mm into the proximal tibial shaft’s bone marrow of the right hind leg.
- Select “ROIs (Target 1 overlay)” and write down mean, minimum, and maximum values in Bq/mL.
- Calculate the maximum standardized uptake value (SUVmax): Divide the maximum value (Bq/mL) by the injected activity and multiply the result by the weight of the animal in grams. Enter the result into the spreadsheet (Figure 2).
7. Determining the tumor-take rate
- To diagnose tumor growth in the right hind leg, repeat MR and PET/CT imaging on day 30 PI, as described above.
NOTE: Tumors will be clearly visible on day 30 PI and feature T2w-hyperintense lesions and clear contrast enhancement in MRI, along with a clearly elevated SUVmax in PET/CT. According to previous experiments, 60%–80% of the animals will develop metastases in their right hind leg.
- Complete the spreadsheet by adding an additional “Tumor” column and enter “1” for every animal that presents metastases, and “0” for every animal without visible tumor burden (Figure 2). Save the spreadsheet as “ImagingFeatures.xlsx” within the Downloads folder.
8. Feature selection
- To determine the most relevant features for prediction of future tumor growth, import the spreadsheet into an open-source data visualization, machine learning, and data mining toolkit18.
- Draw the File-subroutine from the Data menu into the workspace on the right and double-click it. Load the spreadsheet by clicking the “Folder” icon and selecting the file “ImagingFeatures.xlsx”. Select the “Export” worksheet and assign the target-attribute to the variable “Tumor”. Assign the “Skip” function to the animal number (Figure 3).
- Draw the “Rank” subroutine from the Data menu into the workspace and connect the “File” and “Rank” subroutines by drawing a line between them.
- Open the “Rank” subroutine by double-clicking on its icon, and select the “Information Gain” algorithm19.
- From the five acquired parameters, use the top three for further analyses (SUVmax, PE, and AUC).
Note: These parameters reflect metabolic activity (SUVmax) and tissue vascularization (PE and AUC).
9. ML analysis
- Open RStudio 3.4.117 and load the provided TrainModel.R-Script via the “File” menu.
- Install the required libraries (this only has to be done once) by typing: install.packages(c("caret", "readxl", "pROC", "RcmdrPlugin.EZR", "ggplot2"))
- To load the required libraries and set the Downloads folder as the working directory, select the lines 3–5 within the TrainModel.R Script.
- Run the selected code by clicking “Code” within the menu, and then “Run Selected Line(s)”.
10. Training an avNNet ML algorithm
- To train an avNNet algorithm, select the lines 8–39 from the TrainModel.R-Script (see step 9.1).
- Run the selected code by clicking “Code” within the menu, and then “Run Selected Line(s)”.
11. Analyzing the ML algorithm’s results
- To assess standard parameters of diagnostic accuracy (sensitivity, specificity, positive and negative predictive values, and likelihood ratios), select the lines 41–50 from the TrainModel.R-Script.
- Run the selected code by clicking “Code” within the menu, and then “Run Selected Line(s)”.
12. Comparing the final model's Receiver Operating Characteristic (ROC) curve with the ROC curves of its constituent parameters
- To perform DeLong’s tests to compare the model’s ROC curve with the ROC curves of its constituent parameters, select the lines 52–62 from the TrainModel.R-Script (see step 9.1).
- Run the selected code by clicking “Code” within the menu, and then “Run Selected Line(s)”.
The rats recovered quickly from the surgery and injection of the MDA-MB-231 breast cancer cells and were then subjected to MR- and PET/CT imaging on days 10 and 30 PI (Figure 1). A representative DCE analysis of a rat’s right proximal tibia is presented in Figure 2A. The DCE raw measurements were saved by selecting the “Export” button and choosing “DCEraw.txt” as the file name.
Subsequent calculations of the dynamic parameters, AUC, PE, and washout were performed in RStudio with the respective script. The output of the DCE measurements had to be saved as “DCEraw.txt” within the “Downloads” folder, so that the script could be run directly without additional configurations to provide a data table, as depicted in Figure 2B. These data were copied to the provided spreadsheet (Figure 2C). Similarly, the DCE parameters for muscular tissue were determined and transferred to the spreadsheet (Figure 2D,E). These values were normalized by dividing the bone measurements by the muscle measurements; this was performed automatically within the spreadsheet. From the PET/CT, the calculated SUVmax values were subsequently transferred into the table (Figure 2F).
On day 30 PI, all animals were evaluated to determine whether or not they developed metastases, and the table was completed by coding positive tumor burden as “1” and healthy animals as “0” within the rightmost column of the spreadsheet (Figure 2C).
The spreadsheet was imported into the open-source data visualization, machine learning, and data mining toolkit, and feature ranking revealed the SUVmax, PE, and AUC as the top three features for prediction of metastatic disease (Figure 3). These parameters reflect metabolic activity (SUVmax) and tissue vascularization (PE and AUC).
Running the TrainModel.R Script automatically imported the spreadsheet and calculated an avNNet. The optimal hyperparameter combination was determined (Figure 4A) and the final model was then calculated using the optimal hyperparameter combination (Figure 4B). Subsequently, a set of standard diagnostic parameters was calculated (Figure 4C) and an ROC curve of the model was plotted (Figure 4D).
The positive result is shown in Figure 4B–D. A comparison of the model’s ROC curve with the ROC curve of its three constituents (i.e., AUC, PE, and SUVmax) revealed that the model performed significantly better than all of its three constituents (p = 0.01 for AUC, p = 0.003 for PE, and p = 0.007 for SUVmax). The combination of the three selected parameters to an avNNet was more sensitive, thus allowing prediction of macroscopic disease with an overall accuracy of 85.7% (95% confidence interval = 67.3%–96.0%). These results were obtained from an analysis of 28 samples. The confidence intervals can be further narrowed by increasing the number of animals.
The negative results could be obtained as described here. The accuracy measures were highly sensitive to specific types of machine learning algorithms and to steps of data preprocessing. Neural Networks, in particular, tended to perform better when the input data were normalized. This was achieved by the “BoxCox” function in section 10 of the protocol (lines 22 and 36 within the provided TrainModel.R-Script). Refraining from normalization and using a different algorithm (e.g., a neural network not averaged), by changing the method to “nnet” (lines 21 and 35 within the provided TrainModel.R-Script), resulted in an area of 0.594 under the ROC curve (Supplementary Figure 1). Such a model failed to outperform its three constituents significantly (all p > 0.15).
Because the script was optimized for RStudio 3.4.1 and the caret package version 6.0-84, using different software versions might yield different results. Running the provided scripts with the software versions used in this manuscript will give similar results. However, if readers aim to modify the script, add additional variables, change document folders or file names, or modify the machine learning algorithms in greater detail, respective adjustments of the code will be necessary. For these cases, the manual of the caret-package offers in-depth explanations20.
Figure 1: Representative MR and PET/CT images. MR and PET/CT images of the right hind leg of an animal that did not develop metastases over the course of the study (two leftmost columns, with images from day 10 and day 30 PI), and an animal that developed metastases between day 10 and day 30 PI (two rightmost columns, metastases marked with arrows). Note the high signal intensity of metastases in the T2w images (upper row), the contrast enhancement depicted by the increased area under the curve (AUC; second row), and the increased maximum standardized uptake value in the PET/CT (SUVmax; third row). Note that there are no visible differences in the images acquired on day 10 PI (first and third column) between the animal with metastases on day 30 PI and the animal that developed no bone metastases. This figure was modified from Ellmann et al.5. Please click here to view a larger version of this figure.
Figure 2: Assessment of the imaging features and compilation into a spreadsheet. (A) The dynamic contrast enhancement of the proximal tibia’s bone marrow was analyzed with a freeware DICOM viewer15 using a DCE plugin16. The respective measurements were saved, and (B) further analyzed with the provided DCE-Script.R-file in RStudio17. (C) The output was copied into a spreadsheet (see supplementary material for a template). (D) Likewise, the DCE measurement was performed for adjacent muscular tissue, analyzed using RStudio (E), and then copied to the spreadsheet. Data were normalized by dividing the results of the bone measurements by the results of the muscle measurements (C; salmon-shaded cells). (F) The PET/CT measurements were performed with the vendor’s software. The maximum standardized uptake value was calculated by dividing the respective measurement by the injected activity and multiplying it by the animal’s body weight. The result was subsequently copied into the spreadsheet. Please click here to view a larger version of this figure.
Figure 3: Feature ranking. Ranking of the acquired imaging features was performed within an open-source data visualization, machine learning, and data mining toolkit18 by importing the spreadsheet via the “File”-subroutine, and analyzing it via the “Rank”-subroutine. Please click here to view a larger version of this figure.
Figure 4: Representative RStudio output. The machine learning algorithm was developed using RStudio17 with the provided TrainModel.R-Script file. (A) A grid search among different hyperparameter combinations for the model-averaged neural network revealed a size of three neurons and a decay of 0.0005 as an optimum. (B) Using this hyperparameter combination, a full network was trained and cross-validated, reaching an overall accuracy of 85.7%. (C) Standard parameters of diagnostic accuracy, including sensitivity, specificity, positive and negative predictive values, and likelihood ratios, were calculated from a confusion matrix. (D) A representative ROC plot of the cross-validated model revealed an area under the curve (AUC) of 0.917. Please click here to view a larger version of this figure.
Supplementary Figure 1: Negative result. Changing the ML algorithm to a Neural Network without averaging and refraining from normalization of the input parameters led to a drop of the area under the curve of the ROC curve from 0.917 (Figure 4D) to 0.594. Please click here to download this file.
Supplementary Figure 2: Alternative feature ranking. In contrast to the standard method depicted in Figure 3, a random variable was also introduced (“Random”; highlighted in blue), with its importance included in the ranking. This approach confirmed the applied selection of the variables SUVmax, PE, and AUC. Please click here to download this file.
ML algorithms are powerful tools used to integrate several predictive features into a combined model and obtain an accuracy that exceeds that of its separate constituents when used alone. Nonetheless, the actual result depends on several critical steps. First, the ML algorithm used is a crucial factor, because different ML algorithms yield different results. The algorithm used in this protocol is an avNNet, but other promising algorithms include Extreme Gradient Boosting21 or Random Forests. The caret package20 for RStudio provides a plethora of different algorithms (currently >175), and the proposed protocol is highly flexible in terms of switching from one algorithm to another by simply changing single lines of code (e.g., changing method = “avNNet” to method =“rf”) and adapting the TunedGrid-settings to the respective ML algorithm. For details, see the caret github repository22. An excellent overview of different algorithms and their performance with respect to different classification problems was published by Fernández-Delgado et al.23 and could serve as a starting point for other experiments.
Another crucial factor is the choice of relevant features to include in an ML algorithm. This protocol proposes the use of the filter method, “Information Gain”19, to rank the available features in descending order and use the most relevant ones to train the avNNet. Filter methods are based only on general assumptions, such as correlation with the variable to predict, so researchers should preselect features independently of the classifier used24,25. Such methods are particularly effective in computation time and robust to overfitting. However, the cutoff that separates relevant from irrelevant features is defined by the user, making it somewhat arbitrary. For the proposed protocol, the features with the top 75% information gain were used, corresponding to SUVmax, PE, and AUC. This selection, however, can be systematically strengthened by including a random variable that has no relationship to the target, calculating its information gain, and then comparing it to the information gain of the real features (Supplementary Figure 2). This slightly more sophisticated approach additionally confirmed the choice of the three abovementioned features as being the most relevant. Nonetheless, several different filter methods exist, along with other approaches that select features with respect to a particular classifier algorithm, such as feature extraction and wrapper methods. Different feature selection approaches may yield different results.
To ensure generalizability of the ML algorithm and further prevent overfitting, the proposed protocol applies leave-one-out cross-validation (LOOCV). The best approach, however, would be to randomly remove a subset from the entire data set, and treat it as a testing set. The ML algorithm is then trained on the remainder of the data (i.e., the training set) to subsequently predict the outcome of the testing set. However, this approach needs a sufficiently large data set. For smaller sample sizes, application of LOOCV is common because it provides an almost unbiased estimate of a model's true generalization ability26. In LOOCV, the first data point is removed from the data set, and the avNNet is trained with the retained data. Then, the outcome of the formerly withheld data point is predicted and saved. This process is repeated for all data points, so that finally each outcome is predicted with data that were not used for training the algorithm. Other validation approaches include x-fold cross-validations (most commonly 10-fold) and can be easily applied by changing the respective trainControl parameter within the code to method="CV".
From a quantitative point-of-view, medical images are typically evaluated in a very basic way, largely relying on measurements of size and shape of potentially suspicious lesions or areas27,28. However, the advantage of the Digital Imaging and Communications in Medicine (DICOM) standard is that it allows extraction of many features, referred to as radiomics. The term “radiomics“ was initially defined as the high-throughput extraction of large quantities of image features29, but was subsequently extended to include the conversion of images to higher dimensional data30. However, the higher dimensional data are mainly used to identify correlations rather than causes30. The features described in this protocol fall in between classic radiological features, such as size and shape, and radiomics, as they resemble generally accepted parameters of vascularization and metabolic activity. This offers a potential causal relation to the microinvasion of disseminated tumor cells. If desired by the user, an extraction of radiomic features can be performed with different software packages31.
The protocol provided is not restricted to a finite number of features. Thus, it can be used with large radiomics data sets. However, the abovementioned issue of feature selection becomes increasingly important with growing data sets. The presented protocol can also be transferred to different study settings, e.g., from the fields of oncology, infection, or inflammation32.
The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.
This work was supported by the German Research Foundation (DFG, Collaborative Research Centre CRC 1181, subproject Z02; Priority Programme μBone, projects BA 4027/10-1 and BO 3811), including additional support for the scanning devices (INST 410/77-1 FUGG and INST 410/93-1 FUGG), and by the Emerging Fields Initiative (EFI) “Big Thera” of the Friedrich-Alexander-University Erlangen-Nürnberg.
|Binocular Operating Microscope||Leica||NA|
|ClinScan MR System||Bruker||NA|
|Inveon Research Workplace Software||Siemens Healthcare GmbH||NA|
|MDA-MB-231 human breast cancer cells||American Type Culture Collection||N/A|
|Open-source data visualization, machine learning and data mining toolkit.||Orange3, University of Ljubljana||NA||https://orange.biolab.si/|
- D'Oronzo, S., Brown, J., Coleman, R. The role of biomarkers in the management of bone-homing malignancies. Journal of Bone Oncology. 9, 1-9 (2017).
- Ellmann, S., Beck, M., Kuwert, T., Uder, M., Bäuerle, T. Multimodal imaging of bone metastases: From preclinical to clinical applications. Journal of Orthopaedic Translation. 3, (4), 166-177 (2015).
- Braun, S., Pantel, K. Clinical significance of occult metastatic cells in bone marrow of breast cancer patients. The Oncologist. 6, (2), 125-132 (2001).
- Braun, S., Rosenberg, R., Thorban, S., Harbeck, N. Implications of occult metastatic cells for systemic cancer treatment in patients with breast or gastrointestinal cancer. Seminars in surgical oncology. 20, (4), 334-346 (2001).
- Ellmann, S., et al. Prediction of early metastatic disease in experimental breast cancer bone metastasis by combining PET/CT and MRI parameters to a Model-Averaged Neural Network. Bone. 120, 254-261 (2018).
- Bäuerle, T., Komljenovic, D., Berger, M. R., Semmler, W. Multi-modal imaging of angiogenesis in a nude rat model of breast cancer bone metastasis using magnetic resonance imaging, volumetric computed tomography and ultrasound. Journal of Visualized Experiments. (66), e4178 (2012).
- Merz, M., Komljenovic, D., Semmler, W., Bäuerle, T. Quantitative contrast-enhanced ultrasound for imaging antiangiogenic treatment response in experimental osteolytic breast cancer bone metastases. Investigative Radiology. 47, (7), 422-429 (2012).
- Bäuerle, T., et al. Characterization of a rat model with site-specific bone metastasis induced by MDA-MB-231 breast cancer cells and its application to the effects of an antibody against bone sialoprotein. International Journal of Cancer. 115, (2), 177-186 (2005).
- Patel, J., Goyal, R. Applications of Artificial Neural Networks in Medical Science. Current Clinical Pharmacology. 2, (3), 217-226 (2008).
- Naftaly, U., Intrator, N., Horn, D. Optimal ensemble averaging of neural networks. Network: Computation in Neural Systems. 8, (3), 283-296 (1997).
- Bäuerle, T., Merz, M., Komljenovic, D., Zwick, S., Semmler, W. Drug-induced vessel remodeling in bone metastases as assessed by dynamic contrast enhanced magnetic resonance imaging and vessel size imaging: A longitudinal in vivo study. Clinical Cancer Research. 16, (12), 3215-3225 (2010).
- Cheng, C., et al. Evaluation of treatment response of cilengitide in an experimental model of breast cancer bone metastasis using dynamic PET with 18F-FDG. Hellenic Journal of Nuclear Medicine. 14, (1), 15-20 (2011).
- Marturano-Kruik, A., et al. Human bone perivascular niche-on-a-chip for studying metastatic colonization. Proceedings of the National Academy of Sciences of the United States of America. 115, (6), 1256-1261 (2018).
- Sonntag, E., et al. In vivo proof-of-concept for two experimental antiviral drugs, both directed to cellular targets, using a murine cytomegalovirus model. Antiviral Research. 161, 63-69 (2019).
- Horos - Free DICOM Medical Image Viewer | Open-Source. Available from: https://www.horosproject.org/ (2015).
- DCE Tool Plugin. Available from: http://kyungs.bol.ucla.edu/software/DCE_tool/DCE_tool.html (2020).
- RStudio Team RStudio: Inteegrated Development for R. Available from: https://rstudio.com (2015).
- Demšar, J., et al. Orange: Data Mining Toolbox in Python. Journal of Machine Learning Research. 14, 2349-2353 (2013).
- Saeys, Y., Inza, I., Larrañaga, P. A review of feature selection techniques in bioinformatics. Bioinformatics. 23, (19), 2507-2517 (2007).
- Kuhn, M. CRAN - Package caret. Available from: https://cran.r-project.org/web/packages/caret/index.html (2016).
- Chen, T., et al. CRAN: Package xgboost - Extreme Gradient Boosting. Available from: https://cran.r-project.org/web/packages/xgboost/ (2019).
- Available Models | The caret Package. Available from: http://topepo.github.io/caret/available-models.html (2020).
- Fernández-Delgado, M., Cernadas, E., Barro, S., Amorim, D., Fernández-Delgado, A. Do we Need Hundreds of Classifiers to Solve Real World Classification Problems. Journal of Machine Learning Research. 15, 3133-3181 (2014).
- Hira, Z. M., Gillies, D. F. A Review of Feature Selection and Feature Extraction Methods Applied on Microarray Data. Advances in Bioinformatics. 2015, 198363 (2015).
- Sánchez-Maroño, N., Alonso-Betanzos, A., Tombilla-Sanromán, M. Filter methods for feature selection - A comparative study. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 4881, 178-187 (2007).
- Cawley, G. C., Talbot, N. L. C. C. Fast exact leave-one-out cross-validation of sparse least-squares support vector machines. Neural Network. 17, (10), 1467-1475 (2004).
- Forghani, R., et al. Radiomics and Artificial Intelligence for Biomarker and Prediction Model Development in Oncology. Computational and Structural Biotechnology Journal. 17, 995-1008 (2019).
- Jaffe, C. C. Measures of response: RECIST, WHO, and new alternatives. Journal of Clinical Oncology Official Journal of the American Society of Clinical Oncology. 24, (20), 3245-3251 (2006).
- Lambin, P., et al. Radiomics: Extracting more information from medical images using advanced feature analysis. European Journal of Cancer. 48, (4), 441-446 (2012).
- Gillies, R. J., Kinahan, P. E., Hricak, H. Radiomics: Images are more than pictures, they are data. Radiology. 278, (2), 563-577 (2016).
- Nioche, C., et al. Lifex: A freeware for radiomic feature calculation in multimodality imaging to accelerate advances in the characterization of tumor heterogeneity. Cancer Research. 78, (16), 4786-4789 (2018).
- Ellmann, S., et al. Application of machine learning algorithms for multiparametric MRI-based evaluation of murine colitis. PLOS ONE. 13, (10), 0206576 (2018).