-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME.Rmd
128 lines (91 loc) · 7.24 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
fig.dpi = 120, fig.width = 8, fig.height = 6,
out.width = "100%"
)
```
# ovml
<!-- badges: start -->
[](https://www.tidyverse.org/lifecycle/#experimental)
[](https://github.com/openvolley/ovml/actions)
<!-- badges: end -->
## Installation
``` r
## install.packages("remotes")
remotes::install_github("openvolley/ovml")
```
The `ovml` package provides image and video machine learning tools for volleyball analytics. See also [ovmlpy](https://github.com/openvolley/ovmlpy), which provides similar functionality but with Python-based implementations that are currently substantially faster than the libtorch-based implementations in `ovml`.
Currently three versions of the [YOLO](https://pjreddie.com/darknet/yolo/) object detection algorithm are included (versions 3, 4, and 7) and an experimental version of this network specifically for detecting volleyballs. These have been implemented on top of the [torch](https://torch.mlverse.org/) R package, meaning that no Python installation is required on your system.
**NOTE:** the performance of this package is substantially slower than the equivalent native C++/Python implementations. This is a known limitation of the R torch package (see e.g. https://github.com/mlverse/torch/issues/694, https://github.com/mlverse/torch/issues/268) and improvements are expected over time. Nevertheless, this R package provides a convenient mechanism for R users to apply these algorithms without requiring Python (perhaps best suited to analyses of small numbers of images or non-time-critical applications).
Note that you probably can't use `ovml` and `ovmlpy` in the same R session, because of conflicts in shared libraries.
This implementation drew from [ayooshkathuria/pytorch-yolo-v3](https://github.com/ayooshkathuria/pytorch-yolo-v3), [walktree/libtorch-yolov3](https://github.com/walktree/libtorch-yolov3), [rockyzhengwu/libtorch-yolov4](https://github.com/rockyzhengwu/libtorch-yolov4), [gwinndr/YOLOv4-Pytorch](https://github.com/gwinndr/YOLOv4-Pytorch), and [WongKinYiu/yolov7](https://github.com/WongKinYiu/yolov7).
## Example
Use a YOLOv4 network to recognize objects in an image. We use a video frame image (bundled with the package):
```{r ex1}
library(ovml)
img <- ovml_example_image()
ovml_ggplot(img)
```
Construct the network. The first time this function is run, it will download and cache the network weights file (~250MB).
```{r ex2}
dn <- ovml_yolo()
```
Now we can use the network to detect objects in our image:
```{r ex3}
dets <- ovml_yolo_detect(dn, img, conf = 0.3)
dets <- dets[dets$class %in% c("person", "sports ball"), ]
ovml_ggplot(img, dets)
```
Note that this network didn't detect the volleyball (in the process of being contacted by the server). Let's try the experimental volleyball-specific network:
```{r ex4}
dn <- ovml_yolo("4-mvb")
ball_dets <- ovml_yolo_detect(dn, img)
ovml_ggplot(img, ball_dets, label_geom = NULL) ## don't add the label, it obscures the volleyball
```
We can transform the image detections to real-world court coordinates. First we need to define the court reference points needed for the transformation. We can use the `ov_shiny_court_ref` helper app for this:
```{r ex5h, echo = FALSE}
ref <- list(antenna = dplyr::tribble(~image_x, ~image_y, ~antenna, ~where,
0.208, 0.348, "left", "floor",
0.820, 0.353, "right", "floor",
0.823, 0.641, "right", "net_top",
0.200, 0.643, "left", "net_top"),
video_width = 1280,
video_height = 720,
video_framerate = 30,
net_height = 2.43,
court_ref = dplyr::tribble(~image_x, ~image_y, ~court_x, ~court_y,
0.0549, 0.0221, 0.5, 0.5,
0.953, 0.0233, 3.5, 0.5,
0.751, 0.520, 3.5, 6.5,
0.289, 0.516, 0.5, 6.5))
```
```{r ex5, eval = FALSE}
library(ovideo)
ref <- ov_shiny_court_ref(img)
```
`ref` should look something like:
```{r ex6}
ref
```
Now use it with the `ov_transform_points` function (note that currently this function expects the image coordinates to be normalized with respect to the image width and height):
```{r ex7}
court_xy <- ovideo::ov_transform_points(x = (dets$xmin + dets$xmax) / 2 / ref$video_width,
y = dets$ymin / ref$video_height,
ref = ref$court_ref, direction = "to_court")
dets <- cbind(dets, court_xy)
```
And plot it:
```{r ex8}
library(datavolley)
library(ggplot2)
ggplot(dets, aes(x, y)) + ggcourt(labels = NULL, court_colour = "indoor") + geom_point()
```
Keep in mind that `ov_transform_points` is using the middle-bottom of each bounding box and transforming it assuming that this represents a point on the court surface (the floor). Locations associated with truncated object boxes, or objects not on the court surface (players jumping, people in elevated positions such as the referee's stand) will appear further away from the camera than they actually are.