| Life In 19x19 http://www.lifein19x19.com/ |
|
| Reverse engineering neural networks http://www.lifein19x19.com/viewtopic.php?f=18&t=18109 |
Page 1 of 1 |
| Author: | hakuseki [ Tue Mar 23, 2021 7:38 am ] |
| Post subject: | Reverse engineering neural networks |
In the past year or two there's been some exciting research into understanding exactly what neural networks are doing. For example there's this ongoing project to reverse engineer InceptionV1. Essentially they're taking a trained model and thoroughly analyzing it, neuron by neuron, using techniques like feature visualization, and finally designing replacement neurons by hand. I'm thinking about starting a similar project to analyze a Go-playing neural network. I'm not sure how much progress is realistic to expect, since it seems like an enormous task. However, this also seems like something that could give us a new understanding of Go. Rather than just trying to mimic AI, we would be able to fully describe the theory it's using to think about the game. One early decision for this project would be which network to study. I think it makes sense to start small, e.g. with a 6-block network. I really like KataGo for its support of various rulesets, but its extra input features such as ladder status seem like they'd make feature visualization harder. It might make sense to go with some older model, e.g. from the LeelaZero project, or to train a custom KataGo model without these extra input features. Also, I vaguely recall someone working on creating compressed or sparse versions of KataGo networks. Working with a smaller or sparser model in this way may also reduce the amount of work needed to understand it. Another question to consider is how to create the feature visualizations. For example we might start by optimizing positions in a continuous space where each intersection is some linear combination of black, white, and empty summing to 1. This could be visualized using stones colored in continuously varying shades of gray with continuously varying transparency as well. If these visualizations seem too nonsensical we could also try a version using discrete optimization. I'm curious if people have any thoughts on the feasibility of such a project or any useful tips on how to get started. |
|
| Author: | RobertJasiek [ Tue Mar 23, 2021 8:57 am ] |
| Post subject: | Re: Reverse engineering neural networks |
I have made the same basic suggestion several years ago. - If a network is too complex for purely manual analysis of reverse engineering, obviously one also needs automatic analysis. One place to start is with the simplest positions shortly before the game end; slightly modify them and analyse the impact on the neural net. |
|
| Page 1 of 1 | All times are UTC - 8 hours [ DST ] |
| Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group http://www.phpbb.com/ |
|