Update: OpenCL version is partly finished. Now faster on nvidia by about 25%. AMD hardware appears to be reasonably competitive if Siphash-1-3 is used instead of Siphash-2-4. AMD Vegas will not rule as in monero, performance of cards might be closer to what we see in modern games. Older RX 480 and RX 580 will not stand out as it is with ethereum (I don’t have any now, just a guess).
While ASIC can still do siphash cheaply, it would still need expensive GPU-like structure to move edges around quickly. When trimming, my 1080 Ti draws full power (250W) while actively using over 20MB of SRAM, over 5GB of DRAM and has over 50 000 threads in flight to hide latency. I’d say that is really good. While it is fast, it saps lots of power so small devices are not completely out of the game when it comes to performance per watt. And it is not likely that a hidden player will have something twice as effective unless a new algorithm is found.
The most memory saving algorithm needs to move around roughly 20GB of data for each graph solution (computing endpoints on the fly, as suggested by John Tromp). So purely from memory perspective, 2 gps is possible on PC like dual channel DDR4 and 20 gps on top end GPU like memory @512GB/s. Assuming 100% memory utilization and lots of SRAM buffers + (asic/hash) cores to hide DDRXY burst access latency.
1080 Ti - 5.2 gps @ 250W
1080 Ti - 4.0 gps @ 125W (over 50% could be memory power + controller)
i7 ~ 0.5 gps @ 50W
Best mining HW could be next gen nvidia with 16gbps GDDR6 memory. Most power efficient could be something with HBM2 that draws only approx. 30W for 16GB+controller.
Update 2: AMD Vega matches 1080 Ti at low power early in the algorithm (the most difficult part), but gets completely thrashed by nvidia as rounds get progressively thinner. There are multiple ways how to adapt the algorithm to prevent stalling on AMD, but I need to explore them. Having Vega64 2-3x slower is not acceptable.
Update 3: Tested stratum connectivity to Grin node with Urza. Seems to work. Got a helpful tip from OhGodACompany about a strange AMD HW feature that causes the performance drop on AMD. Unfortunately it means I need to rewrite the already verified AMD code, again, and OpenCL version will be pushed back a bit.