Now this should be noted that I have certainly applied the wrong homography matrix to the first image xD.
Right now I am checking the differentiability of
warp and the functions it calls inside it.
warp is itself mutating at some part (which is completely visible), but even apart from that, there are some weird problems I am coming across. I'll give you some examples.
julia> gradient(x->sum(round.(x)),B) (nothing,) julia> gradient(x->sum(round.(x;digits=15)),B) ERROR: MethodError: no method matching iterate(::Nothing)
After having a talk in the
#autodiff channel, Lyndon White told me this -
There is no
rrule(::typeof(round), x; kwargs...) only
@scalar_rule round(x) which generates
rrule(::typeof(round), x) reference.
@scalar_rule doesn't actually accept keyword args, so fixing it would require writing a full rrule. Probably the fix is to add such support here.
Zygote when checking for if there is an
rrule only checks if there are positional arguments than match, it doesn't check kwargs (even though it is aware of them). Not sure if can fix that or not, might be able to. Even if fixed would probably get a different error later though, reference.
I am currently in the process of figuring out a work around where I will be able to differentiate through the function seamlessly and my final goal (for this subpart) would be to train a homography matrix by regressing on the image (optimizing the photometric error).