Requesting help with writing a CFFI wrapper of our secp256k1-zkp fork

Dear all,

As some of you may have already heard, I am working on Mimblewimble implementation in Python. The library already provides some useful functionalities, but it is far from ready.

Current obstacle is access to secp256k1-zkp methods. We have our own fork of secp256k1-zkp library and the way it is used in Rust is via a wrapper.

I am writing a similar wrapper for Python. I have already wrapped some methods, it builds and those methods are accessible using Python. Unfortunately, I am not fluent with C and my work on it is very inefficient. I have prepared a necessary setup to get started

Basically one needs to help me write more definitions in deps.c in a way to avoid type errors during compilation I keep getting. The repo contains everything you need to get started with, once we have the definitions and everything builds (not just few methods I prepared to get started) I will make a Python module out of it and publish it and integrate it with mimblewimble-py.

I also hope ツ Community Council could offer a bounty for someone who is willing to help.
Please leave your feedback!
Cheers!

7 Likes

Sounds reasonable. No idea what a fair amount would be though? Add it to the agenda for the next CC meeting @Cobragrin so we can discuss it.

4 Likes

Really hard to tell, it depends on experience… My suggestion is:

  1. Offer a flat rate prize and state this is assuming X hours of work are needed.
  2. Make it clear that if in the process the problem ends up more challenging the extra time will also be compensated as long as sufficiently documented and there are traces of work over time (GitHub, questions on the forum / keybase etc).
1 Like

I’m interested in working on this one, will fork it tonight and have a look to see what is happening. Do you have a branch that has the problem with the compilation errors?

2 Likes

Hi, so I created a how-to-start branch

You may simply start grabbing definitions from

and put them in defs.c

and then fetch the C source

git submodule init
git submodule update

and run the build

python build.py

and you will start getting parsing errors, type errors etc

Traceback (most recent call last):
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/cparser.py", line 336, in _parse
    ast = _get_parser().parse(fullcsource)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/pycparser/c_parser.py", line 152, in parse
    debug=debuglevel)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/pycparser/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/pycparser/ply/yacc.py", line 1199, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/pycparser/ply/yacc.py", line 193, in call_errorfunc
    r = errorfunc(token)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/pycparser/c_parser.py", line 1861, in p_error
    column=self.clex.find_tok_column(p)))
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/pycparser/plyparser.py", line 67, in _parse_error
    raise ParseError("%s: %s" % (coord, msg))
pycparser.plyparser.ParseError: <cdef source string>:57:5: before: secp256k1_pubkey

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "build.py", line 51, in <module>
    ffi.cdef(_source)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/api.py", line 112, in cdef
    self._cdef(csource, override=override, packed=packed, pack=pack)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/api.py", line 126, in _cdef
    self._parser.parse(csource, override=override, **options)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/cparser.py", line 389, in parse
    self._internal_parse(csource)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/cparser.py", line 394, in _internal_parse
    ast, macros, csource = self._parse(csource)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/cparser.py", line 338, in _parse
    self.convert_pycparser_error(e, csource)
  File "/Users/marek/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/cparser.py", line 367, in convert_pycparser_error
    raise CDefError(msg)
cffi.CDefError: cannot parse "secp256k1_pubkey* pubkey,"
<cdef source string>:57:5: before: secp256k1_pubkey

This is just how to get started. The goal is not to copy all the methods from rustyrussell/secp256k1-py/ but to have all those as well as all the MW zkp methods from GitHub - mimblewimble/secp256k1-zkp: Fork of secp256k1-zkp for the Grin/MimbleWimble project

2 Likes

Thanks Renzokuken.
Started taking a look, it’s quite abit of work. Work that I’m happy to do though :smiley:

2 Likes

Glad you are so motivated! Please keep updating us on your progress.

@Anynomous could you please confirm the bounty has been approved?

3 Likes

@renzokuken,@walkbackgod,@Cobragrin
The bounty proposed in Tuesdays 13th of September’s CC meeting has officially been approved by the CC members:

Bounty for a Python cffi secp256k1-zkp wrapper
-3000$ bounty for significant progress
-10.000$ (includes the $3000) for the finished, well documented and well tested wrapper. MIT license open source, or if possible with the wrapped package, Apache license are required.

We now lock the bounty for you to work on this project @walkbackgod. Keep us informed on your estimate timeline and any progress.

5 Likes

OK have forked and managed to get through one set of errors.
I expect to spend some hours on this later today, will let you guys know how going in next few days.

I suspect there will be sections that are harder :smiley:

Here is the FORK/BRANCH I’m working with.

5 Likes

As you can see from the repo, I completed the first bit a while ago which was to get all the methods from the mw fork working and building. I am working on the next bit which is definitely harder. Trying to get the other methods working… Once i get past the first few it may become easier, but right now it’s difficult. @renzokuken not sure if you want to check the current repo to make sure you are happy with what is there right now?

5 Likes

Thank you! I will do my best to review your code before October 7th, 2022. I am currently travelling for business and it is difficult for me to find a time slot, but I acknowledge I received your request and I added this review to my current TODO list.

3 Likes

Terribly sorry for the delays. @walkbackgod I just added your remote and build from the branch wbg-pythonwrapper. Build properly and without errors. Since I left this project and applied for the funding the bounty you added these following methods:

secp256k1_ec_privkey_tweak_add
secp256k1_ec_privkey_tweak_mul
secp256k1_ec_pubkey_combine
secp256k1_ec_pubkey_create
secp256k1_ec_pubkey_parse
secp256k1_ec_pubkey_serialize
secp256k1_ec_pubkey_tweak_add
secp256k1_ec_pubkey_tweak_mul
secp256k1_ec_seckey_verify
secp256k1_ecdsa_sign
secp256k1_ecdsa_signature_normalize
secp256k1_ecdsa_signature_parse_compact
secp256k1_ecdsa_signature_parse_der
secp256k1_ecdsa_signature_serialize_compact
secp256k1_ecdsa_signature_serialize_der
secp256k1_ecdsa_verify

You are definitely on the right track. I am very happy with the results so far. Keep it up!

3 Likes